We lost lock at 07:41 due to PI29, the time from when the PI started increasing to losing lock was less than 2 minutes.
TITLE: 06/14 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 128Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 13mph Gusts, 8mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.13 μm/s
QUICK SUMMARY:
Taking over from Corey
TITLE: 06/13 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY:
Shift mainly marred by high winds for about 6hrs of shift, but back to Observing during last hour of shift.
LOG:
Heavy winds continue and the green Y-Arm has trouble staying locked (I've stayed at LOCKING ARMS GREEN during the high winds).
Attached is a look at the last 3hrs zoomed in on speeds from 30-54mph. The last 90min have been highest thus far for the shift....but if I squint it might slowly be dropping...but we are still with gusts in the 40-50mph range.
TITLE: 06/13 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 25mph Gusts, 18mph 5min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.11 μm/s
QUICK SUMMARY:
I might have re-saved my original post as a draft after I posted, so I lost my original post---reposting now.
wind, Wind, WIND!!!
Arrived to see H1 after a recent lockloss---Camilla mentioned the winds (I also drove through the gusts and eyed a fire in the Benton City area on the drive in as well.). We've also had a few gusts in the 40mph range in the last 30min.
Currently have OBSERVATORY Mode in the WIND state, have taken ISC LOCK to LOCKING ARMS GREEN, and holding here until the 60min timer alarms for a reassessment. This is because of the loop of losing lock at early states of ISC LOCK.
TITLE: 06/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 131Mpc
SHIFT SUMMARY: Easy relock after Tuesday Maintenance but the wind is forecasts\ to pick up this evening.
LOG:
There is a computer Patrick left turned on in the EY Mech room on HEPI pump controller, Robert powered off the monitor but said that he has previously done research that new style LED monitors are less nosy than computers.
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 13:14 | CDS | Erik | Remote | N | Restarting NUCs OPSLogin0 | 13:22 |
| 14:24 | FAC | Tyler | MidY | N | Slowly move snorkle lift to MidY | 15:10 |
| 15:00 | PCAL | Tony | PCAL lab then CER | N | PCAL measurement prep | 15:27 |
| 15:00 | FAC | Karen | EndY | N | Technical Cleaning | 16:01 |
| 15:07 | FAC | Bubba | Air Handler Room | N | Fans alog 70404 | 17:07 |
| 15:09 | VAC | Travis, | EX, MX | N | Monthly Turbo Pump Checks | 17:54 |
| 15:14 | FAC | Tyler | OSB Recieving | N | Forklift Move | 15:15 |
| 15:15 | COMM | Sheila, Jason | LVEA | N | PSL Racks - turn off sidebands | 15:25 |
| 15:16 | PCAL | Rick, Tony, Julianna | EX | YES | PCAL Measurement, WP 11253 | 17:40 |
| 15:17 | FAC | Kim | EX | YES | Technical Cleaning | 16:17 |
| 15:22 | PSL | Jason | CR | N | Touch up RevCav Alginment alog70402 | 16:12 |
| 15:23 | EE | Fil, Patrick | EY | N | New HEPI Beckhoff, Jim turned off HEPI | 19:06 |
| 15:31 | COMM | Sheila | CR | N | Prep for OMC scans and OMc scans (ETMs misaligned) alog 70409 | 17:44 |
| 15:36 | SEI | (Jim) | EY | N | EY HEPI offline | 19:17 |
| 15:44 | CDS | WAP | All buildings | N | WAP turned on | 19:17 |
| 15:50 | COMM | Daniel | LVEA/CER | N | PSL Racks turn off 117MHz sideband | 15:58 |
| 16:26 | FAC | Karen, Kim | FCES | N | Technical Cleaning | 17:19 |
| 16:29 | SEI | Jim | FCES | n | Feedthrough protection | 17:24 |
| 16:54 | EE | Marc | CER Mezz, EX, EY | N | Check on Kepco Power supplys (not in VEAs) | 19:04 |
| 17:40 | EPO | Mike + 12 | LVEA | N | Tour | 18:13 |
| 17:43 | VAC | Gerardo | FCES | N | length measurements - cables and pipes | 18:18 |
| 17:43 | COMM | Shiela | LVEA | N | Turn on RF sidebands | 17:54 |
| 17:45 | VAC | Janos, Jordan | LVEA, EY, EY | N | Labeling Equipment | 18:57 |
| 17:46 | OPS | Tony | EX | N | Dust Moniter Pump Replacement | 18:54 |
| 18:13 | Tour | Rick + 5 | LVEA | N | Tour | 18:55 |
| 18:26 | PEM | Robert, Bubba | EX | N | HVAC setup | 18:59 |
| 18:48 | SEI | Jim | CR | N | HAM4 TFs | 21:06 |
| 18:10 | SEI | Jim | EY | N | Assist with HEPI Beckoff | 18:40 |
| 18:00 | FAC | Kim, Karen, Cindy | LVEA | N | Technical Cleaning | 18:50 |
| 19:05 | PCAL | Tony | PCAL Lab | N | PCAL lab | 19:45 |
| 19:07 | PEM | Robert | EY | N | Get equipment | 20:02 |
| 19:24 | Betsy | LVEA | N | Walkthrough | 19:36 | |
| 19:31 | VAC | Janos, Jordan | Mx, MY | N | Lableing Equimpent in VEA | 19:58 |
| 19:40 | CDS | Dave | Remote | N | als retarts and DAQ restart | 19:40 |
| 19:52 | PEM | Robert | LVEA | N | Accelerometers near Oplevs | 20:12 |
| 21:03 | PCAL | Tony | PCAL Lab | N | PCAL | ongoing |
Went out of Observing due to SQZ-OPO_PZT_1 getting to edge of it's range 40-110V. No changes have been made to mitigate this as it only changed so rapidly as the LVEA temperature changed, see 70477.
Caused a deliberate Lost lock for Tuesday Maintenance at 15:08UTC (1370704124) alog70399
Relocking was uneventful. Prior to starting initial alignment (while other work was finishing up) I manually touched ETMs and TMSs for better green flashes and then completed an initial alignment.
The lock acquisition itself was completely automatic but we did loose lock twice (TUN_ON_BS_STAGE2 and OFFLOAD_DRMI_ASC). I expect this was due wind > 20mph, plot attached.
NLN at 21:14UTC, Observing at 21:25UTC. Brina accepted a sdf for alog70423: 4:1 Hz MICH1 FM3 filter.
Brina, Sheila,
After taking a look at the UGF for the Michelson loop, we turned back on the 4:1 FM3 filter on 06/13/23 at 21:17:50 UTC and will run it for at least a week to see if this aids in keeping lock from Earthquakes, (alog 70293 thread).
The original filter was in MICH2 FM3 (see 68432) (which is the same as what Brina and I edited MICH1 FM3 to be this morning), and was engaged in the DRMI guardian. I've edited and loaded the DRMI guardian to engage this again, so next time we relock this should be on.
This gets turned off when the IFO unlocks (in sdf revert). To keep this filter on in Observing, the operator should REVERT the lsc model sdf diff for MICH1 FM3 (turning FM3 on) to get into observing.
To get around this we could decide to keep this on during locking or editing the guardian.
WP11256 Upgrade EY HEPI Pump Controller to Beckhoff
Patrick, Fil, Jim, Dave:
Patrick and Fil installed the new Beckhoff HEPI Pump Controller at EY alongside the original "Ben purple box". The new unit has the temporary name h1hpipumpctrley1, Fil ran a new ethernet cable to it from sw-ey-aux port14.
Patrick and Fil got the new unit booted and on the network. Its EPICS IOC is running, there is no overlap in channel names between the old and the new EPICS databases, so they can run at the same time.
The new unit was not put into production today, we are still using the original unit to control EY pumps.
WP11245 TW0 raw minute trend files offload
Jonathan, Dave:
To complete last week's offload of raw minute trend files from TW0 SSD to h1daqframes-0 HDD, the hand edited daqdrc on h1daqnds0 was put into puppet by Jonathan. We verified the configuration when the DAQ was restarted today.
WP11254 Atomic Clock Re-synchronize with timing system
Daniel:
Daniel resync'ed the atomic clock. Please see his alog for details.
WP11257 Remove obsolete filter-modules-with-control parts from h1als[ex,ey]
Daniel, Dave:
Daniel downgraded the H1:LSC-X_ARM_DRIVE and H1:LSC-Y_ARM_DRIVE filter modules on h1als[ex,ey] to standard filter modules. This fixes the issue of the fm-w-ctrl having hardcoded Cin and Cmask attempting to turn on non-existent FM3 and FM5 filters.
Because the MASK PV for these FMs were removed from the INI files, a DAQ restart was required.
DAQ Restart
Dave, Jonathan:
The DAQ was restarted for the above h1als[ex,ey] model changes. Both GDS0 and GDS1 required a second restart to sync their channels lists, other than that it was a good restart and the DAQ configuration is consistently in puppet.
Tue13Jun2023
LOC TIME HOSTNAME MODEL/REBOOT
12:31:30 h1iscex h1alsex
12:32:00 h1iscey h1alsey
12:33:45 h1daqdc0 [DAQ] 0-leg
12:33:55 h1daqfw0 [DAQ]
12:33:55 h1daqtw0 [DAQ]
12:33:56 h1daqnds0 [DAQ]
12:34:03 h1daqgds0 [DAQ]
12:35:11 h1daqgds0 [DAQ] gds0 2nd restart
12:36:53 h1daqdc1 [DAQ] 1-leg
12:37:02 h1daqfw1 [DAQ]
12:37:03 h1daqnds1 [DAQ]
12:37:03 h1daqtw1 [DAQ]
12:37:11 h1daqgds1 [DAQ]
12:38:06 h1daqgds1 [DAQ] gds1 2nd restart
DAQ Frame File Channel List Change
Two slow channels removed from the DAQ Frame today (name, size-bytes, data-rate-Hz)
H1:LSC-X_ARM_DRIVE_MASK 4 16
H1:LSC-Y_ARM_DRIVE_MASK 4 16
Per the checklist T1500386, I made a walk thru of the LVEA. Others were in other VEAs so, JIm did FCES, Ribert EY, and Tony EX.
Ran out of time working on end Y HEPI pump controller upgrade and did not get to WP 11250.
Wrapping up Tuesday Maintenance. Main activities were:
The temperature swing shown in this aLOG is relatively small (~ +/-0.4 deg F), entirely explained by the maintenance day activities on the HVAC system, and recovered to 67 +/- 0.2 deg F by 17:00 PDT. See LHO:70428.
The EY Beckhoff controller has the temporary name h1hpipumpctrley1 (10.105.0.64/24) and I have enabled port 14 of the EY VEA vacuum rack switch (sw-ey-aux) for this computer.
For archive: here are the original EY HEPI Pump Controller Settings.
and here are the CS settings for completeness
This morning Tyler, Chris and myself cleaned the strainer on coil 3 for the chilled water system. This strainer was not as bad as coil 4 which we cleaned 2 weeks ago. It did need cleaning and did improve the flow of the cooling coil. We plan to clean the other 4 strainers as time permits on Tuesdays. We also found the condensate drain plugged on AHU 1 again. We are working to unplug that drain now. The F/B damper on AHU 1 was closed completely so I have manually opened it to 70% for the time being to see if we can reduce some of the condensate from the coils. Coil 1 & 2 may need cleaning sooner than later if the condensate does not start going away soon.
AHU 2 Cooling Coil 3's strainer cleaning has had minimal impact on LVEA temperature value, nor fluctuations. Excellent! (Note, we've been calling these Fans, but I found out in talking with Bubba and Tyler today that this is a liquid strainer that filters the line that feeds the cooling coils. The *numbering* is still legit, they cleaned AHU2 Cooling Coil 4 strainer on May 30 2023 and today they cleaned AHU2 Coiling Coil 3 strainer) Separately, the change to have AHU 1 Damper at 60% is OK (Bubba had adjusted from 70% he mentions in his above aLOG to 60% shortly after posting; comments on its effect in the timeline below). After unplugging the drain of the air handler, and holding the damper open at 60% in "manual" mode (as opposed to servo controlled "auto" mode), the AHU 1 cooling coils 1 and 2 have now restored to much better/cooler, normal temperatures (~47 deg F) that we had lost by May 30th 2023 (the fateful fire-alarm chaos day; see timeline in LHO:70284). Excellent! LVEA temperature excursions in Zones 1A (this BSC2 / Beam Splitter), Zone 4 (Output Arm), and Zone 5 (Input Arm) never exceeded 0.4 deg F outside of 67 deg F range, and restored to normal ~67 deg F temperatures with small diurnal fluctuations within 7 hours. Excellent! Slowly but surely, I think these maintenance activities are good, and not only restoring expected system behavior, but making it better. We now have a much tighter collection of temperatures in the LVEA, cooling coils are operating at a nice low temperature, and more zone heaters are coming alive such that we have the expected "constant heat, constant cool in order to keep the LVEA temperature nice and controlled" behavior. I now have much more confidence that we can continue to do this kind of maintenance and not have it impact the IFO. Timeline of today's work all within today's Tuesday Maintenance: See Today's trend compared against a 7 day and 21 day trend. Jun 13 2023 08:26 PDT Bubba and Tyler start work, bringing down Air Handler 2, valving out AHU2's cooling coil 3's strainer and clean it. Understandably, all zones temperatures start to rise from 67 +/- 0.1 deg F but only max out 67.4 +/- 0.1 deg F. Also while out there, this is when they find that air handler 1 AHU 1 is flooded "again." Jun 13 2023 09:03 PDT Within a half hour, they're done with AHU2 strainer cleaning, AHU1 drain de-clogging, turn AHU2 back on, and temperatures begin to drop accordingly. Upon restoration, though, and back at the control room work station -- they still see AHU 1's cooling coil temperatures high (~ 61 deg F), as its been since May 30 2023. Indeed, also, the damper for AHU 1 is closed at 0%, as it has been doing diurnally since May 30 2023 -- the HVAC servo opens up AHU 1's damper each day (around when outside temperatures exceed 70 deg F), and then gradually ramps closed again by nightfall's temperature drop. Prior to May 30 2023, AHU 1 damper stayed around ~35% open throughout the day and night. Jun 13 2023 09:03 PDT Bubba intuits that there's too much condensation gathering in AHU 1 because its damper is closed too often, and that condensation doesn't get drained out because its drain is clogged. As a mitigation attempt, instead of having the HVAC servo drive the damper open percentage, he switches over to Manual mode and holds it at 70% as described in his aLOG. This drops both cooling coil's 1 and 2's temperature down from 61 to ~47 deg F. Awesome. But, this starts to scare the wimpy scientists (me) who are paying too close attention too quickly, because they see the temperatures in the LVEA drop below the 67 deg F set point. Jun 13 2023 10:02 PDT Zone 4 (input arm) zone heater turns on, and begins to bring the Zone 4 temperature back in line. Jun 13 2023 11:00 PDT Bubba makes a further adjustment of the AHU 1 damper percent open from 70% to 60% open to try to decrease the cooling in the LVEA. Jun 13 2023 12:07 PDT Zone 1A's zone heater collection turns on (human controlled? servo controlled?) for the first time, much like Zones 4 and 5 heaters came on for the first time after AHU2's cooling coil 4's strainer was cleaned, ramping up to ~70% in 5 minutes by 12:15 PDT. This really starts to turn around the LVEA temperatures for the better; the temperatures bottom out turn back up, overshoot a bit, and settle to yesterday's mid-day value. Jun 13 2023 17:00 PDT Temperatures are all restored within to ~67 +/- 0.2 deg F, and again, they never exceeded +/- 0.4 deg F. The "natural experiment" reveals the impulse response of the Zone 1A is about 5 hours, as long at the zone heater for that zone comes on!
LOCKLOSS @ 2:02, had a SRM saturation right before the lockloss. Seeing some movement in INP1 P and some LSC instability as well.
Looking at the WIND and the SRM, the wind seems to gusting between 10-17mph at the same frequency as the oscillations in the SRM starting at 1m,21s before lockloss.
Following up on EJ's alog, I have extended the code to check if the unresponsive filter is under local control by the model. I ran it this afternoon while H1 was in observe, running in 'nice' mode so as to not hammer the frontends with CA requests (delays between requests).
If we disregard locally controlled filters, the number of unresponsive filters is reduced to 3 (see attached). The top one runs on h1lsc, the other two on h1lscaux.
Here is the full list, with locally controlled filters marked with *
Total numer of filtermodules = 13367
num unresponsive 14
h1lsc {'LSC-EXTRA_AI_2': 'FM2 '}
h1lscaux {'LSC-LOCKIN_1_DEMOD_9_I': 'FM1 ',
'LSC-LOCKIN_1_DEMOD_9_Q': 'FM1 '}
h1sqz {'SQZ-RLF_VCXO_SERVO': 'FM2* '}
h1alsex {'LSC-X_ARM_DRIVE': 'FM3* FM5* '}
h1alsey {'LSC-Y_ARM_DRIVE': 'FM3* FM5* '}
h1sussqzin {'SUS-ZM1_M2_COILOUTF_LL': 'FM1* FM6* ',
'SUS-ZM1_M2_COILOUTF_LR': 'FM1* FM6* ',
'SUS-ZM1_M2_COILOUTF_UL': 'FM1* FM6* ',
'SUS-ZM1_M2_COILOUTF_UR': 'FM1* FM6* ',
'SUS-ZM3_M2_COILOUTF_LL': 'FM1* FM6* ',
'SUS-ZM3_M2_COILOUTF_LR': 'FM1* FM6* ',
'SUS-ZM3_M2_COILOUTF_UL': 'FM1* FM6* ',
'SUS-ZM3_M2_COILOUTF_UR': 'FM1* FM6* '}
Turned thses filters off:
h1lsc {'LSC-EXTRA_AI_2': 'FM2 '}
h1lscaux {'LSC-LOCKIN_1_DEMOD_9_I': 'FM1 ',
'LSC-LOCKIN_1_DEMOD_9_Q': 'FM1 '}
Added gain of 1 filters to:
h1sqz {'SQZ-RLF_VCXO_SERVO': 'FM2* '}
h1sussqzin {'SUS-ZM1_M2_COILOUTF_LL': 'FM1* FM6* ',
'SUS-ZM1_M2_COILOUTF_LR': 'FM1* FM6* ',
'SUS-ZM1_M2_COILOUTF_UL': 'FM1* FM6* ',
'SUS-ZM1_M2_COILOUTF_UR': 'FM1* FM6* ',
'SUS-ZM3_M2_COILOUTF_LL': 'FM1* FM6* ',
'SUS-ZM3_M2_COILOUTF_LR': 'FM1* FM6* ',
'SUS-ZM3_M2_COILOUTF_UL': 'FM1* FM6* ',
'SUS-ZM3_M2_COILOUTF_UR': 'FM1* FM6* '}
This should have no effect on anything.
Remove the controls from these filter modules:
h1alsex {'LSC-X_ARM_DRIVE': 'FM3* FM5* '}
h1alsey {'LSC-Y_ARM_DRIVE': 'FM3* FM5* '}
Turned these filters off after a model restart.
