Picket fence was updated to the latest version. As part of the update, the LAIR station was dropped for the DING station.
Ibrahim helped me lock HAM5 & 6 HEPIs this morning. Went as expected, biggest angular offsets after turning loops off were 5-7urad. I've also bypassed the HEPI isolation loops by setting HPI-HAM*_ISO_GAIN to 0 for each chamber, just in case someone tries to turn HEPI back on by mistake. Left the chambers with the ISI damped.
WP11622 In preparation for the vent, Betsy and I removed the two cameras and their housings from the HAM6 +X Door and placed them on ameristat on the cabinets in the SQZ bay.Viewports had guillotines inserted and yellow VP covers attached.
The larger box was from the left VP #3, cable labeled ISCT6_GIGE_CAMERA_2. The smaller box was from the right VP #5, cable labeled ISCT6_GIGE_CAMERA_1.
Tue Jan 16 10:04:24 2024 INFO: Fill completed in 4min 21secs
TCs started high, trip temps were raised to -60C for this fill.
At around 4 this morning I was notified that there was a leak at the LSB building. I arrived shortly after to find soaked carpet throughout the foyer, with standing water in some areas. The peripheral office spaces for Dolly and Bubba were also somewhat effected by the water on the floor as well as the front desk and some of the opposing hallway nearest the kitchen. After a thorough sweep of the entire ceiling spaces at the LSB and given the localization of the water, I am confident that the issue exists in one (partially) failed fire water system sprinkler head. At this time, to avoid an even larger disaster, I have decided to terminate power to the fire water jockey pumps such that if LSB fire water pressures fall below a threshold, the entire system does not dump. In addition, the site fire monitoring system has been moved to "test" for the duration of our normal operating hours. Kustom is presently on site to work on water damage mitigation at the LSB. C. Soike, E. Otterman, R. McCarthy, T. Guidry
This morning I ran the OPLEV charge measurements on both the ETMs before the vent so we can rerun them after the vent to compare.
ETMY's charge is rising on LL but seems fine on the other Quads. ETMX's charge is high but seems to be trending down on all Dofs/Quads except for UR_Pitch.
Tyler reported there was a burst water line in the LSB. The source is a Fire water line in the ceiling (source is near the foyer/lobby main entrance. Kustom Disaster Recovery contractors are on site.
Currently, all potable water supply (i.e. toilets, sinks, etc) has been restored.
TITLE: 01/16 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 3mph Gusts, 2mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.26 μm/s
QUICK SUMMARY:
1548utc/748am H1 had a lockloss effectively ending Observing for Observing Run #4 Part A (i.e. O4a)! We now begin a few weeks of detector work, commissioning, an engineering run and return to Observing with O4c. Jenne was working on H1 remotely this morning to help us transition. Will keep H1 in PREP FOR LOCKING.
It is also Maintenance Day!
Since we lost lock a few mins before 8am (during the in-lock charge measurements), I've set the IFO to DOWN.
Happy end-of-O4a, and congratulations to all on making it such a success!
The O4 break begins this morning, in preparation for an even-better O4b!
The in-lock charge measurements didn't directly cause the 15:48UTC lockloss, lockloss happened during the third injection "L_DRIVE_NO_BIAS" on ITMX, ITMY, ETMY.
Workstations were updated and rebooted. This was an OS packages update. Conda packages were not updated.
TITLE: 01/16 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: The rest of relocking today from when I arrived went smoothly, and we have been Observing since then and have now been Locked for 7.5 hours. The range is still super glitchy and throughout my shift it felt like the DARM FOM was constantly jumping.
I wasn't able to find any reasons for why we've been so glitchy these past few days, but I did notice that hveto had found some auxiliary channels and had used them to veto quite a few triggers (Jan14, Jan15). Namely, on Jan14th H1:CAL-PCALY_RX_PD_OUT_DQ, on Jan15th H1:SUS-ETMY_L3_OPLEV_SUM_OUT_DQ, and on both days H1:PEM-EX_VMON_ETMX_ESDPOWER48_DQ. Tagging detchar
LOG:
00:00 Detector Relocking and at POWER_10W
00:38 NOMINAL_LOW_NOISE
00:44 Observing
Locked for 4 hours and are Observing at 152Mpc
TITLE: 01/16 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
Started relocking Since Jason is done touching up the Ref Cav and gave Oli the updates about the restart the FMC IOC procedure.
Oli is now logged in to fmcs-epics-cds and we went through a practise run to make sure we have the procedure correct.
Dave and I have decided that The last two OPs shifts can handle making sure the FMCS IOC is running manually, and he was going to make a CRON job that restarts it every hour for the OWL shift until Patrick get here in the morning.
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 23:06 | PSL | Jason | Remote | N | Tuning the RefCav and ISS | 23:40 |
TITLE: 01/16 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.35 μm/s
QUICK SUMMARY:
Detector is relocking and in POWER_10W. Wind is low.
Saw the FSS RefCav TPD was down to ~0.44V with the IMC locked, so had asked Tony to let me know if we lost lock so I could tweak the alignment; after the most recent lockloss I took the opportunity (with LLO down for testing) to do this. We had been seeing this drop over the last several days (coinciding with the rapid drop in outside temperature). Jenne performed a remote beam alignment tweak on Friday.
When I started the RefCav TPD was ~0.45V with the IMC unlocked. I was able to get it up to ~0.53V, but could get no more out of it. Tony has begun relocking, hopefully this gets us to tomorrow's start of the O4 break. We'll have to take some time during the commissioning break to get in the PSL enclosure and adjust the FSS beam alignment on the table.
While doing this, I noticed the ISS diffraction % was lower than we tend to like it. It was hovering just below 2%, so I adjusted the ISS RefSignal from -1.94 V to -1.95 V; this brought the diffraction % to ~2.5%, where we like to run it.
Lockloss : https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1389394423
Wind, Primary and Secondary microSeism were all low for this lockloss.
Jason wanted to adjust the Refference Cavity if we had a lockloss today so I'm holding us in Idle while he is taking a moment to touch that up remotely.
11/16 00:44UTC Back to Observing
STATE of H1: Observing at 152Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 5mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.44 μm/s
QUICK SUMMARY:
IMC_LOCK Issue:
IMC_LOCK Is stuck in a loop, between Close_ISS (50) and LOCKED (100)
IMC_LOCK [CLOSE_ISS.run] USERMSG 0: Diffracted power jumped too much, toggling secondloop
Found Camilla's alog about this issue: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=71357
But it seems like it resolved it's self for her relatively quickly when she ran into this issue.
Tried to take IMC_LOCK to CLOSE_ISS and to To MCWFS_OFFLOADED but I still end up in the same loop.
Camilla Suggested maybe changing the IMC_lock code to allow for a bigger Diffracted Power Jump
Line 529 ,in IMC_LOCK.py, I wanted to try this, but was hesitant.
Called up Jason, Who then Mentioned that Keita would know more about the specifics of this particular issue.
Rang Keita, who said changing that line is likely not the real solution.
While we were making a plan to resolve the issue. It resolved it's self after about 2 hours of not working properly.
The issue was very likely: the electronic offset in the board has changed enough that the combination of the 2ndLoop REF Servo H1:PSL-(ISS_SECONDLOOP_REFERENCE_SERVO_OUT16) and 3rdLoop offset can no longer compensate for the changes it needs any longer. Tagging PSL team.
Note to other operators!:
If H1 has a lockloss, contact Jason to touch up the RefCav as this RefCav transmission is too low and liekly a contributing factor to the IMC issue.
FMCS Issues: The FMCS IOC keeps going down and thus the FirePump alerts are going off along with the Temperature sensors not giving us any data through MEDM or NDSCOPES.
Dave is aware of this and has been actively trying to find a way to automate the Restarting process of that IOC to limp us along until tomorrow. so far it's failed twice during this shift. at 9:47UTC and 10:45 UTC. If you have gotten an alert it is likely due to the FMCS IOC errors we have been having. Tagging CDS and FMCS.
See my annotation of infinite loop of 2nd loop enabling-disabling due to mismatch between the electronics offset and the 3rd loop offset. 1st loop is REALLY slow to respond to the change in the 2nd loop board DC output (because the 2nd loop output is added to the already whitened 1st loop sensor signal), which doesn't help either.
The way this works is that the diffraction average is measured just before the board output is enabled, then the guardian waits for 10 seconds (waiting for the reference servo to take care of any remaining electronics offset that the third loop offset could not counteract), enables the 2nd loop servo, wait for a while, measures the diffraction again, and if that's close enough to the original number it's satisfied.
In this case, it seems that the electronics offset drifted enough so 10 seconds is not quite enough for the reference servo to take care of that. The solution would be to tune the 3rd loop offset.
After the conclusion of O4a, I adjusted H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET (was originally 949, now 959.5).
The procedure is really simple.
1. Put the ISS 2nd loop in the configuration shown in the 1st screenshot. Power into IMC doesn't matter, this is only about electronics, but it's better if the 1st loop is working so that you can confirm that your work is not doing any harm. (When I came into the control room the ISS was already in a good state to start the work.)
2. Confirm that H1:PSL-ISS_SECONDLOOP_EXC_MON is not changing much at DC by making a trend. If it's still trending up/down, wait for two minutes. Ignore the change of 0.1/minute, we're talking more about 1/minute.
3. Read H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET value (it was 949). Add the average DC value of H1:PSL-ISS_SECONDLOOP_EXC_MON (was about 11) to make a new number (i.e. 960 in this case).
4. If this is to be done while the IFO is locked, you might want to set H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET_TRAMP to 100. I did that even though it wasn't necessary in this case just to show that doing so will make this procedure almost transparent to the 1st loop (2nd attachment, t~-40min, see the 1st loop readout of the 2nd loop output and the diffraction).
5. Put the new number (960 in this case) into H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET.
6. Wait for ~2 minutes to see that H1:PSL-ISS_SECONDLOOP_EXC_MON goes down to less than 1. You can also fine-adjust it, but it will drift after some time. If this is less than 1 it's already pretty good.
7. If you changed H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET_TRAMP to 100, bring it back to 3.
8. Be happy.
TITLE: 01/14 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Struggling with violins and relocking all day, but we were able to reach NOMINAL_LOW_NOISE twice, but lost lock the first time before we could go into Observing (75381). Currently in NLN and Vicky and Daniel are working on getting the SQZ angle to behave because it keeps trying to move towards 0.
Violin modes to keep an eye on: ITMY05/06, ITMY08, ETMY01, ETMY20 (Rahul's alog with some values that work: 75378)
LOG:
00:00UTC Detector Relocking and in MAX_POWER trying to damp violins
03:24 Continuing up towards NLN
03:36 Reached OMC_WHITENING
04:48 Went into NOMINAL_LOW_NOISE
05:10 Lockloss from NOMINAL_LOW_NOISE (Not Observing) (75381)
05:30 Lockloss from LOCKING_ALS
05:41 Lockloss from CHECK_MICH_FRINGES, starting an Initial Alignment
06:10 Initial Alignment completed, relocking
06:56 Reached MAX_POWER, pausing to damp violins
07:42 Reached NOMINAL_LOW_NOISE
- Had to realign FC2 (72084)
08:11 OBSERVING!
08:15 Dropped out of Observing to troubleshoot SQZ Angle issue
Some SQZ troubleshooting notes:
H1 is now back to observing at 156Mpc following the SQZ troubleshooting described above as of 08:48 UTC.
There is notes on getting FC to lock in the Troubleshooting SQZ wiki. There were lots of LVEA temperature swings this weekend, probably causing trouble in FC alignment.
I'm unsure why H1:SQZ-ADF_OMC_TRANS_PHASE would become unstable when it previously worked for 6+ locks, plot. We could again look at moving the position of the ADF line, but couldn't get the servo stable in 75250.
The first unstable incident was from SQZ_ANG_ADJUST adjusting the angle from 170 to 115deg before the FC had locked, plot. SQZ_MANAGER shouldn't request SQZ_ANG_ADJUST to ADJUST_SQZ_ANG_ADF until it's in FDS and the FC is locked at LOCKED_IR, however at 2024-01-14 07:48:01 UTC it was requested, when SQZ_MANGER was still in SQZ_READY_IFO. This looks like someone requested it so wouldn't happen by itself.
Excellent! I just checked the display and it looks as intended. Thank you very much Erik!