We are power cycling the FMCS EPICS computer (fmcs-epics-cds) as a first try to regain stability.
After the first cell phone alarms were sent, I've bypassed them for a couple of hours.
Wed Jan 17 10:04:57 2024 INFO: Fill completed in 4min 53secs
TCs started high, trip temps were -60C for this fill.
Late entry. The activities carried out on 1-16: - 44" GVs: GV2 and GV7 closed nicely, without issues, however GV5 was a bit stubborn - it needed 50 psi and some time to close. The wiring was also messed up, Fil corrected it - Other GVs: the GVs of the relay tube (RV1, RV2); the GVs between HAM7 and BSC3 (FCV1, FCV2); the GVs along the FCT, after BSC3 (FCV3, FCV4); and the GVs before HAM8 (FCV7, FCV8) are closed - The RGAs for the corner (OMC, HAM6, HAM7) are being pumped. On 1-17 RGA scans. - The leaky HAM7 fiber feedthrough was leak checked, see details here: https://services1.ligo-la.caltech.edu/FRS/show_bug.cgi?id=30141 - The Kobleco is running, seemingly without any issues So far everything went as planned.
For the VAC team to bag and leak check the HAM7 leaky fiber feedthorugh FRS30141, yesterday afternoon I unplugged the HAM7 FC 532nm Fiber from the Feedthrough, covered with a plastic endcap and then replugged in when the VAC team was finished.
Tagging EPO for fiber feedthru pics.
Patrick, Jonathan, Erik Dave:
While the fmcs ioc continues to be unstable, I wrote an auto-restart script which restarts the IOC if its EPICS values flatline for more than 9 minutes.
In order to control the IOC code we moved it from a screen environment to a procServ, and converted the code to a systemd service.
The auto-restart code runs as david.barker on cdsmanager. Every minute it gets the value of the EX chiller yard water temperature channel H0:FMC-EX_CY_H2O_SUP_DEGF.
If the value of this channel does not change for 9 successive minutes, the code restarts the fmcs_ioc.service on fmcs-epics-cds
ssh root@fmcs-epics-cds 'systemctl restart fmcs_ioc.service'
I started the auto-restart code at 23:12 PST Tue night, since that time there have been 3 auto-restarts
Tue 16 Jan 2024 11:35:41 PM PST
Wed 17 Jan 2024 02:29:52 AM PST
Wed 17 Jan 2024 04:32:58 AM PST
Full details can be found in the wiki page h0fmcsbacnet
Attachment shows the three overnight auto-restarts, each preceded by 9 minute flatlines
TITLE: 01/17 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 2mph Gusts, 1mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.25 μm/s
QUICK SUMMARY:
Gerardo found that h0epics2, rack12 in the MSR, was beeping. The beeping later stopped, but the error light is now blinking at approx. 3 Hz.
Powersupplies are both green. I was able to log in. I couldn't find any errors on the system. The EDC has no disconnections.
TITLE: 01/16 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY:
The O4 Break officially started at the beginning of the shift!
Lots of activity today--miainly in prep for upcoming chamber incursions. No volumes were vented (this could possibly start tomorrow).
H1 taken to PLANNED ENGINEERING via Observatory Mode!
LOG:
Per the plan, the O4 break was paused today at both LHO and LLO in order to execute a pre-planned list of repairs, maintenance, updates, measurements, and commissioning tasks.
The schedule is shown in the attached PDF and is an estimate based on work loads and task duration.
Today, many preparations were made which push towards venting the isolatable volumes:
Corner chambers
HAM7 chamber
Relay Tube between HAM5 and HAM7
HAM8 chamber
EX
We plan to do inchamber work in HAM6, 7, 3, and 8, and port work on a few different chambers, some at EX.
FAMIS26165
These haven't been checked since middle of Dec. according to the paper logs. I added 350mL to TCSX and 180mL TCSY. Filters were in good shape.
Picket fence was updated to the latest version. As part of the update, the LAIR station was dropped for the DING station.
Excellent! I just checked the display and it looks as intended. Thank you very much Erik!
Ibrahim helped me lock HAM5 & 6 HEPIs this morning. Went as expected, biggest angular offsets after turning loops off were 5-7urad. I've also bypassed the HEPI isolation loops by setting HPI-HAM*_ISO_GAIN to 0 for each chamber, just in case someone tries to turn HEPI back on by mistake. Left the chambers with the ISI damped.
WP11622 In preparation for the vent, Betsy and I removed the two cameras and their housings from the HAM6 +X Door and placed them on ameristat on the cabinets in the SQZ bay.Viewports had guillotines inserted and yellow VP covers attached.
The larger box was from the left VP #3, cable labeled ISCT6_GIGE_CAMERA_2. The smaller box was from the right VP #5, cable labeled ISCT6_GIGE_CAMERA_1.
Tue Jan 16 10:04:24 2024 INFO: Fill completed in 4min 21secs
TCs started high, trip temps were raised to -60C for this fill.
At around 4 this morning I was notified that there was a leak at the LSB building. I arrived shortly after to find soaked carpet throughout the foyer, with standing water in some areas. The peripheral office spaces for Dolly and Bubba were also somewhat effected by the water on the floor as well as the front desk and some of the opposing hallway nearest the kitchen. After a thorough sweep of the entire ceiling spaces at the LSB and given the localization of the water, I am confident that the issue exists in one (partially) failed fire water system sprinkler head. At this time, to avoid an even larger disaster, I have decided to terminate power to the fire water jockey pumps such that if LSB fire water pressures fall below a threshold, the entire system does not dump. In addition, the site fire monitoring system has been moved to "test" for the duration of our normal operating hours. Kustom is presently on site to work on water damage mitigation at the LSB. C. Soike, E. Otterman, R. McCarthy, T. Guidry
This morning I ran the OPLEV charge measurements on both the ETMs before the vent so we can rerun them after the vent to compare.
ETMY's charge is rising on LL but seems fine on the other Quads. ETMX's charge is high but seems to be trending down on all Dofs/Quads except for UR_Pitch.
STATE of H1: Observing at 152Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 5mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.44 μm/s
QUICK SUMMARY:
IMC_LOCK Issue:
IMC_LOCK Is stuck in a loop, between Close_ISS (50) and LOCKED (100)
IMC_LOCK [CLOSE_ISS.run] USERMSG 0: Diffracted power jumped too much, toggling secondloop
Found Camilla's alog about this issue: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=71357
But it seems like it resolved it's self for her relatively quickly when she ran into this issue.
Tried to take IMC_LOCK to CLOSE_ISS and to To MCWFS_OFFLOADED but I still end up in the same loop.
Camilla Suggested maybe changing the IMC_lock code to allow for a bigger Diffracted Power Jump
Line 529 ,in IMC_LOCK.py, I wanted to try this, but was hesitant.
Called up Jason, Who then Mentioned that Keita would know more about the specifics of this particular issue.
Rang Keita, who said changing that line is likely not the real solution.
While we were making a plan to resolve the issue. It resolved it's self after about 2 hours of not working properly.
The issue was very likely: the electronic offset in the board has changed enough that the combination of the 2ndLoop REF Servo H1:PSL-(ISS_SECONDLOOP_REFERENCE_SERVO_OUT16) and 3rdLoop offset can no longer compensate for the changes it needs any longer. Tagging PSL team.
Note to other operators!:
If H1 has a lockloss, contact Jason to touch up the RefCav as this RefCav transmission is too low and liekly a contributing factor to the IMC issue.
FMCS Issues: The FMCS IOC keeps going down and thus the FirePump alerts are going off along with the Temperature sensors not giving us any data through MEDM or NDSCOPES.
Dave is aware of this and has been actively trying to find a way to automate the Restarting process of that IOC to limp us along until tomorrow. so far it's failed twice during this shift. at 9:47UTC and 10:45 UTC. If you have gotten an alert it is likely due to the FMCS IOC errors we have been having. Tagging CDS and FMCS.
See my annotation of infinite loop of 2nd loop enabling-disabling due to mismatch between the electronics offset and the 3rd loop offset. 1st loop is REALLY slow to respond to the change in the 2nd loop board DC output (because the 2nd loop output is added to the already whitened 1st loop sensor signal), which doesn't help either.
The way this works is that the diffraction average is measured just before the board output is enabled, then the guardian waits for 10 seconds (waiting for the reference servo to take care of any remaining electronics offset that the third loop offset could not counteract), enables the 2nd loop servo, wait for a while, measures the diffraction again, and if that's close enough to the original number it's satisfied.
In this case, it seems that the electronics offset drifted enough so 10 seconds is not quite enough for the reference servo to take care of that. The solution would be to tune the 3rd loop offset.
After the conclusion of O4a, I adjusted H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET (was originally 949, now 959.5).
The procedure is really simple.
1. Put the ISS 2nd loop in the configuration shown in the 1st screenshot. Power into IMC doesn't matter, this is only about electronics, but it's better if the 1st loop is working so that you can confirm that your work is not doing any harm. (When I came into the control room the ISS was already in a good state to start the work.)
2. Confirm that H1:PSL-ISS_SECONDLOOP_EXC_MON is not changing much at DC by making a trend. If it's still trending up/down, wait for two minutes. Ignore the change of 0.1/minute, we're talking more about 1/minute.
3. Read H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET value (it was 949). Add the average DC value of H1:PSL-ISS_SECONDLOOP_EXC_MON (was about 11) to make a new number (i.e. 960 in this case).
4. If this is to be done while the IFO is locked, you might want to set H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET_TRAMP to 100. I did that even though it wasn't necessary in this case just to show that doing so will make this procedure almost transparent to the 1st loop (2nd attachment, t~-40min, see the 1st loop readout of the 2nd loop output and the diffraction).
5. Put the new number (960 in this case) into H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET.
6. Wait for ~2 minutes to see that H1:PSL-ISS_SECONDLOOP_EXC_MON goes down to less than 1. You can also fine-adjust it, but it will drift after some time. If this is less than 1 it's already pretty good.
7. If you changed H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET_TRAMP to 100, bring it back to 3.
8. Be happy.
Jonathan, Patrick, Dave:
The FMCS IOC computer is back online. The restart code is running again. The cell phone alarm bypass has been removed.