We have moved FC1, ZM1, ZM2, ZM3 and OPO suspensions into SAFE state until the CDS issues are resolved.

All suspensions in HAM7 are now back online and into ALIGNED state after Fil reset the DC power supply and Dave restarted the computers.

There's an issue with the IOP for HAM7, dackill and FC1, after talking with Jim I tried to call for CDS help but wasn't able to get anyone (left voicemail). FC1s medm screen appears frozen but when you ndscope OSEM channels they're all oscillating rapidly. We're not able to get in Observing with this current issue, we need a CDS person to look into it.

Explanation of the issue in 73012. As we are now in observing and these are not important channels, we will fix tomorrow.

Camilla confirms it is OK to run without these channels overnight, they will be added back tomorrow when H1 is out of observe.

Camilla installed the new HWS ITMY code at 08:22 while H1 was out of observe. EDC is now green.

I had a chance to test it out a bit more thoroughly after a lock loss around 130 PT. This time I forced it into a situation where it would overshoot resonance and have a very small amount of light in the signal. It then jumped back and got close enough for the fine tuning threshold to move the node into the next state to fine tune.

This situation has been the most frequent cause of the node not finding IR resonance this year, so hopefully this helps reduce assistance required requests. Next steps for this node are to make a "long search" state that will search through the full range for good buildups, and fix the times that it will fine tune indefinitely, circling around the correct value. This latter issue should just be a minor tuning change, but it's challenging to test.

No need to restart the Picket Fence process, after almost exactly 20 minutes at 09:53 PDT the service sprung back to life and is updating normally now with nothing done at our end.

Good to know that the automatic restart is working! Also, very nice detailed interface, I'm a big fan.

It's a bit awkward that it took almost 20 minutes to come back again. I'd prefer it to be closer to 5 minutes, will check the code to see if there's anything amiss. There were no alerts/logs on the copy we're running at Stanford, so I assume this is not a problem from the USGS side.

New code now running on all optics. Took new references on ITMs and ETMX. 

New ETMX reference taken at 16:21UTC with ALS shuttered, after IFO had been down for 90 minutes.  Installed at ITMX, new reference taken at 16:18UTC. Installed at ITMY, new reference taken at 16:15UTC. realized where I previously edited code in /home/controls/hws/HWS/ I should have just pulled the new fix/python3 HWS code I went back did this for ITMs and ETMX. 

ITMX SLED power has quickly decayed, see attached, so I adjusted the frame rate form 5Hz to 1Hz, instructions in TCS wiki page. This SLED was recently replaced 71476 and was a 2021 SLED so not particularly old. LLO has seen similar issues 66713 but over hours rather than weeks. We need to deicide what to do about this, we could try to touch the trimpoint or contact the company. ITMY SLED is fine.

There was two separate errors on ITMY that stopped the code in the first few minutes, a "segmentation fault" and a "fatal IO error 25". We should watch that this code continues the run without issues.

ITMY code keeps stopping and has been stopped for last few hours. I cannot access the computer, maybe it has crashed? There is an orange light on h1hwsmsr.

ITMX spherical power is very noisy. This appears to be because the new ITMX reference has a lot of dead pixels, I've noted them but haven't yet added them to the dead pixels files as cannot edit the read-only file- TCS wiki link.

ITMY- TJ and I restarted h1hwsmrs1 in the MSR and I restarted the code. We were getting a regular "out of bounds" error, attached, but the data now seems to be running fine. The fix/python3 code didn't have all the master commits in it so when we restarted mrs1 some of the channels were not restarted, found by Dave in 73013. I've updated the fix/python3 code and we should pull this commit and kill/restart the ioc and hws itmy code tomorrow.

ITMX - the bad pixels were added and the data is now mcuh cleaner, updated insruciton in the TCS wiki.

Pulled new python3 code with all channels to both ITM and ETMX computers (already on ETMY). Killed and restart softIoc on h1hwsmsr1 for ITMY following instructions in 65966. The 73013 channels are now running again.

Took new references on all optics at 20:20UTC after the IFO had been unlocked and CO2 lasers off for 5 hours. RH settings nominal IX 0.4W, IY 0.0W, EX 1.0W, EY 1.0W.

Thermistors are working.

There is an OSEM pitch shift of OM2 at the end of the maintenance period 3 weeks ago (Aug 29)

Having turned the heater back on will likely affect our calibration.  It's not a bad thing, but it is something to be aware of.

Indeed it now seems that there is a ~5Mpc difference in the range calculations between the front-ends (SNSW) and GDS (SNSC) compared to our last observation time.

It looks like this has brought back the 1.66 Hz comb. Attached is an averaged spectrum for 6 hours of recent data (Sept 20 UTC 0:00 to 6:00); the comb is the peaked structure marked with yellow triangles around 280 Hz. (You can also see some peaks in the production Fscans from the previous day, but it's clearer here.)

To see if one of the Beckhoff terminals for thermistors is kaput, I disconnected thermistor 2 (pins 9, 11, 22 and 24) from Beckhoff at the back of the heater driver chassis.
For a short while the Beckhoff cable itself was disconnected but the cable was connected back to the breakout board at the back of the driver chassis by 20:05:00 UTC.

Thermistor 1 is still connected. Heater driver input is still receiving voltage from the voltage reference.

I checked a 3-hour span starting at 04:00 UTC today (Sept 21) and found something unusual. There is a similar structure peaked near 280 Hz, but the frequency spacing is different. These peaks lie on integer multiples of 1.1086 Hz, not 1.6611 Hz. Plot attached.

Detchar, please see if there's any change in 1.66Hz comb.

At around 21:25 UTC, I disconnected OM2 thermistor 1 (pins 10, 12, 23, 25 of the cable at the back of the driver chassis) from Beckhoff and connected thermistor 2 (pins 9, 11, 22, 24).

Checked 6 hours of data starting at 04:00 UTC Sept 22. The comb structure persists with spacing 1.1086 Hz.

Electrical grounding of the beckhoff systems has been modified as a result of this investigation -- see LHO:73233.

Corroborating Daniel's statement that the OM2 heater power supply was disrupted on Tuesday Aug 29th 2023 (LHO:72970), I've zoomed in on the pitch *OSEM* signals for both 
    (a) when the suspected time of power distruption (first attachment), and 
    (b) when the power and function of OM2 was restored (second attachment).

One can see that upon power restoration and resuming the HOT configuration of TSAMS on 2023-09-19 (time (a) above), OM2 pitch *decreases* by 190 [urad] over the course of ~1 hour, with a characteristic "thermal time constant" exponential shape to the displacement evolution.

Then, heading back to 2023-08-29, we can see a similar shaped event that causes OM2 pitch to *increase* by 160 [urad] over the course of ~40 minutes (time (b) above). Then at 40 minutes IFO recovery from maintenance begins, and we see the OM2 pitch *sliders* adjusted to account for the new alignment, as had been done several times before with the OM2 ON vs. OFF state changes.

I take this to be consistent with:
    The OM2 TSAMS heater was inadvertently turned OFF and COLD on 2023-08-29 at 18:42 UTC (11:42 PDT), and 
    The OM2 TSAMS heater was restored turned ON and HOT on 2023-09-19 18:14 UTC (11:14 PDT).