Olli, Ryan, thanks for noticing this computer by the Xend Pcal system.  While it happens to be located nearby, and we may have used it years ago, we (Pcal team) have not used that computer for some time and don't plan to use it in the future.  As far as I know it is currently in an inoperable state, having been replaced by the CDS laptops that we carry down to the end stations from the corner station.

Thanks Oli. The power supply under the HWS table remains on as it powers the HWS cameras. Moved from in rack power in 2018: 44948 44855.

Tue03Oct2023
LOC TIME HOSTNAME     MODEL/REBOOT
11:30:51 h1daqdc0     [DAQ]  <<< 0-leg restart
11:31:05 h1daqfw0     [DAQ]
11:31:06 h1daqnds0    [DAQ]
11:31:06 h1daqtw0     [DAQ]
11:31:14 h1daqgds0    [DAQ]
11:36:24 h1daqfw0     [DAQ] <<< FW0 2nd restart
11:39:49 h1daqnds0    [DAQ] <<< NDS 2nd restart

11:42:08 h1susauxb123 h1edc[DAQ] <<< EDC restart

11:44:40 h1daqdc1     [DAQ] <<< 1-leg restart
11:44:48 h1daqfw1     [DAQ]
11:44:49 h1daqtw1     [DAQ]
11:44:50 h1daqnds1    [DAQ]
11:44:58 h1daqgds1    [DAQ]
11:47:04 h1daqfw1     [DAQ] <<< FW1 2nd restart
11:47:50 h1daqgds1    [DAQ] <<< GDS1 2nd restart (chan list)
11:49:05 h1daqnds1    [DAQ] <<< NDS1 2nd restart

12:17:28 h1daqfw1     [DAQ] <<< FW1 spontaneous restart
 

DAQ Frame File Channel Changes

Fast channels: no additions nor removals

Slow channels:

no channels removed

3,319 channels added.

I accepted the SDF of the fudge factors of homodyne to go to observe.

Tagging DetChar -- with the CW group in mind. 
After this maintenance day, there might be a change in comb behavior as a result of this work.

This is action taken as a result of Keita, Daniel, and Ansel's work on ID'ing combs from the OM2 heater system -- see LHO:72967.

After Fil was done with the grounding work, I temporarily restored the connection between the beckhoff cable and the heater chassis and used a normal breakout board to measure the voltage between the driver ground (pin13) and the positive drive voltage (pin 6) of D2000212, just like I did on Aug 09 2023 (alog 72061).

1st attachment is today, 2nd attachment is on Aug 09. I see no improvement (OK, it's better by ~1dB today).

After seeing this, I swapped the breakout board back to the switchable one I've been using to connect only a subset of pins (e.g. only thermistor 1). This time, there's no electrical connection between any pins but the cable was physically attached to the breakout board. No connection between the cable shell and the chassis connector shell either. I expect that the comb will be gone, but I'd like detchar to have a look.

The heater driver is driven by the voltag reference on the nearby table, not Beckhoff.

J. Oberling, R. Short

Checking on the laser after a few hours of warm up, I found the cooler to be very warm, and the box housing the DC-DC converter that powers the laser (steps ~11 VDC down to 5 VDC) was extremely warm.  Also, the SUM counts had dropped from the ~7k we started at to ~1.1k.  Seeing as how we just installed a new laser, my suspicion was that the DC-DC converter was failing.  Checking the OpLev power supply in the CER it was providing 3A to the LVEA OpLev lasers; this should only be just over 1A, which is further indication something is up.  Ryan and I replaced the DC-DC converter with a spare.  Upon powering up with the new converter the current delivered by the power supply was still ~3A, so we swapped the laser with another spare.  With the new laser the delivered current was down to just over 1A, as it should be.  The laser power was set so the SUM counts are still at ~7k, and we will keep an eye on this OpLev over the coming hours/days.  Both lasers SN 191-1 and SN 119-2 will be tested in the lab; my suspicion is that the dying DC-DC converter damaged both lasers and they will have to be repaired by the vendor, will see what the lab testing says.  New laser SN is 199-1.

Noticing as the night progresses, the sum counts are slowly going up, starting from ~6200 and now ~7100. Odd.

ITMX OPLEV sum counts are at about 7500 this morning.

Sum counts around 7700 this morning, they're still creeping up

Instead of disabling the crontab, we have decided to reschedule it to run at 05:01 each morning:

01 05 * * * /ligo/home/controls/restart_weather > /dev/null
 

For the 14:41 lockloss, the signals look fine on the lockloss select tool scopes. The violins also look fine, they were all trending downwards. The OMC_DCPD signals looked interesting, they seem to have been diverging before the lockloss.

Right after lockloss DIAG_MAIN was showing:

- PSL_ISS: Diffracted power is low

- OPLEV_SUMS: ETMX sums low

05:41UTC LOCKING_ARMS_GREEN, the detector couldn't see ALSY at all and I noticed the ETM/ITMs L2 saturations(attachment1-L2 ITMX MASTER_OUT as example) on the saturation monitor(attachment2) and noticed the ETMY oplev moving around wildly(attachment3)

05:54 I took the detector to DOWN, and immediately could see ALSY on the cameras and ndscope; L2 saturations were all still there

05:56 and 06:05 I went to LOCKING_ARMS_GREEN again and GREEN_ARMS_MANUAL, respectively, but wasn't able to lock anything - I was able to go to LOCKING_ARMS_GREEN and GREEN_ARMS_MANUAL without the saturations getting too bad but both ALSX and Y evenually locked for a few seconds each and then unlocked and gone to basically 0 on the cameras and ndscope(attachment4). ALSX and Y sometime after this went to FAULT and were giving the messages "PDH" and "ReflPD A"

06:07 Tried going to INITIAL_ALIGNMENT but Verbal kept calling out all of the optics as saturating, ALIGN_IFO wouldn't move past SET_SUS_FOR_ALS_FPMI, and the oplev overview screen showed ETMY, ITMX, and ETMX moving all over the place (I was worried something would go really wrong if I left it in that configuration)

I tried waiting a bit and then tried INITIAL_ALIGNMENT and LOCKING_ARMS_GREEN again but had the same results.

Naoki came on and thought the saturations might be due to the camera servos, so he cleared the camera servo history and all of the saturations disappeared (attachment5) and an INITIAL_ALIGNMENT is now running fine.

Not sure what caused the saturations in the camera servos, but it'll be worth looking into whether the saturations are what ended up causing this lockloss.

As shown in the attached figure, the output of camera servo for PIT suddenly became 1e20 and the lockloss happened after 0.3s. So this lockloss is likely due to the camera servo. I checked the input signal of camera servo. The bottom left is the input of PIT1 and should be same as the BS camera output (bottom right). The BS camera output itself looks OK, but the input of PIT1 became red dashed and the output became 1e20. The situation is same for PIT2, 3. I am not sure why this happened.

Opened FRS29216 . It looks like a transient NaN on ETMY_Y camera centroid channel caused a latching of 1e20 on the ASC CAM_PIT filter modules outputs.

Camilla, Oli

We trended the ASC-CAM_PIT{1,2,3}_INMON/OUTPUT channels alongside ASC-AS_A_DC_NSUM_OUT_DQ(attachment1) and found that according to ASC-AS_A_DC_NSUM_OUT_DQ, the lockloss started(cursor 1) ~0.37s before PIT{1,2,3}_INMON read NaN(cursor 2), so the lockloss was not caused by the camera servos.

It is possible that the NaN values are linked to the light dropping off of the PIT2 and 3 cameras right after the lockloss, as when the cameras come back online ~0.2s later, both the PIT2 and PIT3 cameras are at 0, and looking back over several locklosses it looks like these two cameras tend to drop to 0 between 0.35 and 0.55s after the lockloss starts. However the PIT1 camera is still registering light for another 0.8s after coming back online(typical time for this camera).

Patrick updated the camera server to solve the issue in alog73228.

Thermistors are working.

There is an OSEM pitch shift of OM2 at the end of the maintenance period 3 weeks ago (Aug 29)

Having turned the heater back on will likely affect our calibration.  It's not a bad thing, but it is something to be aware of.

Indeed it now seems that there is a ~5Mpc difference in the range calculations between the front-ends (SNSW) and GDS (SNSC) compared to our last observation time.

It looks like this has brought back the 1.66 Hz comb. Attached is an averaged spectrum for 6 hours of recent data (Sept 20 UTC 0:00 to 6:00); the comb is the peaked structure marked with yellow triangles around 280 Hz. (You can also see some peaks in the production Fscans from the previous day, but it's clearer here.)

To see if one of the Beckhoff terminals for thermistors is kaput, I disconnected thermistor 2 (pins 9, 11, 22 and 24) from Beckhoff at the back of the heater driver chassis.
For a short while the Beckhoff cable itself was disconnected but the cable was connected back to the breakout board at the back of the driver chassis by 20:05:00 UTC.

Thermistor 1 is still connected. Heater driver input is still receiving voltage from the voltage reference.

I checked a 3-hour span starting at 04:00 UTC today (Sept 21) and found something unusual. There is a similar structure peaked near 280 Hz, but the frequency spacing is different. These peaks lie on integer multiples of 1.1086 Hz, not 1.6611 Hz. Plot attached.

Detchar, please see if there's any change in 1.66Hz comb.

At around 21:25 UTC, I disconnected OM2 thermistor 1 (pins 10, 12, 23, 25 of the cable at the back of the driver chassis) from Beckhoff and connected thermistor 2 (pins 9, 11, 22, 24).

Checked 6 hours of data starting at 04:00 UTC Sept 22. The comb structure persists with spacing 1.1086 Hz.

Electrical grounding of the beckhoff systems has been modified as a result of this investigation -- see LHO:73233.

Corroborating Daniel's statement that the OM2 heater power supply was disrupted on Tuesday Aug 29th 2023 (LHO:72970), I've zoomed in on the pitch *OSEM* signals for both 
    (a) when the suspected time of power distruption (first attachment), and 
    (b) when the power and function of OM2 was restored (second attachment).

One can see that upon power restoration and resuming the HOT configuration of TSAMS on 2023-09-19 (time (a) above), OM2 pitch *decreases* by 190 [urad] over the course of ~1 hour, with a characteristic "thermal time constant" exponential shape to the displacement evolution.

Then, heading back to 2023-08-29, we can see a similar shaped event that causes OM2 pitch to *increase* by 160 [urad] over the course of ~40 minutes (time (b) above). Then at 40 minutes IFO recovery from maintenance begins, and we see the OM2 pitch *sliders* adjusted to account for the new alignment, as had been done several times before with the OM2 ON vs. OFF state changes.

I take this to be consistent with:
    The OM2 TSAMS heater was inadvertently turned OFF and COLD on 2023-08-29 at 18:42 UTC (11:42 PDT), and 
    The OM2 TSAMS heater was restored turned ON and HOT on 2023-09-19 18:14 UTC (11:14 PDT).