TITLE: 11/28 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

1. 16:35 UTC - Dave noticed an EY GPS timing error on the CDS overview that was flashing. The PPS A GPS clock was flashing an error due to a timing difference past tolerance. Error disappeared after 16:40 UTC.
2. 17:00 UTC SNEWS alert test T456241 caught on both Diag main and verbal
3. 18:11 UTC HAM7 Watchdog trip, likely due to SQZ rack work that Fil is doing (pulling cables).
4. 18:46 UTC IOC server temp change: While Erik was updating the IOC LVEA temperature, he realized that the entire IOC server needed to be rebooted.
   1. IOC server reboot happened at 18:46 UTC and connection was reestablished at 18:49 UTC.
   2. IOC reboot temporarily took down:
      1. Remote power
      2. Check violins
      3. Wi-Fi
      4. Ops GraceDB Standown - did not restart as expected, TJ turned it back on
      5. Picket Fence - did not restart as expected prompting further investigation by Erik - 
   3. Wi-Fi was separately turned on during reboot in order to maintain control room work.
   4. 19:09 UTC Erik updating IOC temp. Done at 19:16 UTC
5. Two GRB Short Alerts
   1. 19:25 UTC
   2. 19:37 UTC
6. 19:00 UTC (ish) Temp excursion in CER and SUP Rack 1 - still within “nominal/tolerance” but visibly on the come-up. 
   1. 20:10 UTC Eric investigated and found that the lead unit had a fault and so he reset the system. Temperatures visibly on the come-down, both in CER and SUP.
7. 19:46 UTC - during Beckhoff work, 
   1. Beckhoff restart caused a connection error with guardian ALSY node. Guardian 1 seems to have lost connection, despite Beckhoff showing everything as normal. There are 7 SPM diffs, all pertaining to H1:ALSY (and nothing else). 
   2. Apparently it fixed itself at 19:46 UTC and then the same issue happened at 20:27 UTC
   3. Error message reads as: “CONNECTION ERRORS, see SPM DIFFS for dead channels". See screenshot.
   4. The workstations have no problems and everything else seems normal.
   5. Dave and Daniel suggested doing a Guardian reboot, though Dave is looking into it and seeing if it will/won’t help.
   6. 20:45 UTC: This is currently stopping us from locking
   7. 20:51 (ish) UTC - TJ fixed the issue by bringing the guardian node to STOP and then EXEC, instead of reloading
      1. Apparently this is an issue where sometimes when connections are lost and reconnected a complete STOP rather than a PAUSE/RELOAD is necessary to bring the connection back.
      2. No Guardian reboot necessary
8. DAQ Restart: NUC23 isn’t loading and is yielding an error message - Ryan C couldn’t get into it during the previous maintenance Tuesday.
   1. Ryan C is power cycling it and then attempting to connect
   2. Worked fine after restart
9. Locking at 20:53 UTC 
   1. Starting with an initial alignment - Initial Alignment went fine
      1. Had to touch ALS_Y Yaw to get it caught
10. Lock acquisition 1 at 21:48 UTC: Lock loss due to DRMI lockloss at Turning_BS_Stage2
    1. Caused by DRMI_ASC issue that Sheila caught and is working on.
11. Lock acquisition 2 at 21:52 UTC: Sheila and Naoki - DRMI_ASC won’t work until they fix something so guardian only taken to DRMI_Locked_Prep_ASC
    1. Sheila said it was potentially fixed -
    2. Lockloss at the same stage again 21:57 UTC
12. Lock acquisition 3 at 21:59 UTC
    1. IMC put to DOWN at 22:00 UTC while Sheila and Naoki investigate the DRMI ASC issues
    2. Back to lock acquisition at 22:17 UTC
    3. Paused at DRMI_LOCKED_PREP_ASC - Sheila and Naoki noticed ASC SRC1 Pitch and Yaw offset(s) were off by orders of magnitude but they seem to have been fixed during their investigation. See Naoki’s alog 74457.
    4. 22:28 UTC Continuing lock - it worked!
    5. NLN Achieved at 23:04 UTC
    6. LOCKLOSS at 23:14 UTC while ASC clearing SDF Diffs.
       1. TJ noticed noisy power recycling gain
13. 14:54 UTC: FMCS Air Handler 3B (Reheat) - FMCS alarm handler came onto red and then 3 mins later went back to normal (tagged FMP)
14. Lock acquisition 4 at 23:15 UTC
    1. Power recycling gain is being noisy (again).

LOG: