TITLE: 11/29 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Jim
SHIFT SUMMARY:

Locking Recovery After Computer Re-Boot: Not going so well.

• Green Arms
  • X-Arm was a little off (didn't wait to align)
  • Y-Arm ws completely off the flashing (not waiting or that one)
    • ETMX was only re-pitched by ~ 1 uRad
    • ETMY PIT 143.3 -> 146.2    YAW -136.7 -> -135.9
    • TMSY PIT 95.9 ->97.4  YAW -273.4 -> -269.5
• Multiple locklosses at LOCKING_ALS
• After finally making it through LOCKING_ALS IR was not found
  • Neither Comm nor DIFF
• Another attempt made i through LOCKING_ALS Guardian said IR Not Found but both COMM and DIFF were seen on the screen. I had clicked PRMI out of habit and the alignment there was so bad, I ditched and went for IA.
• Handing of to Jim

LOG:

EX Saturations

• 9:36
• 9:58
• 13:14

13:54 Superevent S191129u

Looks like a general core crash, models remain running. Unlike earlier h1seib3 crash, in this case no other systems are affected and only DAQ data from SEI-B1 is down.

14:12 SEI B1 computer went down and reported ITM trips, but H1 remains locked.

• Observatory mode set to CORRECTIVE MAINTENANCE
  • set to OTHER at first. THen realized that was 'ENVIRONMENT'
  • then thought, maybe UNKNOWN under 'OTHER'
  • setlled on CORRECTIVE MAINTENENACE while it's being worked on

14:22 Contact made with Dave Barker - waiting on a return call

While locked and Observing for 7hours, H1 has been increasingly glitchy with only 2 EX saturations

• EX-9:36
• EX- 9:58

The wind is up in excess of 20MPH. There are no other obvious causes available on FOMs. Glitching ranges from half spectrum range (lower) to broad range.

TITLE: 11/29 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Camilla
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 10mph Gusts, 8mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY:

• 03:11 Lockloss. No obvious reason. I was out the room but we had "Commissioning" 5 seconds before "DOWN" on verbal alarms and the DIAG_MAIN user message around that time was "Fast shutter HV not ready"
• 03:17 IR was not found but ISC_LOCK continued anyway. Is that strange?
• 03:22 GRB Long notification received (no need to stand down even if we were observing)
• 03:24 Lockloss #2 at CARM_OFFSET_REDUCTION (lockloss happened here earlier: alog 53552)
• 03:32 Lockloss #3 at FIND_IR

• 22:00 Unlocked
• 00:10  WFS were not catching the ALS Y,  tried resetting ETMY and ITMY to before lockloss and got no flashes at all.
• 00:32 Tried initial alignment but it couldn't lock the green arms, took it to DOWN to manually adjust with ETMY and ITMY.
  • Finally got ALS Y flashes to 0.8 and the WFS grabbed
• 00:36 INITIAL_ALIGN stuck at OFFLOADING_GRREN_WFS even though signals were steady.
  • Tried DOWN, INIT but INITIAL_ALIGN stuck at OFFLOADING_GRREN_WFS again
  • Manually took ALS_X/YARM to UNLOCKED as this is what INITIAL_ALIGN was waiting for. (Did they ever get offloaded?!)
• 00:47 ALIGN_IFO stayed in PRC_ALIGN for 10 whole minutes waiting for PRC1 PIT and YAW to converge to thresh 50. I don't think they did, it just timed out and moved on. A similar thing happened yesterday for Jeff K and I alog 53541 @ 05:21.
• 01:03 Inital Alignment compleate.
• 01:24 Lockloss as CARM_OFFSET_REDUCTION
• 01:30 ALS YARM has fault with message "PHD",  "Green input alignment off"
  • This is the same as happened for Jim ( alog 53549) so I took ISC_LOCK to DOWN and changed H1:ALS-Y_PZT2_TRIG_THRESH_ON  from 0.5 to 0.3 as Keita had done on the phone with Jim. Is this fixing the Problem?  I changed it with caput and it's stayed at 0.3 but with Keita it seemed to go back to 0.5 quickly, maybe I should have changed it back too.
  • Now ALS YARM locking fine with small ETMY tweaks.

You might have noticed the strange gap in the wind trend ndscope covering the time h1seib3 was down (plot 3). This is because when h1seib3's general cores went offline, the mx_stream process (which sends the front end data to the DAQ) stopped running. This in turn caused problems for all the front ends which share the same ethernet port on the DAQ data concentrator. This is why most of the corner station frontend models went into DAQ data error (plot 2), and why it was critical to fix the problem quickly. One of the models so affected was the external EPICS data collector h1edc running on h1susauxh34 (plot 1). So all slow data was also frozen at the last received value.

h1seib3 has suffered a general core crash, meaning the models are still running (h1iopseib3, h1isiitmx and h1hpiitmx) but the DAQ data from itself, and most of the other SUS, SEI and PSL front ends are invalid (see attached). The only recourse is to reboot h1seib3. In preparation for this, I took this machine out of the Dolphin fabric (which killed the lock). Jeff is in the MSR taking a photo of the console and rebooting h1seib3 via the front panel reset button.