J. Kissel, S. Karki

I discovered this morning that H1 PCAL X was no longer churning out any of Sudarshan's thesis generating lines. Though I knew we'd intentionally turned off the 333.9 Hz and 1083.3 Hz lines this past Friday evening (see LHO aLOG 38148), we had intended to continue to run the >2 kHz long-duration sweep lines. They had stopped running at 2017-08-13 01:33 UTC --> 2017-08-12 (Saturday) at 6:33 pm local.

What happened?
When the EX SUS rack lost one leg of its power (see LHO aLOG 38162), the EX:ISC power supply was unnecessarily cycled as well, which killed the timing signal to the ISC I/O chassis, the master of the EX Dolphin network. With the master dead, all EX front-ends went belly up -- including PCALX. Upon restart of the front-ends (see LHO aLOG 38163), the guardian code managing this high frequency line,
   /opt/rtcds/userapps/release/cal/h1/guardian/HIGH_FREQ_LINES.py
did not register that we'd lost lock, the safe.snap had some old setting, and we don't monitor these the frequency or gain of the H1:CAL-PCALX_PCALOSC1 oscillator because it is regularly changed during observation -- so no one noticed that the line was off.

I've edited the guardian code to start at the next data point, at 4001.3 Hz, loaded it, ran the INIT state, which turned on the line at 2017-08-14 20:31:30 UTC (if you want to be picky, it was ramping up from 20:31:30 UTC, and fully ON and stable by 20:31:30 UTC).

Again, because the frequency and gain of this oscillator is not monitored, this guardian reload and settings change did NOT take us out of observation mode.