J. Kissel

We can't have a days worth of model changes, recompiles, reinstalls, restarts, and restores without at least one mystery. Today's casualty is H1SUSETMY's, M0, F1 F2 F3 SD cluster of OSEMs, which appear to be dead, reading out bit noise. I suspect this is an analog electronics failure that is merely coincident today's reboots, because the OSEMs that have failed are all on one satellite amplifer / coil driver chain. All other OSEM sensors appear fine. We'll have to check the analogue electronics in the morning. 

The timeline of the day is as follows:
15:50:11 UTC / 08:50:11 PDT Event -- DC shift in OSEM values, with some ringing appears to be indicative of a ISI trip
18:42:35 UTC / 11:42:35 PDT Signals go dead -- corresponds with h1susetmy model restart at ~11:44 pm 00:02 PDT
19:56 UTC / 12:56 PDT       Signals come back -- corresponds with h1dc0 framebuilder restart at 12:57 pm 00:02 PDT
20:47 UTC / 13:47 PDT       h1susey model restart -- note the large temporal separation between the restart and the signal death.
~21:17 UTC / 14:17 PDT      Signals die
23:19 UTC / 16:19 PDT       h1susey model restart -- note the large temporal separation between the restart and the signal death.
00:04 UTC / 17:04 PDT       h1cd0 frambuilder restart --- signals DO NOT come back
00:53 UTC / 17:53 PDT       h1susey model restart, because I've noticed the problem, found the third bit of ADCO as red, assumed it was a too-speedy "make" and "make-install" 
The last reboot did fix the ADC0 bit error, but it did not restore the OSEM values. 
             
Note I'm retrieving the model restart times from the very useful model_start.log here:
/opt/rtcds/lho/h1/data/startlog/2014/04/29/2014_04_29_model_start.log 
which is a daily log of all model restarts (a new folder and file is automatically created each day).