In the morning we were wondering what the SRM composite mass looks like, in case we want to drive it to find out the frequency and Q of the first internal mode.

https://dcc.ligo.org/LIGO-D1200886

There is a ring shaped holder which has two nubs and a set screw. This holds the 2" optic. This holder is then bolted into the larger aluminum mass. Two views from the solidworks drawings are shows with some parts made transparent. If this drawing is right, I expect that the resonant frequency is in the 500-1000 Hz range.

Saturday 28th

08:47 PSL Diode Chiller monitor signals flat-line (compuer had been running for about 24 hours since last reboot)

10:24 TCS DAC output drops to zero volts (recovered soon after)

17:04 h1nds1 kernel panics (computer rebooted)

23:46 TCS DAC output drops to zero volts (recovered Monday)

Sunday 29th

Monday 30th

11:00 PSL laser power drops to zero (recovered Tuesday)

TCS Y-table FLIR camera

Kiwamu, Dave:

the TCS-Y infrared camera was not responding. Restarting the controller unit at the table appears to have resolved it.

h1nds1 restart

h1nds1 stopped running. Monit was not monitoring it, we manually instructed monit to restart daqd and it is running again.

h1oaf0 power cycle

Following three occurences of the TCS DAC being driven to zero volts, this morning we power cycled the IO Chassis for h1oaf0. Power was removed from the chassis using the front panel switch, it was down for about one minute. h1tcscs was manually burt restored to 09:10 PDT this morning.

roof camera

Richard, Jim, Dave:

We went onto the roof and verified the CDS AUX vlan was operational at the fiber media converter, it was.

J. Oberling, J. Bartlett, P. King (via phone)

Following yesterday's initial investigation into the PSl diode chiller issues (see alogs here), we swapped the control panel for the diode chiller with the one from the chiller we recently removed from service (this is a known working unit, just installed last year).  After installation the chiller restarted without issue and ran for several minutes, also without issue.  The serial number of the control panel we installed in the diode chiller is 44806P605; the faulty unit we removed from the diode chiller had no SN on it.

We then took the time to restart the PSL Beckhoff PC to unstick the frozen diode chiller channels.  According to Dave Barker these channels froze sometime on Saturday morning.  Fortunately the PSL interlocks are not tied to these channels, otherwise we would not have the PSL shutting down when the diode chiller shuts down (as we did yesterday).  Once restarted the channels appeared to be reading OK.  It is possible that these channels freezing for no reason can be used as an early warning sign of imminent chiller failure.  The chiller communicates with the PSL Beckhoff PC via a serial RS-232 interface and we have seen channels freeze when the cables are unplugged and plugged back in but the PC not restarted (which is expected behavior as RS-232 is not hot swappable), but this is the first time I've seen the chiller spontaneously loose communication with the PSL Beckhoff PC.  Jeff Bartlett is setting up a temporary Strip Tool on the PSL monitor in the control room that will monitor these channels.  If anyone sees that these are flatlined (there should always be some variation), please let someone on the PSL team (Peter King, Jeff Bartlett, Ed Merilh, Rick Savage, or myself) know immediately.  Thank you.

We then turned on the HPO, which came up with no problems.  We let it sit and run for a little while, then restarted the rest of the PSL.  As of now, everything is up an running.  We are going to continue to monitor over the next couple days to ensure that everything is working correctly.  The removed control panel will go back to Termotek with the chiller we are sending back and will be replaced as part of the service being done on that chiller.

1420 - 1440 hrs. local -> To and from Y-mid 

Opened exhaust check valve bypass, opened LLCV bypass 1/2 turn ccw -> LN2 at exhaust in 40 seconds -> Closed LLCV bypass, waited a few minutes and closed exhaust check valve bypass. 

Next overfill nominally Wednesday afternoon.

Sheila, Terra, Craig

The PSL is off, is has been since about 17:30 UTC. The chillers don't seem to be tripped. Confusingly the laser screen indicates that the chillers are fine, (two green boxes) while the PSL_STATUS screen has a red box for the crystal chiller (screenshot attached).  Jason will come out to the site to investigate/restart the laser.  

We also noted that the temperature trends for the PSL have been usual since thursday morning's incursion. (2nd screnshot) I went to the controller box and saw that the north AS unit was on, which was probably unintentional (the south unit was off, and they are normally both off in science mode).  I turned it off at noon local time.  Terra noted that the PSL microphone has seen an elevated level of noise in the last few days, which went back to normal as soon as the AC unit was off.  (In the third attached screenshot, blue traces are from the time when the AC was on).  The montors on the RF AM stabilization also changed when we turned off the AC, and some channels on the AM stabilization box seem to have been sensitive to some kind of switching of the PSL HVAC over the last few days.  

It seems like we need some kind of a better way to monitor if the PSL environment settings are correct, maybe adding them to diag main if we can find a good set of tests to write.  It also is suprising to me that our RF system seems to be so sensitive to accoustic pick up in the PSL.  Has anyone in detchar looked at PSL PEM monitors to see if glitches there are correlated to the "RF45" glitches seen durring O1? 

The TCS chillers are also tripped, with the same DAC problem we have been having. (27435)  This happened about 36 hours ago.

(see attached) 
Will investigate Tues.

Evan and I spent most of the day trying to investigate the sudden locklosses we've had over the last 3 days.  

1) We can stay locked for ~20 minutes with ALS and DRMI if we don't turn on the REFL WFS loops.  If we turn these loops on we loose lock within a minute or so.  Even with these loops off we are still not stable though, and saw last night that we can't make it through the lock acquisition sequence. 

2)In almost every lockloss, you can see a glitch in SR3 M2 UR and LL noisemons just before the lockloss, which lines up well in time with glitches in POP18.  Since the UR noisemon has a lot of 60 Hz noise, the glitches can only be seen there in the OUT16 channel, but the UR glitches are much larger.  (We do not actuate on this stage at all).  However, there are two reasons to be skeptical that this is the real problem:

• There is nothing in the witness sensor or OpLev to indicate that the optic is really moving.  These just might not be sensitive enough, but last time we had sus electronics problems with SR3 there was a clear disturbance on the OpLev and witness sensor (21095 and 21081)
• These large glitches have been happening since May10th or so, and we have only started having locking problems in the last 3 days. The RF problem of the last few days seems to corespond better to our locking difficulties, although there aren't signs of problems in the RF readbacks at the exact lockloss times.

It could be that the RF problem that started in the last few days somehow makes us more senstive to loosing lock because of tiny SR3 glitches, or that the noisemons are just showing some spurious signal which is related to the lockloss/ RF problems. Some lockloss plots are attached. 

It seems like the thing to do would be trying to fix the RF problem, but we don't have many ideas for what to do. 

Not clear why. It may correspond to a PSL incursion the same morning.

Also, what used to be hooked up to outputs 7 and 8 on the 9 MHz distribution amplifier in the CER? Sheila and I found some terminated attenuators on these ports, but I recall it used to be cabled up somehow.

Evan H arrived on site this morning to find the TCS chillers tripped, reset them and found the same behavoir as described in alogs 27381 and 27374

On the OAFIOP GDS TP screen, both the DAC and ADC bits (as well as DK) were red, the ADC error cleared with a diag reset but the DAC error would not reset.  We called Dave and he asked us to check that the DAC outputs were all zero by looking at the DAC MON screens (accessed by clicking the blue buttons labeled D0). This means that it is the same DAC problem Nutsinee described.  

To fix the problem, you need to stop and start all the models on the front end, including the IOP model.  This can be done by:

1)maybe check SDF before you kill models (I frogot this step)

2) log in to h1oaf0 as controls 

3) run a script /etc/kill_models.sh, wait for all models to be shut down in the correct order, with the IOP model last

4) run a script /etc/start_models.sh

5) Dave said that for the PEM model only we restore the settings by loading the OBSERVE.snap and hitting LOAD TABLE +EDB.  Since I frogot to check SDF before killing the models I am using the automatic burts to restore the rest of the models. Confusingly, the automatic burts always appear to indicate that there are no diffs, because all channels are no mon in them, so to actually check you need to select full table, select all mon, then set the table back to setting diffs. Time machine doesn't work for the SDF screens, which would be handy in a situation like this.  

Go to the mezzanine and follow instructions here https://lhocds.ligo-wa.caltech.edu/wiki/TCS to restart the chillers, and the laser controllers near the TCS tables.