I forgot to post these yesterday.  Perhaps we should move this task to Wednesday on the OPS checklist so as not to get lost in maintenance day activities.  See attached screenshot for values before reset.

With the AC off in the PSL, and the laser restored, we are back to locking.  Tonight we started to implement the high bandwidth hard loops that we made filters for over the weekend.  The idea here to to make some ASC loops that will be high bandwidth and introduce noise into darm, but be relatively simple to keep stable as we power up.  Then we can worry about a low noise loop only at our final power.  The loop designs using Jenne's model are attached, they are the same filt ers for CHARD and DHARD so only CHARD is attached.  Since the top mass damping was made more uniform today, 27464
 the damping is probably not acurate in the model anymore.   We were able to turn the CHARD yaw loop up to 10 Hz, (gain of -250) and turn the pit loop up to about 5 Hz, but there is something that is unstable when they are both high bandwidth.  We checked that the gain of the MICH loop is not changing as we increase the gain in CHARD. 

• We have some badly rung up violin modes on ETMX, possibly because the guardian was paused when there was still actuation going to the test masses.  Evan worked on some filter settings that will damp single modes for a few of these, the settings that are currently damping them are shown in the 3rd attachment.  I'm hoping to leave the IFO locked with these damping settings, it would be great if we could continue damping in the morning. 
• We reset the soft offsets tonight
• We changed the PUM coil drivers to the high range state

Evan, Rana, Sheila

This afternoon there was a problem with the ITMX M0 BIO.  The TEST/COIL enable indicator was red, and setting H1:SUS-ITMX_BIO_M0_CTENABLE to 0 and back to 1 could not make it green.  Evan tried power cycling the coil driver, which did not work.  We were able to reset this and damp the suspension again by setting the BIO state to 1, it seems that anything other than state 2 works. 

This might be a hardware problem that needs to be fixed, but for now we can use the suspension like this. 

TITLE: 5/31 Eve shift 16:00-0:00 PST

STATE Of H1: Commissioning

SHIFT SUMMARY: Mostly Sheila and Evan trying to lock

INCOMING OPERATOR: Empty chair.

ACTIVITY LOG:

16:20 Gerardo and Manny out of LVEA

19:00 Sheila and I adjust Y arm polarization

Other than that, a lot of lock losses in DRMI/PRMI.

Rana, Evan

The top mass pitch and yaw damping filters for the quads were inconsistent between different suspensions. We cleaned up and regularized the filters (in particular, by turning off the lower-Q "newboo" filter in one of the ETM pitch loops), and settled on the following configurations:

• Pitch: FM1 (plant compensation, confusingly called "rolloff"), FM2 (resonant boost around 0.5 Hz), FM5 (flat gain), FM6 (60 Hz notch), FM8 (+10 dB), FM10 (rolloff filter), gain −1
• Yaw: FM1 (plant compensation, confusingly called "rolloff"), FM2 (resonant boost around 0.5 Hz), FM5 (flat gain), FM6 (60 Hz notch), FM8 (+12 dB), FM10 (rolloff filter), gain −1

The step responses look reasonable (Q of 4 or so). We tuned the gains by increasing them until we saw oscillation, and then we backed off by 6 dB or so. The overall pitch loops are basically the same as before, while the yaw loops have 12 dB more gain.

Chandra, Ken (with help from John & Richard)

Removed the pneumatic actuators from LLCVs on CP 2,3,5,6 and replaced with the electronic version. Upgraded fuses to 2.5 A. 

Will finish up WP 5906 with CP3 tomorrow morning.

CP 2,4,5,6 are in PID mode.

Kiwamu, Nutsinee, Jim, Dave

At noon we power cycled the h1oaf0 IO Chassis to hard reset the 16bit DAC card. We were suprised that both TCS chillers stopped running. During last week's IOP dac-disable events (which zeroed the DAC drives), the chillers kept on running for about an hour before tripping. As a further test, at 14:00 PDT we again powered down h1oaf0 CPU followed by its IO Chassis. At this point the chillers were operational. We then power cycled the AI chassis, which tripped the chillers. We then powered up the IO Chassis and computer, at which point the chillers tripped again. We were unsure if the trip was at the time the IOP model started, or h1tcscs model started driving the DAC channels.

Take home message: before restarting h1oaf0 IOP, CPU, IO Chassis or DAC-AI unit please contact Nutsinee.

Nutsinee, Kiwamu,

Today, we made a final touch on the alignment of the CO2Y table optics. As a result, we got a 5.7 W output power coming out of the table which is twice bigger than the value we measured back in April (26645). The beam profile now looks good -- almost axis symmetrical about the center of the beam area.

The next step: the alignment of the CO2 beam with respect to the interferometer beam.

[Details]

At the beginning, as a coarse alignment between the CO2 beam and the aperture mask, we moved the position of the mask by approximately a few mm in order to improve the horizontal beam profile. We then touched M5 in both pitch and yaw to further refine the alignment. Later we repositioned the beam dump which catches the reflection from the mask. This time, we used the FLIR camera as a reference which is much more sensitive than an image card with the UV light. The attached images are the ones after the fine adjustment.

Once we optimized the intensity profile, we re-aligned the beam through the two irises that have been serving as the fiducial points for projecting the beam to the test mass. After the alignment, we placed a power meter right behind the first iris which read 5.7 W when the rotation stage was at 18 deg (which should give us almost the maximum transmission). Before closing the table, we put the beam dump back in place to block the beam to the FLIR camera.

TITLE: 05/31 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: Jim
SHIFT SUMMARY: After PSL was brought back to life and maintenance activties subsided, we are having issues getting past DRMI locked.
LOG:
 

14:56 Ken to MY working on HVAC

15:06 Cristina and Karen to LVEA

15:35 Fil and Ed pulling cables in beer garden

15:43 Hugh to ends HEPI maint.

15:50 turned of BRS at both ends for maint. work

15:50 Gerardo and Manny to all VEAs, pulling extension cords in prep for power outage

16:12 Kiwamu working on TCS alignment

16:34 Hugh done

16:48 Nutsinee to TCS table

17:00 Karen to EY

17:05 Fil and Ed out

17:54 Fil working on ESD in beer garden

18:00 Gerardo and Manny back from ends, to LVEA

18:02 Jason to diode room

18:05 PSL back up

19:00 Kiwamu done

19:51 Fil to beer garden

20:21 Bubba to LVEA checking 3IFO N2 purge

21:07 Fil out

21:25 Kiwamu and Nutsinee to TCS table

22:27 turned BRS back on at both ends

22:39 Kiwamu and Nutsinee out

Required use of N. crane to stage equipment (~0900 - 0930 hrs. local). 

Temporary aux. pumps removed and HAM4 AIP system in nominal, as found, configuration.

In the morning we were wondering what the SRM composite mass looks like, in case we want to drive it to find out the frequency and Q of the first internal mode.

https://dcc.ligo.org/LIGO-D1200886

There is a ring shaped holder which has two nubs and a set screw. This holds the 2" optic. This holder is then bolted into the larger aluminum mass. Two views from the solidworks drawings are shows with some parts made transparent. If this drawing is right, I expect that the resonant frequency is in the 500-1000 Hz range.

Saturday 28th

08:47 PSL Diode Chiller monitor signals flat-line (compuer had been running for about 24 hours since last reboot)

10:24 TCS DAC output drops to zero volts (recovered soon after)

17:04 h1nds1 kernel panics (computer rebooted)

23:46 TCS DAC output drops to zero volts (recovered Monday)

Sunday 29th

Monday 30th

11:00 PSL laser power drops to zero (recovered Tuesday)

TCS Y-table FLIR camera

Kiwamu, Dave:

the TCS-Y infrared camera was not responding. Restarting the controller unit at the table appears to have resolved it.

h1nds1 restart

h1nds1 stopped running. Monit was not monitoring it, we manually instructed monit to restart daqd and it is running again.

h1oaf0 power cycle

Following three occurences of the TCS DAC being driven to zero volts, this morning we power cycled the IO Chassis for h1oaf0. Power was removed from the chassis using the front panel switch, it was down for about one minute. h1tcscs was manually burt restored to 09:10 PDT this morning.

roof camera

Richard, Jim, Dave:

We went onto the roof and verified the CDS AUX vlan was operational at the fiber media converter, it was.

J. Oberling, J. Bartlett, P. King (via phone)

Following yesterday's initial investigation into the PSl diode chiller issues (see alogs here), we swapped the control panel for the diode chiller with the one from the chiller we recently removed from service (this is a known working unit, just installed last year).  After installation the chiller restarted without issue and ran for several minutes, also without issue.  The serial number of the control panel we installed in the diode chiller is 44806P605; the faulty unit we removed from the diode chiller had no SN on it.

We then took the time to restart the PSL Beckhoff PC to unstick the frozen diode chiller channels.  According to Dave Barker these channels froze sometime on Saturday morning.  Fortunately the PSL interlocks are not tied to these channels, otherwise we would not have the PSL shutting down when the diode chiller shuts down (as we did yesterday).  Once restarted the channels appeared to be reading OK.  It is possible that these channels freezing for no reason can be used as an early warning sign of imminent chiller failure.  The chiller communicates with the PSL Beckhoff PC via a serial RS-232 interface and we have seen channels freeze when the cables are unplugged and plugged back in but the PC not restarted (which is expected behavior as RS-232 is not hot swappable), but this is the first time I've seen the chiller spontaneously loose communication with the PSL Beckhoff PC.  Jeff Bartlett is setting up a temporary Strip Tool on the PSL monitor in the control room that will monitor these channels.  If anyone sees that these are flatlined (there should always be some variation), please let someone on the PSL team (Peter King, Jeff Bartlett, Ed Merilh, Rick Savage, or myself) know immediately.  Thank you.

We then turned on the HPO, which came up with no problems.  We let it sit and run for a little while, then restarted the rest of the PSL.  As of now, everything is up an running.  We are going to continue to monitor over the next couple days to ensure that everything is working correctly.  The removed control panel will go back to Termotek with the chiller we are sending back and will be replaced as part of the service being done on that chiller.

Sheila, Terra, Craig

The PSL is off, is has been since about 17:30 UTC. The chillers don't seem to be tripped. Confusingly the laser screen indicates that the chillers are fine, (two green boxes) while the PSL_STATUS screen has a red box for the crystal chiller (screenshot attached).  Jason will come out to the site to investigate/restart the laser.  

We also noted that the temperature trends for the PSL have been usual since thursday morning's incursion. (2nd screnshot) I went to the controller box and saw that the north AS unit was on, which was probably unintentional (the south unit was off, and they are normally both off in science mode).  I turned it off at noon local time.  Terra noted that the PSL microphone has seen an elevated level of noise in the last few days, which went back to normal as soon as the AC unit was off.  (In the third attached screenshot, blue traces are from the time when the AC was on).  The montors on the RF AM stabilization also changed when we turned off the AC, and some channels on the AM stabilization box seem to have been sensitive to some kind of switching of the PSL HVAC over the last few days.  

It seems like we need some kind of a better way to monitor if the PSL environment settings are correct, maybe adding them to diag main if we can find a good set of tests to write.  It also is suprising to me that our RF system seems to be so sensitive to accoustic pick up in the PSL.  Has anyone in detchar looked at PSL PEM monitors to see if glitches there are correlated to the "RF45" glitches seen durring O1? 

The TCS chillers are also tripped, with the same DAC problem we have been having. (27435)  This happened about 36 hours ago.

(see attached) 
Will investigate Tues.

Evan and I spent most of the day trying to investigate the sudden locklosses we've had over the last 3 days.  

1) We can stay locked for ~20 minutes with ALS and DRMI if we don't turn on the REFL WFS loops.  If we turn these loops on we loose lock within a minute or so.  Even with these loops off we are still not stable though, and saw last night that we can't make it through the lock acquisition sequence. 

2)In almost every lockloss, you can see a glitch in SR3 M2 UR and LL noisemons just before the lockloss, which lines up well in time with glitches in POP18.  Since the UR noisemon has a lot of 60 Hz noise, the glitches can only be seen there in the OUT16 channel, but the UR glitches are much larger.  (We do not actuate on this stage at all).  However, there are two reasons to be skeptical that this is the real problem:

• There is nothing in the witness sensor or OpLev to indicate that the optic is really moving.  These just might not be sensitive enough, but last time we had sus electronics problems with SR3 there was a clear disturbance on the OpLev and witness sensor (21095 and 21081)
• These large glitches have been happening since May10th or so, and we have only started having locking problems in the last 3 days. The RF problem of the last few days seems to corespond better to our locking difficulties, although there aren't signs of problems in the RF readbacks at the exact lockloss times.

It could be that the RF problem that started in the last few days somehow makes us more senstive to loosing lock because of tiny SR3 glitches, or that the noisemons are just showing some spurious signal which is related to the lockloss/ RF problems. Some lockloss plots are attached. 

It seems like the thing to do would be trying to fix the RF problem, but we don't have many ideas for what to do.