Here are the past 3 days trends of the HPO headflow rates at Jason's request due to the LASER trip.
I added 150ml to the xtal chiller after it was restarted.
After a failed startup attempt of the laser, I had to add another 235ml.
TITLE: 07/17 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 0Mpc
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: recovery from 5.8M EQ continues, with some interesting results
LOG:
Cheryl got in contact with Jeff B. regarding a canceled Owl shift tonight. Sheila contacted Keita & myself regarding this.
Jim and I have been adjusting the interferometer alignment for the last few hours.
IMC alignment
We noticed as has been noted by a few people that the IMC alignment is different from before the EQ according to the witness sensors and spots on QPDs (IMC WFS DC, IM4, and ISS QPD). We tried restoring the optics and input PZT using witness sensors, as I am sure people have tried in the last week, and saw that this made it impossible to lock the mode cleaner. Next we went back to the alignment from overnight last night. I moved the uncontrolled DOF (DOF4) to bring the IMC WFS DC spots to where they were before the EQ (this was the idea behind Suresh's effort to control DOF4 several years ago). Jim restored IM1,2,3 to their alignments pre EQ using the osems (the biggest move was IM1 pit). This resulted in restoring the spot positions on IM4 trans and the ISS QPD to their pre EQ positions. (see attached plots of QPDs and range so you can identify times of the EQ and other alignment changes made trying to recover from EQ).
One thing we noticed which Corey has also logged is that there is some kind of mistake in the IMC WFS offlaod script which misaligns the PZT. We could replace this with our generic WFS offload guardian state anyway.
PR3+SR3 osem jumps
This is also something that other people have noted, but there is a large jump in the osem values compared to the oplevs for PR3 +SR3 after the EQ. We spent some time worry about both of these optics because of a problem which turned out to be the fast shutter. PR3 moved after teh EQ, but the osem and oplev do not agree about the size of the move, and it seems like it is some kind of permanent shift in the value of the osem. Cheryl has a nice plot that shows this. It seems like people already knew this and had realinged PR3 correctly using the oplev and reset the SR3 cage servo soon after the EQ.
Fast shutter problem
Jim and I did an initial alignment after the IMC +IM move, but ran into trouble at SRC aling. Cheryl and I spent several hours with this problem, which at first seemed very mysterious. We could not move the beam in single bounce to center it on AS_C, if we tried to do this using either SR2 or SR3 the image on the AS camera would turn into a huge interference/clipping blob, and the power on the AS_C QPD would drop nearly to zero. We tried moving many things with no improvement, but in the end clsing the AS shutter fixed the problem. Our best guess is that the shutter was somehow stuck in a half open state from about 22 UTC July 16th (or earlier) to about 4:30 UTC July 17th. This could have happened as early as this morning's unexplained lock loss. This happened a few more times while we were trying to test the shutter in DRMI. It looks (from the strip tool) like the shutter never actually closes but gets stuck half shut while it is opening.
We are not going to lock for the rest of the night because we don't think the shutter is correctly protecting the OMC. We are trying to contact Jeff B to cancel the OWL shift.
Final note: Ireverted the POP A offsets to what they were before the EQ, in hopes that the IFO alingment is more similar to that now. This worked in DRMI.
Opened FRS ticket 8540 for fast shutter issues.
With the instrument down and earthquake ringing went out on the floor to investigate the Fast shutter. The first thing I noticed was the front panel display was not displaying the normal Capacitor charging 250V. Instead a garbled messeage. (see Pictures) I rebooted the controller and everything appears to be normal. I ran a test on the Shutter and the Ham ISI tripped. Before I could look at light levels the PSL tripped so will have to look once we restore.
Regarding the LCD display on the front of the fast shutter, we have seen this type of symptom before. Once in a while (not often by any means) the large electromagnetic pulse that is radiated out of the shutter driver chassis circuitry exceeds some critical threshold and upsets the communication to the LCD display. Shielding was added to the LCD interface cabling, and it was believed that the addition of this shielding fixed the problem. It appears as though there is still some finite chance that a pulse can upset the display. The good news (if you can call it that) is that the display operation is completely separate from the operation of the shutter, so the garbled LCD is not an indicator of a malfunction in the actual shutter operation inside HAM6. The bad news is that if the shutter has started to behave differently, the likelihood of it being a good thing is essentially zero. Any chance that the beam is hitting the thin wiring leading to the moveable portion of the shutter should be taken seriously. If anything (lock-loss transient etc.) causes a blemish to be formed in the Teflon insulation, the normal flexing behavior will likely change and prefer flexing at the damaged spot leading to fatigue failure of the wire. I would advise great caution and scrutiny be applied to the shutter for a while. The tendency to get stuck in a partly blocking position didn't exist at installation time, so change may be afoot and it is not likely for the better.
TITLE: 07/17 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 0Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
Wind: 8mph Gusts, 6mph 5min avg
Primary useism: 1.20 μm/s
Secondary useism: 0.16 μm/s
QUICK SUMMARY:
TITLE: 07/16 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Initial Alignment
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Lockloss early, spent most of the day aligning, ongoing
LOG:
16:00 Lockloss, Sheila and I start revisiting alignment, ongoing
TITLE: 07/16 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 46Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: Observing for 4 hours. No issues to report.
LOG: None.
Range is ~43 MPc.
Lockloss 9:38 UTC. DHARD loops seemed to be ringing up. ISC_LOCK guardian went into error upon lockloss. It was reporting an issue with OMC_LOCK not being managed, which turned out to be true. I INIT'ed ISC_LOCK. Going to run through an IA to see if it helps the DARM spectrum which was elevated across the entire frequency range.
Back to Observing at 10:54 UTC. DARM spectrum is still elevated at all frequencies. IA did not appear to help. Range is 46 MPc.
TITLE: 07/16 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
Wind: 18mph Gusts, 14mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.05 μm/s
QUICK SUMMARY: Cheryl working on damping violin modes that rang up after last lockloss.
TITLE: 07/16 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Travis
SHIFT SUMMARY:
LOG:
TITLE: 07/16 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 48Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
Wind: 21mph Gusts, 17mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.05 μm/s
QUICK SUMMARY: locked since last night
TITLE: 07/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 48Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
LOG:
21:00 Kyle on site working on VPW
CH2M was supposed to knock down a stack today. Didn't see any evidence of it here on ground sensors or roof cam, but wall FOMs may not include relevant frequency bands?
TITLE: 07/15 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 51Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: Observing for 4+ hours. No issues after IA and relocking.
LOG: None
After a bit of a extensive IA followed by a short lock, we are back to Observing. The range is down from earlier in the week at around 50 MPc.
TITLE: 07/15 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Calibration
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
Wind: 8mph Gusts, 6mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY: Relocking after mystery lockloss post input pointing changes. Currently at DRMI_LOCKED and will attempt to proceed further. Cheryl suggests that if this lock doesn't work out, another IA may be helpful.
TITLE: 07/15 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Calibration
INCOMING OPERATOR: Travis
SHIFT SUMMARY:
LOG: