H1 back up and running for 3hr22min. Livingston is just getting back on the scoreboard but not to low noise yet. Running the script just to "freshen" the alignment for coincidence with LLO.
TITLE: 01/15 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 64.5071Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY: Things have been quiet since Keita and Peter fixed the ISS. The range increased slightly over the past hour and now hanging out at 64 Mpc.
LOG:
03:01 Keita and Peter to LVEA troubleshooting the ISS board
04:58 Back to Observe
The range is still bad, but at least we are back to where we were before the ISS issue. Accepted two SDF differences.
Quick Summary: ISS issue
Hopefully fixed, see alog 33283.
TITLE: 01/15 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC STATE of H1: Corrective Maintenance INCOMING OPERATOR: Nutsinee SHIFT SUMMARY: Every time I got to ROLL_MODE_DAMPING the ITMY roll mode was rung up and I needed to wait for it to damp. Then every time I tried to go to INCREASE_POWER I lost lock. Peter came in to look at the ISS and is still here. He, Nutsinee and I talked to Keita on the phone and ran some tests he suggested. Keita is on his way in now. Robert is also here. The CDS wiki is still down. LOG: 16:13 UTC Starting initial alignment 16:45 UTC Twice PRC locked, converged and then fell out of lock. Stayed locked the third time. 17:03 UTC The IMC keeps losing lock in SRC align and SRC_ALIGN_OFFLOADED 17:07 UTC SRY is now not locking. Took to down to check beam spot on ASC-AS_C. Is well centered. Took back to SRC_ALIGN. Locked this time. 17:12 UTC The IMC seems to lose lock every time the power is lowered in SRC_ALIGN_OFFLOADED. This time when the IMC lost lock and the ALIGN_IFO went from LOWER_POWER to DOWN I requested DOWN and will call the initial alignment done. 17:17 UTC Calling the initial alignment done. 17:39 UTC Guardian has stopped at ROLL_MODE_DAMPING with the diagnostic message 'waiting for roll mode damping'. The histogram shows that ITMY is bad. 17:49 UTC When I attempt to change the roll mode damping gain for ITMY something (Guardian?) sets it immediately back. 17:50 - 18:06 UTC Stepped out of control room 18:07 UTC The ITMY roll mode has been damped by guardian. Made it to DC_READOUT. 18:11 UTC Lock loss in INCREASE_POWER. Same as Ed reported to me. 18:39 UTC Dave called. Is working with Ryan to make wiki accessible. 19:01 UTC IFO is at DRMI_LOCKED. I am going to try to adjust the diffracted power by following alog 31262. 19:08 UTC Done. However the diffracted power is very noisy so it is hard to tell. I didn't seem to need to adjust the input offset to the ac coupling. 19:17 UTC At roll mode damping. ITMY is high again. 19:33 UTC Made it to DC_READOUT again, but this time the ITMY roll mode is still high. Attempting to damp. 20:11 UTC Have left a message with Peter and Keita. 20:28 UTC Turned off output of second loop as a test. Lost lock from DC_READOUT. 22:53 UTC Back to DC_READOUT and waiting for ITMY roll mode to damp. Peter is here investigating the ISS. 23:01 UTC Lock loss at INCREASE_POWER.
Patrick told me that the ISS diffracted power was swinging around more than usual. Indeed the diffraction
minimum and diffraction maximum values were more than usual even with the second loop disabled. After a
quick survey of the ISS MEDM screen, I saw that the calculated diffracted power was inconsistent with the
AOM drive voltage. At one point I saw that the reported diffracted power was ~15% for a reported AOM
drive of less than 0.2 V, which I know to be false.
Some time around 7:17 - 7:18 UTC the output of the AOM driver suddenly dropped. I had a quick look
at some other signals but didn't find anything that matched
I am wondering if the ground reference voltage of the PSL rack has changed relative to the PSL table
surface? There does not appear to be any coincidence with the LVEA temperature, so I cannot think of a
reason why it would change all of a sudden.
For the meanwhile, it is probably worth trying to keep the AOM drive to be ~0.5 V. One does this by adjusting the first loop ISS reference signal (bottom left hand corner slider of the first loop ISS MEDM screen). The adjustment should be done with the second loop ISS off.
Plots of the ISS reference signal and its monitor are attached. The reference signal comes from a DAC. The monitor is read off the ISS board from an OP-27. One thing to note is that the monitor output is somewhat noisier than the reference signal. This might be related to the grounding issue (if there is indeed one).
Problem was tracked down to the loss of -18V in PSL ISS AA chassis. Power cycling seems to have fixed it.
I was called and came to the site at 6PM to work with Peter who was also called earlier.
We have found that some of the ISS 1st loop signals were much smaller than they should be, starting 23:20 local time yesterday. PDA readback was only 0.7V or so when it should be 3.2, PDB was also too small, and ISS reference voltage readback was only -0.1 when it was set to -0.48.
We were first suspicious about analog failure of the ISS chassis, but when we went to the floor, everything looked normal in analog land. We measured the OLTF of ISS 1st loop and the UGF was about 38kHz.
Voltages on one of the DB9 that goes to the AA looked correct.
Then we went to the PSL ISS AA and found that LEDs for negative voltage was off, and the positive voltage LED was dim (will attach picture later). Peter power cycled it and everything seems to be back to normal.
Things to do for Tuesday:
Check PSL ISS AA chassis power board.
Things unrelated to this but are troubling:
We also went to the power supply area in mezzanine, and things look OK except that MANY power supplies have faulty power indicator lamps, some of them were flickering on and off.
Also, oplev power supply has a label saying +18V, but the actual voltage is more like +10V.
The reason why the readback failure matters is because we are now using the 1st loop PDA and PDB values for the power normalization of 2nd loop digital AC coupling.
With the AA chassis failing, H1:PSL-ISS_SECONDLOOP_INPUT_NORM hit the limitter bottom and no power normalization was done. Withrout a proper power normalization this AC coupling is known to cause loss of lock.
Fisrt picture: LEDs of the front panel with negative power failue (top chassis).
Second picture: LEDs of the back panel with negative power failue.
Third picture: UGF of ISS 1st loop.
Lost lock in POWER_UP, same as Ed reported. Something seems wrong with the ISS (see previous alog). I have stopped at DC_READOUT while I investigate, but I'm not making much progress. I have left a message with Keita and Peter.
It seems similar to alog 30335.
Could it be a problem with the ISS AOM power supply? The summary page shows that the voltage has dropped (plot).
I'm not certain, but my guess is that that signal is actuated on to set the diffracted power.
I think it might be a control loop somewhere, because locking and unlocking it can sometimes make the noise go away.
I watched the diffracted power signal and it got noisy as soon as it got to the DIFF portion of FIND_IR.
Peter is on his way to the site.
08:21:45UTC H1 back to Observing