The /opt/rtcds file system disk space used has dropped from 96% to 46% today due to the rotation of the ZFS snapshots out of the reduced look-back time period. On 8th December I compressed most of the target_archive area and reduced the snapshot look-back from 31 days to 19 days. The non-compressed files have now rotated out of the ZFS snapshots.

Next I need to complete the target_archive compression and restore the ZFS look-back to 31 days.

CDS Systems Report (Thu Dec 27 08:00:01 PST 2018):
Thu Dec 27 08:00:01 PST 2018

-- Disk Usage -----------------------------------------
h1fs0:
Size 899 Used 416 (UsedDS 402 UsedSNAP 13 ) Avail 483 Use% 46 %
 

Hang, Dan Brown

• Soft dither is now combined with ENGAGE_ASC_FOR_FULL_IFO. We now directly AC couple the QPD soft loops when they switch on and DC point is set by the dither loops. Convergence now checks for the dither error point and not the soft QPDs. 
• From DARM_TO_RF to CARM_OFFSET_REDUCTION something in EX is nearly saturating, the computer keeps shouting about it
• We updated the CHARD yaw input matrix which has made 25W very stable and 28W look promising although we didn't spend too long there. We pushed up to 30W and lasted ~20 minutes. We checked the LSC loops and they all seem fine and updated PRCL guardian gain scaling.
• CHARD yaw eventually became unstable again at 30W and we lost lock but got enough data to generate some more appropriate input matrices for 30W.

When powering up and thermalising the coupling of PRC2 into CHARD becomes orders of magnitude greater than CHARD back into itself which seems to be the cause of our 3Hz instabilities. At 25W we injected some lines in and looked at how the coupling changed, after it reached some steady state I took 5 minutes of data and processed it offline to compute a new input matrix for PRC2 and CHARD. I used DC5_Y to try out the new input matrix, as can be seen in the DTT window here, blue (new) is looking a lot better than red (old).

We lost lock trying to switch over the matrix, so just update the guardian with these new values. When engaging ful IFO ASC it is a lot more noisy then before, when you get to CHARD_BLEND though everything quietens down and the dither loops catch up.

We put the old sensing matrix values back into DC5_Y to see how they behaved as we powered up in steps, a comparison is shown here. As can be seen the new matrix is overall more noisy than the old, however at 28W PRC2 is much lower than before. At 30W we didn't reach a complete thermal steady state as it got too bad, so we still need to fine tune this more tomorrow. The hope is we can get steady at 30W and then do some TCS tuning.

Hang, Dan Brown

Only EY was down from this dolphin crash so this recovery was quicker than some of the others.

Initial alignment went fine except for PRC align. It seems the PRC1 filter changes made the other day  (46151) made it unstable for initial alignment, probably too much gain. In PREP_PRC_ALIGN Guardian we switched off all the filters apart from FM1 (-20dB), it converged slowly and with FM1 off it still works.

Only one ALS lockloss during LOCKING_ALS, DRMI locked in about 20s.

Took about 2 hours until we were up again - although most of that was checking CO2Y table for leaks and figuring out what was up with PRX initial alignment rather than the Dolphin crash itself.

I expect to me hear for ~90 minutes or so and will make a comment to this entry when I am leaving.

The pressure gauge at X2-8 location near X-end is reading high and might have failed in a new way due to lack of solar power. Adjacent pressures are normal so the control room can ignore this alarm. VAC team will investigate when on site.

Added 350 mL water to TCSY to bring level from 8.6 cm up to 10.1 cm today. This appears to be a higher-than normal drop over the last couple of days, with the caveat that the previous reading was without optimal eyeglasses. At a minimum, it seems wise to keep an eye on the readings.

Also, topped off Wallace with 50 mL.

Unfortunately our 25 day streak of no Dolphin crashes came to an end at: Dec 25 2018 08:56:14 PST

I've captured all the Dolphin and system logs. I'll work on restarting the models soon.

If anyone is in the control room today, please call my cell phone and we can complete the recovery.

This is new - no action needed at this time.

Rana, Hang

After adding the res G's at 0.47 Hz our ASC loops appeared to be quite and stable at 25 W for more than 20 mins. However, as we tried to power up to 28 W, our dithering loops, especially the yaw ones started to drift away. This then triggered the ~ 0.5 Hz oscillation in CHARD YAW and then in other ASC loops. In the second attempt we tried to increase the dithering loops gain as we power up. However this did not kept the dithering loops stable. While the error signal still crosses zero, the oscillation seemed to be growing. The further the soft dithering loops' error signal deviated from 0, the more sever the 0.5 Hz oscillation in CHARD YAW. Or it might be the other way around, as the CHARD YAW RMS increases, the spot motion naturally made the dithering loops oscillates more.

============================================================

In addition to power up, we also tried to go to LOWNOISE at 20 W.

In the lownoise asc state, we reduced the CHARD YAW dc gain to 0.7, giving it roughly 3 Hz UGF. Then we could re-engage the ELP10 (FM2) filter to reduce its noise.

However, we kept losing lock at LOWNOISE_ESD_ETMX, specifically when we tried to transition ETMX to ITMX+ETMY. The ITMX part seemed fine. The ETMY ESDOUTF_DC_OUTPUT was zero before the transition, which seemed weird. In the guardian there also seemed to be no explicit commands setting up ETMY ESD in that state.