For future reference and data quality flags, the total time that PCALX laser was down is as follows (based on trending the channel showing the PCALX laser diode current, H1:CAL-PCALX_LASERDIODECURRENT):
2020-02-19 00:16:57 UTC to 2020-02-25 17:38:49 UTC 
1266106635 to 1266687547
580912.0 seconds
161.36 hours
6.7 days.

Does this green filter look more green than usual, given service life? Are there any substantial particles within the mesh? Wondering how those on site are reacting to the findings of this log.

It seems that there is no apparent large particulate, which may have been precursors to the LLO TCS CO2 laser chiller filter issues (ref. LLO aLOG 25658, LLO aLOG 22766, E1600282). However, thought I'd check to confirm that this is the case.

There isn't any particulate like in the LLO2276, just the usual greenish-grey coloring and sludge of the same color in the wire mesh filter. I don't think that the this greening has been getting worse, but I could anecdotally say that I find the TCSX filter needing to be swapped more frequently (need to check logs at the chiller to verify).

IFO_NOTIFY correctly saw the issue and moved to the ALERT_ACTIVE state at 13:03 UTC, but I didn't receive anything until it went back to WAITING at 14:23 UTC when Sheila fixed it.

2020-02-25_13:03:03.143111Z IFO_NOTIFY [ALERT_ACTIVE.enter]
2020-02-25_13:03:03.218729Z IFO_NOTIFY [ALERT_ACTIVE.run] SQZ OPO REFL DC Power has increased. Sending
2020-02-25_14:23:27.577427Z IFO_NOTIFY JUMP target: WAITING
2020-02-25_14:23:27.578100Z IFO_NOTIFY [ALERT_ACTIVE.exit]

Out of Observing from 1422-1423 UTC

The reason that I didn't get a notification might be that I didn't have guardian notify set to yes.  Screenshots attached.

The system is currently designed not to send Guardian alerts if the IFO is both locked and in observe. This morning when H1 was briefly taken out of observe to reset the squeezer the pending alert was issued.

Design details: if the IFO is not locked+observe and IFO_NOTIFY state transistions to state 30, then GRD alerts are immediately sent out and the system is latched to prevent repeat alerts. The latch is cleared when the IFO goes into locked+observe.

If it is determined that Guardian alerts should be sent irrigardless of the IFO state (e.g. squeezer causing range degradation), I can change the code to do this. We will need to ensure that IFO_NOTIFY does not oscillate between its states too quickly otherwise many alerts could be sent.

Attached is a plot of the DCPD power vs optical gain during this test.  Although these two traces look similar when plotted this way, there is a consistent difference between the two, and fitting for the amount of junk light inboth cases gives 1.09 +/- 0.06 mA in the high modulation depth measurement, and 1.35+/- 0.06 mA in the low modulation depth case.  

While looking at this data I realized that I had made an error in an earlier alog about modulation depths, which is corrected in the comment now.  Based on the change in modulation depth (first scan taken with a 23.4dBm epics setting on driver, which means 0.189 radians modulation depth, second scan taken at 20.3dBm setting which means 0.159 radians), we would expect that carrier light in the interferometer would be reduced by 0.5% and the sideband power injected would be increased by 40% when the modulation depth is decreased. 

This result suggests that the junk light we are seeing is probably not the 9th order 9MHz mode which is near resonance in H1's OMC 46667.  

One explanation for this result could be that I made the test while the interferometer was still thermalizing.  The attached screenshot shows trends while the measurement was taken, the fastest part of the thermal transient was over. 

There was a 0.8% increase in the circulating power in the Y arm, 0.9% increase in the X arm, when the modulation depth was increased, so both of those are sligthly higher than we expected for the change in modulation depth.  It could be that an alignment set point changes when we change the 9MHz modulation depth, which is generating extra carrier junk light. 

The increase in junk light after the 9MHz reduction made Keita and I ask ourselves why we are doing this reduction, and if the tests that motivated the change would have the same outcome in our current configuration.  It seems plausible that the reason this produces junk light (if it does) are related to TCS settings, spot positions and circulating power.

history of 9MHz modulation depth:

• Sept 2016:
  • 9MHz reduction by 6dB added to guardian after power increase.  29983
  • OMC trans image changed with 6dB reduction in 9MHz; suggestion that 900 Hz peak is reduced by 9MHz reduction, 29817 I seem to remember that this change in the 900Hz peak was not repeatable, we were just being confused by noise which came and went.
  • Identification of 9th order 9MHz mode in transmission through OMC: 29395
• Nov 2016:
  • 9MHz reduction which was done after power increase is removed from guardian because of locklosses, we weren't sure if it mattered for sensitivity.
• April 2018:
  • EOM swapped, so dBm settings are not directly comparable before and after this time. 41435
• late Oct 2018:  
  • Refl 9 slew rate limited: 44771
  • The first result in this alog looks dramatic but if you read all the comments and the story becomes much less clear.  The drive levels being compared aren't written in the alog, but I am infering from the plots that these are comparisons between 23dBm drive setting (nominal at the time, 0.184rad) and 17dBm (called redcued here, 0.130 radians) : 44781
• Nov 2018:
  • increasing 9MHz by 6dB compared to nominal setting (of 20dBm) makes DARM worse, lowering it doesn't help 45173
  • I am not finding this discussion in the alog, but my memory is that Koji told us that the EOM driver is more noisy when operating at it's max power, so we probably have more 9MHz RIN when we operate at 26dBm on the driver. 

• Mid Jan 2019: 
  • 46444 we were operating with a 20dBm setting on the 9MHz driver, (0.159 radians 55304) and reported that increasing the drive by 3dBm reduced the range by 10Mpc. (Also estimated sideband/carier ratios on OMC QPDs)

Daniel suggested that we look at what happens to the BS alignment when the modulation depth changes, since AS36 is the signal used to control the BS alignment.  Jenne looked into this and found that the beam splitter doesn't react to the change in modulation depth, but SR2 and SRM both do.  The first attachment shows that this is mostly in pitch, but there is also a yaw reaction. The second attachment shows that the AS-C QPD seems to be the reason why changing this modulation depth changes the alignment of the SRC.  (You can see the change in AS_C before the SRC2 loop brings it back to it's error point by moving the alignment of SR2+SRM).  There is not much happening in the AS72 loop (SRC1), which you would expect if they are well diagonalized. 

This suggests that the amount of junk light we have on the DCPDs can be changed by changing our offset on AS_C.  We plan to try a test of this tomorow.

I've updated the lock-loss-alert DCC document T2000061

I made a REMOTE_OWL_SELECTION.adl that has names listed as cammand buttons (only my name currently). This launches a shell script that calls Dave's owl_op and changes the IFO mode & IFO_NOTIFY state. Should make it easy.

The Pcal laser could have been potentially shut off because of the interlock system that has been down at X-end since Tuesday maintenance (last week). Someone will have a look at it tomorrow to see if that was the cause.

For more details of what we believe happened, see comment LHO aLOG 55285.

Stay tuned for results of today's investigation.

PCALX Laser functionality restored after interlock was repaired today. See LHO aLOG 55291.

Thanks for catching this Evan!! 

I've added an *additional* check to the DIAG_MAIN notification guardian to look at the *error* signals of the PCAL OFS loops, e.g. H1:CAL-PCALX_OFS_ERR_OUT16. If either PCALX or Y's OFS error signal channels are less than -2.0 for more than 120 seconds, DIAG_MAIN throws the error "PCAL: PCAL [X,Y] OFS servo malfunction". We have loaded the new DIAG_MAIN guardian check, and confirmed that it now throws the correct error given this current state.

Instructions for what to do about this (in a "normal" situation) have been updated in the DIAG_MAIN page of the OPS wiki.

Note -- Sudarshan and I tried following those basic instructions, but it appears that PCALX is more broken than just a saturated OFS control loop.

Rick and Sudarshan are actively investigating while I run calibration measurements.

"Wednesday Feb 19 2020" is a bit of a misnomer: a better summary page plot at which to determine the time of failure is the PCALX laser current here, which shows the current drop to 0.0 at 
2020-02-19 00:15 UTC, which is 
       Feb 18 2020 16:15:00 PST
i.e. Tuesday at 4pm local, i.e. only 4 hours after maintenance day closed.

This time is coincident with Keita and Fil's journey to the EX and reboot of the EX ISC IO chassis (see LHO aLOG 55171). *That* effort was a result of the CDS team attempting to mitigate sharp spectral line features earlier in the day, which thwarted some ALS WFS, (see LHO aLOG 55173).

The last sentence of that 55173 aLOG indicates what happened:
"Then we found, after driving back to the corner, that power cycling IO chassis, or maybe disconnecting AA from IO, triggered the safety system at EX to go crazy, the safety system thought that the status was fine but the laser interlock was kept open so the laser couldn't be turned on. This was manually bypassed."

The *ALS* laser system's interlock was bypassed, but the PCALX laser system's interlock was forgotten about.

The CDS/PCAL teams will further investigate if this is what happened during today's maintenance. 

Stay tuned!

Story confirmed. PCALX Laser functionality restored after interlock was repaired today. See LHO aLOG 55291.