Had about 20hrs with no IMC/FSS glitches from 9pm Thurs night until about 340pmPT Fri afternoon. But have now had several for the last 5hrs.
Also for one ~60min stretch, H1 almost made it to NLN, but had rung up violins which kept us at OMC Whitening for 20min until the next IMC glitch.
Here are some running notes for the rough 5hrs filled with glitches (currently have H1 in IDLE).
Notes During Return of IMC/FSS Glitch Locklosses
2340utc (340pmPT) LOCKLOSS with IMC tag was the beginning of IMC/FSS glitch locklosses (where we went almost 20hrs since the last glitch around 5utc (9pmPT).
0111-0208: OBSERVING
0208utc (608pmPT): Attached screenshot shows a coincident glitch lockloss seen on FSS & IMC. Here is the LOCKLOSS page also tagged with IMC glitch.
Starting an FRS Ticket to mark the DOWNTIME (FRS-32644) for 3.5hrs of DOWNTIME so far. Where there was notable part of downtime due to IMC not locking easily.
- 0241 Lockloss at DARM to DC ReadOut looked like a FSS/IMC glitch lockloss---> IMC + FSS lockloss tags.
- 0255 Lockloss at RESONANCE also looked like a FSS/IMC glitch lockloss---> IMC lockloss tag
- 0256 Lockloss at READY ---> FSS lockloss tag
- 0334 lock made it almost to NLN, Yay Yay Yay!!!! BUT it was a little delayed due to....
VIOLINS Rung Up A Bit: (haven't had this since my shift on Tues night!) So were stuck at OMC WHITENING for over 20min before H1 had....
So for the last 3hrs:
- 0208-0256: 50min with a flurry of FSS/IMC glitches causing locklosses at various states also causing IMC locking grief.
- 0256-0357: Hour of NO glitch locklosses, and almost made it to NLN (but rung up violins prevented that). Then another IMC glitch.
- 0420 Went to IDLE due to more locklosses and for a pause
Not sure what is more useful. Keep having ISC LOCK continue to locking, or to leave it in IDLE for IMC/PMC/FSS monitoring....
Fast shutter worked and is working. (I was worried that the mirror was broken off but that's not the case, the shutter still opens and closes without an issue.)
From the guardian log (attached), it failed because AS_B_DC_NSUM was 20.3 when the shutter was closed and the threshold is 15. AS_B was 0.28. Fundamentally, this is because I reduced the gain of this DC path by 20dB and digitally compensated in search for an existing sensor that won't rail when lock is lost, and the same fast shutter test failure was observed (see alog 79104). Back then, I wrote "I adjusted the dark offset while IMC was unlocked but we can probably increase the threshold to 30 or so if it continues to bother us."
The dark offset has drifted enough to bother us at times (i.e. it is about 12 counts or so on average now), so I told Ibrahim to increase the thresholds only for AS_B, i.e.
PdBDarkLo=-40 # dark level low
PdBDarkHi= 40 # dark level high
in FAST_SHUTTER.py.
Since we now have a dedicated diode to look at lock loss power spikes, we can turn down the analog gain of AS_B_DC by 20dB and turn off +20dB filter in FM4 of H1:ASC-AS_B_DC_SEG[1234] filter, and adjust the dark offset again on Tuesday.
The mirror is still attached to the fast shutter and is moving. It actually closed using HV (see GS13 signal) when the test failed (2nd attachment), and later when I opened/closed it with low voltage drive it responded correctly (3rd attachment).