Since yesterday we have not got past LOCKING_ALS alog 67869. Had this issue as soon as we lost lock at 1362369528 (ENGAGE_ASC_FOR_FULL_IFO).
Can get to GREEN_ARM_LOCKED and follow first steps of LOCKING_ALS, until DARM1 gain is turned on. I can turn it on by hand slowly to 30 but higher than this or when turning on ETMX boosts it becomes unstable. Plots of ALS_DIFF locking nominal vs today attached.
I attempted to take a DARM1 OLG measurement when "locked" at DARM1 30 gain. It didn't show anything, see attached. (Excitation amplitude 1000).
There was an MRS temperature excursion but this issue started before that.
Daniel found ALS-C_SHG TEC has had an error flag and not been on since 2022/12/21 16:34 UTC. alog 67879. Both PLL's looked fine.
Dave reminded us that the channels H0:FMC-EX_MAINS_CHAN_{1,2,3}_VOLTAGE can be used for trending to see any mains power glitches. We didn't find any.
Jason and Ryan S checked on the PSL laser.
Koji, Naoki, Jenne
Camilla suggested that we try locking ALS_DIFF using ETMY. We checked that most of the filters were the same, except the L1 'control2' filter in ETMX is not present in ETMY. However, the 'control' that is in ETMY is pretty similar, so we gave that a try.
We turned on some bias for ETMY (so that the bias is maximaly negative). We put a +1 in the LSC output matrix for EY and 0 for EX. We tried both +20 and -20 for the DARM1 gain (half the nominal +40, but trying both signs). Things seemed pretty similar, as if nothing is happening, then once something starts happening we are not able to lock, even with a low gain.
I ran ISC_LOCK through DOWN to Green_arms_manual, so SDF revert should have put back to normal all of the ETMY buttons that we clicked.
Probably Camilla mentioned it in her alog, but every time we had (earlier) tried to lock with the nominal ETMX, when we increase the DARM1 gain from 30 to 40 (nominal is 40, then immediately ramps up to 400), we see a ~1 Hz oscillation. We can make it go away by lowering the gain.
Also, COMM is fine. We can by-hand lock COMM first, then start DIFF (this is the order we used to use).
Koji suggests trying to get up to something like ALS_DIFF's Prepare state, then put in an excitation to watch that indeed the frequencies move around in the way that we expect (i.e. can we follow the signal around the loop).
Last time successfully ran ALS_LOCKED: 2023/03/09 03:47:13 UTC Finished LOCKING_GREEN_ARMS @ 1362368797. Finished ALS_LOCKED @ 1362368851. (Did have one failed attempt before this at 1362368445 but the issue was different, green light just fell away for Y-arm).
First failed time 2023/03/09 04:00:20 UTC: Finished LOCKING_GREEN_ARMS @ 1362369626, Lost lock at LOCKING_ALS @ 1362369638.
Listing these times to make it easier to trend what settings were then verus now.
I have a ndscope template of the green arms and DIFF PLL CTRLMON on ligo/home/camilla.compton/Desktop/ALS_DIFFyaml
Working off Koji's suggestion above with Ibrahim. Keita, Gabriele, Rahul.
Question 1: Is EXC into L1 ETMX going to DIFF_PLL_CTRLMON? Awnser: YES. at 1Hz and 2.5Hz
Question 2: Is 1Hz EXC into L3 ETMX (Drivealign L2L) going to DIFF_PLL_CTRLMON? Answer: No. We then tried putting into a excitation into pitch (Drivealign P2P) and watching Oplevs and saw nothing.)
You can see in attached image since the Tuesday night lockloss the ESD LV MONs have been at 24V. Rahul and I tried turning OFF/ON the EDS high voltage and each quadrants Hi/Li. Nothing Changed. Fil and Rahul are currently investigating at EX.
More details of what we did this morning to diagnose this ALS DIFF / ESD LV issue:
How could we have found it sooner? Maybe a more thorough check of SUS health or a script that can easily check this.
ETMX ESD low voltage chassis replaced, see alog 67899 for details.