Since yesterday we have not got past LOCKING_ALS alog 67869. Had this issue as soon as we lost lock at 1362369528 (ENGAGE_ASC_FOR_FULL_IFO).
Can get to GREEN_ARM_LOCKED and follow first steps of LOCKING_ALS, until DARM1 gain is turned on. I can turn it on by hand slowly to 30 but higher than this or when turning on ETMX boosts it becomes unstable. Plots of ALS_DIFF locking nominal vs today attached.
I attempted to take a DARM1 OLG measurement when "locked" at DARM1 30 gain. It didn't show anything, see attached. (Excitation amplitude 1000).
There was an MRS temperature excursion but this issue started before that.
Daniel found ALS-C_SHG TEC has had an error flag and not been on since 2022/12/21 16:34 UTC. alog 67879. Both PLL's looked fine.
Dave reminded us that the channels H0:FMC-EX_MAINS_CHAN_{1,2,3}_VOLTAGE can be used for trending to see any mains power glitches. We didn't find any.
Jason and Ryan S checked on the PSL laser.
Koji, Naoki, Jenne
Camilla suggested that we try locking ALS_DIFF using ETMY. We checked that most of the filters were the same, except the L1 'control2' filter in ETMX is not present in ETMY. However, the 'control' that is in ETMY is pretty similar, so we gave that a try.
We turned on some bias for ETMY (so that the bias is maximaly negative). We put a +1 in the LSC output matrix for EY and 0 for EX. We tried both +20 and -20 for the DARM1 gain (half the nominal +40, but trying both signs). Things seemed pretty similar, as if nothing is happening, then once something starts happening we are not able to lock, even with a low gain.
I ran ISC_LOCK through DOWN to Green_arms_manual, so SDF revert should have put back to normal all of the ETMY buttons that we clicked.
Probably Camilla mentioned it in her alog, but every time we had (earlier) tried to lock with the nominal ETMX, when we increase the DARM1 gain from 30 to 40 (nominal is 40, then immediately ramps up to 400), we see a ~1 Hz oscillation. We can make it go away by lowering the gain.
Also, COMM is fine. We can by-hand lock COMM first, then start DIFF (this is the order we used to use).
Koji suggests trying to get up to something like ALS_DIFF's Prepare state, then put in an excitation to watch that indeed the frequencies move around in the way that we expect (i.e. can we follow the signal around the loop).
Last time successfully ran ALS_LOCKED: 2023/03/09 03:47:13 UTC Finished LOCKING_GREEN_ARMS @ 1362368797. Finished ALS_LOCKED @ 1362368851. (Did have one failed attempt before this at 1362368445 but the issue was different, green light just fell away for Y-arm).
First failed time 2023/03/09 04:00:20 UTC: Finished LOCKING_GREEN_ARMS @ 1362369626, Lost lock at LOCKING_ALS @ 1362369638.
Listing these times to make it easier to trend what settings were then verus now.
I have a ndscope template of the green arms and DIFF PLL CTRLMON on ligo/home/camilla.compton/Desktop/ALS_DIFFyaml
Working off Koji's suggestion above with Ibrahim. Keita, Gabriele, Rahul.
Question 1: Is EXC into L1 ETMX going to DIFF_PLL_CTRLMON? Awnser: YES. at 1Hz and 2.5Hz
Question 2: Is 1Hz EXC into L3 ETMX (Drivealign L2L) going to DIFF_PLL_CTRLMON? Answer: No. We then tried putting into a excitation into pitch (Drivealign P2P) and watching Oplevs and saw nothing.)
You can see in attached image since the Tuesday night lockloss the ESD LV MONs have been at 24V. Rahul and I tried turning OFF/ON the EDS high voltage and each quadrants Hi/Li. Nothing Changed. Fil and Rahul are currently investigating at EX.
More details of what we did this morning to diagnose this ALS DIFF / ESD LV issue:
- Georgia had checked the ETMX top mass wasn't rubbing early this morning.
- We checked we sill had the same issue this morning.
- We set up an awggui excitation into ETMX_L1_TEST_L (50k amplitude at 2.5Hz) and saw DIFF_PLL_CTRL_OUT was measuring 0.02um for a 4um movement on the L1_OSEM. Attached ndscope.
- Repeated this with 10k amplitude at 1Hz and saw and saw for a 4um movement on the L1_OSEM, the DIFF_PLL_CTRL_OUT was measuring 1um for a 2um movement on the L1_OSEM. Attached ndscope.
- This is a factor of 100 different that initially worried us until Keita reminded us we were looking at L1 movement not L3 so the suspension was just damping the 2.5Hz exc.
- We confirmed that DARM1 in was the same signal as DIFF_PLL_CTRL_OUT
- Jenne told us that the DARM1 feedback goes into L3 and L1.
- We set up a awggui excitation into the ETMX ESD L3_DRIVEALIGN_L2L_EXC. Tried 11Hz and 1Hz amp up to ~100k, saw nothing on DIFF_PLL_CTRL_OUT. We temporarily turned off the 17.6Hz CAL line here to make it simpler for us to see.
- We attempted the same on ETMY. Turned on a BIAS offset but then realized the HV wasn't on and clicking the HV ON/OFF button didn't seem to do anything so we turned this back off and forgot about ETMY. Jeff K later told me we don't expect the green light to stay on so we were just confused.
- We set up a awggui excitation into the ETMX ESD L3_DRIVEALIGN_P2P_EXC. 1Hz amp up to ~100k, saw nothing on DIFF_PLL_CTRL_OUT or the OPLEVs. Rahul came to help.
- Gabriele realized ( above image) that the ESD LV MONs weren't changing and we foudn they have been at 24V since the Tuesday night lockloss.
- Fil and Rahul found the ETMX ESD LV Interface had an issue and swapped it.
How could we have found it sooner? Maybe a more thorough check of SUS health or a script that can easily check this.
ETMX ESD low voltage chassis replaced, see alog 67899 for details.