This is a followup of https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=57094 .
The saturation of MC1 M1 stage came back this morning. There was a huge 19Hz oscillation going on in mostly top three coils, and it was also the case on Friday (first attachment). The oscillation went away when we disabled PIT damping, but half gain was already bad. Nothing apparently wrong in filter definitions and gain setting.
To catch any electronics problem in T2 and T3 coils, after disabling P damping as well as alignment offsets, I injected into H1:SUS-MC1_M1_TEST_P_EXC and measured the transfer function from COILOUTF input of T2 and T3 coil to corresponding FASTIMON and NOISEMON. I did the same thing with Y for LF and RT coils. If everything is working fine, all coils should show the same transfer function amplitude.
Looking at the result (2nd attachment), T3 coil output is about 20dB higher than T2, LF and RT for f>10Hz, as if the z:p=10:1 low pass filter is not turned on. We switched the BIO from state 2 (nominal) to state 0 and back to state 2, but got the same result.
We repeated the same measurement in state 0 and the discrepancy was gone (3rd attachment), T3 result was consistent with T2 and looked quite similar to state 2 T2, RT and LF results (except that the coherence of T3 coil was smaller than the others, so something may still be wrong even with this state).
At this point we convinced ourselves that something about the analog low pass filter for T3 was broken. We left it in state 1 (LPF is turned off) and PIT damping seems to be working OK.
If we have a replacement driver we'd like to swap the existing one.
DTT template is this one: /ligo/home/keita.kawabe/MC1M0T3BIO.xml
You disable P damping as well as alignment offsets, enable P test output and run the template. If T2 and T3 traces are at -31-ish dB for the entire frequency range (there's a green reference trace) they're good.
If you want, you can change EUL2OSEM matrix P column to route the excitation to other OSEMs to make measurements for other coils at once.
Links to the FRS (16045) ticket and IIET issue (16047) is given below,
https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=16045
https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=16047
Keita, Sheila, Corey, Dave, Fill, Eric, Rahul
This morning Fill and Dave power cycled the electronics and IO chassis for for h1sush2a (LHO alog 57113). We reset the watchdogs and IOP dackill for (MC1, PRM, PR3) and then set the GRD state to Damped (from SAFE).
Next we ran the dtt template which Keita made/ran last night ( /ligo/home/keita.kawabe/MC1M0T3BIO.xml ), both for CD state 1.0 and 2.0. The two plots are attached below. Both the plots shows that T2 and T3 traces for MC1 osem are at around -31dB for the entire frequency range, which says that they are now working perfectly fine.
After the measurements were complete, I switched off the P_TEST filter and switched ON the P_DAMP, left the CD state at 2.0 and left the SUS in DAMPED state (and then to ALIGNED state, and PRM to MISALIGNED, after talking to Sheila).
Note:- earlier I was getting error while running the dtt and awggui (and Dave/Eric tried troubleshooting), this happened because I did not restart my dtt after Dave power cycled h1sush2a. For the dtt to run properly we should kill it and open a fresh window, especially when we see such errors.
At 4pm (23:00UTC) found that this saturation had came back- and had been happening since 11:45am(18:44UTC). We retook Keita's test template and found the problem had came back, see attached screenshots.
We left the MC1 M1 analog low pass filter OFF (in state 1) for tonight so it is in a safe state.
Cheryl has found a maybe related problem: see alog 57124
