STATE of H1: Observing at 152Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 5mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.44 μm/s
QUICK SUMMARY:
IMC_LOCK Issue:
IMC_LOCK Is stuck in a loop, between Close_ISS (50) and LOCKED (100)
IMC_LOCK [CLOSE_ISS.run] USERMSG 0: Diffracted power jumped too much, toggling secondloop
Found Camilla's alog about this issue: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=71357
But it seems like it resolved it's self for her relatively quickly when she ran into this issue.
Tried to take IMC_LOCK to CLOSE_ISS and to To MCWFS_OFFLOADED but I still end up in the same loop.
Camilla Suggested maybe changing the IMC_lock code to allow for a bigger Diffracted Power Jump
Line 529 ,in IMC_LOCK.py, I wanted to try this, but was hesitant.
Called up Jason, Who then Mentioned that Keita would know more about the specifics of this particular issue.
Rang Keita, who said changing that line is likely not the real solution.
While we were making a plan to resolve the issue. It resolved it's self after about 2 hours of not working properly.
The issue was very likely: the electronic offset in the board has changed enough that the combination of the 2ndLoop REF Servo H1:PSL-(ISS_SECONDLOOP_REFERENCE_SERVO_OUT16) and 3rdLoop offset can no longer compensate for the changes it needs any longer. Tagging PSL team.
Note to other operators!:
If H1 has a lockloss, contact Jason to touch up the RefCav as this RefCav transmission is too low and liekly a contributing factor to the IMC issue.
FMCS Issues: The FMCS IOC keeps going down and thus the FirePump alerts are going off along with the Temperature sensors not giving us any data through MEDM or NDSCOPES.
Dave is aware of this and has been actively trying to find a way to automate the Restarting process of that IOC to limp us along until tomorrow. so far it's failed twice during this shift. at 9:47UTC and 10:45 UTC. If you have gotten an alert it is likely due to the FMCS IOC errors we have been having. Tagging CDS and FMCS.
See my annotation of infinite loop of 2nd loop enabling-disabling due to mismatch between the electronics offset and the 3rd loop offset. 1st loop is REALLY slow to respond to the change in the 2nd loop board DC output (because the 2nd loop output is added to the already whitened 1st loop sensor signal), which doesn't help either.
The way this works is that the diffraction average is measured just before the board output is enabled, then the guardian waits for 10 seconds (waiting for the reference servo to take care of any remaining electronics offset that the third loop offset could not counteract), enables the 2nd loop servo, wait for a while, measures the diffraction again, and if that's close enough to the original number it's satisfied.
In this case, it seems that the electronics offset drifted enough so 10 seconds is not quite enough for the reference servo to take care of that. The solution would be to tune the 3rd loop offset.
After the conclusion of O4a, I adjusted H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET (was originally 949, now 959.5).
The procedure is really simple.
1. Put the ISS 2nd loop in the configuration shown in the 1st screenshot. Power into IMC doesn't matter, this is only about electronics, but it's better if the 1st loop is working so that you can confirm that your work is not doing any harm. (When I came into the control room the ISS was already in a good state to start the work.)
2. Confirm that H1:PSL-ISS_SECONDLOOP_EXC_MON is not changing much at DC by making a trend. If it's still trending up/down, wait for two minutes. Ignore the change of 0.1/minute, we're talking more about 1/minute.
3. Read H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET value (it was 949). Add the average DC value of H1:PSL-ISS_SECONDLOOP_EXC_MON (was about 11) to make a new number (i.e. 960 in this case).
4. If this is to be done while the IFO is locked, you might want to set H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET_TRAMP to 100. I did that even though it wasn't necessary in this case just to show that doing so will make this procedure almost transparent to the 1st loop (2nd attachment, t~-40min, see the 1st loop readout of the 2nd loop output and the diffraction).
5. Put the new number (960 in this case) into H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET.
6. Wait for ~2 minutes to see that H1:PSL-ISS_SECONDLOOP_EXC_MON goes down to less than 1. You can also fine-adjust it, but it will drift after some time. If this is less than 1 it's already pretty good.
7. If you changed H1:PSL-ISS_THIRDLOOP_OUTPUT_OFFSET_TRAMP to 100, bring it back to 3.
8. Be happy.
