Ops Shift Transition: 02/07/2020, Day Shift 16:00–00:00 (08:00-16:00) - UTC (PT)
State of H1: Locked
Intent Bit: Observing
Weather: 0-15 mph wind
Primary 0.03 – 0.1Hz: 0.01 um/s
Secondary 0.1 – 0.3Hz: 0.2 um/s
Outgoing Operator: Camilla
Quick Summary: Locked and Observing for 13 hours, range has been decreasing for about 4 hours (maybe SQZ related).
Locked 9 hours. Quiet shift so far.
TITLE: 02/07 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 120Mpc
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 4mph Gusts, 2mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.21 μm/s
QUICK SUMMARY: Locked 5 hours.
To be clear, the ETMY damping gains were reverted, but not the Mode20 filterbank changes, as we were worried we would get knocked out of Observe.
TITLE: 02/06 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC STATE of H1: Observing at 120Mpc INCOMING OPERATOR: Camilla SHIFT SUMMARY: Had some difficulty reacquiring lock from the lock loss at the beginning of the shift. No issues since. LOG: Lost lock at 23:45 UTC. Relocking. 00:16 UTC Lock loss from MOVE_SPOTS. 00:36 UTC Lock loss from ENGAGE_SOFT_LOOPS. 00:41 UTC Lock loss from ACQUIRE_DRMI_1F. 00:45 UTC Lock loss from LOCKING_ALS. 01:05 UTC Lock loss from POWER_10W. Starting initial alignment. 01:23 UTC GRB-Short E363135 Ignoring (not locked). 01:29 UTC Initial alignment complete. No problems. Relocking. 01:44 UTC Lock loss from ENGAGE_ASC_FOR_FULL_IFO. 02:11 UTC Lock loss from PREP_DC_READOUT_TRANSITION. 02:25 UTC Pausing at ENGAGE_RF_VIOLINS. 02:26 UTC Pausing at PREP_ASC_FOR_FULL_IFO. 02:27 UTC Pausing at ENGAGE_ASC_FOR_FULL_IFO. 02:35 UTC Pausing at ENGAGE_SOFT_LOOPS. 02:40 UTC Pausing at ENGAGE_DC_VIOLINS. 02:42 UTC Pausing at POWER_10W. 02:45 UTC Pausing at MOVE_SPOTS. 02:53 UTC Requesting NLN. 03:02 UTC NLN. 03:06 UTC Accepted SDF differences. Observing. 03:16 UTC GRB-Short E363145 INJ_TRANS set to INJECT_KILL Swift Trigger duration 1.024 Stand Down 04:32 UTC INJ_TRANS set back to INJECT_SUCCESS.
Jim, Patrick The earthquake response plot linked from seismon has been changed to update as new earthquakes are detected. So instead of opening a plot each time there is an earthquake notification, you can just leave it open and it should update as needed. The new script is at '/opt/rtcds/userapps/release/isi/h1/scripts/animated_eq_plot.py'. This is what the button links to now.
Had some difficulty reacquiring lock from the lock loss at the beginning of the shift. The cause is not immediately clear. Currently standing down for a GRB at 03:16 UTC.
03:06 UTC Cheryl verified that I could accept the attached SDF differences.
Tagging SUS to review the SDF diffs
Yesterday afternoon I installed the latest version of the Lock Loss Alert system. The main new feature is the ability to send repeat alerts with a separate delay time to that of the single alert. I also redesigned the MEDM screens to make them less cluttered (see attachment). The system is documented in DCC-T2000061
While watching the lock acquisition, I was surprised at how long we were in the state MOVE_SPOTS before the A2L gains are actually changed. Looking at the code, it is because there are some sleep statements in there for initializing the TMS centering servo that don't need to be in there.
Recall that we use a servo (that Craig implemented over the October break) to move the TMSs such that we are centered on the IR trans QPDs, and maintain that centering as we engage ASC and change the IFO alignment. The first time the servo is used in ENGAGE_SOFT_LOOPS, we are careful to zero the offsets of the TMS TEST filter banks carefully, if they were somehow left on (they get turned off in DOWN). Then a cds.servo is used, and when we leave the state we leave the TMS in its new pointing.
The servos are started up again when we go to MOVE_SPOTS. But, we seem to be resetting the pointing of the TMS before starting the servos up. The way the reset code was written, it waits 30 seconds to ramp off each TMS offset, but it does it in series, for a total of 2 minutes of sleeping. We shouldn't be resetting the offsets here, because that erases the centering that was done earlier, so I commented out this whole section which also removed the sleeps.
It's possible that the IFO liked this 2 min wait to get some time for convergence and thermalization after going up to 10W. If there are problems at this state, please uncomment lines 3611-3621 in ISC_LOCK to put these sleeps back in (if this is needed, I'll fix them on ~Tuesday so that they are timers and not sleeps, since we're not checking for lockloss during each of these 30 sec sleeps).
Separately, we should probably be resetting the A2L gains to their centered locations in PREP_FOR_LOCKING, rather than waiting until PREP_ASC_FOR_FULL_IFO, since we engage DHARD before prep_asc_for_full_IFO. Seems to not be a big deal, but we are jerking that around on poor DHARD when we set them to their centered values just before turning on the dither lines, so we're changing the amount of DHARD noise injected to DARM.
The lines to uncomment are now 3616-3626 if needed, since I added a few lines in PREP_FOR_LOCKING to center the A2L gains (see alog 54950).
We lost lock once in move spots today (after Jim did an initial alignment). I serached on the lockloss page for locklosses at move spots (506). We've had two lcoklosses here since this change was made, and we had 3 locklosses from move spots in January. I rearranged the order of these guardian states on Jan1st, 54219, which is how some of these pauses got in here. Since we've only had two locklosses, and we made it though for now, I'm not going to make a change, but if we keep having locklosses here it might be that we do need to wait for the TMS servo.
TITLE: 02/06 Day Shift 16:00 – 00:00 (08:00-16:00), all times posted in UTC
STATE of H1: Locking
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Three fast locklosses over the course of the shift, not sure as to the cause.Currently re-locking at ACQUIRE_DRMI_1F.
LOG:
16:57 (08:57) GRB 363095 -- trigger duration too long
17:08 (09:08) Ed to Optics Lab
17:19 (09:19) Ed out of Optics Lab
18:04 (10:04) Fast lockloss
18:34 (10:34) Tyler processing tumbleweeds on X-arm
19:55 (11:55) At NLN, going to Observing
21:41 (13:41) Going out of Observing for 0.5-1 hour of commissioning
21:49 (13:49) Tumbleweed work done for the day
22:07 (14:07) Fast lockloss
22:55 (14:55) At NLN, going to Observing
23:45 (15:45) Fast lockloss
TITLE: 02/06 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 7mph Gusts, 6mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.26 μm/s
QUICK SUMMARY: Lost lock at 23:45 UTC. Relocking.
I haven't found any particular pattern yet of why this is happening, but sometimes we make it through ENGAGE_SOFT_LOOPS the first time around, and sometimes we lose lock there, and then make it up to NomLowNoise the next time. I noticed this because we get to ~300 kPc and then lose lock, which shows up as a small blip up in the range plot during acquisition.
If an on-shift operator has some time, it might be interesting to try to understand why this is happening, so that we can fix it. On several of the times that we make it through on the first time, we are back locked extremely quickly!
The attached screenshot is 3 days of range and guardian state, just showing that sometimes we lose lock there, and sometimes we make it through.
I'm not sure that this is it, but I wonder if the IFO isn't happy when the A2L gains aren't at their 'centered' values (see my side note in alog 54949). Recall that initial alignment is set to the centered values of the optics.
The DOWN state, and through the CARM offset reduction sequence, there are no changes made to the A2L gains for the test masses; they are left at whatever value they had at lockloss. DHARD is turned on using whatever A2L values were there. In PREP_ASC_FOR_FULL_IFO we set the A2L gains to the values that will center the beam on the optic, then we turn on the ADS lines. In Engage_ASC_FOR_FULL_IFO we start servoing the PRM to make the spot centered on ITMY, and then in ENGAGE_SOFT_LOOPS we start moving the arms to center the spots on the ETMs.
Since the A2L gains are set to their centered values before we start engaging the full IFO ASC, I'm having trouble coming up with a reason that this should be causing egregious problems. However, it feels awfully suspicious that we seem to be making it to NomLowNoise the *second* attempt pretty regularly, where the first attempt will always be with the high power A2L values (if we lost lock from NomLowNoise, or anywhere after MOVE_SPOTS), and the second attempt (if we lost lock after PREP_ASC_FOR_FULL_IFO) will be with the centered values.
I have put a few lines of code into prep for locking to set the A2L gains to their centered values so that we always start with the same situation every lock. If this is bad, operators can comment out lines 427-430, near the center of PREP_FOR_LOCKING in ISC_LOCK.
Tagging PSL for future reference/search.