TITLE: 03/22 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ed
SHIFT SUMMARY: Less than successful evening for locking/observing. Made it to NLN briefly a few times. Struggling to get past LOWNOISE_ESD_ETMX. Commissioners investigating cause.
LOG: Previous aLogs.
Sharan Banagiri, Anamaria Effler, Philippe Nguyen, Kara Merfeld, Robert Schofield
23:00 - 01:30 UTC
2:43 - 2:54 UTC
We completed magnetic injections at EX and we made magnetic injections with the large coil at the corner station, some shaker injections on HAM6, and a PSL table injection. We're behind and expecting that we will need to make injections the week of April 14.
Unknown cause, but PEM crew is doing injections in CS.
[Jenne, Sheila, Anamaria]
It took some convincing to get the IFO to lock today. A big part of the reason that we've struggled to get through the CARM offset reduction sequence is the 60Hz noise. Sheila unplugged the Agilent from the IMC board (alog 47741), and then we were able to get to high power. Recall that we found this problem in January (alog 46465), but somehow having it plugged in the past few weeks hadn't been a problem, so we've been leaving it there. Perhaps something changed on Tuesday, after which we started noticing the 60Hz lines more? But they haven't been a blocker for locking until ~yesterday.
After we were able to get to 35W, we started to have trouble with the same 4.2 Hz resonances that Georgia and Craig were fighting last night (alog 47737).
Sheila measured CHARD_P, since that sees this 4.2 Hz quite strongly, and we determined that it had extremely marginal phase. CHARD is not currently a noise limiter (DHARD is), so we decided to change the cutoff to give more phase, but allow a bit more drive around 20 Hz. This is new cutoff is now in the guardian for both CHARD P and CHARD Y, although as a consequence we have put CHARD injections back onto the noise budget list to ensure that they aren't limiting us.
We were also suspicious of LSC-MICH, since it is one of the loops that sees the 4.2 Hz oscillation grow just before lockloss. However, the lower side of its phase bubble is at 5 Hz not 4.2 Hz. In any case, it doesn't look like a LSC-MICH instability, but it also does still look as though if we lower the MICH UGF much further (this measurement was done just before lownoise length control), we're going to have problems. The LOWNOISE_LENGTH_CONTROL state was reducing the MICH gain from 2.8 to 2.5, which would put the MICH UGF at 7 Hz with less than 20 degrees of phase. As long as MICH FF takes care of MICH noise in DARM, it should be fine to leave the UGF higher.
We also wanted to put the ETMX spot position back to where Anamaria had found the most circulating power, but as was discovered last night this puts too much drive onto the L2 stage of the suspension for the lowest noise coil driver state. At LLO, they run with the intermediate-level coil driver state for the L2 stages, so we've changed the COIL_DRIVERS state to do the same for us. So far we don't see an obvious noise impact, but we should check to confirm. This lets us have the better spot position without having the large A2L coupling that existed last night.
I used the script that Joe B adapted from Shivaraj (44473) to look at data from a broadband injection that Craig did using PCAL Y last night.
The script used is attached, it is also in aligocalibration/trunk/Runs/O3/H1/Scripts/CALCS_FE As Joe said it is only correcting DELTAL for the whitening, and multiplying GDS calib strain by the arm length to calibrate it into displacement.
The sign of the GDS strain channel agrees with the sign of CAL DELTAL. (First attachment).
The sign of both CAL DELTAL and GDS calib strain are wrong (second and third attachments). Craig is injecting on PCAL Y: at DC applying pressure to the highly reflective surface of the optic makes the Y arm longer, which make DARM more negative. We are making this measurement at frequencies above the suspension resonances, which give a 180 degree phase shift. So, if the sign of the calibration were correct, the phase between PCALY excitation and DARM should be 0 degrees. Since we are measuring 180 degrees for both DELTAL and GDS calib strain, we flipped the sign.
We flipped the sign in the inverse sensing function gain, and the actuation sign we flipped for each stage in the CAL CS TM_OUTPUT_MTRX. The sign was flipped at 23:02 UTC March 21st.
This undoes the sign flip from 47574. Since we still think that the logic that led us to flip the "arm sign" filter was correct and it is in agreement with the calibration groups documentation of their sign convention, we are leaving the "arm sign" as -1 for the x arm and +1 for the y arm.
The first attached plot shows the trend of relative phases between PCAL & DELTAL and PCAL & GDS since February 1, 2019 to till date. I sampled one data point per day (at 00:01:00 UTC) and also kept only those points during which the IFO was in low noise. In the plot, I have plotted both H1 and L1 for comparison purposes. I have also added a plot (second png) showing similar comparison during end of O2. These plots suggest that the current sign of both detectors follow the expected sign convention of Lx-Ly. In the first plot we can also see the interim change in the sign of H1 that happened a few weeks ago. I have used the same code base as in the above alog, but just looked at the relative phase at high frequency calibration lines (the ones used for calculating sensing function).
Note: The line frequencies were changed at LLO between O2 and O3 and hence we expect to see small differences due to that.
These changes were accepted into CAL-CS safe.snap, namely, H1:CAL-CS_DARM_ERR_GAIN = 1.03 H1:CAL-CS_DARM_TM_OUTPUT_MTRX_2_2 = -1.0 (ETMX UIM to dL_UIM) H1:CAL-CS_DARM_TM_OUTPUT_MTRX_3_3 = -1.0 (ETMX PUM to dL_PUM) H1:CAL-CS_DARM_TM_OUTPUT_MTRX_4_4 = -1.0 (ETMX TST to dL_TST) We're still working on the documentation to make sense of all this sign stuff.
TITLE: 03/21 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Locking
INCOMING OPERATOR: Travis
SHIFT SUMMARY: IR comm was difficult, comm offset needed to be set by hand, DRMI ASC was also a constant problem
LOG:
17:00 Robert to CER
17:00 Lockloss because someone did something in ASC
17:30 Lockloss
18:00 Lockloss
18:20 Lockloss, to initial alignment
20:15 Lockloss, follow by long period struggling with IR comm and DRMI asc
21:48 Lockloss
Shift was a bit rough - spent 11am to 4pm on relocking/alignment with Jenne and Sheila. Back into locking, Anamaria and the Robert crew jumped on PEM injections.
ITMX and ITMY both have St2 sensors that could be addressed. If I recall correctly this has been the case for a couple of weeks now.
Details: https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20190311
Topped off Crystal Chiller with 150mL. Diode Chiller OK. Filters look fine (mentioned slight yellow tint of Crystal filter to Jeff B & he said it is OK).
We have just banked a bit over 6 hrs of OBS time, but need to make some commissioning changes due to some egregious low f noise, as well as start PEM injections scheduled for the day. Sheila is starting some short commiss work, Robert and crew are on deck for PEM inj at 10:30am.
TITLE: 03/21 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 98Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
Wind: 3mph Gusts, 2mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.21 μm/s
QUICK SUMMARY:
TITLE: 03/21 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 98Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: Trouble in the beginning but Craig and Georgia figured it out. After that I played with the CO2Y power a few steps. I had meant to choose the new mask (in the annular location), but apparently it didn't switch when I requested it and I hadn't noticed. I brought it back to its usual 0.0W before I left, but it will need a bit of time to cool down.
LOG:
0729 Lock loss from Nominal Low Noise
0833 Lock loss from Nominal Low Noise
1005 - 1006 Out of Observing
1012 - 1015 Out of Observing
1101 - 1103 Out of Observing. CO2Y laser lost its lock point, recovered.
1136 - 1137 Out of Observing to adjust CO2Y power from 0.0 to 0.1W
1232 CO2Y power from 0.1 to 0.2 W
1338 CO2Y power from 0.2 to 0.3W
1427 CO2Y power from 0.3 to 0.2W
1500 CO2Y power from 0.2 to 0.0W
1500 Bubba to EY chiller yard
We have been having continuing problems with a 4.18 Hz oscillation saturating ETMX L2 within 10 mins of reaching NLN. Yesterday afternoon Sheila and Jenne removed the ITMs from DHARD P to help with a similar oscillation (but at 3.2 Hz) that was also saturating EX L2. Sheila put the ITMs back in to DHARD P this evening (we've accepted this in SDF). The 4.18 Hz caused another 4 locklosses since that change. The ETMX L2 P2L gain was 4.0 for all of these locks.
We suspected the oscillation was related to the pro-spring detuning of DARM, and tried some SRCL offsets, but this did not work. In the last lock we lowered the ETMX L2 P2L gain to 1, this seems to have fixed the problem. To keep the dither loop maintaining the same old spot position, we added a 0.25 offset in the PIT4 loop, just for this lock. This will compensate for the offset added by the P2L gain change. This might need to be reverted by hand tomorrow.
We also lost lock several times this evening around CARM_TO_ANALOG. When we tweaked up alignment of PRM, SRM, PR2, and SR2 at PREP_AS_SHUTTERS we made it through ok. Before we pass CARM_TO_ANALOG the 60 Hz, 120 Hz and harmonics visible in the DARM FOM seemed much bigger than last week, unsure why.
Our lockloss tool isn't working at the moment (Jamie and others are working on fixing this), so I went through and found the approximate lockloss times that I think Georgia is talking about:
The 60 and 120 Hz noise was very high durring the final stages of the CARM offset reduction, as Georgia says, this has been causing us saturations and locklosses since Tuesday. We found that the aliglent was plugged into the IMC board excitation points, although the excitation switch was off disconnecting the cables seems to have fixed at least the 60-120 Hz problems in the CARM offset reduction.
Investigations of CHARD P and the 4 Hz problem last night are ongoing.
Craig and Georgia were able to get the IFO stable at low noise.
Attached relevant SDF diffs I accepted.
Had to go out and into observing a few times to get AWG lines set up and TCS. Times out of observing:
1005 - 1006 UTC
1012 - 1015 UTC
1101 - 1103 UTC Out of Observing. CO2Y laser lost its lock point, recovered.
We got a notification of a 6.2 earthquake in Vanuatu this morning. When we started seeing effects in the ASC, we switched SEI_CONF to the EARTHQUAKE state, even though my plot put this earthquake barely into the LARGE_EQ state, the EARTHQUAKE state was the right response, I'll adjust this a bit. This knocked us out of observe, and we got some glitching in DARM below 100hz, so Jenne turned off the ADS. We may want to do this for future transitions to make the lock more robust, but it's unclear how it should be handled, that probably should knock us out of Observe. Eventually we lost lock, probably from some Sheila doing test on ETMX HEPI.
The SDF diffs that knocked us out of lock were HAM ISI sencor next chans, BSC ISI ST1 Z blend cur and next fms, and HEPI senscor next chans, Z only for the BSCs, X Y & Z for HAM1. I'll try to unmonitor these, so future transitions don't knock us out.
Also, the BS ST1 blend guardian didn't initially change the Z blend successfully, I had to init that guardian to get it to try again. It complained that it hadn't successfully switched, but didn't seem to be doing anything.
Re-requesting the state should force it into the correct configuration. Running INIT in an unknown configuration would just bring the node to a DOWN state because it doesn't know where to go. If this didn't work, please let me know.
WP 8131
Fans for the follwing units are now being powered off external power supplies:
1. ISCEX IO Chassis
2. ISCEY IO Chassis
3. LSC IO Chassis
4. ASC IO Chassis
Part of the noise hunting seen in DARM, alog 47575.
F. Clara, R. McCarthy
The timing of h1lsc0 was glitched during its fan power upgrade. I checked the IO Chassis was seen by the computer and then restarted all the models with /etc/startWorld.sh.
Despite h1lsc0 being the timing master for h1oaf1, it did not appear that h1oaf1 needed a restart, (and so its models were not restarted).
Opened/Closed FRS Fault ticket 12567
