We took about 20 minutes for some squeezer measurements, which are a follow up on the squeezing angle scan taken in 54822.
[Jenne, Sheila]
I have added some logic to the INJECT_SQUEEZING state of ISC_LOCK to reset and then retry squeezing, if the squeezer guardian is stuck. This is very similar to the reset and retry of the OMC locking that I put in October of 2018 (alog 44481) so that we didn't sit unneccessarily at PREP DC Readout Transition.
This lock acquisition the squeezer was stuck again, thinking that it was locked, but it was not. Sheila noted that this is a third pathology with the squeezer that we've seen in recent days, although the fix has been basically the same every time: turn it off and turn it on again. (Actually, Sheila has been doing the steps in guardian by hand, but I suspect that it'll be fine if we let guardian retry on its own).
This code has been added to the ISC_LOCK guardian, but not yet loaded. Operators, if ISC_LOCK is stuck at INJECT_SQUEEZING, please try re-loading the ISC_LOCK guardian. The guardian will likely be upset (since it'll already be in the Run state, but the timer variable is initialized in the Main state), but you should be able to take ISC_LOCK to Manual, click INJECT_SQUEEZING, then take ISC_LOCK back to Auto, which will force the guardian to restart the state from the beginning and initialize that variable. If that doesn't work, you can take ISC_LOCK to Manual, click INIT, load the guardian, click INJECT_SQUEEZING, then take ISC_LOCK back to Auto, and that should really force it.
If, after loading the guardian, this has caused any problems, please comment out the following lines, save, then reload the guardian (to take out my changes). Comment out line 4846 and lines 4860-4864.
We had yet another fast lockloss, so I loaded this change.
I've been looking at some of the locklosses to see if there were any things that looked like thermal drifts. So far I haven't found anything like that. Looking with Sheila, we thought we had found something suspicious on the ETMs, but it's only the HEPI tidal accumulations bleeding off. First attached plot are the ETMX HEPI ISC input (in nm) and the ETMX oplev pit and yaw. While the ISC input is ramping down to zero, the oplev sees an apparent angle change. This is a known issue, so not unexpected. What I think is not know is if the tidal bleed off is causing issues for us when we are relocking. It's been suggested that maybe we should wait for the tidal offsets to go to zero before ISC_LOCK returns to ready. This could mean we are waiting for a couple of extra minutes before we start trying to lock the arms.
Unfortunately, I don't think we can push the HEPI bleed off any faster. Looking at the T240 counts on the ETM ISIs, both get up to 20000, or over, counts, out of 32000 allowed. We haven't had any problems with ISI trips, but if we try to speed the hepi reset up, we could start tripping ISIs on lockloss.
At 17:28 utc we were knocked out of Observe for a second, because I clicked the wrong button. I accidentally hit the load coefficients button on SEIPROC. No filters were changed, no controls in use were actually touched, I just hit the wrong button on the GDS screen. We were out of Observe for about 2 minutes while I explained to Niko what happened, then we were back to Observing. Sorry 'bout that.
Looking through the lockloss site, I was able to find some examples of locklosses where ISC_LOCK_STATE_N transitions to LOCKLOSS a long time after the refined lockloss times we derive from our indicator channels. These may indicate states where the Guardian code needs to be changed to keep up with the state of the IFO.
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/events/1263/651643/indicators_WIDE.png
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/events/1264/540297/indicators_WIDE.png
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/events/1264/153709/indicators_WIDE.png
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/events/1264/440335/indicators_WIDE.png
This is good info to see. My biggest concern is with OFFLOAD_DRMI_ASC (index = 111), has a sleep in the main() method that reads whatever the MICH TRAMP used to be and then sets the sleep to that. It will set the sleep timer to 20seconds because of the ENGAGE_DRMI_ASC state in ISC_DRMI. I have removed this odd referencing to an old TRAMP time and changed it to just wait for the ramping to be done.
The other ones could use some time to look at as well, but it iwll have to wait for now.
When I set up the nonlinear gain calibration, I used a different calibration than LLO, calibrating the RF6 into an equivalent normalized nonlinear interaction strength x value. (X=sqrt(Power/threshold power) I didn't want to change to the LLO calibration in part because this makes more sense to me, and in part because the way that squeezing is estimated in the noise budget uses this channel. I've reverted this calibration now.
On nuc30 (centre FOM display on main control room wall) I've made the H1_DARM_FOM.xml file be a symbolic link to the userapps/release/isc/h1/scripts file. I patched and rebooted nuc30. It is now showing Sheila's 6th February (i.e. today) reference.
Added 150 mL to crystal chiller. X filter looks a little darker than D
Ops Shift Transition: 02/06/2020, Day Shift 16:00–00:00 (08:00-16:00) - UTC (PT)
State of H1: Locked
Intent Bit: Observing
Weather: 0-10 mph wind
Primary 0.03 – 0.1Hz: 0.01 um/s
Secondary 0.1 – 0.3Hz: 0.3 um/s
Outgoing Operator: Jeff
Quick Summary: Locked and Observing for 14.5 hours, low wind, microseism slowly declining
Other than a few minor earthquakes, it has been a quiet shift so far.
TITLE: 02/05 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC STATE of H1: Observing at 120Mpc INCOMING OPERATOR: Jeff SHIFT SUMMARY: One lock loss just before the start of the shift. Had some trouble with the FSS and squeezer (see Sheila's alog) while relocking. LOG: 23:58 UTC Lock loss. FSS unlocked. Toggled the FSS autolocker. Did not seem to work. FSS then later locked on its own. 00:06 UTC Chandra to mid Y. 00:10 UTC Lock loss at start of LOCKING_ALS. 00:22 UTC Chandra back. 00:41 UTC Lock loss from DARM_TO_DC_READOUT. 01:19 UTC NLN. Sheila had to intervene to lock the squeezer. Cheryl loading violin monitor filters. 01:33 UTC Cheryl done. Observing.
I noticed that in the previous lock (5 Feb 23:50UTC) mode 18 peak was not in DARM, yet the monitor filter was flatlined at around 2.6, and the mode 20 peak was in the spectra and flatlined at around 3.25. Looking at the monitor filters, they were both centered on the peaks (mode 18 at 1000.296Hz, mode 20 at 1000.307Hz). The gain at the frequencies was 120dB, and each filter overlapped the other frequency at around 112dB. I moved the monitor filters appart, which lowered the overlap gain to be around 86dB. Attached is an ndscope trend of the previous lock and current lock, which shows that in this current lock mode 18 monitor filter is now decreasing as mode 18 is damped, and mode 20 is flatlined at around 2.93, which accurately reflects both peaks in DARM.
Attachments:
Have remained locked and in observing since recovering from the lock loss at the beginning of the shift. No issues.
At 15:18 (07:18) dropped out of Observing when CALINJ turned on, creating an SDF DIFF. Called LLO, no one there started them. Called LHO support and left message. At 15:25 (07:25) the injection stopped. The SDF DIFF cleared and went back to Observing.
This is the last of DetChar injections scheduled a few weeks ago (see alog 54500). These aren't suppose to knock the IFO out of observing. I suspect that the H1:CAL-INJ_TRANSIENT_GAIN channel has been somehow added to the channels that are being monitored. Can someone there ensure that this channel isn't being monitored.
This channel is currently not being monitored, but the rest of that filter bank is. I also don't see anything in the INJ_TRANS guardian node for that time (Feb 5 15:18UTC, log attached). This node also shows a DETCHAR injection at 9:59 UTC, and we did not get kicked out of Observing then. Looking at the gain for those times, we see it change for only the 9:59 time.
If this wasn't seen by INJ_TRANS, what else could be changing something in CALINJ?
Ah. It isn't something turning on, it's something turning off. The CW injections stopped at that time and then restarted themselves, as they should. There was also a DIAG_MAIN message stating exactly this.
Sheila, Richard,
Filiberto and Richard installed new noise monitor circuits on the ETM PUMs last week, this morning I measured some transfer functions of them.
Attached are the dtt templates. Unfortunately I measured these TFs with the coil driver in state 3, which means that the low pass is on and the acq filter is off (the acq filter is after the noisemon pick off). In the digital noisemon filters I used the ANTILP filter zpk([0.4915;240.0583],[5.4697;21.8761],1,"n").
I'm also attaching the ITM measurements, which were taken in the same configuation 54614
The transfer function for ETMY LL seems significantly different from the rest. Richard and I set the test/coil enable to 0 (seting the driver into test mode, which means no input from DAC, I'm not sure what it means about terminating or floating inputs) and looked at spectra, and LL also looks different (3rd screenshot).
Richard is heading to the end station to check if there is something like a capacitor that wasn't swapped in that channel.
The EE team did find a bad solder joint in the ETMY LL channel, and repaired it before the end of maintence. Richard also re-ran the set of transfer functions for the noise monitors for EY, the attached file has his new set of measurements. The quadrants all seem to be the same now.
This ETMY measurement was taken with coil drivers in state 2, but the antiLP filter was on. That means that to compare these TFs to the others, we need to remove the filter zpk([0.4915;240.0583],[5.4697;21.8761],1,"n").
Attached are filters to calibrate the noisemon outputs in DAC counts.
They are designed by fitting the above-measured transfer functions (using IIRrational) and then inverting them. The antiLP filter was removed from the ETMY data. Poles at very low frequencies were nudged up to 1 Hz to limit the DC gain.
Attachment 1: plots of the data and fitted filters.
Attachment 2: filter install script
Thank you very much Chris.
I have just run this script and turned on the new filters, because we are out of observing for ~40 minutes of commisioning time. The new filters are turned on as of 20:22 UTC Feb 13, so the noise mon channels are all calibrated into DAC counts now.
