Shift Summary: Ran A2L DTT check – Pitch and Yaw are a bit elevated. As LLO is down, dropped out of Observing to run corrective script. A2L script finished with errors. Back to Observing. Spoke with Keita about the A2L Script. See aLOG #34763 for details. Successfully ran the script from a Controls terminal. Pitch and Yaw are now below the reference.
Remained lock for the entire shift. No issues or problems to report.
A good first half of the shift for data collection. No problems or issues to report. Successfully ran the A2L script from a Controls terminal. Keita is looking into the permissions needed to run the script from the Ops terminal.
I am on site this morning to inspect the access roads down the arms and found that Y arm is partially blocked, X are is mostly blocked. I am trying to reach the landscapers at the moment to bring them out tomorrow at noon to start baling on the Y arm. This should give them ample time to clear that road by Tuesday morning and allow the required liquid nitrogen delivery. I have informed Keita of this plan.
There were errors in the A2L script when I ran it this morning. TJ had reported a problem with the script a couple of weeks ago. I had put it on the Ops white board. As the A2L line had been erased from the Ops white board, and there were no aLOG entries concerning the script, I assumed it had been fixed. After running the script with errors, the A2L is now worse than before the script ran. The low frequency noise is elevated. Range is still over 65 Mpc, so will leave it to the Monday crew. The errors are captured in the two attachments.
The problem TJ had was a result of the ASC model changes. Those were fixed that day, and the a2l script had been run successfully since.
It looks like the user account you're using doesn't have permission to write those figure files, so the script is failing on some random trial value of the a2l gain.
Try running the script from a personal account, not ops or controls - that is usually successful, and we can check on the permissions on Monday. If that fails, revert all the values to what they were...that is probably better than whatever trial value the script is leaving in there. Maybe TJ can help me change the script such that it reverts if there is some problem.
After speaking with Keita and reviewing Jenne's comments, I ran the A2L Script from a Controls terminal with no problems. The Pitch and Yaw signals are back to below the reference. Up to about 20 Hz the low frequency noise looks better.
TITLE: 03/12 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 66Mpc
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Quiet shift. Not much going on. LLO is down.
Still Observing. LLO is also up.
TITLE: 03/12 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Recent lockloss.
OUTGOING OPERATOR: Travis
CURRENT ENVIRONMENT: Wind: 15mph Gusts, 9mph 5min avg Primary useism: 0.02 μm/s Secondary useism: 0.72 μm/s
QUICK SUMMARY: Looks like wind and useism is low enough for me to lock the IFO. Accepted SDF differences (see attachment) and back to Observe.
Okay just before I was about to post this alog PI mode rung up so fast I couldn't damp it in time. Lost lock because of that.
Back to Observe. I hope it stays this way this time.
TITLE: 03/12 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY: Pretty rough shift with high wind and extremely high microseism for the duration. PSL tripped and was reset at the beginning of the shift. Since then, almost every known problem has occurred (many not logged since they are 'common' problems and I was more concerned with fixing the issues than recording their times). The most frequent issue seems to be the PSL FSS PZT oscillating, which I had to deal with ~20 times during the shift. We made it as far as COIL_DRIVERS once, so I'm pretty confident that the main issues are with the environment and not the IFO. If it makes anyone feel better, LLO has been down for the entire shift as well.
LOG: See previous aLogs.
Struggling to get back to locking. Lots of FSS PZT oscillations since the PSL was brought back up. ETMX RMS watchdog has tripped several times. Got a few "Tidal X/Y error" messages, but not for an hour or so. We have made it as far as INCREASE_POWER a couple of times, but during the power up steps, got "OMC DCPD Saturation" notifications and lock was lost. Microseism is still very high and wind is picking back up. Looks like it will be a rough night.
The tidal alarms have been continuing so I called Keita. We looked at the tidal arm servos which seemed to be railing at their limits causing the alarm. I kept an eye on them and X arm had another alarm when it reached its limit of 10. At Keita's suggestion, I increased the limit to 12 and the signal ramped to -11.145 and stopped, so it appears it just needed a bit more range. I'll leave it at this value for tonight since we seem to be struggling with microseism which is still hovering around 1 um/s.
Another X arm tidal alarm. This time I increased the limit to 15 and the signal ramped to -13.679. I also now have a DIAG_MAIN message of 'TIDAL_LIMITS:X_COMM_CTRL output within 10 percent of limi'.
Reset the laser without any problems. I also reset the injection locking counter which was 19. By the time all the other servos had re-acquired, the injection locking counter had reached 3. Had to enter the LVEA to reset the noise eater. The trip was caused by the drop in flow rate in head 3. Note that the flow rate in head 3 dropped close to the trip limit and appeared to recover before really crashing approximately 10 minutes (~594 s) later.
Shift Summary: Ran A2L script – Pitch is at or below the reference; Yaw is over 0.7. Will run the A2L script if the opportunity presents.
Wind and seismic activity has been rising since mid-shift. Several gusts near 40 (at least one over). Primary microseism in X & Y is around 0.09um/s and secondary microseism in Z is up to 2.0um/s. Lost lock just after big wind gust (over 40), PI-Modes 23 & 27 ringing up, and microseism high. Shortly after the lock loss, the PSL went down with a Flow1 error. Peter is coming in to reset the PSL.
TITLE: 03/12 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Other
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
Wind: 32mph Gusts, 27mph 5min avg
Primary useism: 0.07 μm/s
Secondary useism: 1.32 μm/s
QUICK SUMMARY: PSL is currently down. PeterK is on site attempting a reset. Wind and microseism are still very high.
Good observing shift so far. Range stands at 65.6 Mpc. One Mag 5.5 EQ that did not cause any difficulties. The wind and secondary microseism have kicked up a bit. The A2L script needs to be run at the next opportunity.
Miriam, Hunter, Andy A subset of blip glitches appear to be due to a glitch in the ETMY L2 coil driver chain. We measured the transfer function from the ETMY L2 MASTER channel to the NOISEMON channel (specifically, for the LR quadrant). We used this to subtract the drive signal out of the noisemon, so what remains would be any glitches in the coil drive chain itself (and not just feedback from DARM). The subtraction works very well as seen in plot 1, with the noise floor a factor of 100 below the signal from about 4 to 800 Hz. We identified some blip glitches from Feb 11 and 12 as well as Mar 6 and 7. Some of the Omega scans of the raw noisemon signals look suspicious, so we performed the subtraction. The noisemons seem to have an analog saturation limit at +/- 22,000 counts, so we looked for cases where the noisemon signal is clearly below this. In some cases, there was nothing seen in the noisemon after subtraction, or what remained was small and seemed like it might be due to a soft saturation or nonlinearity in the noisemon. However we have identified at least three times where there is a strong residual. These are the second through fourth plots. We now plan to automate this process to look at many more blip and check all test mass L2 coils in all quadrants.
In case someone wants to know, the times we report here are:
1170833873.5
1170934017
1170975288.38
I have noticed similarly caused glitches on the 10th March, in particular for the highest SNR Omicron glitch for the day:
Looking at the OmegaScan of this glitch in H(t) and then the highest SNR coincident channels which are all the quadrants of H1:SUS-ETMY_L2_NOISEMON:
Hi Borja,
could you point us to the link to those omega scans? I would like to see the time series plots to check if the noisemon channels are saturating (we saw that sometimes they look like that in the spectrogram when it saturates).
I am also going to look into the blip glitches I got for March 10 to see if I find more of those (although I won't have glitches with such a high SNR like the one you posted).
Thanks!
Hi Miriam,
The above OmegaScan can be found here
Also I noticed that yesterday the highest New SNR glitch for the whole day reported by PyCBC live 'Short' is of this type as well. The OmegaScan for this one can be found here.
Hope if helps!
Hi Miriam, Borja,
While following up on a GraceDB trigger, I looked at several glitches from March 1 which seem to match those that Borja posted. The omegascans are here, in case these are also of interest to you.
Hi,
Borja, in the first omega scan you sent, the noisemon channels are indeed saturated. In that case it is difficult to tell apart if that is the reason for the spectrogram looking like that or if indeed it might be a glitch in the coil drive. Once Andy has a more final version of his code, we can check on that. In the second omega scan, the noisemon channels look just like the blip glitch looks in the calib_strain channel, which means the blip was probably already in the DARM loop before and the noisemon channels are just hearing it. Notice also that, besides the PyCBC_Live 'short', we have a version of PyCBC_Live that is dedicated specifically to find blip glitches (see aLog 34257), so at some point we will be looking into times coming from there (I will keep in mind to look into the March 10 list).
Paul, those omega scans do not quite look like what we are looking for. We did look into some blip glitches where the noisemon channels looked like what you sent and we did not find any evidence for glitches in the coil drive. But thanks for your omega scans, I will be checking those times when Andy has a final version of the subtraction code.