Betsy, Sheila, Travis, Evan
Something about the following coils appears to be unhealthy: PRM M1 RT&SD, PR3 M1 T1&T2. See attached OSEM sensor spectra.
According to Betsy, this high-frequency junk started around 2015-08-29 15:00:00 Z. On the control side, this junk dominates the rms of the PRM M1 LF drive (it is about 10000 ct).
Probably this is related to the problem that Kiwamu saw last night with PRM M1 DAMP L.
The first plot below is a trend of PRM DAMP L showing the start of the noise - the noise started in the middle of the lock stretch from Sat morning. The second plot shows the 4 OSEM Sensors - it's hard to see it in any of the sensor trends, except PRM RT.
For Richard: yes, all 4 of these noisy signals are on the same cable set and Sat box line.
It looks like this high-frequency noise is due to the shadow sensors. We paused in LOCK_DRMI_1F with the PRM aligned, and turned off the top stage damping. In this state there were no digital signals going to the coil driver (MASTER_OUTs were zero), and the NOISEMON and FAST_IMON readbacks were flat. But the inputs from the RT and SD OSEMs had the high-frequency noise. See attached spectra. So, it doesn't look like it's a bad coil driver (this would have been three in as many weeks)...maybe it's an issue with the satellite box?
With the top stage damping enabled, the noise is large enough that it passes through the damping filters and shakes the M1 stage, but it's so high frequency that I don't think we are shaking the optic. There's no sign of the noise peaks in the PRCL error signal. No reason to think we can't run like this until Tuesday maintenance.
We have seen this before at LLO back in July 2011: https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=1262
Nairwita Mazumder, Rich Abbott A few days back Jim noticed (alog ) that the "Bumbling line" which varies over a large frequency range is again back on ETMX seismic channels . This was first noticed on March and disappeared before ER7 and again was seen from 4th August. One can see the lines at all the horizontal and vertical sensors on ETMX. I have attached a pdf containing some follow up work done during Rich's recent visit to LHO. The first plot in the pdf is the spectrogram of ETMX GS13 on 26th August. It can be seen that there are multiple wandering lines having a fixed offset. We were suspecting that some magnetometers at the End X might be the culprit (as we could not find any correlation between temperature fluctuation with the line ). The second and third plots are the spectrum of H1:PEM-EX_MAG_EBAY_SEIRACK_Z_DQ and H1:ISI-ETMX_ST2_BLND_Z_GS13_CUR_IN1_DQ for 2nd August and 26th August respectively. The red one is for 2nd August when the bumbling line could not be found and the blue one is the recent data (26th August). It is clear that the peaks appearing on ISI-ETMX_ST2_BLND_Z_GS13 after 3rd August are correlated with the peaks of the spectrum (which also appeared around the same time) of SEIRACK magnetometer . The plots on the second page shows the coherence between GS13 and the magnetometers in the VEA and SEIRACK. It looks like the magnetometer on the SEI rack has stronger coherence with GS13 sensors than the magnetometer located at VEA . I have marked two points (blue and red cross) in the coherence plots to highlight two of the many peaks.
Adding to Nairwita's comments, the signal seen in the GS13 spectra is also present in the magnetometer data. This being the case, it's most likely that the harmonic series points to an electromagnetic artifact associated with the HEPI pump variable frequency drive. The fact that the same signature does not exist at the other end station (I assume this to be true, but have not verified) may point to an enhanced susceptibility in the X-end electronics for some reason. No reason to panic much yet, but duly noted.
I have attached the coherence plots computed between PEM-EX_MAG_SEIRACK and GS13 , ST1 CPS and ST2 CPS over the frequency range 0.4Hz-900Hz to check the following two points: (1) If there exists any coherence between CPS and the Magnetometer at frequency above 256 Hz (2) What the low frequency behavior is I can be seen that the coherence between CPS and the magnetometer above ~25Hz is pretty low compared to GS13, but them have relatively high coherence with PEM-EX_MAG_SEIRACK near 20Hz .
(All time in UTC)
15:00 Take over from TJ. IFO not locked.
15:32 DRMI wouldn't lock after 10 mins. I was about to do the PRMI when I started seeing AS90 flashing. I gave it a few more minutes and it managed to lock.
Lost lock at REDUCE_CARM_OFFSET soon after. Looks like ETMX was saturating. Wind speed ~20 mph.
15:45 Lockloss again at REDUCE_CARM_OFFSET. Turning ISI BSC WIndy blend filters on for BSC1,2,3.
18:23 Lockloss. Power outage. Guardian couldn't turn the ETMX ESD back on. We called Richard for help. See alog21030.
21:22 Switched the Windy blends off.
22:50 Begins the initial alignment after trouble relocking.
23:47 We're running into an issue with MICH_DARK_LOCKED. The beam would diverge away. I've tried setting the ALIGN_IFO to DOWN and waited a little bit before trying again. Didn't work. Sheila is working on it.
We looked into two of the locklosses that we had durring high winds. (screen shots attached). In both cases PRM was saturating or close to saturating before the lockloss. Evan has increased the PRM and SRM offloading gains by a factor of 2.
There were no indication of anything was going bad right before the lockloss. No glitches in the spectrum, no earthquake, nothing. The saturation alarm yelled ITMX and DRMI was unlocked. The SUS OMC SW watch dog tripped but seems to have uptripped itself. After that the Guardian wasn't able to turn ETMX ESD driver on and the reset switch kept flickling. Sheila is on her way to the site so I'm waiting for her.
I noticed the light in the control room flickered at the exact time of the lockloss. I suspect a power glitch may have caused this lockloss.
I set the Operating Mode to UNKNOWN.
We went to reset the ESD box at both end stations. The high voltage supply was 0. We reversed the input back to where it was and solved the problem. ETMY railed after the power reversed so we had to unplug and re-plug the DAC to fix it.
Soon after we got the interferometer back, wind started picking up speed.
Note, Sheila clarified that they called Richard who walked them through a set of commands on the front panel in order to get the power supplies back up while at both end stations. ('Always a good idea to call Richard after a power glitch anyways.)
After several locklosses at REDUCE_CARM_OFFSET, I took it very slowly. I stayed at DRMI_LOCKED until the saturation alarm complained less about ETMX. Even though the wind speed is only 15-20 mph but PRM seems to wiggle a lot so I turned on the 90 mHz windy blend filters for all the corner station BSCs. The intent bit has been set to observing.
The OMC-DCPD RMS is a about an order of magnitude lower than the threshold. I turned on the 2nd stage OMC whitening. Intent bit was still set to Observing. The BNS range is slightly above 70 Mpc.
ER8 days 12 and 13. No restarts reported.
Times in UTC
925 - Took the IFO to DOWN for Kiwamu to work on more calibration measurements
1016 - H1SUSETMX Timing error cleared
1208 - NOMINAL_LOW_NOISE, but BS bounce was rung up and Kiwamu tried to damp it to no avail
1219 - Itbit set to Undisturbed
1416 - Lockloss (Wind is picking up, 20mph, but that shouldn't be a problem)
1431 - Lockloss @ DC_READOUT (everything looked good, not sure what happened unless it really is the wind)
1438 - Gardeners here for tumbleweed cleanup
1444 - Lockloss @ DARM_WFS
1454 - Lockloss @ CARM_ON_TR
1500 - Handing if off to Nutsinee
I don't know if this is a known issue, but today I noticed that when PRM was in the pre-defined misalgined position (set by Guardian), the M1_DAMP_L loop oscillated and kept saturating the DAC output in LF and RT coils all the time. While this does not pose any issue for lock acqusition or low noise lock, this may be an indication of some potential issue or something bad. Since I have no energy to further investigate this issue tonight, I am leaving it as it is.
I have completed a set of measurements that is necessary for nailing down the ETMY actuation scale factors. This is a third round of the measurements in order to study repeatability and systematic errors. The measurement data can be found at the follwong places.
aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_L3toDARM_LVLN_LPON_FullLock.xmlaligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_L2toDARM_FullLock.xml/aligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_L1toDARM_FullLock.xmlaligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMX_toDARM_FullLock.xmlaligocalibration/trunk/Runs/ER8/H1/Measurements/FullIFOActuatorTFs/2015-08-29/2015-08-29_H1SUSETMY_PCALYtoDARM_FullLock.xmlaligocalibration/trunk/Runs/ER8/H1/Measurements/PCAL/2015-08-29_PCALY2DARMTF_7to1200Hz.xmlaligocalibration/trunk/Runs/ER8/H1/Measurements/DARMOLGTFs/2015-08-29_H1DARM_OLGTF_7to1200Hz.xmlaligocalibration/trunk/Runs/ER8/H1/Measurements/ALSDIFF/2015-08-29/2015-08-29_ALSDiff_ETMX_L3_HVHN.xml/aligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1SUSITMX_L2_State2_MICH.xmlaligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_MICH_OLGTF.xmlaligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1MICH_freeswingingdata.xmlaligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1SUSITMX_L2_State2_XARM.xmlaligocalibration/trunk/Runs/ER8/H1/Measurements/FreeSwingMich/2015-08-29/2015-08-29_H1SUSETMX_L3_HVHN_XARM.xmlJ. Kissel, K. Izumi I've process the results from the above aLOG using the same model I had used for LHO aLOGs 21015, just to get something out the door for review at today's calibration meeting. The results look encouragingly consistent with Wednesday's (LHO aLOG 20940) and Friday's (LHO aLOG 21005) results. Again, the next steps are to - functionalize the analysis scripts so we can easily process future measurements, - have those functions spit out their individual measurement results so they can be compared with each other (the comparison of which is another function to write), and - make sure that all the scripts are using the output of the latest DARM model and parameter set, instead of recreating a naive model from the matlab QUAD's [m/N] and knowledge of the electronics chain. - finish processing results from all the of the electronics chain measurements to identify if frequency-dependent systematics lie in there - Update / refine the DARM model parameters to squash the frequency-dependendent residual to as small as possible, such that we can be confident in professing number with properly quantified uncertainty. What has NOT been included in the naive actuation model: - 16k IOP up-sampling filter - Analog AA filter - Any of the un- or poorly-compensated poles and zeros of the ESD drivers - The recently found mis-match between expected and reality regarding the UIM driver's poles and zeros (LHo aLOG 20846) I have a high degree of confidence that these are the reasons for the residual frequency dependence. In other words, these be resolved once I switch to using the full DARM model's actuation chain. That being said, suggestions are welcome as to what else the discrepancies might be.
After the 40mph wind storm died down, I began the locking procedure. No full initial alignment was required, just some touching up of the PRMI SUSes. The only hurdle was detailed in aLog 21016, which took around 30 minutes to overcome. Kiwamu has been running Calibration excitations for a good portion of the lock stretch.
0:52 Started locking procedure
2:24 Locked on Low Noise
4:12 Started PCal/DARM measurement at Kiwamu's request
4:31 Switched to Observing
4:45 Switched to Calibration
6:13 Lockloss, PRM saturation
6:48 Locked Low Noise
Over the last ~5 days, nearly 100 SDF channels have gone into alarm. We should probably link these to the INTENT bit, since many of these are actually real configuration control changes of the IFO, yet currently do not dictate any GO/NO GO of observation mode. When connected to the intent bit, it will force commissioners and operators to address these changes in realtime, rather than let them stack up for me to reinvent every 5 days.
Since super-winds (and calibrations) have the IFO down, I've taken the opportunity to work on these - so far I've cleaned up:
- ETMX and ETMY L1 and L2 damping loop outputs enabled last Tuesday around the maintenance period - maybe the BURT pushed these buttons. I verified that the SAFE file sets the gain of the loop to be zero and the IN and OUTPUTS are ON. Guardian then controls the gain.
- Accepted a few pages of offset changes, and the extra AS_B_RF45_AWHITEN_SET2 due to Evan/Stefan running the dark offsets script/commissioning last Thur night (alog 20976). I'll wait for them to comment that the 1st attached SDF diffs are 1) their intended gain changes as it's vague in their alog, and 2) if they are ready to keep/accept them.
- Since SDF couldn't write the very small ~ 10e-19 CALCS epics values to the SAFE.snap file (commissioned AUG 11 alog 20452), I wrote them in by hand and then hit LOAD TABLE on the CALCS SDF. All of the values cleared, since now they match the very small values loaded in their respective medms. I've committed the CALCS safe.snap to svn.
- ALS X and Y - I need someone to point me to an alog regarding why the ALS_WFS_DOF_3 gains changed on both X and Y in the middle of the night on Thur night (2am Fri - see second SDF diff attachment.)
Today, Evan pointed out that the DARK OFFSETs script sets these gains of the ASC loops to 1.0, but then does not reset the gains back to their nominal settings. This was surprising to Evan, so he is reworking the script. We have reverted the first 16 gain changes in the attached ASC SDF screen because of this.
Then, the next 4 gain setting diffs (ASC-AS_B_RF45) were from commissioning by Sheila/etal (20961), so we've ACCEPTED those in SDF.
The existing script was, to put it mildly, not suitable for use as a control room tool. As Betsy says, it simply wrote a gain of 1 to every segment of every AS WFS. In fact, it did this twice: once inside a for loop, and then again immedately afterward (using a string of 32 caput commands). Moreover, there is no reason for such a script to touch the segment gains at all.
This script (along with the analogous REFL script) was summarily overwritten with a new python script: userapps/asc/common/scripts/dark_offsets/WFS_offset_AS. Usage is given in the first few lines as a comment.
Jeff, Evan
There is now a new DMT viewer template for viewing the BLRMS of the corner and end-station STSs. It is userapps/isc/h1/scripts/Seismic_FOM_STS.xml.
This is meant to replace the Guralp DMT viewer template.
Right now it only displays the lowest two BLRMS (0.03 to 0.1 Hz and 0.1 to 0.3 Hz).
[As an aside, the frequency bands implied by the channel names in DMT viewer don't seem to match up with the front-end channels. For example, the front-end EX-X channels are named as follows:
H1:ISI-GND_STS_ETMX_X_BLRMS_100M_300M
H1:ISI-GND_STS_ETMX_X_BLRMS_10_30
H1:ISI-GND_STS_ETMX_X_BLRMS_1_3
H1:ISI-GND_STS_ETMX_X_BLRMS_300M_1
H1:ISI-GND_STS_ETMX_X_BLRMS_30_100
H1:ISI-GND_STS_ETMX_X_BLRMS_30M_100M
H1:ISI-GND_STS_ETMX_X_BLRMS_3_10
while the DMT channels are named as follows:
H1:ISI-GND_STS_ETMX_X_DQ_0p03-0p1Hz_48h
H1:ISI-GND_STS_ETMX_X_DQ_0p1-0p2Hz_48h
H1:ISI-GND_STS_ETMX_X_DQ_0p2-0p35Hz_48h
H1:ISI-GND_STS_ETMX_X_DQ_0p35-1Hz_48h
H1:ISI-GND_STS_ETMX_X_DQ_1-3Hz_48h
Are these BLRMS distinct from what's being computed in the frontend?]
What was wrong with the old plot?
The STS is more sensitive than the Güralp in the frequency band where we typically watch for earthquakes (30 mHz to 100 mHz).
Looks like we should keep the old plot then, since everyoe already knows it.