typed in -3 gain, restoring damping
[Jenne, Cheryl, Rick]
I glanced at the spectrum on the wall, and noticed that the PcalY lines weren't on even though we were observing. This is highly unusual, since the calibration lines should always be on when we are in Observation.
I opened up the CAL_LINES overview screen (sitemap -> top cal button -> CAL_LINES) and saw that H1:CAL-PCALY_OPTICALFOLLOWERSERVOOSCILLATION was high at about 9.5 (text on the screen says that it should be below 0.01) and had a big red box around it. I called Rick for advice, and he suggested the ol' level-0 thing to try: turn it off and on again.
Cheryl and I took the IFO out of Observing, I turned off the PcalY optical follower servo (H1:CAL-PCALY_OPTICALFOLLOWERSERVOENABLE, the Loop Enable button on the PcalY overview screen), and then turned it back on. It seemed happy when it came back on, with the oscillation value back down to nearly zero, and also the H1:CAL-PCALY_OFS_PD_OUTMON wasn't railed flat. The Pcal lines in the DARM spectrum came back on, so we declared the IFO fixed, and went back to Observing.
By trending that oscillation channel, it looks like it was oscillating since about 21:48 UTC. This is about halfway through the time that the CDS team was at EY doing some corrective maintenance on the Beckhoff system down there (alog 48209 and comments). Daniel thinks that perhaps it could be that this was the time when some cables were unplugged or something?
In the attached screenshot, you can see that the blue trace was bad for a long-ish time, including some time when the red trace (which is the OBSERVE bit) was 1. The tick down in the red trace is my taking the IFO out of Observe, you can see that the oscillation went away, then we immediately went back to Observe.
The data during that time should be entirely valid and good for astrophysical searches, but note that the calibration lines were not on, so some diagnostics may report bad / funny values.
In order to prevent this in the future, I propose that we add a "test" to DIAG_CRIT (and maybe also DIAG_MAIN) that will make a notification of a problem with either Pcal servo. DIAG_MAIN is useful, since it is what we look at on the wall, but it is on purpose ignored by the Observe bit. So, it needs to be in DIAG_CRIT so that we *can't* go to Observe without the Pcal lines being on and healthy. As long as there is enough of an error message in DIAG_CRIT that an operator can find and fix the problem, then maybe it doesn't have to be replicated in DIAG_MAIN.
Rick suggests that if the servo goes into oscillation again (say, if it wasn't caused by the Beckhoff work at EY), we could perhaps try turning the gain (H1:CAL-PCALY_OPTICALFOLLOWERSERVOGAIN) down by 3dB until we get a chance to remeasure the servo loop gain next Tuesday. Recall from alog 48206 that we measured the loop earlier today and it had a UGF of about 84 kHz with 50 degrees of phase margin, so should be plenty stable, but perhaps things 'settled' after we left?
We made a measurement of the actuator model installed in CALCS (we have actually done this several times). (see png attachment).
The second attached plot is a comparison of what we measured to the pyDARM model. To make the comparison, I multiplied the individual stages by the pyDARM output actProd.calcs2gdsResidActFiltOut_NoDAQDownSample. This is a filter which contains the corrections to the CALCS model that will be applied in GDS. There are three versions of the calcs2gds, two versions with _NoDownSample and calcs2gdsResidActFiltOut_NoDAQDownSample_removeCALCSdelay.
Since the measurement of the total was taken after the delay in CALCS is applied, while the individual stages are measured before the delay, so I applied calcs2gdsResidActFiltOut_NoDAQDownSample_removeCALCSdelay to the total actuation.
These match up well with the unknown scale factor of 0.95 applied in CALCS, the only remaining thing is a small phase shift in the ESD at around 100 Hz.
TITLE: 04/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 102Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
Wind: 24mph Gusts, 18mph 5min avg
Primary useism: 0.13 μm/s
Secondary useism: 0.20 μm/s
QUICK SUMMARY: H1 in Observe for a few minutes without PCAL-Y, Jenne caught this, H1 is back in Observe with PCAL-Y
Kara M., Philippe N., Robert S., Sharan B.
Kara has written a script that will make 6 magnetic injections in a row with the parameters that the PEM investigations so far have indicated are optimal. The script is intended to be run by operators every Tuesday morning at 7:45AM. It is called PEM_weekly_magnetic_injection.py, and it is saved in:
/opt/rtcds/userapps/release/sys/h1/scripts/
It can be run with the command
./PEM_weekly_magnetic_injection.py
The script runs for about 7 minutes, making injections through these channels in the following order:
H1:PEM-CS_GDS_0_EXC
H1:PEM-EX_GDS_1_EXC
H1:PEM-EY_GDS_1_EXC
We request that these channels remain exclusively for PEM magnetic injections for the duration of O3. At the moment, the CS channel sends its excitation to the wall-mounted coil in the LVEA (described in 43406), but the EX and EY channels have not been connected to any injection equipment yet (See 46388 for proposed magnetic injection coil plans).
In its current configuration, the script sends a 10-100Hz broadband excitation through all three channels consecutively, then a 100-1000Hz broadband excitation through all three channels consecutively (similar to Figure 4 of 47881). Upon completion, the script will say "Done with all injections."
The script can be interrupted with "Control+C", which stops any ongoing injection and cancels any remaining injections.
The script logs its injections in .txt files containing the parameters for each of the 6 injections in the following format: channel,start_time,end_time,lower_frequency,higher_frequency,gain.
These logs will be saved to
/ligo/www/www/exports/pem/WeeklyMagneticInjection/logs
TITLE: 04/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 102Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
EY issues to with PCal maintenance and then Beckhoff later in the afternoon. High winds for the past 3 or so hours greatly inhibited getting past find IR. Things finally quieted down enough to get back into the game. 16 CALCS diffs to accept everytime before Observe.
LOG:
15:58 Karen driving her vehicle to the woodshop and also to the OSB shipping area
15:59 Nichole, Elizabeth, Christina, and Betsy out to MX for property search
17:44 Betsy out to LVEA W Bay
18:07 Rick et. al are done and leaving EY
18:08 have been holding at POWER_30W for PCal work and some LVEA incursion - Obs Mods CORRECTIVE MAINTENANCE - just switched to LOCK ACQUISITION for remaider after PCal team reported finished at EY
20:34 winds in excess of 40mph. Settled on USEISM_MOD_WIND to get green arms to settle down, particularly Y-arm.
21:00 Locking is futile. Winds forecast to increase over the next couple of hours. DOWN
21:30 Fil and Marc out to EY to resolve a Beckhoff issue WP#8151.
22:00 FIl and Marc back
22:14 winds are slower and locking is commencing. fingers crossed.
22:54 Back to NLN/Observing
I've just noticed that the corner weather station is not reporting any wind speeds. It stopped reporting at 08:11 PDT Tuesday 26 March (one maintenance period ago).
Hardware seems to be working. Software is reporting an error. Looks like the receiving stream has fallen out of sync. The timeout is 10 sec, so it isn't helping. Maybe we can try to disconnect the RS232 cable for 20 sec and reconnect.
Whoa! This actually worked. Fixed...
We are seeing intermittent Beckhoff errors at EY, Daniel is investigating.
We are temporarily loosing communication with terminals M11 through M17 of the end station 2 chassis. M11 is a a EL9410 power supply terminal. This could indicate that this module is failing. It could also indicate that the prior terminal, EL4134, has a problem communicating with the next terminal.
Problem started 4/3/2019 19:39 UTC.
Fil and Marc swapped the power supply terminal.
Fault: https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=12659
JenneD, NikoL, SharanB, RickS
In response to Evan Goetz's observation of increased modulation harmonics impacting CW searches (see LHO log 48192), we went down to Yend to address the decrease in diffracted power we have known about for some time (JeffK noticed that the 14 Hz harmonic of the 7 Hz excitation had increased weeks ago).
As found, the power incident on the Pcal AOM was about 1.88 W and the diffracted power (with the OFS loop unlocked) was about 1.01W (54% diffraction efficiency).
We checked that the beams were in their nominal positions at the Rx power sensor aperture (see first attached image). Then we carefully adjusted the alignment apertures in the Pcal transmitter module. Then, we adjusted the rotation and height of the AOM. Nothing was gained by rotating the AOM, but raising it turning the height adjusters on the 4-axis NewFocus stage by almost one full turn gave us about 1.56 W diffracted power (83% diffraction efficiency).
We adjusted the OFF gain to 36 dB from 43 dB. The OLTF of the OFS loop is attached (UGF about 100 kHz with about 50 deg. of phase margin).
We adjusted the OFS offset from 2.725 to 3.8.
We adjusted the two Pcal beam positions on the Rx power sensor aperture (see attached photo).
We then looked at the harmonics of the 7 Hz Pcal exctiation in the Rx PD signal. It has been reduced by about a factor of ten (see attached ASD plot).
Nice work! Thanks for addressing this. I attach below a spectrum comparison of the before and after work PCAL Y transmission PD output. Blue is before the work and red is after the work. We can see the reduction in a number of peaks and other features, but they are not all gone completely (and a few are slightly higher than before, but hopefully not in a region the IFO is maximally sensitive). A watchful eye should be kept on the Pcals, perhaps also as part of the the DetChar DQ rota, so that we catch any potential problems like this going forward.
Commencing with 03
SDF Diffs accepted are as follows:
Ooops, hadn't switched snsor correction back on at EY.
18:28 Senscor back on at EY
Time to start the observation run restart reports for O3. Reminder of colour scheme: red=unexpected restart, purple=restart to fix unexpected problem, green=planned restart.
Monday 01apr2019: No restarts
Tuesday 02apr2019:
2019_04_02 11:48 h1edc
2019_04_02 11:52 h1broadcast0
2019_04_02 11:52 h1dc0
2019_04_02 11:52 h1fw0
2019_04_02 11:52 h1fw1
2019_04_02 11:52 h1fw2
2019_04_02 11:52 h1nds0
2019_04_02 11:52 h1nds1
2019_04_02 11:52 h1tw0
2019_04_02 11:52 h1tw1
2019_04_02 11:52 h1tw3
2019_04_02 12:03 h1guardian1
Maintenance day, installed external edcu for first time (h1edc), DAQ restart for this and Beckhoff slow controls work. Rebooted guardian machine to commence O3 from clean restart.
Summary: Pcal Y seems to have a bad comb structure that is likely producing contaminating lines in h(t). The behavior of this Pcal is not similar to Pcal X or either of LLO's Pcals. I urgently suggest a check is done on H1 Pcal Y to remedy the situation. Details: I was concerned that H1 low-frequency spectrum is so much more contaminated with lines than L1, so I started looking at some easy to check things, like the Pcal. Looking at summary pages, you see things like Figure 1 H1 Pcal Y. Compare that with figure 2 (H1 Pcal Y) or 3 and 4 (L1 Pcal X and Pcal Y). Comparing H1 Pcal Y spectrum to GDS-CALIB_STRAIN, I find that many lines are very close to the noise floor. This is very concerning for long duration searches (CW and Stochastic). Figure 5 is a broad band spectrum comparison Figure 6 is a zoom on 10 - 200 Hz Figure 7 is a zoom on 70 - 100 Hz I don't think this is due to clipping on Pcal periscope because this is on the transmission module PD (so, light that actually reaches the ETM). I hope it is not that hard to solve this issue.
J. Kissel for R. Savage & N. Lecoeuche In fact, this problem has been going on for quite some time now: see for example its first discovery LHO aLOG 47372 and FRS Ticket 12491. We think this is indicative of the PCAL Y laser heading out to the pasture. An attempt to reduce the non-linear intensity noise was made on March 11 (see LHO aLOG 47444). Rick and team will be going down to the Y end station today to attempt to address the issue. We were hoping that a tune-up will help, but we have several back-up options: (1) The optical plant is much more stable than it used to be, and specifically the optical spring is stable, and a factor of 2 lower frequency than it in O2. Thus, we can consider turning off the 7.93 Hz calibration line. (2) Further, there have been some R&D efforts in the LSC to monitor the optical spring with the higher-frequency actuator reference line (currently at ~35 Hz), see appendix A of T1700106. (3) Finally, we can also divert to using the X-end pcal system, but that will take a good bit of effort "transferring" the standard through measurement, and our measurement analysis code will likely be confused for a day or three while we figure out where all the hidden sign bugs are. (Examples of us clearing out the bugs include 48131 and 47574)
@Jeff, Rick, and Niko, thanks for working on this quickly! Hopefully it can be resolved fast so that groups don't have to reject too much data. I wonder if some contamination might also be linked to Pep's study on bi-linear couplings (see LHO aLOG 48161). It might also be worth checking the Pcal spectrum to see if the noise seen in Pep's study is linked to this as well. Maybe that also will further clean up the Pcal PD spectrum...?
Mostly fixed by hardware adjustments on the Y-end Pcal (see LHO aLOG 48206). We should keep a watchful eye on this during O3.
Sheila, Daniel, Nutsinee
We injected broad band noise into DARM through various point in SQZ system: OPO, CLF, and LO common mode board. All injections were made after the boosts (exc A on the common mode board). Only CLF and LO injections showed up in DARM. We increased OPO error signal by two orders of magnitude and didn't see much (if anything at all).
Only data points where witnesses (LO, CLF, OPO) and DARM sees a factor of 1.5 increase during injections get plotted. The projected ambient SQZ noise is calculated using:
ambient = witness_ref.* ( sqrt(DARM_exc.^2-DARM_ref.^2)./ sqrt(witness_exc.^2-witness_ref.^2) )
The m/rad of DARM/LO and DARM/CLF ASD ratio are only a factor of 2 different, this could suggest that wherever we inject (CLF or LO), the coupling mechanism is the same. Injecting noise through CLF added noise back to the pump laser as LO tries to suppress it (and inject that noise back to the laser). This results in the final phase difference between the green and the carrier, which is squeeze angle phase noise. On the other hand, injecting directly to LO the injection gets suppressed by the LO loop (UGF ~10kHz). The reason why CLF doesn't have much of a loop shape could be because CLF loop UGF is very low (<1kHz).
The best suspect for the coupling mechanism is the backscatter of the IFO beam (squeezer backscattered measurement) from the OPO. As CLF noise shows up in LO loop, LO tries to suppress it by sending the noisy control signal to the laser, which then makes the laser noisy, which makes the green pump noisy. The noisy green light gets inside the OPO, induced change in OPO crystal temperature which then induced length noise via dn/dT. When leakage IFO beam sees that, it brought back to OMC DCPD noisy signal which gets mixed in the clean carrier, causing the noise we saw.
Daniel, Nutsinee
The plots above have wrong calibration: here I attached a more properly calibrated one. All of them (OPO, CLF, LO) agree with the old plots taken from SR785 within a factor of 2.
We also found that during the CLF injection data used in the original plot we saturated the read back, so we found the time where things weren't saturated and used that data here.
Mystery: We don't get the same amount of LO per CLF excitation. These plots are calibrated in terms of sqz angle. CLF noise has been multiplied by a factor of 2. LO control signal is calibrated using VCO calibration.
I also changed DARM_exc/DARM_ref cutoff to a factor of 1.7.
