The IFO just got to NLN, so I switched to Observe. Shortly after, the ring EX ring up reappeared, so we have dropped out of Observe again, and are tuning some ASC to try to settly the IFO down.
Not a cause for alarm. Just noting for commissioners interest that after doing initial alignment tonight, I went straight to Lock PRMI, which acquired quickly, but the transition to DRMI failed. This is just the first round of trying to relock for tonight. DRMI just acquired for a second time, then promptly dropped the whole IFO out. Looking to be a long night.
J. Kissel While trying to characterize problems with HEPI / TIDAL / ISI / ASC (LHO aLOGs 23676 and 23676), I tried taking a swept-sine transfer function with three data points between 0.001 and 0.005 to charaterize the HEPI / UIM cross-over frequency (recall that the TIDAL crossover frequencies were never calibrated beyond "roughly," LHO aLOG 16255). However, upon starting the excitation, DTT gave the IFO a good kick on lots of ASC and LSC DOFs. It did *not* break the lock, but it certaintly came close, and any higher an excitation amplitude would have certaintly lost it. From a look at the time serious, it looks like DTT *tried* to ramp the signal, but chose a much faster ramp than is necessary for such a waveform. Attached are the start of the waveform, a zoom in on the ramp, and then a screenshot of the DTT params.
After the maintenance of GraceDB this afternoon (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23665) I restarted the ext_alert process on h1fescript0 using Monit.
Generator will run continuously until Wednesday afternoon to supply power for the bake-out of the new ion pump -> Will be in @ ~0100 hrs. local tonight to refuel
Title: 11/23 Day Shift 16:00-24:00 UTC (8:00-16:00 PST). All times in UTC.
State of H1: Locked at NLN but in Commissioning
Shift Summary: After a couple of EQ related locklosses, we ran into an issue with the ISIs using 45mHz blends causing them to ring up, resulting in lockloss. Evan, Hugh, JeffK, and presumably Jim (when he arrives for shift) working on resolving this. We could revert back to 90 mHz blends, but with microseism increasing, that might not last long either.
Incoming operator: Jim
Activity log:
16:37 Joe and Chris to X arm for beamtube sealing
18:52 Lockloss
19:25 Observing
20:11 Joe and Chris back
23:00 Lockloss
A few short locks that I did not log as we were looking in to the osc. issue.
Evan, Hugh
As seen by JimW and others, the ETMX was moving ALS excessively especially wrt ETMY. Attached are the in line outputs from the Isolation loop for EX and EY. This two hour trend plot shows an EQ that took the IFO out of lock around 1/3 of the wasy through. The ring up of the ISO X at ETMX is very clear in the second have of the plot. The second attachment shows uncalibrated spectra of the X Drive two hours ago and during the ring up where it appears to be elevated in the 40mHz region. Oh what the heck, I'll put them on the same snap.
We deisolated the DOF and then reisolated. This action seems to have addressed the problem for now. Well not for long. Rung up again with IFO locked. Took it out of Observing and tried turning of boosts and adjusting gains.
E. Hall, J. Kissel, T. Sadecki, H. Radkins, J. Warner
The IFO had lost lock again from the same problem about ~20 minutes later. We noticed LLO was down, so we tried pushing buttons hoping to rectify the problem.
We tried several things, all on hunches we'd had from all signals we saw oscillating at ~30-40 [mHz], which included HEPI / ST1 ISI / PRC1 Yaw and TIDAL -- so this is NOT just an "ISI ETMX Loop goes unstable" problem, its a nasty, slow, cross-coupled interaction of which the ISI is only one of the players.
Things we "know" that informed our hunches and button mashing:
- This only shows up when the ISIs are on 45 [mHz] blends.
- Symptoms point us toward ETMX, because it stays rung-up after lock-loss
- The above mentioned channels are visible wobbling at these 30-40 [mHz] frequencies
- There seems to be a 6-8 minute period envelope to the oscillations
Our hunches include
(1) Bad IMC_F-to-UIM or UIM-to-HEPI tidal cross-over when ISIs are in 45 [mHz] blends, i.e. the tidal offload is at a higher frequency than the ISI's inertial blend frequency, so the ISI's isolation loops are fighting actuation from tidal
(2) PRC1 Yaw ASC loop (which keeps PRM aligned to the PRC's cavity axis) unstable, which modulates the power into the arms. This modulates the classical radiation pressure on the ETMs in common, which excites CARM / Tidal. This then puts excess motion into the Tidal feedback and offload.
(3) Excess motion on HEPI from tidal is causing excess tilt, because HEPI tilts when you request it to drive in longitudinal. With the ISI ST1 in 45 [mHz] blends, and the RX&RY blend still in the 250 [mHz] blend configuration, the excess tilt from HEPI is coupling into the ST1 CPS, tilting the ISI, and fooling the X&Y DOFs into thinking there's excess X&Y motion, and drives excess X&Y. This pushes the test mass in X&Y, exciting some DARM / CARM action, which then spills back to HEPI.
Note that these are all sort of half-coupling mechanisms that, if I waved my hands hard enough, you might believe. But, we have no measurements (i.e. Sys ID, Loop Transfer Function measurements) to prove any of these statements.
Based on these hunches, we twiddled the following knobs:
- Reducing UIM to HEPI offload UGF (H1:LSC-${X,Y}_TIDAL_CTRL_UGF) from 0.002 to 0.001 [Hz] --> no noticable effect, but we were watching a 10 [minute] strip tool, so out patience may not have been high enough. The Oscillation was present, but not ringing up.
- Reducing UIM to HEPI offload UGF by half again to 0.0005 [Hz] --> same result
- Returning UIM to HEPI offload UGF to 0.001 [Hz], and reducing the IMC-F to UIM offload UGF (H1:LSC-${X,Y}_COMM_CTRL_UGF) --> We might have started to see a reduction in amplitude.
- Reducing the PRC1 loop gain (via the POP A DC to PRC1 ASC input matrix H1:ASC-INMATRIX_Y_3_21) from 1.0 to 0.5 --> This seems to reduce the oscillations in all signals.
While Evan and Jim coninued to twiddle knobs, I and LLO was down, I suggested we try to characterize some of these low frequency loops. However, a third of the way through my initial characterization of the UIM to HEPI tidal offload (which had already had a bad start, see LHO aLOG 23680), the oscillations had come back to a point of no return. Unclear whether they were because we just didn't reduce the original oscillation fully, or whether the bad start to the excition kicked everything enough to restart a bigger oscillation, or if it was my measurement itself slowly driving things around that re-triggered the oscillation enough to eventually saturate Tidal / the UIM. However, it was apparent that it was the 6-8 minute envelope that brought things over the edge, not the 30-40 [mHz].
After that lock-loss, Jim had found IR in the arms to be pretty bad, so he has since gone into initial alignment.
Stay tuned...
I attach some visual aides that I'd used to discuss the hunches with Hugh / Evan / Travis in case it helps anyone else. (THC = Tilt Horizontal Coupling).
Went to Commisioning Mode at 22:48 UTC because ETMx and ETMy ISIs began oscillating. Evan and Hugh tried a few things, but to no avail and lock was lost.
Laser Status:
SysStat is good
Front End power is 31.47W (should be around 30 W)
Frontend Watch is GREEN
HPO Watch is RED
PMC:
It has been locked 5.0 days, 21.0 hr 50.0 minutes (should be days/weeks)
Reflected power is 1.726Watts and PowerSum = 25.12Watts.
FSS:
It has been locked for 0.0 days 0.0 h and 37.0 min (should be days/weeks)
TPD[V] = 0.9939V (min 0.9V)
ISS:
The diffracted power is around 8.366% (should be 5-9%)
Last saturation event was 0.0 days 0.0 hours and 37.0 minutes ago (should be days/weeks)
Performed a run spanning the time period from 1131828057 (Nov 17 2015 20:40:40 UTC) to 1132088085 (Nov 20 2015 20:54:28 UTC)
The following parameters were used for this run:
No scheduled injections were found for this period.
There were only two single-IFO injections found; both were CAL-INJ resets in H1:
Both injections were found to occur only in ODC HOFT and GDS HOFT.
No notable anomalies were found.
ISI blends switched to 45mHz due to increasing microseism during the lockloss interval. Evan and Hugh diagnosed and remedied an ISI issue that was keeping us from locking ALS.
(Note to operators: Don't touch modulation depth, it doesn't do anything good while making it difficult for everybody to analyze the glitches.)
Summary
1. Output level of RF AM stabilization went back and forth between two levels since our last incursion to the PSL room.
2. Looking at RF AM glitch in LSC-ASAIR_B_RF90, LSC-ASAIR_A_RF45 and ASC-AS_RF45, it seems like the RF level going to EOM is constant but the modulation phase changes.
The first attachment shows a jumbo RF45 glitch reported in alog 23618 at around 11-21 2015 11:44 UTC. Nobody was touching the modulation depth slider.
Top left shows that the RF AM control signal jumped up by about 0.2% of its DC value and then came back after about 4 seconds. During this 4 second window, the signal was full of large glitches. This was felt by the control and error signal of the loop (top and middle, left ) but not by out-of-loop AC and DC (top and middle, middle row).
This was also felt by the spare unit in the CER (middle right) and the demodulator at the ISC rack that Evan installed (alog 23567, top right). For the CER unit, the jump was about 2E-5 of the DC level. For Evan't demodulator, it's AC-coupled, but using the gain of 1000 for SR560 and using 175mV DC level before amplification, the jump seems to be 5mV/175mV/1000~3E-5, which is consistent with the CER unit. Since these are much smaller than what the RF AM stabilization see (0.2%), it seems to me that the glitch in CER and ISC rack is a result of something bad in the EOM modulation path propagating back to the CER.
Interestingly, ASAIR_A_RF45_I_ERR jumps but Q_ERR doesn't (bottom, middle). The latter should be dominated by the DARM length offset. If the RF level coming out of the EOM driver jumps, I would expect both Q and I jump. This seems to me that the RF AM stabilization is working fine, but there's some phase jump associated with the AM glitch and the effect is more evident in the phase that has smaller signal.
Similarly, ASAIR_B_RF90_Q_ERR jumps but I_ERR doesn't (bottom, left), ASC-AS_A_RF45_I_SUM does but Q_SUM doesn't.
The second attachment shows the second jumbo glitch, but this was not reported by the operators because it happened when IFO is not locked. You can see that the LSC-MOD_RF45_AM_CTRL_OUT jumped by similar amount as the first event, but it never came back. No IFO signals were available of course, but you can see that it's not like there's a huge electronics coupling for LSC-ASAIR_B and ASC-AS_A_RF45.
It seems like there are two states, whatever they are, and the RF AM stabilization is responding to the transition from one to the other (or maybe the stabilization is going crazy). It could be an impedance change somewhere but I'm not sure.
Whatever the cause is, it cannot be after the EOM driver. If something is going on after the driver, the reflection should propagate back to the rms detector of in-loop and out-of-loop differently, and I cannot imagine the out-of-loop channels look flat as they do here.
So, something is going on, it could be anywhere between the distribution amplifier for this specific path in CER to the EOM driver itself, including these two end points. We could swap the distribution amplifier spigot of the spare and PSL in CER, I doubt that it does anything but we can further narrow down the trouble spot.
The third attachment shows that this going back and forth behavior was present before. This is from a week ago noted in Ed's alog. The amplitude was much smaller, but we can see that the output level jumped at around 77 seconds, then back at around 123 sec.
WP 5623 The psinject process running on h1hwinj1 is no longer controlled by Monit. If the psinject fails, it will not be started automatically and will require operator intervention. The condition of failure is indicated by a RED status for excitation on the CDS overview screen for the H1CALCS model. We will work toward installing an alarm for the condition. According to David Shoemaker, the restarts of the CW injections are disruptive to observing. If the psinject fails, it should be restarted during single-ifo time or downtime, or at the time of relocking.
SRM saturation alarms. It appears to be an EQ that was not announced by Terramon. USGS shows a 5.5 in Mexico @ 20:41 UTC. Not sure why there is so much lag with Terramon.
Terramon finally updated, ~8 minutes after the predicted R-wave arrival time. Surprise!
LLO had called and said they would be down for awhile. I'm going to finish doing a swept-sine through the INJ_HARDWARE filterbank from 5-500Hz. Previously I had only finished 500-2000Hz. The IFO intent mode has been turned off. This is WP#5595.
Previous swept-sine measurement was: aLog 23307 The following was done: * psinject turned off at 5:20 UTC * swept sine measurement started at 5:21 UTC * swept sine measurement ended at 5:48 UTC * psinject turned on at 5:49 UTC I've attached a plot of the transfer function and coherence between INJ_CW and DELTAL_EXTERNAL_DQ. WP#5955 is now complete.
Attached are results of computing a transfer function from H1:CAL-INJ_CW_OUT to H1:GDS-CALIB_STRAIN, using the same method used for the higher-frequency swept sine on Nov 11. The transfer function is generally flat and well behaved below 500 Hz and above ~35 Hz. The feature at 167 Hz appears to be an artifact of the swept sine measurement at that point being artificially split into several points. Figure 1 - Spectogram of H1:CAL-INJ_CW_OUT during swept sine Figure 2 - Spectogram of H1:GDS-CALIB_STRAIN during swept sine Figure 3 - Transfer function and coherence More detailed results can be found here.
The DTT template for doing the 5 to 500Hz swept-sine measurement is in the SVN here: template_cw_inj_sweptsine_5to500_20151121.xml The DTT template for doing the 500 to 2000Hz swept-sine measurement is in the SVN here: template_cw_inj_sweptsine_500to2000_20151111.xml
Performed a run spanning the time period from 1129593617 (Oct 23 2015 00:00:00 UTC) to 1131828544 (Nov 17 2015 20:48:47 UTC)
The following parameters were used for this run:
Only two injections were found to be scheduled during this period:
However, neither of these injections were found to actually occur within the examined time period.
There were no normal injections that occurred within the examined period. There were a number CAL-INJ resets, all single-IFO with the same curious pattern to the frame flags as denoted in the immediately prior HWInj report (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23485). There was a single-IFO burst injection in H1
BURST 1129673117.000 (H1)
with the only anomaly being that it is UNSCHEDULED. There was a single UNKNOWN injection in L1
UNKNOWN 1129673117.000 (L1)
This injection was confirmed to occur only within the CAL-INJ channel of RAW frames with the TRANSIENT bit indicating an injection but none of the specific injection-type bits, CBC, BURST, DETCHAR, or STOCHASTIC, indicating an injection. It was confirmed to not occur in ODC HOFT, ODC RDS, ODC RAW, or GDS HOFT frames.
It is interesting that the single-IFO injections BURST 1129673117.000 (H1) and UNKNOWN 1129673117.000 (L1) are found to occur at the exact same time but within different IFOs. Even further, the fact the BURST injection in H1 is perfectly normal, other than being UNSCHEDULED, while the UNKNOWN injection in L1, in addition to being UNSCHEDULED, is inconsistent. HWInjReport currently does not check the HW bit or the CW bit in the CAL-INJ channel for RAW frames for the presence of injections; however, upon examining these bits directly, it was found that while the HW bit indicated an injection, along with the TRANSIENT bit, the CW bit did not indicate an injection (my first thought was that this may be a CW injection only occurring in L1). This means this was a truly UNKNOWN, UNSCHEDULED injection occurring only with L1. Given both injections occur at the exact same time and the L1 injection is a truly UNKNOWN type (practically a ghost injection, by the bits), it is likely this was intended to be a normal network injection; however either user-error or a catastrophic software error corrupted the proper setting of the bits.
There was an additional UNKNOWN injection that occurred at 1131397646 in both H1 and L1 (as noted by Peter Shawhan) with a signature similar to the one reported here. However, for some reason HWInjReport missed it. I will have to investigate why HWInjReport missed that particular injection, but, unlike the last session of bug-fixing, I am not going to cease production of reports. It is my estimation that HWInjReport is currently operating with reasonable confidence to find and report most true anomalies. If reporting errors or omissions are found, then those errors should be correctable and a new report on the relevant time period can be generated.
I didn't see any follow-up on this but I did some checks: The injection at 1129673117 is the stochastic hardware injection that Joe and Chris did. This was out of science mode. The injection at 1131397646 should be the injection in the schedule file from line: 1131397639 1 1.0 coherentbbh0_1126259455_ This CBC injection was not logged properly for the searches because tinj was setting CAL-INJ_TINJ_TYPE instead of CAL-PINJX_TINJ_TYPE. So I would guess that was the problem that you're seeing here. The issues have since been fixed though.
Now that I finally have my diagnostic tool working (it's a ripped-down version of HWInjReport called HWInjCheck that simply returns raw output from FrBitmaskTransitions to allow rapid verification of anomalies found in HWInjReport), I re-examined the missing UNKNOWN injection at 1131397646. Firstly, HWInjReport can only identify injections as UNKNOWN if they occur in the CAL-INJ. This is because, as far as I can determine, only the CAL-INJ channel has an independent excitation bit that indicates whether a transient injection has occurred, regardless of type (as well as an independent HW bit indicating the occurrence of an injection of any type). This does not seem to be true for ODC-MASTER and GDS-CALIB channels. So, in CAL-INJ, if the TRANSIENT bit is "off" (indicating the presence of a transient injection) but the CBC, BURST, DETCHAR, and STOCHASTIC bits, which indicate the type of injection, are "on", then it is clear we have an UNKNOWN type transient injection occurring. However, for ODC-MASTER and GDS-CALIB, if the type bits are not "off", there is nothing to indicate excitation of a transient injection with an unspecified type. If 1131397646 occurs with an unspecified type in the ODC-MASTER or GDS-CALIB channel but not in the CAL-INJ channel, then HWInjReport will completely miss the injection as it won't show up at all in any of the bits and channels that HWInjReport checks.
Using HWInjCheck to check the output from FrBitmaskTransitions, I have not, yet, found any indication of the occurrence of 1131397646, that is, there is no indication of excitation in the excitation bits. I will check again more carefully, but, at this point, it does seem HWInjReport is working properly; this is an expected missing injection. To be clear, there may still have actually been an excitation, there just doesn't seem to be any record of it in the bits and channels that HWInjReport checks.
Getting ready to go back into Observe, Jenne had me reduce the gains on PRC1 Y with a -20dB filter, and reduce the gains on DHARD P (from 10 to 7) and Y(from 15 to 10). The low frequency ground motion is coming up, so we may be getting an earthquake.