ISI blends switched to 45mHz due to increasing microseism during the lockloss interval. Evan and Hugh diagnosed and remedied an ISI issue that was keeping us from locking ALS.
(Note to operators: Don't touch modulation depth, it doesn't do anything good while making it difficult for everybody to analyze the glitches.)
Summary
1. Output level of RF AM stabilization went back and forth between two levels since our last incursion to the PSL room.
2. Looking at RF AM glitch in LSC-ASAIR_B_RF90, LSC-ASAIR_A_RF45 and ASC-AS_RF45, it seems like the RF level going to EOM is constant but the modulation phase changes.
The first attachment shows a jumbo RF45 glitch reported in alog 23618 at around 11-21 2015 11:44 UTC. Nobody was touching the modulation depth slider.
Top left shows that the RF AM control signal jumped up by about 0.2% of its DC value and then came back after about 4 seconds. During this 4 second window, the signal was full of large glitches. This was felt by the control and error signal of the loop (top and middle, left ) but not by out-of-loop AC and DC (top and middle, middle row).
This was also felt by the spare unit in the CER (middle right) and the demodulator at the ISC rack that Evan installed (alog 23567, top right). For the CER unit, the jump was about 2E-5 of the DC level. For Evan't demodulator, it's AC-coupled, but using the gain of 1000 for SR560 and using 175mV DC level before amplification, the jump seems to be 5mV/175mV/1000~3E-5, which is consistent with the CER unit. Since these are much smaller than what the RF AM stabilization see (0.2%), it seems to me that the glitch in CER and ISC rack is a result of something bad in the EOM modulation path propagating back to the CER.
Interestingly, ASAIR_A_RF45_I_ERR jumps but Q_ERR doesn't (bottom, middle). The latter should be dominated by the DARM length offset. If the RF level coming out of the EOM driver jumps, I would expect both Q and I jump. This seems to me that the RF AM stabilization is working fine, but there's some phase jump associated with the AM glitch and the effect is more evident in the phase that has smaller signal.
Similarly, ASAIR_B_RF90_Q_ERR jumps but I_ERR doesn't (bottom, left), ASC-AS_A_RF45_I_SUM does but Q_SUM doesn't.
The second attachment shows the second jumbo glitch, but this was not reported by the operators because it happened when IFO is not locked. You can see that the LSC-MOD_RF45_AM_CTRL_OUT jumped by similar amount as the first event, but it never came back. No IFO signals were available of course, but you can see that it's not like there's a huge electronics coupling for LSC-ASAIR_B and ASC-AS_A_RF45.
It seems like there are two states, whatever they are, and the RF AM stabilization is responding to the transition from one to the other (or maybe the stabilization is going crazy). It could be an impedance change somewhere but I'm not sure.
Whatever the cause is, it cannot be after the EOM driver. If something is going on after the driver, the reflection should propagate back to the rms detector of in-loop and out-of-loop differently, and I cannot imagine the out-of-loop channels look flat as they do here.
So, something is going on, it could be anywhere between the distribution amplifier for this specific path in CER to the EOM driver itself, including these two end points. We could swap the distribution amplifier spigot of the spare and PSL in CER, I doubt that it does anything but we can further narrow down the trouble spot.
The third attachment shows that this going back and forth behavior was present before. This is from a week ago noted in Ed's alog. The amplitude was much smaller, but we can see that the output level jumped at around 77 seconds, then back at around 123 sec.
WP 5623 The psinject process running on h1hwinj1 is no longer controlled by Monit. If the psinject fails, it will not be started automatically and will require operator intervention. The condition of failure is indicated by a RED status for excitation on the CDS overview screen for the H1CALCS model. We will work toward installing an alarm for the condition. According to David Shoemaker, the restarts of the CW injections are disruptive to observing. If the psinject fails, it should be restarted during single-ifo time or downtime, or at the time of relocking.
SRM saturation alarms. It appears to be an EQ that was not announced by Terramon. USGS shows a 5.5 in Mexico @ 20:41 UTC. Not sure why there is so much lag with Terramon.
Terramon finally updated, ~8 minutes after the predicted R-wave arrival time. Surprise!
GraceDB will be going down for emergency maintenance this afternoon (15:30 CST) for up to 3 hours. Information below from the DASWG list. Incident Type -- Emergency Maintenance Start Date and Time — 2015-11-23 15:30 CST (UTC-5) End Date and Time -- 2015-11-23 ~18:30 CST Service(s) Affected — GraceDB needs to be temporarily shut down in order to recover backup data. The service will be unavailable during the outage."
Patrick reported that h1isiham3 is seeing a single CRC error Link
Last Tuesday I reconfigured the DAQ to trend these CRC channels since we have been seeing them recently. This ISIHAM3 is the first error seen since Tuesday. The error was raised yesterday evening
Nov 22 2015 17:08:43 PST
The GDS-TP screen for ISIHAM3 shows no loading events at this time (system has been stable since Oct 20th).
This error is saying that for one 1/16th of a second block of data sent from ISIHAM3 to the DAQ data concentrator, the CRC check sum for the data block as computed by the front end did not match that computed by the data concentrator.
The next time H1 is out of observation mode, the CRC can be reset with the "DAQ clear accumulated CRC" button on the CDS overview screen.
No obvious cause. VerbalAlarms stated that ITMy was saturating simultaneously with lockloss. Lockloss tool plots attached.
SudarshanK, RickS
The plots attached is looking at the calibration line at 332 Hz at both H1 and L1 for X03 and X02 data for a single day (24 hr period). The data was filtered using GDS state vector greater or equal to 32767. The residual plots between X03 and X02 are auto-ranged to include all the points and the residual are pretty small for both H1 and L1
The linked matlab figure includes all 9 days worth of data and all plots are autoscaled again. The residuals are very small, except for a few outliers that are all, or mostly all, in the X02 data, not the X03 data. So, from the perspective of our SLM tool analysis, it looks like the X03 frames are an improvement and the GDS/Pcal calibration ratios at the ~330 Hz lines are very stable, near or at the 1% level.
Attached are 7 day trends of OpLev pitch, yaw, and sum for all active H1 oplevs.
An alog describing the new GDS calibration filters has been posted to the LLO alog #22939.
The plots being presented here were made by an upconversion/downconversion finding tool called NCOR. An explanation of the tool and how to interpret the resulting plots can be found HERE.
To make the plots presented in this post all "analysis-ready" lock times of O1 have been included. That is all data from Sept. 14th - Nov. 22nd when the "analysis-ready" flag was active. Each point in these plots is 32 seconds of data. In general all 60 Hz harmonics have been notched out, as well as the violin mode peaks in the higher frequency bands. Robert Schofield has given me valuable help with the analysis, including channels to look at, and the list of channels searched at both sites can be found here: LHO LLO .
Some of the most interesting upconversion bands are shown below:
.1 - .3 Hz -> 55 - 65 Hz
.1 - .3 Hz -> 85 - 110 Hz
.1 - .3 Hz -> 50 - 200 Hz
.3 - 1 Hz -> 55 - 65 Hz
.3 - 1 Hz -> 10 - 50 Hz
.3 - 1 Hz -> 85 - 110 Hz
.3 - 1 Hz -> 50 - 200 Hz
10 - 30 Hz -> 85 - 110 Hz
30 - 50 Hz -> 85 - 110 Hz
30 - 50 Hz -> 121 - 200 Hz
30 - 50 Hz -> 1000 - 2000 Hz
The directories with the full results, including all band-pairings, can be found here: LHO LLO .
I did search for downconversion as well (from the 700 - 900 Hz band and the 700 - 1200 Hz band) but I did not find too much of interest. Those results can be seen in the full results directories. If anyone has any specific frequency bands or channels they would like looked at I am happy to analyze them.
TITLE: 11/23 Owl Shift 08:00-16:00UTC (00:00-08:00 PST), all times posted in UTC
STATE Of H1: Observing
SUPPORT:
LOCK DURATION:
INCOMING OPERATOR: Travis
END-OF-SHIFT SUMMARY: Other than one lockloss, it's been a quiet night. Hardly any wind. No noticable seismic activity in EQ band. Useism ~2e-1 um/s.
ACTIVITY LOG: See alog23653 and comments therein.
In response to the behaviour of the ISS diffracted power at Livingston (https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=22928) the attached plots are generated. In short, the large fluctuations seen at Livingston do not appear to be present.
Cause is unclear. USGS reported a 4.9M earthquake in South of Africa at 11:18 UTC. Terramon predicted the R-wave arrival time to be 4:40 UTC with 0.0404561 um/s. That shouldn't have dropped us out of lock though. ASC signal and tidal looked stable on the FOM before the lockloss. Wind speed ~2mph.
Relocked and Observing again at 12:35 UTC. The butterfly mode near 2500Hz amplitude is currently at 2e-16 m/sqrt(Hz) but seems to be coming down slowly. I had to touch COMM VCO and DIFF OFFSET to maximize LSC TRX and Y normalized transmission during FIND_IR. And had to engage ISS 2nd loop by hand several times before succeeded.
The BNS range has been slowly dropping since about an hour ago. I know we lose some range due to the local traffic but losing 10 Mpc seems a bit much....
Performed a run spanning the time period from 1129593617 (Oct 23 2015 00:00:00 UTC) to 1131828544 (Nov 17 2015 20:48:47 UTC)
The following parameters were used for this run:
Only two injections were found to be scheduled during this period:
However, neither of these injections were found to actually occur within the examined time period.
There were no normal injections that occurred within the examined period. There were a number CAL-INJ resets, all single-IFO with the same curious pattern to the frame flags as denoted in the immediately prior HWInj report (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23485). There was a single-IFO burst injection in H1
BURST 1129673117.000 (H1)
with the only anomaly being that it is UNSCHEDULED. There was a single UNKNOWN injection in L1
UNKNOWN 1129673117.000 (L1)
This injection was confirmed to occur only within the CAL-INJ channel of RAW frames with the TRANSIENT bit indicating an injection but none of the specific injection-type bits, CBC, BURST, DETCHAR, or STOCHASTIC, indicating an injection. It was confirmed to not occur in ODC HOFT, ODC RDS, ODC RAW, or GDS HOFT frames.
It is interesting that the single-IFO injections BURST 1129673117.000 (H1) and UNKNOWN 1129673117.000 (L1) are found to occur at the exact same time but within different IFOs. Even further, the fact the BURST injection in H1 is perfectly normal, other than being UNSCHEDULED, while the UNKNOWN injection in L1, in addition to being UNSCHEDULED, is inconsistent. HWInjReport currently does not check the HW bit or the CW bit in the CAL-INJ channel for RAW frames for the presence of injections; however, upon examining these bits directly, it was found that while the HW bit indicated an injection, along with the TRANSIENT bit, the CW bit did not indicate an injection (my first thought was that this may be a CW injection only occurring in L1). This means this was a truly UNKNOWN, UNSCHEDULED injection occurring only with L1. Given both injections occur at the exact same time and the L1 injection is a truly UNKNOWN type (practically a ghost injection, by the bits), it is likely this was intended to be a normal network injection; however either user-error or a catastrophic software error corrupted the proper setting of the bits.
There was an additional UNKNOWN injection that occurred at 1131397646 in both H1 and L1 (as noted by Peter Shawhan) with a signature similar to the one reported here. However, for some reason HWInjReport missed it. I will have to investigate why HWInjReport missed that particular injection, but, unlike the last session of bug-fixing, I am not going to cease production of reports. It is my estimation that HWInjReport is currently operating with reasonable confidence to find and report most true anomalies. If reporting errors or omissions are found, then those errors should be correctable and a new report on the relevant time period can be generated.
I didn't see any follow-up on this but I did some checks: The injection at 1129673117 is the stochastic hardware injection that Joe and Chris did. This was out of science mode. The injection at 1131397646 should be the injection in the schedule file from line: 1131397639 1 1.0 coherentbbh0_1126259455_ This CBC injection was not logged properly for the searches because tinj was setting CAL-INJ_TINJ_TYPE instead of CAL-PINJX_TINJ_TYPE. So I would guess that was the problem that you're seeing here. The issues have since been fixed though.
Now that I finally have my diagnostic tool working (it's a ripped-down version of HWInjReport called HWInjCheck that simply returns raw output from FrBitmaskTransitions to allow rapid verification of anomalies found in HWInjReport), I re-examined the missing UNKNOWN injection at 1131397646. Firstly, HWInjReport can only identify injections as UNKNOWN if they occur in the CAL-INJ. This is because, as far as I can determine, only the CAL-INJ channel has an independent excitation bit that indicates whether a transient injection has occurred, regardless of type (as well as an independent HW bit indicating the occurrence of an injection of any type). This does not seem to be true for ODC-MASTER and GDS-CALIB channels. So, in CAL-INJ, if the TRANSIENT bit is "off" (indicating the presence of a transient injection) but the CBC, BURST, DETCHAR, and STOCHASTIC bits, which indicate the type of injection, are "on", then it is clear we have an UNKNOWN type transient injection occurring. However, for ODC-MASTER and GDS-CALIB, if the type bits are not "off", there is nothing to indicate excitation of a transient injection with an unspecified type. If 1131397646 occurs with an unspecified type in the ODC-MASTER or GDS-CALIB channel but not in the CAL-INJ channel, then HWInjReport will completely miss the injection as it won't show up at all in any of the bits and channels that HWInjReport checks.
Using HWInjCheck to check the output from FrBitmaskTransitions, I have not, yet, found any indication of the occurrence of 1131397646, that is, there is no indication of excitation in the excitation bits. I will check again more carefully, but, at this point, it does seem HWInjReport is working properly; this is an expected missing injection. To be clear, there may still have actually been an excitation, there just doesn't seem to be any record of it in the bits and channels that HWInjReport checks.
