Set OIB/OOM to Commissioning/Maintenance for Richard to go into CER and check cables for 1Hz comb in PRCL
Wind calm. Sharp rise in ITMY_Z_BLRMS_30M_100M possibly due to 4.7mag in Central America an hour ago but it doesn't seem to be fazing us at all. IFO still locked and in good shape. In fact, in the last half hour it appears that the range may have taken a very slght upward trend from 60 to 62Mpc. This segment has been glitch free, so far. We've had about 3 hours of coincidental locking with LLO. Steady as she goes. Full steam ahead.
Set at 8:43UTC. 60Mpc
The ODCMASTER in SDF will show 1 diff during this segment due to a change that had to be made that was blocking the Observation Ready bit to Green and keeping he INtent bit from being set. CALCS will also show diffs during this segment. Commissioners aren't certain what these mean.
[Sheila, Dan, Evan, Jenne, others]
All afternoon (basically, since we started relocking after maintenence day) we have seen a 1 Hz comb (offset by 0.5 Hz) in the DARM spectrum, as well as in many, many other places. This seems like the same thing as the final bullet point that was reported in aLog 20790 a few days ago, but with much taller peaks.
Much of the day was spent looking into this, by several people. The lines are very clear in all of the length error signals, as well as the ASC error signals.
One thing that we tried, just to exonerate it, was swap the PRC length actuation from the PRM to PR2, and then turn off the PRM M3 coil driver. As expected, PR2 affects the cavity length twice as much as the PRM, so we needed a gain of 0.5 in the M3 LOCK filter bank for PR2, rather than the 1 that PRM has. We were able to turn off the PRM M3 coil driver, but saw no change in the length signals' spectra. We put the actuation back on PRM, but when we tried to turn off the PR2 coil driver, PR2 tripped and we lost lock. So, the new PRM coil driver is certainly not to blame for this new noise. (PR2 was untripped, and its coil driver turned back on) Also, we haven't seen the glitches due to the old PRM coil driver at all today, so hopefully they're gone for good, and not just hidden.
At Richard's suggestion, we tried restarting the models that were restarted earlier today in case it is some kind of synchronization problem. We restarted h1prm, h1pr2, h1lsc, h1asc. This didn't change anything - the lines are still there loud and clear after relocking.
The 1 Hz comb is still present, and we are running low on ideas for what it could be, and what to do about it. DetChar friends, and anyone with ideas, please let us know if you have a suggestion of where to look.
--------------
I don't know if this has the same cause as the 1Hz comb, but there are also some unusually large lines at ~838 Hz and ~855 Hz. These lines are a few Hz wide, and are coherent with magnetometer signals. But, the magnetometers have seen these lines at the same (or larger) amplitude, in the past, without seeing the lines in DARM. Why would the coupling from magnetic fields have changed so drastically??
Tagging DetChar so they're aware of this entry.
The various noise features that appeared after maintenance today -- the 1Hz comb below 70Hz, some broadband noise above 80Hz, and the peaks around 850Hz -- are all coherent with the DRMI length error signals, and are much louder (compared to quiet references) in PRCL and MICH. Something is different in the corner station...hopefully all three things have the same source.
The 1-Hz comb with 0.5-Hz offset is prevalent in magnetometer channels. Looking at a typical magnetometer FSscan spectrogram is not for the faint of heart, but there are plenty of examples to be found here. One example from EY_MAG_EBAY_SUSRACK_Y is attached. It's not unusual.
Since the reports stated it shows up in PEM channels I disconnected the LO to the antenna Demod board that resides in the PEM rack and was coincidentally connected yesterday and the comb disappeared. WE need to verify the RF level needed for these radio signals. Pick some attenuators and re-installed the LO signal. I put terminators back on the outputs from the distribution chassis.
This could also have been due to a ground loop. A balun should be added to the lines.
I thought this might be the ITM HWS. It looks like the filter on the power supply (which is filter out the 1Hz comb) was turned off yesterday.
A small point about the PRM/PR2 experiment: when we switched to PR2, there was a huge 45 Hz resonance (visible in DARM and the vertex LSC signals) that was rung up. It was distinct from the usual 41 Hz HSTS roll mode resonance that we usually see when passing through the noise tuning step in full lock. It did not ring down on its own.
We made sure that the PRCL OLTF did not change before and after the swap, and anyway the loop appeared to have enough phase (more than 30°) at this frequency, so I don't think it was gain peaking. By increasing the loop gain, we were able to bring the resonance down a little bit. I then added 4 dB of resonant gain around 45 Hz, which somehow squashed the resonance completely. (I put it in the PR2 M3 lock L SFM).
It's not clear to me what was going on here.
IFO locking appears to have recovered well from maintenance but investigations are ongoing into new excess noise. UTC (Pacific) 23:33 (16:33) Dave to roof to look at antennas to update drawings 23:39 (16:39) Dave back 00:03 (17:03) Nutsinee to OSB high bay mezzanine to check for distilled water for TCS chillers 00:25 (17:25) Nutsinee back. She also cleaned the TCS chiller filters. 05:08 (22:08) Sheila and Jenne to flip switch on PRM M3 LL coil driver 05:15 (22:15) Eric T. running injections 05:48 (22:48) Eric T. done 06:06 (23:06) Sheila turned off PR2 coil driver, tripped PR2 watchdog, broke lock, turning it back on 06:14 (23:14) Sheila, Evan, Jenne restarting PRM, LSC, ASC and PR2 models
Patrick, Sheila, Jenne, Eric For the first part of the test, we injected our fiducial CBC waveform (same one used in ER7) and tried raising the LIMIT value on the hardware injection block in order to address saturation problems observed in ER7. During ER7, the LIMIT was 200. We raised it to 400. The first injection did not go through: 1124601535 1 1.000000 cbctest_1117582888_ intent bit off, injection canceled Patrick, Sheila, and Jenne tried to turn on the intent bit, but there was some sort of problem, which will be alog'ged separately. As a temporary work-around, we turned off the tinj intent-bit check and injected again: 1124602724 1 1.000000 cbctest_1117582888_ successful Patrick determined that the injection produced a maximum |amplitude| of 15 counts coming out of the injection block, which seemed to indicate that the original LIMIT value of 200 was sufficient. However, an alarm went off to indicate that there was saturation at ETMY. Thus, the saturation problem cannot be solved by tinkering with the INJ block in MEDM. Rather, the problem is occurring downstream on the ETM actuators. We request that Jeff K, Adam M, et al. look into options for avoiding saturation at the ETMs. Next we tried a blind injection using the new blind injection code. The blind injection code does not log injections in EPICS so they are not automatically picked up in the segment database. 1124603111 1 1.000000 cbctest_1117582888_ successful The blind injection was clearly visible. The ETM saturation warning went off again. The injection was logged correctly in the blind injection blindinj_H1.log: current time = 1124603049... Attempting: awgstream H1:CAL-INJ_BLIND_EXC 16384 /ligo/home/eric.thrane/O1/Hardw areInjection/Details/Inspiral/H1/cbctest_1117582888_H1.out 1 1124603111 Injection successful. All of these injections were carried out with scale factor = 1; (that's the 1.000000). The injection file, described in a comment below, is a 1.4 on 1.4 BNS, optimal orientation, at D=45 Mpc. It is the same waveform used in previous ER tests.
It looks like the injection actually does hit the 400 count limit (plot 1). It saturates right at the end when the injection chirps up to high frequency. There's some kind of ringing as well (plot 2). From the spectrogram (plot 3) and the zoom (plot 4) this looks like a feature at just above 300 Hz. I thought it might be a notch for the PCal line, but that's 331.9 Hz. So someone will have to check the inverse actuation filter and see what's happening at that frequency. It's possible to see the overflow from the first injection in the ETMY L3 MASTER channel (plot 5). It happens at -131072 counts, and the injection is trying to push it past -200000. The blind injection caused an overflow as well, but since this channel is only recorded at 2048 Hz, it looks like it falls short of overflow (plot 6). There's a faster readback whose name escapes me at the moment. Unless the blind injection is made a factor of about 10 smaller, or rolled off at high frequency, it will be trivial to detect it by looking at the drive to the ETM.
FYI, the injected waveform was fiducial waveform from ER7: https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=16125 It's a 1.4-1.4 BNS at 45 Mpc, optimal orientation.
There are a couple of things to watch out for when performing CBC hardware injections, based on iLIGO experience:
For the ER7 injection we used an SEOBNRv2 waveform that has a ringdown at the end, hoping that this turn off would not trigger an impulse. However, for BNS masses, the turn off and ringdown is pretty sharp. I've asked Chris check that there are no "whooper" effects with the SEOBNRv2 waveform, but we haven't had chance to do this yet. For a SpinTaylorT4 waveform (the other waveform CBC wants to inject), there will definitely be a step, so this needs to be checked and rolled off carefully.
One other comment on the test: what scaling in awgstream did you use? That waveform looks monstously loud (eyeball SNR > 20). That's much louder than would be useful for a blind injections, but good for helping us find whooper effects.
Duncan, the scale factor is 1.
Just for completeness, because I didn't see it posted, here's an Omega scan of the injections in h(t). The first is the non-blind injection, the second is the blind injection. I think the glitch ten seconds after the blind injection is unrelated. I thought it might be a filter turning off or being reset, but it's not on a GPS second (it's at 1124603210.28). It does cause an overflow of the ETMY ESD DAC.
I verified that the blind injection was correctly recorded in the raw frame file.
Sheila, Patrick, Jenne, Eric Thrane on the phone.
Eric wanted to do an injection, and since his code requires the intent bit to be set, Patrick attempted to set it. This however proved impossible. This is probably related to the work described in 20870. According to the CDS overview, there are no excitations on except those from calibration, which will need to be on while the intent bit is set. (screenshots attached of the CDS overview and some ODC screens).
Side note (sheila's opinion only):
I found the ODC screens confusing to navigate, and it is very unclear to me which bits go into calculating other bits to find out what the problem is. It seems to me like it is a bad idea for us to make ourselves dependent on ODC to set the intent bit, since people on site don't have control over it.
Evan deciphered the ODC screen that was apparently telling us that TCS was not observation ready. We put a 0 in the H1:ODC-MASTER_TCS_2_MASK, and this made the observation bit ready, so Ed has set the intent bit.
I think this has uncovered a bug/issue: There may be a problem with the connection between the TCS ODC and the ODC-MASTER models at LHO. I'm debugging it now and will reply with what needs to be fixed. On a separate note: I believe site leads (at least Fred) said that operators would be trained on how to read and use the ODC with the changes made in https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=20870 and Duncan M. and Stefan B. are the experts on site to teach people how to read this screen, but I'll walk through the way to read it for this example: Evan correctly read off the bit that was preventing obs-ready from being green from the ODC MASTER overview screen. To do this, you just trace across the row that you care about (OBS-READY) and find the red bit, then count across the top list of subsystems to find which subsystem that is coming from (I would appreciate it if anyone can show us how to turn those text boxes so we can align them with the columns better). The next step would be to look down one step further by clicking on the TCS button to see that the 30th bit of the TCS ODC is the only bit that can affect the OBS-READY bit. From the comment, this was done correctly as well, and you can read off the bitmask for the relevant row in the matrix on the right. You can also see this by looking at the large black/green matrix to find the green bits that affect each row, which tell you which bits of the subsystem ODC are mapped to which row in ODC MASTER. At this point, you will see that the whole TCS ODC at the top of that screen is red, however, which I believe is a bug somewhere in the FE code. If it were behaving normally and you wanted to learn what the bits meant in the vector at the top: The last step would be to click on the relevant subsystem from the ODC-SITE-OVERVIEW screen, and in this case you would learn that bit 30 is the Excitation bit, as expected.
I have found the issue: The h1odcmaster.mdl was locally edited and then committed to SVN on Aug. 18, as recorded in alog: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=20622 This occured after we had circulated the new h1odcmaster.mdl that was used for alog: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=20870. Duncan M. is going to ensure that everything is up to date with the SVN, then re-implement the changes he made on Aug. 18 in a copy of h1odcmaster.mdl and then generate the ECR to fix the bug. For the time being, I have set all the bitmasks of form H1:ODC-MASTER_TCS_XX_MASK to 0x0 so the bad connection cannot affect ODC-MASTER.
Ryan, this does not adequately explain the situation. Why does the bit mask in TCS affect READY? We were told that READY was just GRD-IFO_OK and NO EXC. How was it that TCS was somehow wired into that logic?
Darkhan, Craig, Sudarshan
We took DARM Open Loop Transfer function and Pcal sweep one after each other.
We had a hard time keeping the ETMY ESD from saturating at frequencies above 800 Hz and at the same time obtaining reasonable coherence. So this measurement is good only below 800 Hz and upto 7 Hz. The calibration lines are turned back on after the measurement was done. (At around 2015-08-26 5:05:00 UTC).
Attached is DTT screenshot of both the measurement 1. DARM OLGTF, 2. Pcal2DARM TF and conlog file that contains filter information during the time of measurement.
Analysis to follow.
One of the difficulties we had after maintence day today was caused by the DRMI ASC not being completely offloaded to the suspension top stage before the gaurdian attempted to offload the ASC to the alingment sliders. I've added a check to the guardian, and there is a function called asc_convergence_checker in ISC_library that can be used for this in other states as well. It would be a good idea to add these checks durring inital alingment and possibly before the power up.
Daniel, Sheila
We went to the racks near HAM6 and measured the response of each quadrant of the AS WFS, both for 45 and 36 channels. We sent the excitation in on test in and read it back using RF mon on each quadrant's demod. The notches for 91 MHz are generally mistuned, (they are between 87 and 90 MHz). An image is attched for one 36 MHz channel and one 45 MHz channel, the data for all the channels are in the attached zip file.
| magnitude at 45.5MHz (dB) | at 36.4MHz (dB) | at 91 MHz (dB) | |
| AS A 45 ch1 | -36.2 | -53.6 | |
| ch2 | -37 | -55.2 | |
| ch3 | -36.8 | -54.8 | |
| ch4 | -36.5 | -55.7 | |
| AS A 36 ch1 | -38.6 | -54.8 | |
| ch2 | -39.3 | -56.4 | |
| ch3 | -39.4 | -55.9 | |
| ch4 | -38.8 | -56.8 | |
| AS B 45 ch1 | -37.2 | -55.4 | |
| ch2 | -36.9 | -56 | |
| ch3 | -37.4 | -55 | |
| ch4 | -37.5 | -57.4 | |
| AS B 36 ch 1 | -39.3 | -56.4 | |
| ch2 | -38.9 | -57.3 | |
| ch3 | -40 | -56.4 | |
| ch4 | -39.4 | -58.1 |
J. Kissel, P. Thomas Given that we've had so many examples of rocker switch death this evening (see LHO aLOG 20839), I was able to identify that my new(ish) warning message about this failure mode on the MEDM overview screen (see LHO aLOG 20281) had a bug. The MEDM logic was watching L2 for all three stages, L2, L1, and M0. I've committed the bug-fix to the SVN such that LLO can receive its benefits.
What is the MEDM Overview Screen? Is this the overall QUAD Overview? If so, where is the new(ish) warning message? Are you talking about the Guardian message window (upper right corner) on the QUAD overview?
On the top-level screen for each quad, you should see a red rectangle appear around the top, UIM, or PUM coil output filters that says "rocker switch death", as in this screenshot from Jeff's alog.
11:38UTC Glitched down to 36Mpc. It doen't appear to be related to the typical ETMY saturation. Should have kept quiet about it!