edmond.merilh@LIGO.ORG - posted 01:43, Wednesday 26 August 2015 - last comment - 01:56, Wednesday 26 August 2015(20896)
Observation Intent Bit/ Observing Mode
Set at 8:43UTC. 60Mpc
Set at 8:43UTC. 60Mpc
[Sheila, Dan, Evan, Jenne, others]
All afternoon (basically, since we started relocking after maintenence day) we have seen a 1 Hz comb (offset by 0.5 Hz) in the DARM spectrum, as well as in many, many other places. This seems like the same thing as the final bullet point that was reported in aLog 20790 a few days ago, but with much taller peaks.
Much of the day was spent looking into this, by several people. The lines are very clear in all of the length error signals, as well as the ASC error signals.
One thing that we tried, just to exonerate it, was swap the PRC length actuation from the PRM to PR2, and then turn off the PRM M3 coil driver. As expected, PR2 affects the cavity length twice as much as the PRM, so we needed a gain of 0.5 in the M3 LOCK filter bank for PR2, rather than the 1 that PRM has. We were able to turn off the PRM M3 coil driver, but saw no change in the length signals' spectra. We put the actuation back on PRM, but when we tried to turn off the PR2 coil driver, PR2 tripped and we lost lock. So, the new PRM coil driver is certainly not to blame for this new noise. (PR2 was untripped, and its coil driver turned back on) Also, we haven't seen the glitches due to the old PRM coil driver at all today, so hopefully they're gone for good, and not just hidden.
At Richard's suggestion, we tried restarting the models that were restarted earlier today in case it is some kind of synchronization problem. We restarted h1prm, h1pr2, h1lsc, h1asc. This didn't change anything - the lines are still there loud and clear after relocking.
The 1 Hz comb is still present, and we are running low on ideas for what it could be, and what to do about it. DetChar friends, and anyone with ideas, please let us know if you have a suggestion of where to look.
--------------
I don't know if this has the same cause as the 1Hz comb, but there are also some unusually large lines at ~838 Hz and ~855 Hz. These lines are a few Hz wide, and are coherent with magnetometer signals. But, the magnetometers have seen these lines at the same (or larger) amplitude, in the past, without seeing the lines in DARM. Why would the coupling from magnetic fields have changed so drastically??
Tagging DetChar so they're aware of this entry.
The various noise features that appeared after maintenance today -- the 1Hz comb below 70Hz, some broadband noise above 80Hz, and the peaks around 850Hz -- are all coherent with the DRMI length error signals, and are much louder (compared to quiet references) in PRCL and MICH. Something is different in the corner station...hopefully all three things have the same source.
The 1-Hz comb with 0.5-Hz offset is prevalent in magnetometer channels. Looking at a typical magnetometer FSscan spectrogram is not for the faint of heart, but there are plenty of examples to be found here. One example from EY_MAG_EBAY_SUSRACK_Y is attached. It's not unusual.
Since the reports stated it shows up in PEM channels I disconnected the LO to the antenna Demod board that resides in the PEM rack and was coincidentally connected yesterday and the comb disappeared. WE need to verify the RF level needed for these radio signals. Pick some attenuators and re-installed the LO signal. I put terminators back on the outputs from the distribution chassis.
This could also have been due to a ground loop. A balun should be added to the lines.
I thought this might be the ITM HWS. It looks like the filter on the power supply (which is filter out the 1Hz comb) was turned off yesterday.
A small point about the PRM/PR2 experiment: when we switched to PR2, there was a huge 45 Hz resonance (visible in DARM and the vertex LSC signals) that was rung up. It was distinct from the usual 41 Hz HSTS roll mode resonance that we usually see when passing through the noise tuning step in full lock. It did not ring down on its own.
We made sure that the PRCL OLTF did not change before and after the swap, and anyway the loop appeared to have enough phase (more than 30°) at this frequency, so I don't think it was gain peaking. By increasing the loop gain, we were able to bring the resonance down a little bit. I then added 4 dB of resonant gain around 45 Hz, which somehow squashed the resonance completely. (I put it in the PR2 M3 lock L SFM).
It's not clear to me what was going on here.
IFO locking appears to have recovered well from maintenance but investigations are ongoing into new excess noise. UTC (Pacific) 23:33 (16:33) Dave to roof to look at antennas to update drawings 23:39 (16:39) Dave back 00:03 (17:03) Nutsinee to OSB high bay mezzanine to check for distilled water for TCS chillers 00:25 (17:25) Nutsinee back. She also cleaned the TCS chiller filters. 05:08 (22:08) Sheila and Jenne to flip switch on PRM M3 LL coil driver 05:15 (22:15) Eric T. running injections 05:48 (22:48) Eric T. done 06:06 (23:06) Sheila turned off PR2 coil driver, tripped PR2 watchdog, broke lock, turning it back on 06:14 (23:14) Sheila, Evan, Jenne restarting PRM, LSC, ASC and PR2 models
Patrick, Sheila, Jenne, Eric For the first part of the test, we injected our fiducial CBC waveform (same one used in ER7) and tried raising the LIMIT value on the hardware injection block in order to address saturation problems observed in ER7. During ER7, the LIMIT was 200. We raised it to 400. The first injection did not go through: 1124601535 1 1.000000 cbctest_1117582888_ intent bit off, injection canceled Patrick, Sheila, and Jenne tried to turn on the intent bit, but there was some sort of problem, which will be alog'ged separately. As a temporary work-around, we turned off the tinj intent-bit check and injected again: 1124602724 1 1.000000 cbctest_1117582888_ successful Patrick determined that the injection produced a maximum |amplitude| of 15 counts coming out of the injection block, which seemed to indicate that the original LIMIT value of 200 was sufficient. However, an alarm went off to indicate that there was saturation at ETMY. Thus, the saturation problem cannot be solved by tinkering with the INJ block in MEDM. Rather, the problem is occurring downstream on the ETM actuators. We request that Jeff K, Adam M, et al. look into options for avoiding saturation at the ETMs. Next we tried a blind injection using the new blind injection code. The blind injection code does not log injections in EPICS so they are not automatically picked up in the segment database. 1124603111 1 1.000000 cbctest_1117582888_ successful The blind injection was clearly visible. The ETM saturation warning went off again. The injection was logged correctly in the blind injection blindinj_H1.log: current time = 1124603049... Attempting: awgstream H1:CAL-INJ_BLIND_EXC 16384 /ligo/home/eric.thrane/O1/Hardw areInjection/Details/Inspiral/H1/cbctest_1117582888_H1.out 1 1124603111 Injection successful. All of these injections were carried out with scale factor = 1; (that's the 1.000000). The injection file, described in a comment below, is a 1.4 on 1.4 BNS, optimal orientation, at D=45 Mpc. It is the same waveform used in previous ER tests.
It looks like the injection actually does hit the 400 count limit (plot 1). It saturates right at the end when the injection chirps up to high frequency. There's some kind of ringing as well (plot 2). From the spectrogram (plot 3) and the zoom (plot 4) this looks like a feature at just above 300 Hz. I thought it might be a notch for the PCal line, but that's 331.9 Hz. So someone will have to check the inverse actuation filter and see what's happening at that frequency. It's possible to see the overflow from the first injection in the ETMY L3 MASTER channel (plot 5). It happens at -131072 counts, and the injection is trying to push it past -200000. The blind injection caused an overflow as well, but since this channel is only recorded at 2048 Hz, it looks like it falls short of overflow (plot 6). There's a faster readback whose name escapes me at the moment. Unless the blind injection is made a factor of about 10 smaller, or rolled off at high frequency, it will be trivial to detect it by looking at the drive to the ETM.
FYI, the injected waveform was fiducial waveform from ER7: https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=16125 It's a 1.4-1.4 BNS at 45 Mpc, optimal orientation.
There are a couple of things to watch out for when performing CBC hardware injections, based on iLIGO experience:
For the ER7 injection we used an SEOBNRv2 waveform that has a ringdown at the end, hoping that this turn off would not trigger an impulse. However, for BNS masses, the turn off and ringdown is pretty sharp. I've asked Chris check that there are no "whooper" effects with the SEOBNRv2 waveform, but we haven't had chance to do this yet. For a SpinTaylorT4 waveform (the other waveform CBC wants to inject), there will definitely be a step, so this needs to be checked and rolled off carefully.
One other comment on the test: what scaling in awgstream did you use? That waveform looks monstously loud (eyeball SNR > 20). That's much louder than would be useful for a blind injections, but good for helping us find whooper effects.
Duncan, the scale factor is 1.
Just for completeness, because I didn't see it posted, here's an Omega scan of the injections in h(t). The first is the non-blind injection, the second is the blind injection. I think the glitch ten seconds after the blind injection is unrelated. I thought it might be a filter turning off or being reset, but it's not on a GPS second (it's at 1124603210.28). It does cause an overflow of the ETMY ESD DAC.
I verified that the blind injection was correctly recorded in the raw frame file.
Sheila, Patrick, Jenne, Eric Thrane on the phone.
Eric wanted to do an injection, and since his code requires the intent bit to be set, Patrick attempted to set it. This however proved impossible. This is probably related to the work described in 20870. According to the CDS overview, there are no excitations on except those from calibration, which will need to be on while the intent bit is set. (screenshots attached of the CDS overview and some ODC screens).
Side note (sheila's opinion only):
I found the ODC screens confusing to navigate, and it is very unclear to me which bits go into calculating other bits to find out what the problem is. It seems to me like it is a bad idea for us to make ourselves dependent on ODC to set the intent bit, since people on site don't have control over it.
Evan deciphered the ODC screen that was apparently telling us that TCS was not observation ready. We put a 0 in the H1:ODC-MASTER_TCS_2_MASK, and this made the observation bit ready, so Ed has set the intent bit.
I think this has uncovered a bug/issue: There may be a problem with the connection between the TCS ODC and the ODC-MASTER models at LHO. I'm debugging it now and will reply with what needs to be fixed. On a separate note: I believe site leads (at least Fred) said that operators would be trained on how to read and use the ODC with the changes made in https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=20870 and Duncan M. and Stefan B. are the experts on site to teach people how to read this screen, but I'll walk through the way to read it for this example: Evan correctly read off the bit that was preventing obs-ready from being green from the ODC MASTER overview screen. To do this, you just trace across the row that you care about (OBS-READY) and find the red bit, then count across the top list of subsystems to find which subsystem that is coming from (I would appreciate it if anyone can show us how to turn those text boxes so we can align them with the columns better). The next step would be to look down one step further by clicking on the TCS button to see that the 30th bit of the TCS ODC is the only bit that can affect the OBS-READY bit. From the comment, this was done correctly as well, and you can read off the bitmask for the relevant row in the matrix on the right. You can also see this by looking at the large black/green matrix to find the green bits that affect each row, which tell you which bits of the subsystem ODC are mapped to which row in ODC MASTER. At this point, you will see that the whole TCS ODC at the top of that screen is red, however, which I believe is a bug somewhere in the FE code. If it were behaving normally and you wanted to learn what the bits meant in the vector at the top: The last step would be to click on the relevant subsystem from the ODC-SITE-OVERVIEW screen, and in this case you would learn that bit 30 is the Excitation bit, as expected.
I have found the issue: The h1odcmaster.mdl was locally edited and then committed to SVN on Aug. 18, as recorded in alog: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=20622 This occured after we had circulated the new h1odcmaster.mdl that was used for alog: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=20870. Duncan M. is going to ensure that everything is up to date with the SVN, then re-implement the changes he made on Aug. 18 in a copy of h1odcmaster.mdl and then generate the ECR to fix the bug. For the time being, I have set all the bitmasks of form H1:ODC-MASTER_TCS_XX_MASK to 0x0 so the bad connection cannot affect ODC-MASTER.
Ryan, this does not adequately explain the situation. Why does the bit mask in TCS affect READY? We were told that READY was just GRD-IFO_OK and NO EXC. How was it that TCS was somehow wired into that logic?
Darkhan, Craig, Sudarshan
We took DARM Open Loop Transfer function and Pcal sweep one after each other.
We had a hard time keeping the ETMY ESD from saturating at frequencies above 800 Hz and at the same time obtaining reasonable coherence. So this measurement is good only below 800 Hz and upto 7 Hz. The calibration lines are turned back on after the measurement was done. (At around 2015-08-26 5:05:00 UTC).
Attached is DTT screenshot of both the measurement 1. DARM OLGTF, 2. Pcal2DARM TF and conlog file that contains filter information during the time of measurement.
Analysis to follow.
One of the difficulties we had after maintence day today was caused by the DRMI ASC not being completely offloaded to the suspension top stage before the gaurdian attempted to offload the ASC to the alingment sliders. I've added a check to the guardian, and there is a function called asc_convergence_checker in ISC_library that can be used for this in other states as well. It would be a good idea to add these checks durring inital alingment and possibly before the power up.
Daniel, Sheila
We went to the racks near HAM6 and measured the response of each quadrant of the AS WFS, both for 45 and 36 channels. We sent the excitation in on test in and read it back using RF mon on each quadrant's demod. The notches for 91 MHz are generally mistuned, (they are between 87 and 90 MHz). An image is attched for one 36 MHz channel and one 45 MHz channel, the data for all the channels are in the attached zip file.
| magnitude at 45.5MHz (dB) | at 36.4MHz (dB) | at 91 MHz (dB) | |
| AS A 45 ch1 | -36.2 | -53.6 | |
| ch2 | -37 | -55.2 | |
| ch3 | -36.8 | -54.8 | |
| ch4 | -36.5 | -55.7 | |
| AS A 36 ch1 | -38.6 | -54.8 | |
| ch2 | -39.3 | -56.4 | |
| ch3 | -39.4 | -55.9 | |
| ch4 | -38.8 | -56.8 | |
| AS B 45 ch1 | -37.2 | -55.4 | |
| ch2 | -36.9 | -56 | |
| ch3 | -37.4 | -55 | |
| ch4 | -37.5 | -57.4 | |
| AS B 36 ch 1 | -39.3 | -56.4 | |
| ch2 | -38.9 | -57.3 | |
| ch3 | -40 | -56.4 | |
| ch4 | -39.4 | -58.1 |
The attached plot shows the uptime for all three EtherCAT computers. A restart is occurring, when the value jumps to zero from a value smaller than 49.7 days. If it reaches 49.7 days (2^32 in ms), it simply wraps around back to zero. Most of the restarts were due to software upgrades. The only 2 crashes in the past 64 days were in end X due to a now fixed configuration error. End Y has been up and running for more than 50 days without interruption.
Maintenance Day Summary:
DAQ Reconfiguration WP5456
Peter F, Hugh, Jeff K, Keita, Daniel, Dave:
GDS broadcaster was modified, details in ealier alog. h1calex modified to write IRIGB channel to science frame. Channels reduced/removed from science frame from models h1oaf and h1asc. Following model restarts the DAQ was restarted.
New Frame Writer Install WP5455
Carlos, Keith, Dave, Jim:
new frame writer is being built today. Installation is ongoing.
Guardian autoburt.req
Dave, Patrick:
the autoBurt.req file for guardian was regenerated to match new nodes. A new channel list was applied to conlog.
Start of HWS EPICS IOC
Ellie, Nutsinee, Dave:
Ellie and Nutsinee started the HWS EPICS IOC, the DAQ EDCU reconnected to these channels.
Timing System
Richard, Filiburto, Dave
The reference GPS receiver in the MSR was hooked up to its roof mounted antenna. Its 1PPS signal was connected to the third port of the MSR comparator.
I am updating the timing as-built drawings, which required touching timing cables to determine routing/naming and going onto the roof.
The H1BROADCAST0.ini file was modified this morning to:
fix a typo in one channel
remove obsolete channels which no longer exist in the H1 system.
Added 19 channels. Removed 16 channels. The following channels remain unmonitored: H1:GRD-SYS_DIAG_LOGLEVEL H1:GRD-SYS_DIAG_MODE H1:GRD-SYS_DIAG_NOMINAL_S H1:GRD-SYS_DIAG_REQUEST H1:GRD-SYS_DIAG_REQUEST_S H1:GRD-SYS_DIAG_STATE_S H1:GRD-SYS_DIAG_STATUS H1:GRD-SYS_DIAG_TARGET_S
The SYS_DIAG node was removed this morning, so those channels will never exist.
There is now a DIAG_MAIN node, so new channels: H1:GRD-DIAG_MAIN...
Dave updated the guardian autoBurt files and I rescanned the channel list. + H1:GRD-DIAG_MAIN_LOGLEVEL + H1:GRD-DIAG_MAIN_MODE + H1:GRD-DIAG_MAIN_NOMINAL_S + H1:GRD-DIAG_MAIN_REQUEST + H1:GRD-DIAG_MAIN_REQUEST_S + H1:GRD-DIAG_MAIN_STATE_S + H1:GRD-DIAG_MAIN_STATUS + H1:GRD-DIAG_MAIN_TARGET_S + H1:GRD-SR3_CAGE_SERVO_LOGLEVEL + H1:GRD-SR3_CAGE_SERVO_MODE + H1:GRD-SR3_CAGE_SERVO_NOMINAL_S + H1:GRD-SR3_CAGE_SERVO_REQUEST + H1:GRD-SR3_CAGE_SERVO_REQUEST_S + H1:GRD-SR3_CAGE_SERVO_STATE_S + H1:GRD-SR3_CAGE_SERVO_STATUS + H1:GRD-SR3_CAGE_SERVO_TARGET_S - H1:GRD-SYS_DIAG_LOGLEVEL - H1:GRD-SYS_DIAG_MODE - H1:GRD-SYS_DIAG_NOMINAL_S - H1:GRD-SYS_DIAG_REQUEST - H1:GRD-SYS_DIAG_REQUEST_S - H1:GRD-SYS_DIAG_STATE_S - H1:GRD-SYS_DIAG_STATUS - H1:GRD-SYS_DIAG_TARGET_S inserted 16 pv names deleted 8 pv names
Kiwamu, Patrick Neither of us can get the medm screen for all the ISC_LOCK guardian states to open.
That almost always means there's a syntax error with the ISC_LOCK guardian node:
jameson.rollins@operator1:/opt/rtcds/userapps/release 130$ guardutil print ISC_LOCK
ifo: H1
name: ISC_LOCK
Traceback (most recent call last):
File "/ligo/apps/linux-x86_64/guardian-1485/bin/guardutil", line 396, in <module>
args.func(args)
File "/ligo/apps/linux-x86_64/guardian-1485/bin/guardutil", line 34, in sprint
cli.print_system(system)
File "/ligo/apps/linux-x86_64/guardian-1485/lib/python2.7/site-packages/guardian/cli.py", line 90, in print_system
system.load()
File "/ligo/apps/linux-x86_64/guardian-1485/lib/python2.7/site-packages/guardian/system.py", line 403, in load
module = self._load_module()
File "/ligo/apps/linux-x86_64/guardian-1485/lib/python2.7/site-packages/guardian/system.py", line 291, in _load_module
self._module = self._import(self._modname)
File "/ligo/apps/linux-x86_64/guardian-1485/lib/python2.7/site-packages/guardian/system.py", line 165, in _import
module = _builtin__import__(name, globals, locals, fromlist, level)
File "/opt/rtcds/userapps/release/isc/h1/guardian/ISC_LOCK.py", line 1339
@nodes.checker().switch('SUS-PR2_M1_LOCK_P', 'INPUT', 'ON')
^
SyntaxError: invalid syntax
jameson.rollins@operator1:/opt/rtcds/userapps/release 1$
This is just a quick report about one of today's calibration activities.
I made a measurement to help understanding the OMC DCPD response by having intensity modulated light on OMC in a simple configuration. At the beginning I was going to use ISS PD arrays to calibration the intensity noise, but it turned out that they were not accurate enough to get 1 % accuracy due to mismatched dewhitening filters in the digital system. Seeking for alternatives, I finally ended up with ASC AS_C. Data and analysis to come later.
The frequency response of OMC DCPDs (A and B) are checked using ASC-AS_C as a intensity noise monitor in the single bounce configuration. Even though the response of ASC-AS_C is not well known, the result shows an almost flat response as expected with a deviation of 6% at most which is encouraging.
In order to improve the accuracy of the measurement, we should prepare a well-calibrated photo detector, probably placed at ISCT6, and make the same comparison measurement with the OMC DCPDs.
Method:
The interferomter was in a single-bounce configuration where the beam bounces off of ITMX and goes to the AS port without any recycling. Also ETMs are misaligned as well in order to avoid flash from the arm cavities. The PSL power was intentionally set to the maximum in order to get the maximum signal to noise ratio everywhere. The OMC is locked to the carrier 00 mode with a OMC-LSC_GAIN of 20. I forgot the UGF of the OMC length loop, but it was on the order of 100 Hz, I believe. The OMC alignment was done by the QPD loops with a overall gain of 0.1. For some reason, I needed to engage DC centering loops for the AS WFS A and B, otherwise it would saturate the OMC suspension. The photo current on each OMC DCPD was about 17 mA, which is a bit too high compared with the nominal full lock photo current of 10 mA.
I injected swept sine signal to the ISS inner loop from 7 kHz to 10 Hz. Then I measured a relative response between ASC-AS_C segment 3 and two OMC DCPDs. One trick in doing this is that I had to use an IOP signal to measure the response of ASC-AS_C because the ASC front end runs at only 2 kHz. Also, since I used the IOP signal, I was not able to grab summed QPD signals, and that's why I ended up with one of the QPD segments. Segment 3 happened to receive the highest power and therefore I chose this for the final analysis.
The dtt file and its ascii formated data are checked into the SVN at
/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Measurements/OMCDCPDs/2015-08-24_omc_dcpd_and_asc_pd.xml
/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Measurements/OMCDCPDs/2015-08-24_asc_to_omc_dcpds_tfs.txt
/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Measurements/OMCDCPDs/2015-08-24_asc_to_omc_dcpds_coh.txt
Also the analysis code can be found in SVN at
CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/OMCDCPDs/analyze_asc_to_omc_dcpds.m
Results:
The result is shown in the attached pdf. It is relative gain (or transfer function) between OMC DCPDs and ASC-AS_C. The red dots are for DCPD A and blue for DCPD B. The following frequency responses are taken into account:
Note that since AS_C is acquired at 64 kHz without any downsampling, I did not apply any digital low pass for it. Aside from these known parameters, I had to make some assumptions as follows.
Obviously, the key points to success this method is to reduce the number of the assumptions, but this time I simply attempted with ASC-AS_C, which I had to make multiple assumptions, to get some idea of how the measurement would go.
As shown in the upper panel, the absolute magnitude is almost flat with a trend in which the DCPD response higher at high frequencies. The error bars are placed by using the usual coherence technique (see alog 10506). The low frequency part below 20 Hz is clearly limited by low coherence, but it seems to consistently show lower response at low frequencies. If we take a peak to peak of the variation, it is going to be roughly 1.04 / 0.94 ~ 10 %. Looking at the phase, shown in the lower panel, the phase seems to diverge as it goes to high frequencies such as AS_C response. I don't think it can be explained by mismatch in the analog anti-aliasing filters because they are usually well matched. At this point, we can not really say if the high frequency deviation is from the real DCPD response of some artifact from unidentified uncompensated AS_C response.
J. Kissel, P. Thomas Given that we've had so many examples of rocker switch death this evening (see LHO aLOG 20839), I was able to identify that my new(ish) warning message about this failure mode on the MEDM overview screen (see LHO aLOG 20281) had a bug. The MEDM logic was watching L2 for all three stages, L2, L1, and M0. I've committed the bug-fix to the SVN such that LLO can receive its benefits.
What is the MEDM Overview Screen? Is this the overall QUAD Overview? If so, where is the new(ish) warning message? Are you talking about the Guardian message window (upper right corner) on the QUAD overview?
On the top-level screen for each quad, you should see a red rectangle appear around the top, UIM, or PUM coil output filters that says "rocker switch death", as in this screenshot from Jeff's alog.
The ODCMASTER in SDF will show 1 diff during this segment due to a change that had to be made that was blocking the Observation Ready bit to Green and keeping he INtent bit from being set. CALCS will also show diffs during this segment. Commissioners aren't certain what these mean.