A comparison of narrow lines in 133.5 hours of O2 DARM to those seen in 107 hours of ER10 DARM shows very similar structures and strengths. On the whole, the low-frequency lines are generally higher in the O2 data than in ER10, but the upconversion around the quad violin modes & harmonics is better in the O2 data, consistent with the significant mitigation of low-frequency doublets with a splitting of 0.0467 Hz, suspected to be due (somehow) to beating of the 2nd quad violin harmonics 1009.4414 and 1009.4881 Hz. For reference, zipped tar files of sub band spectra up to 2000 Hz are attached, where the orderings of the two overlain curves (ER10, O2's 1st 13 days) are reversed, to make more apparent in which data sets individual lines are stronger.
Lost lock twice after PI mode 26 rang up. Nothing I tried seemed to affect it. Called Terra while the mode was ringing up the second time. She is now logged in remotely to see if she can help diagnosis it. TCSY chiller flow is glitching.
When I called during the second lock, Patrick had walked through all the usual things to try and check but we couldn't get a response; I assume the mode was too rung up by then to turn easily and we were having to be careful with gain to not saturate. After the second lock loss, I logged in and was able to damp remotely at the very first sign of ring up. I moved Mode26 bandpass filter up by 0.5 Hz, but the original bandpass had still been encompassing the peak and the PLL had been engaging. Not sure why damping hadn't been seeming to have an effect. I was able to damp by flipping the sign of the gain and then decreasing the phase in ~10 degree steps every 10 min or so as the mode creeped back up.
C. Cahillane Reported are the 1-sigma (68%) uncertainties in the calibration parameters, and the plots of these parameter's impact on the sensing, actuation, and response function uncertainties. These are the uncertainties due entirely to MCMC fits of the calibration parameters given the November 12 reference measurements. This does not take into account unmodeled changes in the calibration, i.e. systematic errors. Those are characterized by the Gaussian Process method and will be reported later in the total uncertainty + systematic error budget. Plot 1 shows the LHO sensing function parameters and cornerplots. Parameter 68% and 95% confidence intervals are posted below:= = = = = = = = == Median ===== - 68% CI == + 68% CI == - 95% CI == + 95% CI Optical gain = +1.1527e+06 -0.088 % +0.088 % -0.174 % +0.174 % [cnts/m] Cavity pole = +3.4674e+02 -0.390 % +0.389 % -0.766 % +0.777 % [Hz] Time delay = +2.3216e-06 -49.15 % +49.54 % -96.2 % +97.4 % [usec] Spring frequency = +7.3837e+00 -1.101 % +1.078 % -2.150 % +2.123 % [Hz] Spring Inverse Q = +4.5749e-02 -8.57 % +8.62 % -16.9 % +17.0 %Plot 2 shows the sensing function Nov 12 measurement and its model fit given the MAP parameters. Plot 3 is the UIM actuation parameters cornerplots. Parameter confidence intervals:= = = = = = = = Median = = = - 68% CI = + 68% CI = - 95% CI = + 95% CI Gain UIM = +8.1807e-08 -0.220 % +0.218 % -0.434 % +0.431 % [N/ct] Time delay = +9.3546e-05 -44.226 % +45.798 % -81.585 % +90.190 % [s]Plot 4 is the UIM actuation Nov 12 measurement and model fit given MAP parameters. Plot 5 is the PUM actuation parameters cornerplots. Parameter confidence intervals:= = = = = = = = Median = = = - 68% CI = + 68% CI = - 95% CI = + 95% CI Gain PUM = +6.8377e-10 -0.034 % +0.034 % -0.067 % +0.068 % [N/ct] Time delay = +9.5548e-06 -8.855 % +8.849 % -17.395 % +17.405 % [s]Plot 6 is the PUM actuation Nov 12 measurement and model fit given MAP parameters. Plot 7 is the TST actuation parameters cornerplots. Parameter confidence intervals:= = = = = = = = Median = = = - 68% CI = + 68% CI = - 95% CI = + 95% CI Gain TST = +4.3880e-12 -0.044 % +0.044 % -0.087 % +0.087 % [N/ct] Time delay = +4.8062e-06 -12.882 % +12.788 % -25.492 % +25.184 % [s]Plot 8 is the TST actuation Nov 12 measurement and model fit given MAP parameters. Plot 9 is the response functionR(f) = 1/C + D*Aplotted with its components. This shows what dominates where. Plot 10 is the relative response function uncertainty due entirely to the MCMC results. It is < 0.5% and < 2 degrees everywhere between 5 and 5000 Hz. This is not the total uncertainty budget, just those due to the uncertainty in calibration parameters. This is interesting because it tells us a couple of things about calibration parameters. (1) They are extremely well-determined, and (2) marginalizing over just calibration parameters may not be enough to capture the total uncertainty budget. Stay tuned What remains to be done: (1) Take out time-dependence from the measurements. (2) Run Gaussian Process on all measurements with time-dependence removed. (3) Compile total uncertainty budget including systematic error. For a similar budget at LLO, see LLO aLOG 30440
Commented out in configuration file and committed to svn.
Josh, Andy, David
After stumbling on the ringing phones in the LVEA in O2 (entry 32503) we sent this request, to find other times in h(t) that look like phone ringing glitches, to GravitySpy citizen scientists. Though it's dirty laundry airing this is a way to engage a network of many interested folks to find issues.
The reply from user @EcceruElme pointed to two similar glitches in O1/ER10/O2.
One is in Livingston at 1132398529.14 (Nov 24th 11:08 UTC - link) and looks like an ER10 instance of engaging violin mode damping while in observation ready but before SDF checks were being enforced, so ok.
One is in Hanford at 1128269436.32 which is 2015-10-07 at 16:00 during a long nice science mode lock (summary page for that day). This sounds a little like a fax machine then some bangs and a human voice through a loudspeaker and some more bangs. In the hour around that time there are also some other loud bangs. We have no idea what it could be but it happened in otherwise unbroken excellent O1 data so perhaps folks on site know what it is and if it can be (or has been) turned off.
Attachments:
- Audio file of Input Optics Microphone with no filtering.
- Spectrogram of Input Optics Mic (it was also loud in BS Mic, others in LVEA but quiet in PSL room Mics) and Strain
- OmegaScan of Strain (from GravitySpy)
I listened to this in Quicktime with the bass at max and treble at min, and it sounds like a crane being actuated on/off multiple times, which I hear in the clicking. I think the screaching is metal to metal contact as the crane moved. I hear one voice.
Listening with trebble at max and bass at min gave the same results for me.
inputoptics-mic-2015-10-07.wav sounds to me like an accidental dial into the public address system in the OSB/LVEA with feedback coming from standing near one of the overhead speakers throughout the building.
Is it possible to dial into PA systems in the VEAs? Seems that should definitely be disabled during science runs. I can see that for safety it may need to be turned on for e.g., Tuesday maintenance, but it is such an easy way to corrupt the data with rich signals that it needs to be off bounds. May I also ask that the card reader records at the time of the Event (audio event that is) be inspected, in order to determine if anyone entered the LVEA around that time?
Card reader shows no activity in the LVEA between Oct /07/2015 00:00:00 and Oct/08/2015 00:00:00 Pacific (that should be Oct/07/2015 07:00:00 to Oct/08/2015 07:00:00 UTC).
The event time is around Oct/07/2015 16:09:43 UTC.
TITLE: 12/14 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 76.3748Mpc
OUTGOING OPERATOR: Ed / TJ
CURRENT ENVIRONMENT:
Wind: 20mph Gusts, 14mph 5min avg
Primary useism: 0.06 μm/s
Secondary useism: 0.20 μm/s
QUICK SUMMARY: Back to observing. Diag reset to clear timing error on h1iopasc0.
(Filling in for Ed till Patrick comes on shift)
The HAM5 CPS issue seems to be fixed, so we are on our way back up.
16:29 Fil , J. Batch ad John Worden out to LDAS.
16:42 Jim. Johm and Fil back from LDAS. Electrical panels are tripped. Electrician is on his way to investigate. Still not sure what order things went down.
16:46 Lockloss EQ
16:50 Switched ISI config for Large EQ in Northern California.
17:04 Switch ISI config back to Windy after quick decay of EQ
17:20 GRB alert. H1 not locked
17:34 H1 locked and observing. Monitoring f1 violin modes which are slightly rung up.
19:14 Fil reports that the LDAS power panel trips were due to the room, first, overheating. Queries into remote monitoring of LDAS room temps are being made.
19:17 filed FRS ticket for noisy ops station computer fan.
20:12 lockloss HAM5 and 6 ISI tripped
20:20 HAM5 CPS glitching. Jim out to power cycle chassis in CER.
20:34 Jim and Fil out to LVEA to physically "exercise" CPS connections.
21:30 Recieved Hanford Release advisory. Decided that I should get my non-snow ready vehicle home. TJ is there as well as Patrick who is the evening shift operator.
Assuming Ed meant to mark this entry for 12/14.
Assuming Ed meant to mark this entry for 12/14.
2:20 pm local
Took 30 sec to overfill CP4 at 70% open using Patrick's new automation code. See aLOG https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=32573
More testing Friday.
Chandra lifted the lead for TE252A. Kyle put the CP4 LLCV into PID and disabled. The script to fill CP4 was run: vacuum@vacuum1:/ligo/home/patrick.thomas/svn/scripts$ python cp4_fill.py 70 -60 3600 Starting CP4 fill. TC A error. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 30 seconds. LLCV set back to 35.0% open. Chandra put back the lead for TE252A. Chandra lifted the lead for TE202B. Kyle put the CP3 LLCV into PID and disabled. The script to fill CP3 was run: vacuum@vacuum1:/ligo/home/patrick.thomas/svn/scripts$ python cp3_fill.py 50 -30 3600 Starting CP3 fill. TC B error. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 110 seconds. LLCV set back to 19.0% open. Chandra put back the lead for TE202B. Both scripts are in svn under /trunk/scripts in the projects repository. Revision 4023.
Done under WP 6402 and 6403.
Updated scripts: Added check for exceptions on returning LLCV to initial position. Changed loop period from 10 seconds to 1 second. Revision 4024.
Moved both scripts to trunk/cds/h1/scripts in the cds_user_apps repository.
The h1fw0 frame writer was restarted about 12:30 PM PST. The cause of the shutdown was a power failure caused by a cooling system failure in the LDAS room where the file system for frames is located. The h1fw1 frame writer ran without interruption, so no data was lost.
J. Kissel, for the people actually doing the work: J. Warner, F. Clara We've lost lock because HAM5 ISI capacitive position sensors (CPS) have suddenly started going terribly glitchy and saturating. Jim and Fil are investigating, but this'll likely involve some electronics power cycling, if not swapping. Stay tuned for further details.
Some detail on finding and fixing the problem.
HAM5 suddenly tripped, presumably causing the lockloss, when I went to the overview several of the HAM5 CPSs were saturating nearly constantly, but the GS-13s didn't seem very perturbed, so I didn't believe the CPS were reading real motion. I couldn't reset the watchdog because of this, and when I bypassed the CPS to force a reset, the ISI rang up and tripped immediately.
While Jeff did the paperwork, Fil and I went first to the CER to try power cycling the AA and CPS power. Several rounds of this showed no improvement (several CPS were still glitching over 20k counts), so we then went out to the chamber. There we reseated the power and data cables, disconnected and reconnected the probes and cards in the Corner 1 & 2 satellite rack, but saw no improvement.
We then went to the corner 3 rack, and as soon as Fil grabbed the power cable to try reseating it (first thing he was going to try), the CPS settled down. Doing some "tap" tests on the ground showed believable motion on the live CPS signals in dataviewer (i.e. stomping on the ground showed "real" motion, mostly in Z). We were then able to reset the watchdog and the ISI was able to isolate. We went out again to make sure all of the cables were tight, but while we were there the CPS started glitching again. We then did another round of checking cables on the Corner 3 box and this time when Fil unscrewed the retaining screws on the data cable, the CPSs started behaving again. He carefully reseated the screws and we gently walked away. It's not clear which tweak to what cable fixed the problem, and we didn't find anything obviously wrong before touching stuff. I did find one of the data cables a the CER rack loose, but didn't fix anything by securing it.
This whole process was kind of hampered by not being able to open the sitemap on the mac book we were using. I could get dataviewer, but I could not get the ISI overview to see when we tripped the platform or reset the watchdogs. We were forced to use the workstation in the CER for MEDM.
We'll monitor this for the next couple of days. This does not seem to be exactly the same failure that Hugh found on HAM3 a few weeks ago (in alogs 31564, 32076, 32079). The attached watchdog plot from HAM5 doesn't seem to show the single glithches that Hugh saw on HAM3. Instead the noise on V2, H3 and V3 seem to just suddenly get worse. Kind of worrying.
Second CPS unit (slave) for HAM5 needs to be investigated further. Possible grounding or connector issue, as glitchy/saturation in channels comes and goes with moving of cables for this unit.
Interesting to note: After I noted all of the pertinent information for the California quake. USGS updated and inserted 3 more quakes in between the one reported here and the previous one in California. (2 that could have contributed to the detriment of the lock) See far right attached image.
OOPS! USGS didn't update in the way that I thought it did. I was zoomed on California, so those were the only showing in the margin. Sorry about the mis-post.
I'm attaching a few lockloss plots. The first two are the "stock" lockloss plots, one from about ten minutes before the earthquake and one at the lock loss. These plots are a number of useful angular signals from the IFO. Not much to say, but it looks like the arm angular controls are the ones that get the most upset (ASC C/DHARD/SOFT). They seem to go up by ~10x or more, while PRC/SRC etc go up by a factor of 4 or so. The point where the are ASC gets fuzzy at -10 seconds on the second plot looks to be of interest. Did we saturate some sensor here?
The next two plots are of different length drives for different suspensions, some ISI sensors and some ground sensors. Third plot is for 10 minutes before, the fourth is for the lockloss. I also included the MICH freeze signal (LSC_CPSFF), which looks like it could be interesting to monitor during an earthquake. Most of the time this seems to be a couple hundred nanometers peak to peak, but during the earthquake this grows to a couple thousand. We didn't get much warning for this earthquake (it was in California), so I don't know what good monitoring local sensors would have done this time around.
I have increased the heat in the LVEA in sections 1A and 5 by 1 ma each.
I also raised the heat at End X from 9.5 to 10ma.
End X is slow to respond so I have raised the heater to 11.5 ma and increased fan flow SF01 to 12 ma and ~11500 cfm.
I assume these temperature increases were well motivated, but I hope the affects of such increases were considered before hand, especially considering we're in the middle of a run. Small temperature variations can have noticable impact on the IFO, such as changing test mass suspension positions and/or acoustic mode frequencies.
It looks like "temperature" difference units are given in amps? What is the temperature as a functon heater current?