Interesting to note: After I noted all of the pertinent information for the California quake. USGS updated and inserted 3 more quakes in between the one reported here and the previous one in California. (2 that could have contributed to the detriment of the lock) See far right attached image.
OOPS! USGS didn't update in the way that I thought it did. I was zoomed on California, so those were the only showing in the margin. Sorry about the mis-post.
I'm attaching a few lockloss plots. The first two are the "stock" lockloss plots, one from about ten minutes before the earthquake and one at the lock loss. These plots are a number of useful angular signals from the IFO. Not much to say, but it looks like the arm angular controls are the ones that get the most upset (ASC C/DHARD/SOFT). They seem to go up by ~10x or more, while PRC/SRC etc go up by a factor of 4 or so. The point where the are ASC gets fuzzy at -10 seconds on the second plot looks to be of interest. Did we saturate some sensor here?
The next two plots are of different length drives for different suspensions, some ISI sensors and some ground sensors. Third plot is for 10 minutes before, the fourth is for the lockloss. I also included the MICH freeze signal (LSC_CPSFF), which looks like it could be interesting to monitor during an earthquake. Most of the time this seems to be a couple hundred nanometers peak to peak, but during the earthquake this grows to a couple thousand. We didn't get much warning for this earthquake (it was in California), so I don't know what good monitoring local sensors would have done this time around.
TITLE: 12/14 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 72.2908Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:
H1 had a good night (only out for ~30 for an alog), LDAS did not.
LOG:
Below are the spectra for the BSC & HAM ISI CPS's. The CPS's all look good for their high freq ranges. This closes FAMIS 6876.
Andy L. contacted us via TeamSpeak to let us know that we have a local issue with LDAS (& "cluster"?). On the status page (https://monitor.ligo.org/ldg),
Also noticed that fw0 is INVALID on the CDS Overview (but fw1 appears to still be operational), and so I believe I remember Dave saying as long as we have one of these writers still operational, we should be OK----when they BOTH go down, it's an emergency.
The main 200 amp breakers for the computer power are tripped, so there is no power other than lighting.
The h1fw0 frame writer daqd has been stopped until the disk system in LDAS can be brought up again.
Greg called me at 7:30amPST. Asked me to head out to the LDAS room (at the VPW), because he feared we had an HVAC failure because he could not ping any of his computers. He wanted me to see if the LDAS room was hot. I was not able to open the door with my card. I did hear a high pitch tone from within. I was also able to reach Greg on his cell (it must have been off when I tried calling earlier).
I notified Bubba and John about the situation and they will check out the room.
Greg is going to contact Dan Moraru to have him check out their system.
I gave Ed (incoming operator) an update on the status.
11:53utc: LOCKLOSS: So Far Reason For Drop is Unknown
12:24utc: Back to OBSERVING
There appears to be a problem with Condor on the Caltech cluster this morning, and that killed GWIstat. I'll watch for the problem to be fixed and restart GWIstat then.
TITLE: 12/14 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC STATE of H1: Observing at 73.359Mpc INCOMING OPERATOR: Corey SHIFT SUMMARY: Recovered well from earthquake except for trouble damping violin mode (See Jeff K.'s alog). Locked for last ~ 3.5 hours. I silenced the CP4 alarm on the control room alarm handler. The two channels should probably be removed. There were also a couple of dust alarms in the PSL enclosure. It is not clear if a sweep has been done since maintenance this morning. LOG: 01:24 UTC Changed sign of gain for PI mode 28. 01:47 UTC EY saturation. Happened to look at CDS overview and saw a number of scattered red squares under end Y that quickly went away. 02:39 UTC (18:39 PST) Lock loss. Earthquake. HAM6 ISI tripped. See attached screenshots for USGS, Terramon and BLRMS info. 04:04 UTC Stopping at DC_READOUT. Waiting for 5.3 magnitude earthquake in Solomon islands to arrive. 04:28 UTC Earthquake did not show up. Moving on. 04:35 UTC NLN. Jeff K. is damping ETMY violin mode at 1009.44 Hz. 04:38 UTC restarted video4. 05:08 UTC Changed sign of gain for PI mode 27 05:14 UTC Observing. Leaving violin mode damping on (Jeff K. accepted SDF differences). 05:17 UTC Changed phase for PI mode 27 05:35 UTC Changed phase for PI mode 27
J. Kissel, P. Thomas We tried using settings in the Violin Mode Table to damp what the table says to be 1009.44 ETMY(?) violin mode that has rung up by 2 orders of magnitude in recent lock stretches. Those settings are to use MODE9 bank, with FM1 (1008.45-1009.02), FM4 (100dB), gain = +100. Unfortunately, no success (as gauged by exponential reduction of control signal out of the bank) once past the initial turn of of the filter. Also tried - moving the phase of the filter +/- 60 deg with FM2 or FM3, but nothing budged. - the filter specifically targeted for 1009.44 Hz in that bank (FM10), but no phase around the unit circle caused an improvement. We would have explored further, but LLO was up and running, so we punted by accepting the settings in the OBSERVE SDF to have the Violin Mode Table settings (FM1 (1008.45-1009.02), FM4 (100dB), gain = +100 -- where it was entire OFF before -- in hopes that something might happen for the better in the long run. (Continuing to watch as I write this log indicates still no change, though). This has not been coded into guardian or anything, so the filter bank gain will be zeroed upon the next lock loss, and show up as a DIFF in the H1 SUS ETMY SDF once we get to OBSERVE. We should accept having it OFF. I attach an ASD of the mode. Pink is from Oct 18th (some good reference time, nothing special), Black is when I got started this evening, and Red is where has ended up after attacking it. No change.
Back to observing after lockloss during 5.8 magnitude earthquake in the Northern Mariana Islands. Jeff K. has left the filters on to damp the ETMY violin mode at 1009.44 Hz and accepted the differences for these in SDF.
Looking at a few tasks on the calibration to-do list, I ended up down the rabbit hole. Sadly, it's getting late and will have to return to this tomorrow. Investigations: 1) Working on fixing the PCAL --> CAL-CS_DELTAL_EXTERNAL calibration (see LHO alog 31994). Kiwamu's original script computed a transfer function accounting for time delays caused by the DAQ and the CAL-CS whitening filter. However, this is already taken into account in the clock cycles delay in the CAL-CS model when adding the sensing and actuation paths. Since CAL-CS corrects for the phase delay caused by AA/AI filtering--analog and digital--using the delayed actuation path, then what one really wants is to correct CAL-CS for the magnitude changes of the AA/AI filtering and any CAL-CS whitening on this channel while also correcting the PCAL_RX_PD channel for the effect of AA and that we also have not applied a filter with two poles at 1 Hz (for the free-mass response): DELTAL_EXT W * [Derr/C_foton + A*Dctrl*delay] ---------- = ---------------------------------- PCAL_RX_PD m * f^2 * AA_a * AA_d m DELTAL_EXT [C_real/C_foton] / [W * abs(AA_a * AA_d * uncompensatedOMCpoles)] --- = ------------------------------------------------------------------------------ m PCAL_RX_PD / [f^2 * AA_a * AA_d] Note that the correction factors applied in the numerator of the final equation are really only to be applied to the sensing side. However, the corrections are only at high frequency (low-frequency corrections are simply unity), so the numerator term is dominated by the sensing function. Thus the calibration to be applied is thus: [C_real/C_foton] * [f^2 * AA_a * AA_d] ---------------------------------------------- [W * abs(AA_a * AA_d * uncompensatedOMCpoles)] Unfortunately, this results in a large deviation in phase at higher frequencies (~60 degrees at 1 kHz, see attached figure) while recent measurements using an incorrect calibration filter do not suffer from this large phase deviation. The phase deviation is due to application of the analog and digital AA on the Pcal. Why? 2) I constructed an O2 version of the front-end calibration time delay diagrams based on the model (see attached), but when I look at the GDS correction factors to be applied to the CAL-CS channels, I do not find matching delays. In addition, the values obtained to not mesh with the recent work done at LLO (see LLO alog 29899). This will have to be investigated further tomorrow also.
The AA (analog and digital) had to be taken into for DELTAL_EXT also, including their phase. And also we need a cycle advacne for DELTAL_EXT. Attached plot show the same comparison with the mentioned modifications and we see that the old correction factor was good to use. I have also attached the script used to make the plot. Most of these correction factors are calculated in computeSensing.m of DARM model code, so I just get these factors from there.
Thanks to Shivaraj for pointing out the 7 clock cycles adjusted the actuation path to have consistent phase with the sensing path, but that a signal measured by back CAL-CS_DELTAL_EXTERNAL would have an apparent delay of 117.6 usec (see PDF attachment to original alog entry above) + 61 usec for the delay between OMC user model and CAL-CS user model. The apparent 117.6 usec on the sensing side is a combination of optical response, AA filtering, and uncompensated, super-Nyquist OMC DCPD poles. The correct math is as follows: DELTAL_EXT W * [Derr/C_foton + A*Dctrl*delay] ---------- = ---------------------------------- PCAL_RX_PD m * f^2 * AA_a * AA_d m DELTAL_EXT * [C_foton/C_real] / [W * AA_a * AA_d * OMCpoles * OMCtoCALCSdelay * lightTimeDelay * unknownSensingDelay] --- = ----------------------------------------------------------------------------------------------------------------------- m sign * PCAL_RX_PD / [f^2 * AA_a * AA_d] So the calibration transfer function is: [C_foton/C_real] * f^2 ------------------------------------------------------------------------------------------- [W * sign * uncompensatedOMCpoles * OMCtoCALCSdelay * lightTimeDelay * unknownSensingDelay] A comparison of the new transfer function is attached below. Note that the magnitude is basically unchanged, but there are changes in phase at the ~5 degree level or less from 10 Hz to 1 kHz. The new transfer function is stored at: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Scripts/ControlRoomCalib/pcal2darm_calib.txt. I also attach a comparison of DTT transfer functions using the old calibration versus the new calibration. The magnitude of the transfer function is basically unchanged. The phase is modestly affected. Unfortunately, since the transfer function wasn't perfect before and it remains imperfect (darn). I saved 2016-11-30_H1_PCAL2DARMTF_4to1200Hz_fasttemplate.xml with the new calibration transfer function.
The broad-band calibration has also been updated. The transfer function is also created by the same script, /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Scripts/ControlRoomCalib/H1_pcal2darm_correction.m The broad band transfer function is stored at: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Scripts/ControlRoomCalib/caldeltal_calib.txt I have updated the most recent DTT session with this changed calibration (see attached): /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER10/H1/Measurements/PCAL/2016-11-30_H1_PCAL2DARMTF_BB_5to1000Hz.xml Note that the DTT session calibrates the DELTAL_EXTERNAL and PCAL_RX_PD channels separately. The PCAL_RX_PD channel is calibrated by zpk([],[1;1],1,"n"); therefore, the DELTAL_EXTERNAL needs to have all the other calibration applied to it: [C_foton/C_real] ------------------------------------------------------------------------------------------- [W * sign * uncompensatedOMCpoles * OMCtoCALCSdelay * lightTimeDelay * unknownSensingDelay]
J. Kissel, E. Hall, J. Driggers Back in July 2016, Evan Goetz and Evan Hall redesigned the ETMY L2/L3 (PUM/UIM) actuation hierarchy filters to improve the rolloff of the PUM stage (see LHO aLOG 28746), as a result of the calibration group's critique (G1501372). As a refresher, I attach two select .pdfs from Evan's entry that shows the difference that the new crossover makes. These filters were installed and turned on the same day as the design aLOG (July 29 2016 21:57 UTC), and ran until late August (Aug 27 2016 10:57 UTC -- Friday Night / Saturday Morning at 3:57a local time), when they were somehow reverted back. Sadly, the configuration with the better filters seems to then come and go for the next few months, - Initial Install Jul 29 2016 21:57 UTC (Friday , Jul 29th, at 02:57pm PDT) - lost on Aug 27 2016 10:57 UTC (Saturday, Aug 27th at 03:57am PDT) - fixed again on Sep 10 2016 19:32 UTC (Saturday, Sep 10th, at 12:32pm PDT) - lost again on Sep 16 2016 03:09 UTC (Thursday, Sep 15th, at 08:09pm PDT) - fixed again on Sep 30 2016 15:12 UTC (Friday, Sep 30th, at 08:12am PDT) - lost again on Oct 01 2016 17:50 UTC (Saturday, Oct 1st, at 10:50am PDT) and it has stayed this way since. Notice that none of these correspond to a maintenance Tuesday; they correspond to afternoon/evening/late-night commissioning times. It looks like, currently, the old configuration (see attached for MEDM ScreenShot aide showing this bad configuration) is currently hard-coded into the run portion of the ISC_LOCK guardian's LOWNOISE_ESD_ETMY state, 3540 yl2da = ezca.get_LIGOFilter('SUS-ETMY_L2_DRIVEALIGN_L2L') 3541 yl3da = ezca.get_LIGOFilter('SUS-ETMY_L3_DRIVEALIGN_L2L') ... 3567 yl2da.only_on('INPUT', 'FM2', 'OUTPUT', 'DECIMATION') 3568 yl3da.only_on('INPUT', 'FM3', 'FM4', 'OUTPUT', 'DECIMATION') #KI ellptic FM7 added Aug-17-2015 ... 3587 self.yl2da = ezca.get_LIGOFilter('SUS-ETMY_L2_DRIVEALIGN_L2L') ... 3592 self.yl2da.switch('FM7', 'FM8', 'ON') The latter two lines are switched on after a counter is set. I attach a 60-day trend of the bitword that represents the status of the filter banks, Good Bad L2 L2L 37984 (FM6, FM7) 38082 (FM2, FM7 ,FM8) L3 L2L 37912 (FM4, FM5) 37900 (FM3, FM4) Unclear why L2, in its bad state, indicates a filter change -- as far as the current guardian code goes, the (bad) filter configuration does not change anywhere in the acquisition sequence. But, it helps identify when the changes were made. The trend is also annotated Naturally, this is too far back in time for a viable aLOG search (I tried keyword searching for a few relevant buzz words and looking for mention of such things in the aLOG for a few days surrounding Aug 27), so I asked Jenne, who didn't remember. As of yet, it's unclear if this was a conscious reverting and hard-coding because the IFO was non-functional in the new configuration, or if this was accidental, but it is clearly related to commissioning activity and guardian code. During commissioning phases, guardian code is not committed and quick / "small" changes are made without aLOG all the time, so I don't expect we'll ever be able to find out what happened unless someone remembers. But, I suggest we switch back to the good configuration right after the holiday break, if and when we have an engineering run to recoup after the holidays. Note -- the lack of upgrade has been correctly accounted for in any/all O2 calibration models, so there's no problems there. It's just sad.
WP6398 remove CP4 from cell phone alarms.
Dave:
CP4 pump level channel and its error channel were removed from the cell phone alarms system. We noticed that the ERROR_CHAN signal for CP4 is ERROR, and that for CP3 is OK. Team-VAC were not sure why this would be the case. Also, CP4 ERROR_CHAN went OK briefly late Monday morning before returning to ERROR.
WP6401 add LN2 dewar channels to cell phone alarms.
Dave:
All vacuum LN2 dewar percent-full channels, along with their error channels, were added to the cell phone alarms system. Current alarm levels are 20%-and-below, 99%-and-above.
Upgrade cdsfs1 raid controller card firmware
Carlos, Ryan.
cdsfs1's raid controller firmware was upgraded. We are testing on cdsfs1 before installing on cdsfs0.
h1tw0 file system turned off
Jim:
due to file system errors, daqd was turned off to stop alarms. This in preparation for a full file system rebuild on the SSD raid. Both NDS servers are serving h1dmt1's raw minute trend data.
LVEA dust monitor comtrol issue
Jeff B, Carlos, Fil, Jim:
Missing dust monitor signals were tracked to a powered down comtrol unit in the LVEA.
A coherent bbh injection has been scheduled for 1:05 UTC (17:05 PST). Here is the schedule file update:
1165712717 H1L1 INJECT_CBC_ACTIVE 1 1.0 Inspiral/{ifo}/bbhspin_hwinj_snr24_1163501502_{ifo}_filtered.txt
I have increased the heat in the LVEA, Zone 2A. This heater was off so I energized 1 stage of the heating elements.
Since the start of O2, we have had some pesky dropouts of OBSERVING due to SDF channel changes for some ALS channels from a Beckhoff computer at each of our End Stations (sysecatx1plc2 & sysecaty1plc2). Below, I wanted to list each of the occassions, the amount of time we were out of Science mode, channels involved, and their current state.
Should UNMONITOR (3) ALS-Y Channels
Above we can see all of the channels we noticed/caught/UNMONITORED. I would suggest we take the preemptive action of UNMONITORING (if not already done so) the following channels for sysecaty1plc2 next time we are out of OBSERVING:
Done.
Filed an FRS for this (#6932). I CLOSED it, but there is always the possibility that rogue channels which haven't been addressed can drop us out in the future.