nutsinee.kijbunchoo@LIGO.ORG - posted 11:13, Monday 28 September 2015 - last comment - 13:08, Monday 28 September 2015(22023)
Lockloss 18:11 UTC
PSL tripped.
PSL tripped.
re Int Issue 1127, WP 5521.
Have modified the WHAM5 Top Level Model to terminate the output of the ADC and ground the input of the Coil Driver Corner2 Voltage monitors.
This will eliminate the false Rogue Excitation WD Trips of which this platform has been suffering. BTL identified the cause finding the bad monitor output.
This is temporary until we swap out the driver and look for a cause in the monitor circuit.
On Tuesday 29 Sept, the model will be installed and the FE restarted after taking the platform down to safe or offline.
After restart, the SDF will have to be repointed to the OBSERVE.snap
Attached is an image of the modified model area: the two don't look like the others.
Modified model commited to svn rev 11742.
The glitch rate hasn't been doing very well in the past hour (since I relocked the ifo) so I dropped the intent bit and reduced the modulation index (17:37 UTC) by 1 dB. The H1:LSC-MOD_RF45_AM_RFSET value is now 22.2 instead of 23.2. I went back to observing with this configuration and see if it helps with the glitch rate. So far the DMT Omega has been looking much cleaner in the past 10 minutes.
Lockloss after 60 hours locking at NOMINAL LOW NOISE (beat the old record of 46 hours from three days ago). There's a 5.4M earthquake in Argentina at a depth of 198.5km happened right around the time. Terramon suggested this earthquake shouldn't drop us out of lock but LLO has also dropped lock at the same time. USGS reported no bigger earthquake.
Back to Observing 16:27 UTC.
TITLE: 09/28 [DAY Shift]: 15:00-23:00 UTC (08:00-16:00 PDT), all times posted in UTC
STATE Of H1: Observing at ~64Mpc. The ifo has been locked for 58 HOURS!
OUTGOING OPERATOR: TJ
QUICK SUMMARY: Nominal seismic activity in earthquake band. Rising in seismic activity in anthropogenic band. RF45 behaving okay. Wind below 5 mph. No broadband loud glitches in the past half an hour.
TITLE: 09/28 [Owl Shift]: 07:00-15:00 UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Observing, ~70 MPc
SHIFT SUMMARY: Really quiet first half, then the RF45 Noise would intermittently appear. I would have to adjust H1:LSC-MOD_RF45_AM_RFSET down to fix it. Definitely seems to be getting worse.
INCOMING OPERATOR: Nutsinee
LOG:
The RF Noise isn't quite as bad this time, for the past 10min or so it will breifly get noisy, relax for min, then get noisy againa and saturate ETMY.
LLO is down right now so I will take that opportunity to try and fine tune the RF45Set again.
Observing
H1:LSC-MOD_RF45_AM_RFSET == 21.2
The Coherence below 2Hz in H1:LSC-DARM_IN1_DQ is almost 0 with the RFSET this low, and AS90 is at about 400.
The RF Noise is back again, I will try to tweak H1:LSC-MOD_RF45_AM_RFSET again.
Back to Observing.
Tried to find a good balance of adjusting the RF Set value to bring the noise down, and keeping AS90 up. Ended up with:
Back to just above 70Mpc.
I followed alog21716 and brought H1:LSC-MOD_RF45_AM_RFSET down to 22.2 and checked that H1:ASC-ODC_AS_A_DC_LT_TH was set to -11000. This dropped the noise down to a level much closer to what it originally was (at least according to the dtt H1_45MHz_Demod_FOM.xml). I tried tweaking the RFSET slightly to see it I could improve it further but it is too hard to tell if it gets better or worse.
Range is back up to ~70Mpc. Screenshot attached of dtt session.
I will hopefully only take it out a few minutes to follow Stephans alog21716 to adjust the modulation index on H1:LSC-MOD_RF45_AM_RFSET .
If this does not fix it I will make some calls.
Just as I starting writing about how uneventful it has been, the RF Noise started. Patrick reported that the noise fixed itself last shift, so I will wait a little before taking any action.
TITLE: 09/28 [Owl Shift]: 07:00-15:00 UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Observing at ~70 MPc
OUTGOING OPERATOR: Patrick
QUICK SUMMARY: H1SUSETMX Timing error being reported on the CDS Overview, I'll reset it when LLO or us drops. Environment is calm, wind speeds are less than 10 mph, and we've been locked for almost 50 hours!
H1SUSETMX Timing error cleared by hitting Diag Reset at 07:53:00 UTC
TITLE: 09/27 [EVE Shift]: 23:00-07:00 UTC (16:00-00:00 PDT), all times posted in UTC STATE Of H1: Observing, ~72 MPc SHIFT SUMMARY: Rode through four earthquakes in the Northern Mid-Atlantic Ridge. 45 Hz RF noise came and went a couple of times. A timing error has appeared on H1SUSETMX. Had a number of SUS ETMY saturations. Had a couple of OMC DCPD saturations. Wind is below 5 mph. INCOMING OPERATOR: TJ SUS E_T_M_Y saturating (Sun Sep 28 1:16:40 UTC) SUS E_T_M_Y saturating (Sun Sep 28 1:16:42 UTC) OMC DCPD Saturation (Sun Sep 28 1:16:42 UTC) SUS E_T_M_Y saturating (Sun Sep 28 1:16:48 UTC) SUS E_T_M_Y saturating (Sun Sep 28 2:30:11 UTC) SUS E_T_M_Y saturating (Sun Sep 28 3:22:20 UTC) SUS E_T_M_Y saturating (Sun Sep 28 3:23:31 UTC) SUS E_T_M_Y saturating (Sun Sep 28 3:56:10 UTC) SUS E_T_M_Y saturating (Sun Sep 28 3:56:12 UTC) OMC DCPD Saturation (Sun Sep 28 3:58:30 UTC) SUS E_T_M_Y saturating (Sun Sep 28 3:58:33 UTC) SUS E_T_M_Y saturating (Sun Sep 28 5:35:33 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:9:35 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:9:41 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:9:45 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:9:47 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:9:49 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:9:51 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:9:54 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:9:56 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:10:2 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:10:26 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:10:28 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:10:30 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:10:32 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:10:48 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:10:53 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:10:55 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:10:57 UTC) SUS E_T_M_Y saturating (Sun Sep 28 6:10:59 UTC)
Throughout ER8 to present we have seen many examples of the 45 MHz EOM driver acting up. I did a study of this noise between 11th Sept - 23rd Sept using a BLRMS of H1:LSC-MOD_RF45_AM_CTRL_OUT_DQ and from this study, found a threshold on the BLRMS on which to create a data quality flag (details of exactly what I did can be found on this page). This is the flag the analyses use to completely remove this bad data from their searches (CAT1 veto)
From 12th Sept 00:00 UTC - 26th Sept 03:00 UTC the flag marked 23789 seconds (6.61 hours) of LHO data (2.93% of H1:DMT-ANALYSIS_READY). This corresponds to 17611 seconds (4.89 hours) of coincident data (3.14% of {H1,L1}:DMT_ANALYSIS_READY).
From 26th Sept 03:00 UTC - 28th Sept 00:00 UTC the flag marked 25992 seconds (7.22 hours) of LHO data (17.5% of H1:DMT-ANALYSIS_READY). This corresponds to 10970 seconds (3.05 hours) of coincident data (12.6 % of {H1,L1}:DMT_ANALYSIS_READY)
The EOM driver seems to have been acting up more in the last few days - the flag has marked more time as bad in the last few days as 2 weeks of data taking!
00:03 - 00:08 UTC Stepped out of control room 00:51 UTC Peter K. here eclipse hunting at LSB parking lot ~ 02:00 - 02:10 UTC Stepped out of control room 02:20 UTC heard buzzing from speaker in control room, turned it off ~02:20 UTC RF noise starting 02:39 - 02:43 UTC Stepped out of control room RF noise stopped SUS ETMX timing error on CDS overview
Peter K. done at 9:25 pm local time (04:25 UTC).
We tested a set of waveforms to see if they would saturation the ETMY ESD. Test the 15Hz waveform Previously we have tested a waveform that began at 30Hz, see aLog 21838. It was requested we try it from 15Hz instead of 30Hz. Here are the commands I ran for testing the 15Hz waveform: ezcawrite H1:CAL-INJ_TINJ_TYPE 1 awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 0.1 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 0.25 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 0.5 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 1.0 -d -d >> log3.txt The start times of the injections from the log file (log3.txt): SIStrOpen: Waveform starts at GPS=1127276638, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127276799, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127277026, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127277293, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127277517, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127277840, epoch=0, sample=0 I did not notice any ESD saturations. John's waveforms John V. has also provided 10 waveforms that I tested, see aLog 21964. Cautiously testing the first waveform Here are the command I entered as I slowly scaled up the first waveform's amplitude: awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.2 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.5 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.75 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.85 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 1.0 -d -d >> log3.txt The start times for these injections: SIStrOpen: Waveform starts at GPS=1127278105, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127278209, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127278748, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127279093, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127279520, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127279644, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127280123, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127280433, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127281232, epoch=0, sample=0 The injection that began at 1127280433 was near an ETMY alert from the robot voice. The ETMY alert happened a bit after the injection happened. SO I waited, I repeated the same injection, and there was no alert. I did one of these injections with a padding of 10 minutes so if the search groups are going to followup one of these injections in more detail this would be a good choice: awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 1.0 -d -d >> log3.txt # gave this guy a bit of time The start time of this injection is: SIStrOpen: Waveform starts at GPS=1127281956, epoch=0, sample=0 Testing the other nine waveforms Then I tested the remaining 9 waveforms by beginning with a scale factor of 0.3333 and incrementing to 1.0. I did these with little spacing between the injections so I do not think this set is worth the analysis groups following up. However, I think the injections with scale factor 1.0 should be looked at for ESD saturation (I believe Andy has a script for that): awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior1_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior1_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior1_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior2_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior2_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior2_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior3_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior3_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior3_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior4_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior4_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior4_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior5_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior5_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior5_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior6_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior6_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior6_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior7_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior7_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior7_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior8_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior8_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior8_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior9_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior9_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior9_1126259455.out 1.0 -d -d >> log3.txt The start times of these injections: SIStrOpen: Waveform starts at GPS=1127282013, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282066, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282120, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282173, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282226, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282278, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282326, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282377, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282422, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282476, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282517, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282577, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282631, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282681, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282727, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282770, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282819, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282964, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283007, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283048, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283089, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283160, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283204, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283245, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283292, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283338, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283440, epoch=0, sample=0 As I did these injections I had the SUS-ETMY_L3_ESDOUTF_LL_OUT channel open in dataviewer. I did not see any injection go above the 90000 counts. The highest was posterior0*.out which was near 80000 counts.
I am going to follow up the injections I passed to Chris. Assuming the "start time" is the time at the 1st sample of the injection file (and not the 1st non-zero sample) the end times of the increasing amplitude posterior0 injections should be around
The only ETMY overflows during the injection periods were: 1127278436.8 1127280154.9 (also overflowed ETMY L2 and OMC DCPD) 1127282847.5 None were related to the injections.
Sitting in the control room I heard something about a lock loss. I looked at the monitor that displays various cavity transmissions and noticed that the laser had tripped. I went into the diode room. The Beckhoff computer indicated that the laser switched off due to the power watchdog being kicked. The power watchdog performed as it should. However we should make it routine to reset the power watchdog every maintenance day. Something we have mentioned but perhaps were not as diligent about it as we should be. All the PSL related servos were up by the time I returned to the control room. Attached is a plot of the laser power since the last time (I believe) the watchdog was reset back on Sept. 1st.Locked again at NOMINAL_LOW_NOISE. RF45 MODE reduced to 22.2 dB after it was reset to 23.2 by Guardian after the lockloss. Currently not Observing due to Praxxair (a really big truck) on site.