Mike asked me to turn it off to maintain the configuration. So I went off observing for a bit to switch off the boost filter. I accepted the change in SDF and back to observing. All this took less than a minute.
TITLE: 09/26 [DAY Shift]: 15:00-23:00 UTC (08:00-16:00 PDT), all times posted in UTC
STATE Of H1: Observing. The range fluctuates between ~70-40Mpc most likely due to RF45 glitches.
OUTGOING OPERATOR: Ed M.
QUICK SUMMARY: RF 45MHz has been glitching. Wind speed seems to be increasing (now ~30 mph). Seismic activity in earthquake band increasing coresponding to the wind speed. Last earthquake report from Terramon was about an hour ago.
TITLE: Sep 26 OWL Shift 23:00-07:00UTC (16:00-00:00 PDT), all times posted in UTC
STATE Of H1: Observing
LOCK DURATION: Entire Shift
SUPPORT: Mike Landry
INCOMING OPERATOR: Nutsinee
Activity log:
13:59 After tweaking, 45Mhz RFAM held stable for ~15 minutes. Still glitching but not as frequently.
14:05 looks like a quake is incoming. Engaging Sheila’s DHARD_Y boost.
14:25 Wind on the X arm is picking up to around 16 to 20mph.
14:28 accepted DHARD_Y boost change in SDF and going to go back to observing.
14:31 Intent bit set to Undisturbed
End-Shift Summary: Last half of my shift has been dominated by a continuing loss of range due to constant low frequency glitching in DARM. Mike visited me and we realized that it was the 45Mhz RFAM issue. We toggled the gain a bit as per Stefan’s aLog and then reverted it back. This seems to have lessened the occurrences although it will still happen sporadically. IFO was locked for the entire shift. ETMY glitched twice around 08:00UTC and didn't glitch again for about 4 hours. Then they came at the typical frequency. Mexican EQ showed on the graph so while we were out of Observing I switched on Sheila’s DHARD_Y boost and accepted the change to be able to return to Undisturbed. Neither Terramon nor the USGS site showed the quake before our graph did. nice. Handing off to Nustinee.
Now that the multi-processing version of HWInjReport is operational and returning results (multi-processing code is an absolute bear!), this is the first of daily reports of analysis of output from HWInjReport. Currently, HWInjReport still has to be run manually and requires one to checkout the current schedule file from the SVN (currently at https://svn.ligo.caltech.edu/svn/dac/hwinj/Details/tinj/schedule; if this has changed, please let me know so I can modify the run script appropriately). Automatic execution of HWInjReport is soon to come. I've attached copies of the output report file and the schedule file used. NOTE: the report file is very wide due to the number and size of the columns in the network injections tables of the report. To examine the report, you will need to either zoom out or change the font size in your browser/text editor to 10pt or less (I'm looking into compressing the columns in the network injections tables in future updates).
The daily run performed with the following parameters:
GPS Start Time = 1127163296 # Beginning of time span, in GPS seconds, to search for injections
GPS End Time = 1127249696 # Ending of time span, in GPS seconds, to search for injections
Check Hanford IFO = True # Check for injections in the Hanford IFO frame files.
Check Livingston IFO = True # Check for injections in the Livingston IFO frame files.
IFO Coinc Time = 0.01 # Time window, in seconds, for coincidence between IFO injection events.
Check ODC_HOFT = True # Check ODC-MASTER_CHANNEL_OUT_DQ channel in HOFT frames.
Check ODC_RAW = True # Check ODC-MASTER_CHANNEL_OUT_DQ channel in RAW frames.
Check ODC_RDS = True # Check ODC-MASTER_CHANNEL_OUT_DQ channel in RDS frames.
Check GDS_HOFT = True # Check GDS-CALIB_STATE_VECTOR channel in HOFT frames.
Report Normal = True # Include normal (coherent, consistent, and scheduled) injections in report
Report Anomalous = True # Include anomalous (incoherent, inconsistent, or unscheduled) injections in report
NOTE: coherent -> coincident. Missed changing that in the code where it outputs the report.
The schedule file only has injections spanning 1125280499 - 1126450499. This is outside the range of times checked by HWInjReport, so there are no occurring or non-occurring scheduled injections reported.
No normal injections, as defined above for HWInjReport, were reported for the network injections. All injections found were reported as UNSCHEDULED, and all injections occurring were reported as CBC injections.
Two H1-L1 coincident injections were found: CBC 1127175853.757(H1), 1127175853.764(L1) and CBC 1127179822.757(H1), 1127179822.764(L1). Both injections were reported as UNSCHEDULED but, otherwise, had no other reported anomalies.
H1 had only 1 single-IFO injection, CBC 1127173426.757 (the report file shows 3, but only 1 is actually an H1-only injection. There is apparently a bug due to the multi-processing code that is not propagating the association of the other 2 injections with their corresponding L1 injections. It's basically a problem of how memory works for a multi-processing environment.)
L1 had 5 single-IFO injections:
The first three injections have the anomaly that they occur in the ODC hoft and GDS hoft frames but not in the ODC raw or ODC rds frames. The remaining 2, other than being UNSCHEDULED, had no other anomalies reported.
ADDENDUM: I was able to successfully fix the data propagation bug. I've attached a copy of the resulting "fixed" report that correctly shows the single-IFO injections for H1 and L1.
Peter Shawhan and I examined the anomalies more closely and found they are not anomalies. The missing injections in RAW and RDS for L1 do actually occur, but HWInjReport missed them. My current working hypothesis is that the code missed these injections because of how it has to separate the list of files to pass to FrBitmaskTransitions into chunks of no more that 4090 files. This is to prevent the number of arguments passed to FrBitmaskTransitions, one for each file, from exceeding the number of arguments supported by the OS (I actually ran into this issue at one point with the RAW frame files). HWInjReport merges the output from the chunks into a single continguous internal list, however, it currently is not accounting for the occurrence doubled transitions (two "off" or "on" transitions consecutively placed) during the process of merging the transitions internally. This may cause the code to become misaligned when finding the injections, based on the bit transitions, and so it completely misses it.
I am reasonably convinced this is the case because when I performed a run on a time-span around the anomalies, 1127162120 - 1127162970, the anomalies do not occur. But, this is because the list of files is much smaller and only needs one chunk, instead of several, to be passed to FrBitmaskTransitions.
This also brings another point which is that I need to include all the output files, the report generated, the schedule used, and the log file when I upload files with my alog summaries of HWInjReport, because the log file has a lot of information regarding the internal activity to HWInjReport. I built it that way because the code has some unexpectedly complex logic in places, which has made debugging a total bear, and it only got worse with the transition to multi-processing.
I just realized there is another bug in HWInjReport, though this one is somewhat benign. While HWInjReport is specified to cover only a certain time-span, it actually ends up covering a larger time-span due to the fact that FrBitmaskTransitions processes entire frame files and HWInjReport is processing the resultant transitions into injection events. This means that HWInjReport can receive from FrBitmaskTransitions a set of transitions that lie well outside the specified time-span and, consequently, generate injection events that lie outside the time-span. It does not have this issue with the scheduled injections, because it trims those to the specified time-span before doing any further processing. The fix, fortunately, is simple: just trim the transitions from FrBitmaskTransitions to within the specified time-span. However, the bug does have the effect of potentially creating injections just outside the beginning or ending of the specified time-span that are flagged as UNSCHEDULED, because the scheduled injections to which they may correspond were trimmed.
14:31UTC Intent bit set to Undisturbed
Ed, Mike L
13:50UTC addressing RF45 AM issue that has been ongoing for about 4.5 hours. Mike L and I decided to try a reduction in the LSC-MOD_RF_45_AM gain by 1dB to as per Stefan's aLog https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=21716
Mid-Shift Summary: IFO Locked @ 64Mpc (lower than normal due to funny business visible in DARM). No obtrusive Seismic or wind activity. 2 ETMY glitches. Some increasing activity in the lower DARM frequencies are causing drops in range. It just keeps bouncing up and down (see screenshots) Is this a combination of bounce and roll modes ringing up? This was the same activity that I noted in my transition summary. Is it from anthropogenic noise?
So this was due to the 45Mhz RFAM issue that H1 has been experiencing. With Mike L's help, I got an education on what it is and how to try and tame it in the days prior to it being rectified once and for all.
TITLE: Sep 26 OWL Shift 07:00-15:00UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Observing
OUTGOING OPERATOR: Patrick
QUICK SUMMARY:. IFO Locked at 68Mpc/22.9W. No injections are running. All lights are off in LVEA, MID, END and PSL. Wind is <10mph. Microseism seems to be riding steady at ~.1um/s for the last few hours following an earthquake. EQ seismic took a little hop from ~ .2 um/s to .1um/s due to a small earthquake in the Portugal area. It appears that at 07:09UTC there was some raised level of noise activity in DARM from 25Hz and below. This only lasted about 1 minute.
TITLE: 09/25 [EVE Shift]: 23:00-07:00 UTC (16:00-00:00 PDT), all times posted in UTC STATE Of H1: Observing, ~67 MPc SHIFT SUMMARY: Recovered from two lock losses. The cause of the first is unclear. The second was likely from a 6.2 magnitude earthquake in Chile. After recovering from the earthquake I put it into commissioning mode to allow Chris to do hardware injections while LLO was down. He is done and we are back to observing. INCOMING OPERATOR: Ed
Terramon and GW Stat on nuc1 are down due to some kind of authentication issue.
We tested a set of waveforms to see if they would saturation the ETMY ESD. Test the 15Hz waveform Previously we have tested a waveform that began at 30Hz, see aLog 21838. It was requested we try it from 15Hz instead of 30Hz. Here are the commands I ran for testing the 15Hz waveform: ezcawrite H1:CAL-INJ_TINJ_TYPE 1 awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 0.1 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 0.25 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 0.5 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/coherenttest1from15hz_1126257408.out 1.0 -d -d >> log3.txt The start times of the injections from the log file (log3.txt): SIStrOpen: Waveform starts at GPS=1127276638, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127276799, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127277026, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127277293, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127277517, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127277840, epoch=0, sample=0 I did not notice any ESD saturations. John's waveforms John V. has also provided 10 waveforms that I tested, see aLog 21964. Cautiously testing the first waveform Here are the command I entered as I slowly scaled up the first waveform's amplitude: awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.2 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.5 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.75 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 0.85 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 1.0 -d -d >> log3.txt The start times for these injections: SIStrOpen: Waveform starts at GPS=1127278105, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127278209, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127278748, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127279093, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127279520, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127279644, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127280123, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127280433, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127281232, epoch=0, sample=0 The injection that began at 1127280433 was near an ETMY alert from the robot voice. The ETMY alert happened a bit after the injection happened. SO I waited, I repeated the same injection, and there was no alert. I did one of these injections with a padding of 10 minutes so if the search groups are going to followup one of these injections in more detail this would be a good choice: awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior0_1126259455.out 1.0 -d -d >> log3.txt # gave this guy a bit of time The start time of this injection is: SIStrOpen: Waveform starts at GPS=1127281956, epoch=0, sample=0 Testing the other nine waveforms Then I tested the remaining 9 waveforms by beginning with a scale factor of 0.3333 and incrementing to 1.0. I did these with little spacing between the injections so I do not think this set is worth the analysis groups following up. However, I think the injections with scale factor 1.0 should be looked at for ESD saturation (I believe Andy has a script for that): awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior1_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior1_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior1_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior2_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior2_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior2_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior3_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior3_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior3_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior4_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior4_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior4_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior5_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior5_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior5_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior6_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior6_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior6_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior7_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior7_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior7_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior8_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior8_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior8_1126259455.out 1.0 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior9_1126259455.out 0.3333 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior9_1126259455.out 0.6666 -d -d >> log3.txt awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 ../H1/posterior9_1126259455.out 1.0 -d -d >> log3.txt The start times of these injections: SIStrOpen: Waveform starts at GPS=1127282013, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282066, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282120, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282173, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282226, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282278, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282326, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282377, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282422, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282476, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282517, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282577, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282631, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282681, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282727, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282770, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282819, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127282964, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283007, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283048, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283089, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283160, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283204, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283245, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283292, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283338, epoch=0, sample=0 SIStrOpen: Waveform starts at GPS=1127283440, epoch=0, sample=0 As I did these injections I had the SUS-ETMY_L3_ESDOUTF_LL_OUT channel open in dataviewer. I did not see any injection go above the 90000 counts. The highest was posterior0*.out which was near 80000 counts.
I am going to follow up the injections I passed to Chris. Assuming the "start time" is the time at the 1st sample of the injection file (and not the 1st non-zero sample) the end times of the increasing amplitude posterior0 injections should be around
The only ETMY overflows during the injection periods were: 1127278436.8 1127280154.9 (also overflowed ETMY L2 and OMC DCPD) 1127282847.5 None were related to the injections.
John V. has also provided 10 waveforms that I used for testing. The waveform single-column ASCII files that contain the h(t) time series can be found in the hardware injection SVN here: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/Inspiral/${IFO}/posterior*.out
And the corresponding XML parameter files can be found here: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/Inspiral/posterior*.xml
I will post time series plots of the waveforms as a comment later.
06:20 UTC Chris is done with hardware injections, back to observing mode
I'm done with the hardware injection tests. More details in upcoming aLog entry.
04:20 UTC Taking out of observing to allow Chris to test hardware injections
L1 is down at the moment. I am beginning some single-IFO hardware injection tests at H1 now.
03:28 UTC locked on DRMI briefly, loss lock, tripped BS ISI stage 2 GS13 04:09 UTC Back to low noise 04:16 UTC Back to observing mode
03:15 UTC lockloss, 6.2 Mag Earthquake, 30km SW of Ovalle, Chile, 02:51:18 UTC