Now that the multi-processing version of HWInjReport is operational and returning results (multi-processing code is an absolute bear!), this is the first of daily reports of analysis of output from HWInjReport. Currently, HWInjReport still has to be run manually and requires one to checkout the current schedule file from the SVN (currently at https://svn.ligo.caltech.edu/svn/dac/hwinj/Details/tinj/schedule; if this has changed, please let me know so I can modify the run script appropriately). Automatic execution of HWInjReport is soon to come. I've attached copies of the output report file and the schedule file used. NOTE: the report file is very wide due to the number and size of the columns in the network injections tables of the report. To examine the report, you will need to either zoom out or change the font size in your browser/text editor to 10pt or less (I'm looking into compressing the columns in the network injections tables in future updates).
The daily run performed with the following parameters:
GPS Start Time = 1127163296 # Beginning of time span, in GPS seconds, to search for injections
GPS End Time = 1127249696 # Ending of time span, in GPS seconds, to search for injections
Check Hanford IFO = True # Check for injections in the Hanford IFO frame files.
Check Livingston IFO = True # Check for injections in the Livingston IFO frame files.
IFO Coinc Time = 0.01 # Time window, in seconds, for coincidence between IFO injection events.
Check ODC_HOFT = True # Check ODC-MASTER_CHANNEL_OUT_DQ channel in HOFT frames.
Check ODC_RAW = True # Check ODC-MASTER_CHANNEL_OUT_DQ channel in RAW frames.
Check ODC_RDS = True # Check ODC-MASTER_CHANNEL_OUT_DQ channel in RDS frames.
Check GDS_HOFT = True # Check GDS-CALIB_STATE_VECTOR channel in HOFT frames.
Report Normal = True # Include normal (coherent, consistent, and scheduled) injections in report
Report Anomalous = True # Include anomalous (incoherent, inconsistent, or unscheduled) injections in report
NOTE: coherent -> coincident. Missed changing that in the code where it outputs the report.
The schedule file only has injections spanning 1125280499 - 1126450499. This is outside the range of times checked by HWInjReport, so there are no occurring or non-occurring scheduled injections reported.
No normal injections, as defined above for HWInjReport, were reported for the network injections. All injections found were reported as UNSCHEDULED, and all injections occurring were reported as CBC injections.
Two H1-L1 coincident injections were found: CBC 1127175853.757(H1), 1127175853.764(L1) and CBC 1127179822.757(H1), 1127179822.764(L1). Both injections were reported as UNSCHEDULED but, otherwise, had no other reported anomalies.
H1 had only 1 single-IFO injection, CBC 1127173426.757 (the report file shows 3, but only 1 is actually an H1-only injection. There is apparently a bug due to the multi-processing code that is not propagating the association of the other 2 injections with their corresponding L1 injections. It's basically a problem of how memory works for a multi-processing environment.)
L1 had 5 single-IFO injections:
- CBC 1127162126.764
- CBC 1127162532.764
- CBC 1127162961.764
- CBC 1127181622.764
- CBC 1127183422.764
The first three injections have the anomaly that they occur in the ODC hoft and GDS hoft frames but not in the ODC raw or ODC rds frames. The remaining 2, other than being UNSCHEDULED, had no other anomalies reported.
ADDENDUM: I was able to successfully fix the data propagation bug. I've attached a copy of the resulting "fixed" report that correctly shows the single-IFO injections for H1 and L1.
Peter Shawhan and I examined the anomalies more closely and found they are not anomalies. The missing injections in RAW and RDS for L1 do actually occur, but HWInjReport missed them. My current working hypothesis is that the code missed these injections because of how it has to separate the list of files to pass to FrBitmaskTransitions into chunks of no more that 4090 files. This is to prevent the number of arguments passed to FrBitmaskTransitions, one for each file, from exceeding the number of arguments supported by the OS (I actually ran into this issue at one point with the RAW frame files). HWInjReport merges the output from the chunks into a single continguous internal list, however, it currently is not accounting for the occurrence doubled transitions (two "off" or "on" transitions consecutively placed) during the process of merging the transitions internally. This may cause the code to become misaligned when finding the injections, based on the bit transitions, and so it completely misses it.
I am reasonably convinced this is the case because when I performed a run on a time-span around the anomalies, 1127162120 - 1127162970, the anomalies do not occur. But, this is because the list of files is much smaller and only needs one chunk, instead of several, to be passed to FrBitmaskTransitions.
This also brings another point which is that I need to include all the output files, the report generated, the schedule used, and the log file when I upload files with my alog summaries of HWInjReport, because the log file has a lot of information regarding the internal activity to HWInjReport. I built it that way because the code has some unexpectedly complex logic in places, which has made debugging a total bear, and it only got worse with the transition to multi-processing.
I just realized there is another bug in HWInjReport, though this one is somewhat benign. While HWInjReport is specified to cover only a certain time-span, it actually ends up covering a larger time-span due to the fact that FrBitmaskTransitions processes entire frame files and HWInjReport is processing the resultant transitions into injection events. This means that HWInjReport can receive from FrBitmaskTransitions a set of transitions that lie well outside the specified time-span and, consequently, generate injection events that lie outside the time-span. It does not have this issue with the scheduled injections, because it trims those to the specified time-span before doing any further processing. The fix, fortunately, is simple: just trim the transitions from FrBitmaskTransitions to within the specified time-span. However, the bug does have the effect of potentially creating injections just outside the beginning or ending of the specified time-span that are flagged as UNSCHEDULED, because the scheduled injections to which they may correspond were trimmed.