FYI: 1. Note that the good news is I see H1 low latency hoft making it to CIT, and it is being aggregated. Also it seems the low-latency analysis can determine the lock segments from the hoft frames. Thus, the problems listed below are something we can investigate Monday morning. An email has been sent to DASWG and the relevant persons about this. 2. However, the redundant copy of hoft written by the DMT seems to have stopped today at: Apr 30 11:21 PDT. Much earlier today, the DMT logs report Input/output error like this: $ tail logs/H1_rename_Frames-2017.04.25-09.43.log ‘/gds-h1/dmt/frames/hoft/H1/.temp/H-H1_DMT_C00-1177574268-4.gwf’ -> ‘/gds-h1/dmt/frames/hoft/H1//H-H1_DMT_C00-117757/H-H1_DMT_C00-1177574268-4.gwf’ mv: cannot move ‘/gds-h1/dmt/frames/hoft/H1/.temp/H-H1_DMT_C00-1177574268-4.gwf’ to ‘/gds-h1/dmt/frames/hoft/H1//H-H1_DMT_C00-117757/H-H1_DMT_C00-1177574268-4.gwf’: Input/output error 3. Also, thus summary pages show dropout of the H1 lock segments https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20170430/ and nagios reports errors like, https://dashboard.ligo.org/cgi-bin/nagios3/status.cgi?host=all&servicestatustypes=28 "Known segments in frames that aren't in database" Thus, low-latency data is making it to CIT, I think with the correct locked segments, but these are not making into the segment database. This is something we'll investigate Monday morning.
This is not related to previous problems with the calibration code not reporting lock after a data dropout.
This recent frame,
bash-4.4$ ls -l /scratch/frames/cache/hoft/H1/H-H1_DMT_C00-117764/H-H1_DMT_C00-1177648628-4.gwf
-rw-r--r-- 1 40829 40829 392952 Apr 30 21:37 /scratch/frames/cache/hoft/H1/H-H1_DMT_C00-117764/H-H1_DMT_C00-1177648628-4.gwf
bash-4.4$ tconvert -l 1177648628
Apr 30 2017 21:36:52 PDT
reports a H1:GDS-CALIB_STATE_VECTOR of
67010559 = 0b11111111100111111111111111
which says H1 is in analysis ready mode.