(manual run of the scripts, crontabs were not operational)
Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 40 seconds. TC B did not register fill. LLCV set back to 18.0% open.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 630 seconds. TC A did not register fill. LLCV set back to 42.0% open.
I've reconfigured the crontab for user vacuum on vacuum1, autofill should run automatically on Wednesday.
Increased CP4 to 43% open.
CP3 log file DOES NOT exist! CP4 log file DOES NOT exist!
The regular Monday 11am LN2 auto-overfill of CP3 and CP4 did not run this morning, presumably related to OS updates on Friday. Will run it manually for today.
Having moved the STS2-C (HAM5) from Roam2 (near NE BSC8{H2 ITMY} chamber leg/SEI Pier) to Roam3 (~2/3 distance to BSC4 {H2 BS}) on Friday during EQ downtime, some weekend wind has given a good comparison to examine. Reference Traces are from the quiet period (0805utc May 1) and the current traces are during the windy time beginning at 2300utc April 30.
See the first plot attached showing the wind velocity at the corner station. 180 degrees on the anemometer is pretty much right out of the +X to -X direction, i.e., no Y component and this may likely explain why the Y dof comparison of the ASD is nearly identical. The coherence shows stronger frequency dependence. On the X dof, the STS2-C at Roam3 is certainly noisier than STS2-B at its home location and a bit more so during the higher wind. However, comparing the same plots from aLog 35848, Roam3 is a much less noisier location than Roam2. Caveat: the wind may be a bit quieter (few mph) and the direction is nearly opposite. But, we might feel the largest deal is ~doubling the distance from the south wall. Still, the North wall which is getting the direct impact of the wind here is much farther (at least 3 times farther) than the South wall which was getting the direct hit from the aLog 35848 wind storm. On the Z dof comparisons, the low wind time equity extends to lower frequencies and during high wind that breaks down around 10 or 20mHz.
Conclusion, Roam3 is a better position than Roam2 location, at least for this wind direction but the STS2-B location remains best. Need a good SW wind (from +Y) to really fill in some holes.
We had two more lock ups over the weekend. Luckily they were on non-dolphined corner station machines (SUS-AUX) and I was able to work remotely with the operator on getting these machines reset via the front panel reset button.
Here is a summary of the lockups since Wednesday night:
h1susauxb123 | Mon 11:24 01 mar 2017 UTC (04:24 PDT) |
h1susauxh2 | Sun 20:20 30 apr 2017 UTC (13:30 PDT) |
h1susex | Fri 12:43 28 apr 2017UTC (05:43 PDT) |
h1seiex | Thu 13:10 27 apr 2017 UTC (06:10 PDT) |
Extended table, and added recent h1seib2 lockup:
computer | lock-up time (local) | computer uptime at lockup | timer reset date-time (local) |
h1seib2 | Mon 13:05 5/1 PDT | 215 days | Wed 22:07 4/26 PDT |
h1susauxb123 | Mon 04:24 5/1 PDT | 215 days | no data |
h1susauxh2 | Sun 13:30 4/30 PDT | 214 days | Wed 22:18 4/26 PDT |
h1susex | Fri 05:43 4/28 PDT | 209 days | Wed 23:34 4/26 PDT |
h1seiex | Thu 06:10 4/27 PDT | 209 days | Wed 21:53 4/26 PDT |
John Zweizig has fixed the problem reported here,
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=35924
which was caused by the DMT RAID box running out of inodes.
John has fixed the problem by deleting older data to free inodes.
TITLE: 05/01 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Environment
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
Wind: Calm
Primary useism: EQ
Secondary useism: EQ
QUICK SUMMARY: ops lazy script isn't doing transition (-t).
Shift Summary: Run A2L check script. Pitch is below reference, Yaw a bit elevated. Received GRB. Both sites in observing. Spoke with LLO. In one hour hold until 12:12 (05:12).
FE computer H1IOPSUSAUXB123 crashed. Spoke to Dave B. – Reset FE computer to recover. Back into observing.
Received an EQ warning and shortly lost lock. USGS showing a Mag6.5 in Alaska. Both Primary and Secondary microseism are off the top of the charts. Tripped the WD on most SUS, all, ISIs, several HEIPs, and all four IMs. Have the IFO in DOWN until things settle down a bit. Have been resetting tripped WD as necessary. A dozen after socks in Alaska have been recorded. Primary and Secondary microseism are still above 1.0um/s.
As the seismic was ringing down tried relocking. After relocking ALS Green, hit with another Mag6.3 earthquake near Skagway Alaska. Seismic Primary and Secondary are off the top of the charts. Put the IFO back into DOWM and started resetting the SUS and SEI watchdogs.
6.5 Magnitude EQ Near Skagway, Alaska at 12:31utc, (05:31PT).
The Mag6.5 EQ was followed by 13 aftershocks. After seismic was starting to settle down tried relocking. However, at 14:18 USGS reported another Mag6.3 earthquake near Skagway Alaska. Primary and Secondary microseism are climbing again. Put the IFO back into down until this one passes. Most US, ISI, HEPI WD are tripping.
Locked for 58.75 hours. All looking good at this time.
5.3 Magnitude EQ Triple Junction Region, Galapagos at 07:17utc, (00:17PT).
TITLE: 05/01 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 61Mpc
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Been locked for almost 55 hours. LLO joined us briefly but down again. Wind has pretty much died down. No issue to report.
FYI: 1. Note that the good news is I see H1 low latency hoft making it to CIT, and it is being aggregated. Also it seems the low-latency analysis can determine the lock segments from the hoft frames. Thus, the problems listed below are something we can investigate Monday morning. An email has been sent to DASWG and the relevant persons about this. 2. However, the redundant copy of hoft written by the DMT seems to have stopped today at: Apr 30 11:21 PDT. Much earlier today, the DMT logs report Input/output error like this: $ tail logs/H1_rename_Frames-2017.04.25-09.43.log ‘/gds-h1/dmt/frames/hoft/H1/.temp/H-H1_DMT_C00-1177574268-4.gwf’ -> ‘/gds-h1/dmt/frames/hoft/H1//H-H1_DMT_C00-117757/H-H1_DMT_C00-1177574268-4.gwf’ mv: cannot move ‘/gds-h1/dmt/frames/hoft/H1/.temp/H-H1_DMT_C00-1177574268-4.gwf’ to ‘/gds-h1/dmt/frames/hoft/H1//H-H1_DMT_C00-117757/H-H1_DMT_C00-1177574268-4.gwf’: Input/output error 3. Also, thus summary pages show dropout of the H1 lock segments https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20170430/ and nagios reports errors like, https://dashboard.ligo.org/cgi-bin/nagios3/status.cgi?host=all&servicestatustypes=28 "Known segments in frames that aren't in database" Thus, low-latency data is making it to CIT, I think with the correct locked segments, but these are not making into the segment database. This is something we'll investigate Monday morning.
This is not related to previous problems with the calibration code not reporting lock after a data dropout.
This recent frame,
bash-4.4$ ls -l /scratch/frames/cache/hoft/H1/H-H1_DMT_C00-117764/H-H1_DMT_C00-1177648628-4.gwf
-rw-r--r-- 1 40829 40829 392952 Apr 30 21:37 /scratch/frames/cache/hoft/H1/H-H1_DMT_C00-117764/H-H1_DMT_C00-1177648628-4.gwf
bash-4.4$ tconvert -l 1177648628
Apr 30 2017 21:36:52 PDT
reports a H1:GDS-CALIB_STATE_VECTOR of
67010559 = 0b11111111100111111111111111
which says H1 is in analysis ready mode.
Been locked and Observe for almost 52 hours. The wind is still~30 mph.
Amplitude of 32761 Hz PI (Mode23 in LHO damping scheme) correlates with 3 - 30 Hz seismic motion. There is also a corresponding comb that rises in the PI OMC channel at 2*(Nyquist - PI frequency), where Nyquist = 32768 Hz and PI freq ~ 32761 Hz.
Thanks to the summary pages, I'd noticed that the 28 - 32 kHz PI band somewhat correlated with SenseMon range. I narrowed this correlation down to 3 - 30 Hz seismic motion (which we now know correlates with range due to input beam scattering) and PI ETMY Mode 23 aka ~32761 Hz. Correlation seems to be only with this mode, ruling out electronics coupling since a mode sensed right next door in frequency world has no hardware difference - both are sensed with H1:OMC-PI_DCPD_64KHZ_AHF_DQ. First attachment is 32 hr stretch showing correlation between narrow OMC-PI band, seismic, and range.
Kiwamu then did a drive/damp test of this mode and found narrow lines appeared in DARM while driving/damping well below saturation. Looking at the OMC channel, he actually saw just the first peak of a 14.5 Hz comb seen in OMC-PI, as well as 64, 78.5, 142.5 Hz lines. Second attachment shows comb while he was driving. This ~15 Hz comb is visible in OMC-PI anytime Mode23 is rung up enough (haven't quantified 'enough' yet) and first peak is always 2x the frequency difference of 32768 Hz (OMC-PI's Nyquist) and Mode23 frequency; I've confirmed this to within .01 Hz. Third attachment shows presence of first peak in comb during a typical day's Mode23 non-driven ring up. I haven't seen the other peaks Kiwamu saw (64, etc.) any other time.
This is an actual mode ringing up (as opposed to just changing in sensing at OMC) as it's seen in the TransMon QPD's (and all feed forward is off for this mode). These have much lower SNR for PI so mode has to be unusually high to see. Fourth attachment shows a recent day with high 3 - 30 Hz seismic activity where amplitude correlations are visible everywhere. Fifth attachment shows spectrum during the peak just after 5 hours in previous attachment. Note that in TransMon channel, signal has been demoded with 32800 Hz, so two peaks are (32800 - 32761) = 39 Hz and (32800 + 32761 - 65536) = 25 Hz.
Evan G., Jeff K. Summary: The calibration measurement data that was collected yesterday has now been analyzed using our Markov Chain Monte-Carlo (MCMC) methods. We detail the results below. Nothing abnormal was found, and we find that the time varying factors can track changes in sensing gain and coupled cavity pole and actuation coefficients. Details: We analyzed the data collected during yesterday's calibration measurements, see LHO aLOG 35849. We have simplified the process for analyzing the data; rather than running two separate Matlab script to generate the MCMC results, we can now run just one script to get the final model results. For the sensing function, we run ${CALSVN}/trunk/Runs/O2/H1/Scripts/SensingFunctionTFs/runSensingAnalysis_H1_O2.m while for actuation, we run ${CALSVN}/trunk/Runs/O2/H1/Scripts/FullIFOActuatorTFs/analyzeActuationTFs.m. Note that for the actuation calibration script, we have not yet converted it to an IFO agnostic script. Once it is converted, this script will be renamed. Below are the table of values and their associated uncertainties for yesterday's measurements. Also, for comparison, are the modeled values from the reference measurement 3 Jan 2017: Reference Value MAP (95% C.I.) MAP (95% C.I.) 2017-01-03 2017-01-03 (MCMC) 2017-04-27 (MCMC) Optical Gain K_C [ct/m] 1.088e6 1.088e6 (0.0002e6) 1.124e6 (0.0002e6) Couple Cav. Pole Freq. f_c [Hz] 360.0 360.0 (7.6) 343.4 (2.6) Residual Sensing Delay tau_C [us] 0.67 0.67 (6.7) -1.8 (1.8) SRC Detuning Spring Freq. f_s [Hz] 6.91 6.91 (0.1) 7.4 (0.04) Inv. Spring Qual. Factor 1/Q_s [ ] 0.0046 0.046 (0.016) 0.009256 (0.0069) UIM/L1 Actuation Strength K_UIM [N/ct] 8.091e-8 8.091e-8 (0.2%) 8.0818e-8 (0.18%) PUM/L2 Actuation Strength K_PUM [N/ct] 6.768e-10 6.768e-10 (0.02%) 6.795e-10 (0.08%) UIM/L3 Actuation Strength K_TST [N/ct] 4.357e-12 4.357e-12 (0.02%) 4.537e-12 (0.07%) UIM/L1 residual time delay [usec] n/a n/a 29.1 (35.5) PUM/L2 residual time delay [usec] n/a n/a 7.7 (3.1) TST/L3 residual time delay [usec] n/a n/a 10.2 (1.8) These values are derived from MCMC fitting to the data values. The attached plots show these results for the sensing and multiple-stage actuation functions. We have added a new feature to the MCMC analysis, modeling the residual time delay in actuation. We expect to have zero usec of residual time delay, provided the model accurately captures all dynamics of the actuation. Deviations from zero can reveal un-modeled dynamics. For example, the PUM and TST residual time delays are inconsistent with zero usec, but we expect that this is due to imperfect modeling of the complicated violin resonances of the quad suspension. The time varying factors are doing a good job tracking the changes between the reference model and the currently measured parameters (see time varying factors summary page). For reference, the parameter files were used as follows: ${CALSVN}/trunk/Runs/O2/H1/params/2017-01-24/modelparams_H1_2017-01-24.conf (rev4401, last changed 4396) ${CALSVN}/trunk/Runs/O2/H1/params/2017-04-27/measurements_2017-04-27_sensing.conf (rev4596, last changed 4596) ${CALSVN}/trunk/Runs/O2/H1/params/2017-04-27/measurements_2017-04-27_ETMY_L1_actuator.conf (rev4596, last changed 4596) ${CALSVN}/trunk/Runs/O2/H1/params/2017-04-27/measurements_2017-04-27_ETMY_L2_actuator.conf (rev4596, last changed 4596) ${CALSVN}/trunk/Runs/O2/H1/params/2017-04-27/measurements_2017-04-27_ETMY_L3_actuator.conf (rev4596, last changed 4596)
A typo in the reference values for 1/Q_s above. It should be 0.046 (not 0.0046 as typed above).
For a bigger picture look at these 2017-04-27 actuation function measurements, I attach a plot of the model against measurement *before* dividing out the frequency dependence. This helps discern what the overall phase of the actuator stages should be relative to each other during O2.