The wind fence sensors in End X are currently offline. This was found by looking at the H1:SYS-ETHERCAT_X1DEVICE0_ERROR_FLAG. The [CXY]1DEVICE[0-6] channels are monitoring the Beckhoff hardware and typically this would indicate a serious error.
Dave and Jonathan,
The external alert system is not connecting to GraceDb. Looking back at the logs it has not been working since 28 Feb 2019. Queries to gracedb are failing with SSL errors.
In tracking this down we moved the external alerts to a new Debian 9 vm (ext-alert.cds.ligo-wa.caltech.edu) from h1fescript0. This was to put it under better configuration management and to get to a newer set of python and ssl libraries. This has not solved the problem. We are in communication with the GraceDb developers.
I heard back from the GraceDb developers.
When GraceDb moved to being hosted on AWS the load balancer that put in front of the system was changed. The new proxy could not understand impersonation proxy certificates. Simply removing a -p option in the call to ligo-proxy-init when generating the x509 certificate used to authenticate to GraceDb.
TITLE: 03/29 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 110Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: ~29hr lock. In and out of Observing from CAL/SQZ measurements and then a few random SDF channels that were not monitored.
LOG:
1616 Out of Observing to start Cal meas.
1733 Back to NLN, starting SQZ measurement
1739 Restarted h1edc
1740 Observing
1837 Out of Observing due to TCSY laser losing lock
1839 Observing
1840 Out of Observing, Large scatter shelves in DARM
1842 Observing
1943 Out of Observing, squeezer
1945 Observing
2034 Out of Observing for CAL measurement
2056 Jason, Rick to MY to grab 3IFO stuff
2056 Observing
2156 Observing
2200 Jason, Rick back
2204 Out and into Observing
I made some updates to the squeezer BLRMS, mostly fixing the bandpass filters to get rid of lines and making sure they are all similar gains, while doing so I accidentally knocked us out of observing twice.
The BLRMs are for bands around 16kHz (BLRMS1) 300Hz, (BLMS2), 764 Hz (BLRMS 3) and 4.68kHz (BLRMS5). They should all be roughly working now.
Summary: There may be two other combs at low frequency in H1 data: 5.048889 Hz with 3.365555 Hz offset and 11.394444 Hz with 0.0 Hz offset. These should be investigated/mitigated and potentially be added to the tracked combs. Details: After Pep reported that the 1 Hz comb was hopefully now mitigated (see LHO aLOG 48032), I wanted to check the current status of combs in the low-frequency, sub-100 Hz, region. I also wanted to regain familiarity with some of tools used to hunt for combs and to see if they would work on Fscan outputs, which I believe they should do. Fscans are useful because the data is normalized so the spectrum is essentially flat. Noise-weighted ASDs are also useful in this regard but also down-weight noisy frequency bands. Using the Fscan output from March 28 (see plot here and data here), I used some of Ansel's very nice python tools modified to accept Fscans as input. I saved a version here, which takes two arguments: $ bash fscan2bokeh.sh {output dir} {input Fscan txt file} so I analyzed this: $ bash fscan2bokeh.sh ./H1 /home/pulsar/public_html/fscan/H1/daily/H1Fscan_coherence/H1Fscan_coherence/fscans_2019_03_28_17_00_02_PDT_Thu/H1_GDS-CALIB_STRAIN/spec_0.00_100.00_H1_1237766420_1237852820.txt The normal output does not show any lines, so I added the currently tracked lines (with the argument inside the bash script --tagcombs "0.996795,0" "1,0" "1,0.5"). There is not much evidence for these combs in 1 day of Fscan data, but we need to monitor the situation in case things change during the O3 run. Ansel's FineTooth suite has a program to search for additional combs (on LHO cluster ~aneunzert/FineTooth/findComb.py). Invoking Ansel's virtual environment (~aneunzert/gwpybokeh2), I used the findComb.py program: $ python findComb.py --inputfile=/home/evan.goetz/public_html/CW/H1/dat_converted.hdf5 --fmin=5 --fmax=100 --threshold=1 This gave indication that there were two additional combs 5.048889 Hz with 3.365555 Hz offset and 11.394444 Hz with 0.000004 Hz offset (I've rounded this offset to zero in the plotting). Attached figure shows these combs plotted from Ansel's findComb algorithm. I'll check out L1 next and ping Ansel/Keith/Pep in case they have any additional thoughts.
The 11.39 Hz comb is an old friend from O2. It was also listed in the known lines/combs list made by Pep, Ansel, Bryn, and myself with inputs from Keith and Pat.
Richard, Dave:
The analog camera system was not functioning. Richard power cycled the comtrol and analog-video server units, I restarted the code on h0epics2. It is now working again.
Out at 2034
Back to Observing 2056 UTC
I had a look at the damping trends of ETMX violin mode 9 which was ringing up for 2 consecutive nights (as reported by Jeff B in his LHO alog 47969, ) and bumping the IFO into commissioning. This mode sits at 516.781 Hz, which is very near to ETMX mode 7 which sits at 516.678 Hz. I compared the ndscope (figure shown below) and found that ETMX mode 9 rings up very high when Guardian chooses not to damp it - at the same time ETMX mode 7 is being continuously damped. However, once damping is applied to both the modes, ETMX-mode 9 decreases significantly over time.
I would encourage on-shift operators to apply damping to ETMX mode 9 (GAIN - negative3) even if guardian chooses to switch it off (this usually happens at the beginning of the lock).
Locked for ~26hrs, but with CAL and SQZ measurements. We were kicked out of Observing a few times due to squeezer unlocking, scatter shelves, and TCS laser unlocking.
J. Kissel After resolving some problems with CAL-DELTAL_EXTERNAL yesterday (LHO aLOG 48012), we left off with - 2% / 4 deg systematic error in PCAL2DELTAL transfer function below 40 Hz. - achieved by scaling down the overall actuator function reproduction by 4% (i.e. gain of 0.96), and - some poorly understood confusion with the 2019-03-28 reproduction of the sensing function. - leaving the relative path delay at 7 clock cycles (though the model predicts 7.7 clock cycles) This morning, after some sleep and a bit more exploring, we are now better: - Systematic error within 1% / 1 deg in PCAL2DELTAL transfer function from 20 Hz to 500 Hz (and above, only 2% / 1 deg from 500-1000 Hz). - No time-dependent change in the PCAL2DELTAL transfer function over the course of 16 hours -- the interferometer's TDCFs must be *very* stable - A direct one-to-one map of the 2019-03-28 sensing function model, with no gain adjustment (i.e. gain of 1.0) - A 5% scale down (gain of 0.95) of the overall actuator function. - leaving the relative path delay at 7 clock cycles. Now I'm confident we no longer need to change the CAL-CS DELTAL EXTERNAL, and it's systematic error (+/- 1%) is within a factor of 2 of the stated current PCAL's uncertainty (0.5% at 68% confidence interval). Attached are some more broadband PCAL2DELTAL transfer functions, demonstrating - the exploration of whether the 2%-below-40Hz systematic error was because of a *relative* gain mismatch between actuator stages, or if it was an overall actuator scale error. The answer is as discussed above -- we only needed a minor adjustment of overall scale from 4% to 5% (i.e. gain of 0.96 to 0.95). - the "BEST YESTERDAY" and "BEST TODAY" -- measurements 16 hours apart, in the same lock stretch in the same CAL-CS configuration. This demonstrates that, within a single lock stretch at least, all time-dependent corrections must be very stable, much better than O2. What remains: - Understand the necessary 5% scale of the actuator (OFFLINE WORK) - Update the pyDARM loop model parameters to reflect this scale difference, and call it the reference model (OFFLINE WORK) - Fix the recreation of reference model values at calibration line frequencies used to create the front-end computed time-dependent correction factors. (OFFLINE WORK) - Create GDS correction filters from the reference model, and update the filters to create GDS-CALIB_STRAIN - Compare outputs of GDS and DELTAL EXTERNAL against PCAL and confirm systematic errors are understood at the +/- 1% level - Compare outputs of time-dependent correction factors computed by GDS and CAL-CS, and confirm they make sense at the +/- 1% level - Start the run!
while we were out of observing, I cleared h1edc CFC bit by restarted the model. It was once again being raised due to an apparent empty H1EPICS_GRD.ini file, which happened several times between 4am and 5am this morning (PDT). TJ verfied that the guardian code should not be emptying this file, he is investigating this further.
2019_03_29 10:41 h1edc
TJ, Rahul
This morning we have unmonitored all the violin mode damping gain changes from the SDF for all four QUADs. This will prevent the interferometer dropping out of OBSERVE mode, when guardian is make changes to the damping GAIN.
1740 UTC
Out of Observing from 1837 - 1839 UTC due to TCSY laser losing lock.
Then again, briefly out of observing from 1840 - 1842 UTC. Jenne noticed a large scatter shelf and turned off the ADS lines, this caused a SDF diff. Once settled we went back in.
Out of Observing from 1943 - 1945 UTC squeezer lost lock.
These injections are scheduled to begin at 05:00:00 PDT 29 March 2018, the first injection should occur at 05:00:10 PDT 29 March 2018, and the entire excitation should last only 91 sec. The expected GPS times of all these injections can be found in GraceDb via: https://gracedb.ligo.org/search/?query=H1+Test+HardwareInjection+gpstime%3A+1237896018+..+1237896120+HWINJREQ&query_type=E&results_format=S
Schedule file was updated and INJ_TRANS was reloaded to pick up this injection. Unfortunately, this caused H1 to drop from observing.
The injections were successful. I've attached a CSV file with the expected injection paramters. They can also be discovered in GraceDb via: https://gracedb.ligo.org/search/?query=Test+H1+HardwareInjection+HWINJOK+1237896018+..+1237896118&query_type=E&results_format=S
These injections are scheduled to begin at 02:00:00 PDT 29 March 2018, the first injection should occur at 02:00:10 PDT 29 March 2018, and the entire excitation should last only 91 sec. The expected GPS times of all these injections can be found in GraceDb via: https://gracedb.ligo.org/search/?query=H1+Test+HardwareInjection+gpstime%3A+1237885218+..+1237885320+HWINJREQ&query_type=E&results_format=S
Schedule file was updated and INJ_TRANS was reloaded to pick up this injection. Unfortunately, this caused H1 to drop from observing.
The injections were successful. I've attached a CSV file with the expected injection paramters. They can also be discovered in GraceDb via: https://gracedb.ligo.org/search/?query=Test+H1+HardwareInjection+HWINJOK+1237885218+..+1237885318&query_type=E&results_format=S