J. Kissel

I've continued effort towards converging on an uncertainty estimate for the 2020-01-03 calibration loop model, and thus the uncertainty in h(t) that has used this model (i.e. GDS-CALIB_STRAIN data products after 2020-01-13, but we'll likely use this model to cover all of O3B when we produce DCS-CALIB_STRAIN products.)

We last left off with LHO aLOG 54907, where the broadband transfer functions between PCAL and GDS-CALIB_STRAIN are consistently outside of the estimated 68% confidence interval (see this plot from that aLOG).

The comment to that aLOG (LHO aLOG 54911) left several paths forward, one of which included a better understanding of how to account for the known 0.43% systematic error in PCAL displacement estimates prior to 2019-11-12 (LHO aLOG 53188).
However, in the interest of "seeing what the answer is if you just ignore the data you don't understand," here I show what the results look like when one only includes data *after* 2019-11-12.

See the attached plots,
    [5] The "final answer" uncertainty budget compared against the two versions of measurements which should be a cross-check and equivalent. These are PCAL transfer functions to the following response channels:
         (a) Against GDS-CALIB_STRAIN: 2020-02-07_H1_PCAL2GDS_BB_meas1262990871_on_2020-01-13_vs_model_uncertainty_for_2020-01-03.pdf
         (b) Against "corrected***" CAL-DELTAL_EXTERNAL 

I again note that for [5](b), the CAL-DELTAL_EXTERNAL transfer function data has been corrected for the expected time-independent differences between CAL-DELTAL_EXTERNAL and GDS-CALIB_STRAIN (see T1900169), but the data has *not* been corrected for time-dependence (i.e. no TDCFs for the data have been applied to properly compare the reference model time on 2019-12-26 against the measurement times).

One can see in [5](a) and [5](b), that -- while the uncertainty estimate and median systematic error agree much better with the data below 40 Hz, are are closer to the data above 125 Hz -- there is still some discrepancy between the median line and the data. However, given that I've excluded all data prior to 2019-11-12 (i.e. sensing function data between 2019-10-31 and 2019-11-11, and all actuation data from 2019-11-12 all the way back to 2019-03-29), the uncertainty on the estimate has grown larger.

As such, with this larger uncertainty, 
    (i) Yes, you can say that the data is now within the 68% confidence interval, but 
    (ii) there is still evidence for discrepancy right around 90-110 Hz.

So -- I'd like to 
    (I) continue to assess how to apply the PCAL systematic error in a way that makes us confident in using data prior to 2019-11-12,
    (II) Get more GDS-CALIB_STRAIN broad band data from Aaron, to make sure this one measurement is not a fluke (though the CAL-DELTAL_EXTERNAL data that I do have implies that it's not),
    (III) build in the functionality in to the pyDARM universe to correct the CAL-DELTAL_EXTERNAL data for both GDS / CALCS corrections *and* user defined TDCFs.

I'll add the script locations and further plots that went in to creating this data in the comments.
    

In doing so, the estimated 68% confidence interval

The CS HEPI Pump Servo program froze ~1258utc today.  Recovery was easy and typical.  Details:

Following the OPS WIKI, https://cdswiki.ligo-wa.caltech.edu/wiki/SEI, wiki text is verbatum in blue and my comments are in red:

HEPI Pump Stations

You are here likely because an Alarm on the HEPI Differential Pressure. Couple things: is the Alarm Invalid or other? Is it Momentary, or Ongoing?

• Momentary: The pressure sensors can be noisy but Alarm parameters should make these alarms rare. Make a log and get back to better things.

• Ongoing: This indicates a trend in the pressures which will be caused by the servo likely being stalled. If not white boxed, there is likely a fluid level shutdown: follow the Other than Invalid Alarm guidance below.

Invalid Alarm This was our situation this morning

If the alarm is Invalid, the Pump Controller medm is likely white boxed; look for this under the Sitemap HEPI pulldowns. A restart of the servo is required. Don't panic though, the system may still actually be running and might be able to be restarted without losing the HEPI Platform's Isolation state. If the HEPI Platforms have not tripped, this is likely the situation. If power to the servo computer wasn't interrupted, the output from the servo to the VFD may be steady and keeping the Pressure near Okay.

Again, exactly the case this morning.  Went to the pump stations and the dial gauge on the pumps were reading ~80psi, normal.  The computer appeared to be running but the heartbeat was flatlined.  I did not restart the computer.

Restarting the White Boxed Servo: If your servo is not running (white boxed medm) restart it with the following sequence:

If the HEPI platforms are still isolating (not tripped,) trend the VOUT channel: H1:HPI-PUMP_L0/EX/EY_CONTROL_VOUT for use in restarting. The conditions would indicate the servo computer has latched its output to the VFD and the Pump Motors are running 'open loop' but with the servo dead, you don't know the drive level. The trend will tell you what it was when it stopped, and when. Now

The trends showed that the last output from the PID to the VFD was ~1085

continue with the following sequence to restart the servo:

1. ssh controls@h1hpipumpctrll0 For the Corner Station

• ssh controls@h1hpipumpctrlex For the EndX Station

• ssh controls@h1hpipumpctrley For the EndY Station See Secrets for passwords or ask Hugh

2. cd target-new For the Corner Station

• cd target-ex For the EndX Station; cd target-ey For the EndY Station

3. sudo ./run

Logging in was trivial but the medm took a good while to come completely alive but it did.  During this period, the IFO dropped out of observing as two fields changed: 1) the PID Setpoint level, I could hardcode that into the dB, and 2) the Servo Control Auto/Manual button.  This must come up in manual mode so the program does not write anything to the VFD.  This is what allows the HEPI system to keep running in this situation. If it were in Auto, the PID would push a zero before it had a chance to calculate and process a reasonable value.  Maybe a smart dB programmer could work this out but for now, this is what we (I) do.

This should restart the servo and the medm should come alive. Do not push any buttons, you don't want the servo to process any records. Note the pressures, these are valid and hopefully not too far from nominal. However, the servo will not have processed any records yet and the 0 counts output to the VFD is not valid. The servo starts in a safe state, most importantly Manual Control and no Burt Restore is requied. Now using the VOUT value from the trends before the drop out, put this value into EPICS:

caput H1:HPI-PUMP_L0/EY/EX_CONTROL_VOUT value. caput the 1085 gleaned from the trends above.

• This will put control back to the servo setting the VOUT to that value and things should carry on as before. Now, using the Manual Output [SetPoint, Output is incorrect, I'll fix the WIKI] Tweak, adjust the differential pressure to 70 psi and  [this actually doesn't apply, will remove from WIKI guiadance.] Using the SetPoint Tweak, set the SetPoint to 70 psi; and finally, change Servo Control to Auto. The UGF is quite low so this is pretty harmless. Return the Tweak to 0 and things should be good.  This is what was done and the results were good. The PID changed the output to the VFD a count or two but it is always doing that.  The restart was perfect, a perfect ping.

Okay, that's it.  Sure, look at the 6 hour trend plot attached.

16:20 The intent bit was kicked out of Observing by Hugh's restarting of the HEPI software on cpmputer h1hplpumpctrll0. This action caused 2 SDF diffs (see image)

16:24 H1 back to Observing.

TITLE: 02/10 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 120Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY: Only out of observing briefly at the start of the shift to unlock and relock the squeezer. Lost the EPICS connection to h1hpipumpctrll0. Hugh is investigating.
LOG:

08:11 UTC - 08:15 UTC Noticed range degrading so unlocked and relocked the squeezer per the instructions in alog 54967. Caused us to go out of observing. Seems to have fixed it. Back to 120 MPc.

12:55 UTC Alarm handler raised an invalid alarm for lost connection to h1hpipumpctrll0. It is pingable.

TITLE: 02/10 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 121Mpc
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.42 μm/s
QUICK SUMMARY:

No issues since unlocking and relocking the squeezer at the beginning of the shift.

FAMIS 13325

Trends attached. All four channels show some non-zero values in the past 3 months.

TITLE: 02/10 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 120Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 9mph Gusts, 8mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.39 μm/s 
QUICK SUMMARY: Noticed range degrading so unlocked and relocked the squeezer per the instructions in alog 54967. Seems to have fixed it. Back to 120 MPc. No other issues.

TITLE: 02/10 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:  locked all shift
LOG:

• NCALX: computer went down, 56 channels not in H1EDC, called Dave, he took a look, fixing tomorrow
• reduced violin damping gains and saw a reduction of the 1.5K peaks, all violin damping restored
• range drop at the end of the shift, handing off to Patrick

H1EDCU has 56 NCALX channels not connected. 

TITLE: 02/10 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 120Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 5mph Gusts, 4mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.40 μm/s
QUICK SUMMARY: locked in Observe, update as of 4:00UTC, locked in Observe

TITLE: 02/09 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 48Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY:

• Shift was quiet until the Lockloss at 21:16
  • there were e few issues re-locking including but not limited to:
    • ALSX locked at low power and not moving to INCREASE FLASHES for ~8 minutes (well after ALSY was locked and cruising
    • PRMI wouldn't lock and I broke that lock by trying CHECK MICH FRINGES
    • PRMI (2nd attempt took a PRM and BS tweaking to get H! to DRMI 1F
    • there were a couple more failed attemps getting back up
• 23:19 H1 back up and Observing
• 23:53 Squeezer unlocked and kicked intent bit
• 23:55 H1 back to Observing

Handing Locked/Observing H1 off to Cheryl

LOG:

Cause unknown - 27hrs 36min

TITLE: 02/09 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
OUTGOING OPERATOR: Camilla
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 4mph Gusts, 1mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.50 μm/s
QUICK SUMMARY:

TITLE: 02/09 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY: Very quiet shift.

TITLE: 02/09 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 121Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 18mph Gusts, 15mph 5min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.46 μm/s
QUICK SUMMARY: Locked 14hours 30m. Some wind and high-ish microseism still.

TITLE: 02/08 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ed
SHIFT SUMMARY: Locked until 10 minutes before Ed takes over! Microseism and wind and some ground motion maybe were too much for H1. Handing off with green arms in INCREASE_FLASHES.
LOG:

• 13:05 BBH event!
• 15:38 SEI_CONF to EQ from 6.1Mag in Sandwich Islands
• 15:51 Lockloss Not sure why, it didn't seem like anything was getting rung up from the EQ