The error does not go away when I request to go back to WINDY or reload the code. I am unable to go back to observing while it persists. I have left a message on Jim's cell phone and am looking at the code.

02:56 UTC Fixed, back to observing.

I found the relevant code in /opt/rtcds/userapps/release/isi/h1/guardian/sei_config/manager.py. I ran an svn diff on it and found uncommitted changes that looked like they could be causing the problem. I copied manager.py to manager.py.backup and ran svn revert on manager.py. I then reloaded the code from the guardian screen. The error went away and I was able to go back to observing.

[Jenne, TJ, Camilla, Patrick]

We've made some more modifications to the violin damping guardian, this time successfully making it so that a person can change the damping gain values while we are still in the nominal Damping_on_DC state, so that we can stay in Observe while damping violing modes. 

Note from the weekend that these 1kHz modes on ETMY that are rung up don't seem to have been made worse by the new logic from last week, but this change will enable operators to work on trying to damp them without taking us out of observing. 

The logic from last week was modifying the reference amplitude, and then checking against this updated value to see if the mode was ringing up.  If a mode was ringing up according to the most recent reference value, the damping gain would be set to zero.  But, since the reference value kept getting updated, there was a risk that modes would never get turned off when they needed to be.  So, now the guardian will update the reference values twice, but then will stop updating the reference value and so will leave the damping gain at zero.  This allows some modes to continue getting damping, in the case where they are a bit higher after lock than when we initially acquire, but for which their damping settings do work well.  But, if they have damping settings that do not work well they should have their gains set to zero.

The new logic that was added today was to allow the guardian to determine if a human is trying to change the damping gain.

• If the current damping gain is different from the last value that the guardian thinks it wrote, then it assumes a human is working on the damping, and the guardian will not do anything to that violin mode.  It will not check to see if it is ringing up, it will not turn the gain to zero.  So, if a person is working on damping a mode, then they must be watching that mode.  In practice, this is already what our operators are doing anyway. 
• If at any time the gain value returns to the value that the guardian last set, then the guardian will take back control of that mode.  There is a notification (that blinks annoyingly - I'll try to follow TJ's suggestion on how to make it not blink) that is also written to the log file of what that last guardian value was.  So, if you put the gain back to that value, the guardian then goes back to monitoring the mode and adjusting the gain.  By doing this, you can give control back to guardian while staying in Observe.
• Since all of the reference values are reset in the main part of the DAMPING_ON_DC guardian state, another way to give control back to guardian would be to toggle over to DAMPING_ON_SIMPLE, and then back to DAMPING_ON_DC.  However, this will take the interferometer out of Observing, so is not preferred.

So far, things seem to be working; TJ, Camilla, and Patrick are working on damping the EY modes that are high right now, and we're in Observing!

Vlad:

Measured an extra swept sine of the UIM to DARM dynamics, and Pcal-Y to DARM, to aid in filling in gaps in the transfer function of the UIM dynamics between 200 and 1000 Hz.

The data has been committed to the CalSVN here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/

2019-12-04_H1SUSETMX_L1_iEXC2DARM_HFDynamicsTest_200-1000Hz_A_PCALYRX_B_DARMIN1_SweepSine_coh.txt
2019-12-04_H1SUSETMX_L1_iEXC2DARM_HFDynamicsTest_200-1000Hz_A_PCALYRX_B_DARMIN1_SweepSine_tf.txt
2019-12-04_H1SUSETMX_L1_iEXC2DARM_HFDynamicsTest_200-1000Hz_SweptSine.xml
2019-12-04_H1SUSETMX_L1_PCAL2DARM_HFDynamicsTest_200-1000Hz_A_PCALYRX_B_DARMIN1_SweepSine_coh.txt
2019-12-04_H1SUSETMX_L1_PCAL2DARM_HFDynamicsTest_200-1000Hz_A_PCALYRX_B_DARMIN1_SweepSine_tf.txt
2019-12-04_H1SUSETMX_L1_PCAL2DARM_HFDynamicsTest_200-1000Hz_SweptSine.xml

Due to misconfiguration of the sweep, I have slightly driven the violin modes despite trying to deliberately not sweep through them.

The intention is to complete making the UIM dynamics transfer function as and independent filter that can be applied in both foton and pyDARM in the near future.

We made some A2A and A2L to DARM transfer function measurements, few weeks ago,  using broadband excitations and noticed some phase mismatch between the measurement and the model (LHO alog #53627). In order to make sure the phase mismatch was not due to some non-linear coupling due to the high coherence excitations, we made additional measurements using sinusoidal exciations with few different amplitudes. Initial observation shows that the phase mismatch still exists but will post more detail information (with plots) soon.

NCAL Measurements:

During the calibration measurements, I use the NCAL to take a sensing sweep. Using the PCALX frequency sweep points as a template and calulated the spin frequency required to spin the NCAL such that the 2F signal is as close to the PCALX frequency as possible.

First, I took an OLG measuement using frequency sweep points equal to that of the expected NCAL 2F and 3F signals:

After the OLG sweep was completed, from the control room, the NCAL was spun at the given NCAL counts given in the table below. At least 120 seconds of intergration time was allowed in order to ensure sufficient SNR.

Once the NCAL sweep was completed, a PCALX and PCAL Y to DARM dtt template with the following drive counts and frequencies was run with the following PCAL counts drive and frequencies:

The NCAL electronics has been laid to rest:

Times given below are approximatly the time the task was done.

1259537718  (4 Dec 2019 23:35 UTC ) -> Unplugged AC power, ethernet and DC power from NCAL Beckhoff Motor Controller
1259537838 (4 Dec 2019 23:37 UTC ) -> Unplugged BNC cable from EE mid bay chassis
1259537958 (4 Dec 2019 23:39 UTC ) -> Unplugged ethernet from NCAL Beckhoff PC

Summary

I have added the NCAL measurement to the CALSVN at revision number 8883. These files have been extracted and have been extracted in a way that should enable pyDARM to read in the files and produce a sensing function result for NCAL. Some modifications will be required to fold in the DARM OLG, PCALX, PCAY and NCAL sweeps.

Files added:

SVNDIR = /ligo/svncommon/CalSVN/aligocalibration/trunk/Projects/NewtonianCalibrator/measurements

^/20191204-SensingMeasurements
^/20191204-SensingMeasurements/2019-12-04_H1_DARM_OLGTF_NCAL_5-30Hz_A_DARMIN2_B_DARMEXC_COH.txt
^/20191204-SensingMeasurements/2019-12-04_H1_DARM_OLGTF_NCAL_5-30Hz_A_DARMIN2_B_DARMEXC_TF.txt
^/20191204-SensingMeasurements/2019-12-04_H1_DARM_OLGTF_NCAL_5-30Hz_A_DARMIN2_B_DARMIN1_COH.txt
^/20191204-SensingMeasurements/2019-12-04_H1_DARM_OLGTF_NCAL_5-30Hz_A_DARMIN2_B_DARMIN1_TF.txt
^/20191204-SensingMeasurements/2019-12-04_H1_DARM_OLG_NCAL_5-30Hz.xml
^/20191204-SensingMeasurements/2019-12-04_H1_PCALX2DARMTF_NCAL_5-30Hz_A_PCALXRX_B_DARMIN1_COH.txt
^/20191204-SensingMeasurements/2019-12-04_H1_PCALX2DARMTF_NCAL_5-30Hz_A_PCALXRX_B_DARMIN1_TF.txt
^/20191204-SensingMeasurements/2019-12-04_H1_PCALX2DARMTF_NCAL_5-30Hz_A_PCALXRX_B_DELTALEXT_COH.txt
^/20191204-SensingMeasurements/2019-12-04_H1_PCALX2DARMTF_NCAL_5-30Hz_A_PCALXRX_B_DELTALEXT_TF.txt
^/20191204-SensingMeasurements/2019-12-04_H1_PCALX2DARM_NCAL_5-30Hz.xml
^/20191204-SensingMeasurements/2019-12-04_H1_PCALY2DARMTF_NCAL_5-30Hz_A_PCALYRX_B_DARMIN1_COH.txt
^/20191204-SensingMeasurements/2019-12-04_H1_PCALY2DARMTF_NCAL_5-30Hz_A_PCALYRX_B_DARMIN1_TF.txt
^/20191204-SensingMeasurements/2019-12-04_H1_PCALY2DARMTF_NCAL_5-30Hz_A_PCALYRX_B_DELTALEXT_COH.txt
^/20191204-SensingMeasurements/2019-12-04_H1_PCALY2DARMTF_NCAL_5-30Hz_A_PCALYRX_B_DELTALEXT_TF.txt
^/20191204-SensingMeasurements/2019-12-04_H1_PCALY2DARM_NCAL_5-30Hz.xml
^/20191204-SensingMeasurements/NCAL_OLG_points.txt
^/20191204-SensingMeasurements/PCAL_NCAL_Sweep.txt

The systematic error in h(f) during the time of this measurements is discussed in LHO aLOG 57887.

added 500ml to the Diode chiller

I'd agree the SDF diffs are from INCREASE_FLASHES. The state just saves the TEST offset values and then puts them back in when it's done, so I'm surprised that the values it put back in seem rounded from the originals. I'll look into it.

What was happening that made you think that the bootstrapping was the issue?

Is this supposed to run automatically at LOCKING_ARMS_GREEN? I had to manually request it.

After a waiting period it will run automatically. I think its 5min? Perhaps too long, but I was keeping consistent with the intervention sheet.

Opened FRS13893

Outage from 05:10:42 UTC to 05:16:38 UTC (21:10 - 21:16 PST).

Evidence is now very strong that this was a freeze up of the Cisco network switch in the CER. This switch serves the PSL Diode Room (Beckhoff computer), the PSL LVEA enclosure Axis cameras and all the digital video cameras. All three systems exhibited a network error between 21:10 and 21:16. The digital video centroids are not directly trended by the DAQ, but some are copied to the end station and the received data shows the freeze (see plot).

Keita, Patrick, Dave:

What to do if this happens again and does not come back after 6 minutes.

The big question is if this were to happen again, with the PSL network down, H1 locked out of OBSERVE and the network not quickly coming back. Keita has agreed that we can make a guardian execption to remove the PSL Diode IOC from the OBSERVE veto and clear any associated SDFs. Patrick is looking at the code to see how that can be achieved. This will only be used it the network is down for at least 30 mins.

In the case of another network outage lasting more than 10 minutes I would like the operator to:

1) try to ping the switch (ping sw-lvea-aux) to see if it is running.

2) go into the CER and take a photograph of the switch, seeing if its error LEDs are lit. The switch is the Cisco with the WAP ethernet port. Only do this if it can be done safely (e.g. using LLO ops for budy).

I think we would need to add "LASER_PWR" to the EXCLUDE_NODES list in /opt/rtcds/userapps/release/sys/h1/guardian/IFO_NODE_LIST.py and reload the IFO node. The IFO node can be accessed from the 'GRD IFO' button at the very top left of the GUARD_OVERVIEW medm screen.

If the LASER_PWR node is in error, then ISC_LOCK will not have its OKAY channel as True since ISC_LOCK manages LASER_PWR. I don't think that then placing ISC_LOCK on the exclude list is a good idea.