Lost lock at ENGAGE ASC PART3 twice in a row. I've attached the log and the lockloss plots of both locklosses (and the ASC Striptool from the recent lockloss). Sorry about the terrible screenshots of the log (only showing the DOWN state).

I called Kiwamu to make sure the ASC loop gains were correct. Turned out INP and PRC1 (pitch and yaw) gains were 0 when they're supposed to be 1. I put in the gains by hand before realized that PRC1 had no ramp time at all. There was a spike from PRC1 yaw and the ifo lost lock shortly after. So I gave PRC1 ramp time of 10 s.

Nutsinee, Sheila, Kiwamu,

Nutsinee had a number (5-ish times) of lock losses in ENGAGE_ASC_PART3. We blame TMSY for these lock losses.

Nutsinee went through the initial alignment and it corrected TMSY in pitch by 2 urad. See the attached screenshot showing 2 day trend of TMS angles. A large step shown in the middle of the screenshot must be the temporary correction that Corey introduced in the last night (alog 22535). You can see that the TMSY angle is now back to where it was 2 days ago. Tonight, the symptom was that when we engaged the SOFT loops, something started running away and decreasing the cavity power everywhere. We think that TMSY was pulling the SOFT loops to some kind of bad alignment point due to the different TMSY angle. It was nothing to do with the 20 dB boost in the SOFT loops (for example, alog 21587) which we suspected at the very beginning of the investigation. After the initial alignment, everything seems to be going as smooth as usual and we made it back to low noise (alog 22579). We are happy.

In additon, there was another issue with INP1 and PRC1 where they were not engaged in ENGAGE_ASC_PART1 and PART2. This was fixed by reloading the ISC_LOCK guardian since it had been edited in the last evening but was not reloaded since then.

I took a quick look at this particular lockloss and it seems like PRC2 P was growing prior to the lockloss. The zoomed-in version of the same plot doesn't tell me much except that DHARD might be glitchy prior to the lockloss. Looking at the LSC channel list it seems like POPAIR RF90 and ASAIR_A_LF glitched right before the lockloss. The Witness channel list tells me that SR3 is likely responsible for the glitch.

I don't really have a conclusion here. Just reporting what I observed.

Operators:  Do not run A2L script until Jenne troubleshoots script.  (had "Permission denied" errors, so maybe there's an issue of running script logged in as ops?).  

Will put this "pause" in running A2L in the new Ops Sticky Note page.

I was able to run the a2l script on my account. The script cleared all the SDF after it's done as Jenne advertised. All is well.

For lack of a better place to write this, I'm leaving it as a comment to this thread. 

The problem was that the results directory wasn't write-able by the Ops accounts, although it was by personal accounts.  I've chmod-ed the directory, so the A2L script should run no matter who you are signed in as.

Please continue running it (instructions) just before going to Observe, or when dropping out of Observe (i.e. for Maintenence).

I implemented an automatic restart system using monit for the GRB alert code  See LLO entry 21671.  For unknown reasons, this new method of running the script (as a detached process from an init script, not in a terminal window) also made all the GraceDB comm errors go away.  Implementation details are posted in the log entry if LHO wishes to follow.

Jeffrey K, Kiwamu, Sudarshan, Darkhan

Some conclusions based on the analysis of the DARM OLG TF and the PCAL to DARM TF measurements taken after flipping the ESD bias sign:

• Together with the ESD bias sign flip we also inverted the sign of the ESD DriveAlign gain, this means that overall L3 stage actuation function sign was not affected by today's sign flip activites;
• Since L3 stage actuation function sign was not affected, CAL-CS front-end calibration and EPICS values for calculating kappas are still valid¹;
• Actuation function analysis showed that flipping the ESD bias sign resulted in reduction of the ESD actuation stregth by 7-8% (LHO alog 22558);
• Change in ESD actuation strength was reflected in mean κ_tst calculated from calibration lines over 43 minutes of undisturbed data after the ESD bias sign flip; comparison of the most recent actuation and sensing function measurements vs. H1 DARM model corrected with kappas showed residuals at the same level as measurements vs. kappa corrected models before the sign flip (see attached plots).

¹although the total L3 stage actuation function sign did not change, the ESD bias sign flip was reflected in two of the SUSETMY L3 stage model EPICS replicas in CAL-CS model: H1:CAL-CS_DARM_ANALOG_ETMY_L3_GAIN and H1:CAL-CS_DARM_FE_ETMY_L3_DRIVEALIGN_L2L_GAIN.

Below we list kappas used in the kappa corrected parameter file for the measurements taken after the ESD bias sign flip on Oct 15. We used 43 minute mean kappas calculated from 60sec FFTs starting at GPS 1128984060 (values prior bias sign flip are given in brackets, previously they were reported in LHO alog comment 22552):

κ_tst = 1.004472 (1.057984)
κ_pu = 1.028713 (1.021401)
κ_A = 1.003843 (1.038892)
κ_C = 0.985051 (0.989844)
f_c = 334.512654 (335.073564) [Hz]

An updated comparison script and parameter files for the most recent measurements were committed to calibration SVN (r1685):

/trunk/Runs/O1/H1/Scripts/DARMOLGTFs/CompareDARMOLGTFs_O1.m
/trunk/Runs/O1/H1/Scripts/DARMOLGTFs/H1DARMparams_1128979968.m
/trunk/Runs/O1/H1/Scripts/DARMOLGTFs/H1DARMparams_1128979968_kappa_corr.m

Comparison plots were committed to CalSVN (r1684):

/trunk/Runs/O1/H1/Results/DARMOLGTFs/2015-10-15_H1DARM_O1_cmp_AfterSignFlip_*.pdf

As a double check we created Matlab file with EPICS values for kappas using H1DARMparams_1128979968.m parameter file and made sure that the values agree with the ones currently written in the EPICS. The logs from calculating these EPICS values was committed to CalSVN (r1682):

/trunk/Runs/O1/H1/Scripts/CAL_EPICS/D20151015_H1_CAL_EPICS_VALUES.m
/trunk/Runs/O1/H1/Scripts/CAL_EPICS/20151015_H1_CAL_EPICS_*

For the record, we changed the requested bias voltage from -9.5 [V_DAC] (negative) to +9.5 [V_DAC] (positive).

This is actually 10/15 & Jim's DAY shift.  :)

J. Kissel

DARM OLTGF Is Complete. Results saved and committed to 
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Measurements/DARMOLGTFs/
2015-10-15_H1_DARM_OLGTF_7to1200Hz_BeforeBiasFlip_A_ETMYL3LOCKIN2_B_ETMYL3LOCKEXC_coh.txt
2015-10-15_H1_DARM_OLGTF_7to1200Hz_BeforeBiasFlip_A_ETMYL3LOCKIN2_B_ETMYL3LOCKEXC_tf.txt
2015-10-15_H1_DARM_OLGTF_7to1200Hz_BeforeBiasFlip_A_ETMYL3LOCKIN2_B_ETMYL3LOCKIN1_coh.txt
2015-10-15_H1_DARM_OLGTF_7to1200Hz_BeforeBiasFlip_A_ETMYL3LOCKIN2_B_ETMYL3LOCKIN1_tf.txt
2015-10-15_H1_DARM_OLGTF_7to1200Hz_BeforeBiasFlip.xml

J. Kissel

PCAL to DARM Transfer Functions complete. We're going to quickly run an A2L measurement, then open the beam diverters. After which, we'll try to preserve the lock by transitioning over to ETMX for DARM control. However, this will require turning the ETMX HV ESD driver back ON, and we're on 50% confident that the IFO lock can survive that turn on transient. Wish us luck!

PCAL 2 DARM results have been committed to 
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Measurements/PCAL/
2015-10-15_PCALY2DARMTF_7to1200Hz_BeforeBiasFlip_A_PCALRX_B_DARMIN1_coh.txt
2015-10-15_PCALY2DARMTF_7to1200Hz_BeforeBiasFlip_A_PCALRX_B_DARMIN1_tf.txt
2015-10-15_PCALY2DARMTF_7to1200Hz_BeforeBiasFlip.xml

DARMOLGTF model vs. meas. comparison plots from measurements prior to ESD bias sign flip are attached to this report.

Parameter files for these measurements and updated comparison scripts are committed to calibration SVN (r1673):

/trunk/Runs/O1/H1/Scripts/DARMOLGTFs/CompareDARMOLGTFs_O1.m
/trunk/Runs/O1/H1/Scripts/DARMOLGTFs/H1DARMparams_1127083151_kappa_corr.m
/trunk/Runs/O1/H1/Scripts/DARMOLGTFs/H1DARMparams_1128956805.m
/trunk/Runs/O1/H1/Scripts/DARMOLGTFs/H1DARMparams_1128956805_kappa_corr.m

Plots and a .MAT file with the models are committed to calibration SVN (r1673):

/trunk/Runs/O1/H1/Results/DARMOLGTFs/2015-10-15_H1DARM_O1_cmp_bfSignFlip_*.pdf
/trunk/Runs/O1/H1/Results/DARMOLGTFs/2015-10-15_H1DARM_O1_cmp_bfSignFlip_DARMOLGTF.mat

In the kappa corrected parameter file for the measurements taken on Oct 15 we used average kappas over 1 hour calculated from SLM tool starting at GPS 112894560. The values are listed below:

κ_tst = 1.057984
κ_pu = 1.021401
κ_A = 1.038892
κ_C = 0.989844
f_c = 335.073564 [Hz]

For kappas used in parameter files for Sep 10 and Sep 23 see LHO alog comment 22071.

For the record, we changed the requested bias voltage from -9.5 [V_DAC] (negative) to +9.5 [V_DAC] (positive).

Here is a comparison of the calibrated DARM spectrum from times when the HVAC was ON and OFF, in the frequency band that was affected.

I plotted glitchgrams and trigger rates during this time. Doesn't seem to have made a noticable change.

https://ldas-jobs.ligo-wa.caltech.edu/~jordan.palamos/detchar/HVAC/glitchgram_HVAC_1128909617.png

https://ldas-jobs.ligo-wa.caltech.edu/~jordan.palamos/detchar/HVAC/rate_HVAC_1128909617.png

Attached are ASDs of DARM and one of the PEM seismometer channels (corner station Z axis) for all of the times when the HVAC was turned on and off (not including the times of transition). In general, the noise level between 40-100 Hz is lower during the times when HVAC was off. The peak around 75 Hz was better during the second two off times, but not in the first segment. (1128910297 to 1128910697)

More PEM seismometer channels are here: https://ldas-jobs.ligo-wa.caltech.edu/~marissa.walker/O1/Oct15HVACtest/

(note: the seismometer calibration from pem.ligo.org is only valid from 0-20 hz)

epics freezes never fully went away completely and are normally only a few seconds in duration. This morning's SUS ETMX event lasted for 22 seconds which exceeded Guardian's timeout period. To get the outage duration, I second trended H1:IOP-SUS_EX_ADC_DT_OUTMON. Outages are on a computer basis, not model basis, so I have put the IOP Duotone output EPICS channel into the frame as EDCU channels (access via channel access over the network). When these channels are unavailable, the DAQ sets them to be zero.

For this event the time line is (all times UTC)

The investigation of this problem is ongoing, we could bump up the priority if it becomes a serious IFO operations issue.

To be clear, it sounds like there was a lockloss during acquisition that was caused by some kind of EPICS drop out.  I see how a lockloss could occur during the NOMINAL lock state just from an EPICS drop out.  guardian nodes might go into error, but that shouldn't actually affect the fast IFO controls at all.

Sorry, I meant that I can not see how a guardian EPICS dropout could cause a lock loss during the nominal lock state.

There was a request to see how the inverse correction filters for the sensing chain were rolled off at high frequencies in these filters.  This is done with a simple smooth roll-off above ~6500 Hz.  I've attached plots that zoom in on the 5000-8192 Hz range to show how the rolloff causes the GDS filters to differ from the ideal inverse correction filters for the sensing chain.