Evan, Dan, Jeff

The SR3 M1 T2 actuator is...unwell.

We started to have trouble staying locked after power up about four hours ago.  At first it looked like an ASC problem in the SRC, and since the AS36 loops are the scapegoat du jour, Evan started to retune the SRC1 loop.  He observed a recurring transient in the beam spots at the AS port, and saw the same thing in the SR3 oplevs.  We turned off the SR3 cage servo and the kicks kept on coming.

Eventually we looked at the SR3 M1 voltmons and found the first plot attached, which is for eight hours.  The T2 OSEM and voltmon started to get ratty about three and a half hours ago.  The noise is a series of slow transients with a several-second rise and a steep decay.  See Fig. 2.

We're pretty convinced that it's an actuator problem, something between the DAC and the voltmon readback in the coil driver box.  We have unlocked the IFO and turned off the SR3 top-stage damping and we still see the pattern of noise.  We shall leave the pleasure of power-cycling the AI chassis and coil driver to others.

As an amuse bouche for the maintenance day team, we also discovered that the SR3 M1 T1 voltmon is complete nonsense.  The T1 voltmon time series is a collection of step functions and spikes, 100x larger than the other M1 voltmons.

The SR3 M1 damping has been turned back on.

8/17 EVE Shift:  23:00-7:00UTC (16:00-0:00PDT)

Shift started with Travis taking H1 GRD-ISC_LOCK to:  NOISE_TUNINGS

• Kiwamu installing a roll-off freq filter for ETMy to decrease the "ETMy Saturations"
• Sudarshan investigating ISS (because of locklosses around ENGAGE_ISS_2ND_LOOP)
• Travis mentioned issues at INCREASE_POWER due to IMC (so Guardian stops and one will Pause Guardian, and then proceed to next step of COIL_DRIVERS)
• May also have to skip the ENGAGE_ISS_2ND_LOOP to get to Low Noise

H1 Guardian Operations for Tonight:

There are two states of Guardian where H1 is having issues:

1. INCREASE_POWER:  Due to PMC drift, H1 gets stuck checking power.  So, to get past this state, we have been hitting MANUAL (on the Lock State medm), selecting COIL_DRIVERS, and then hit AUTO, and go to NOISE_TUNINGS
2. Next problem spot appears to be the engaging the ISS 2nd Loop.  So, once in NOISE_TUNINGS, hit MANUAL, and select NOMINAL_LOW_NOISE.

Other Activities:  UTC (PDT)

• 23:50 (16:50) - 0:40 (17:40):  Dave working on conlog & will need to restart
• 0:41 (17:41):  All Calibration lines are off
• 3:55 (20:55):  HEPI EX Diff Pressure alarm (looks like just a random data point strayed out of range
• 4:45 (21:45):  Contacted LLO Operator to gauge their status for the evening.  They are extending Commissioning until ~8:00UTC for parametric instability work.
• ~5:00UTC (22:00PT), when Commissioning was to end, attempted to lock H1 for the night....but had trouble getting to NOMINAL NOISE.
• 5:20 (22:20):  Waited 18 min for DRMI, and then had lockloss due to BS Coil Drivers
• 6:58 (23:58):  Evan's investigating possible ASC issues with H1 & I'm handing off a pesky H1 to Jeff B.

J. Kissel, J. Oberling, B. Weaver

Up to bat for the Recovery Team tomorrow:
Operator: T. Sadecki
DetEng: B. Weaver, J. Oberling
Commissioning: J. Driggers

Attached is tomorrow's Maintenance Plan. The major activities (i.e. those with the greatest impact) will be 
- an adjustment of the PSL's PMC Temperature to try to restore the full power output, and
- moving the violin mode monitors out of the QUAD models and into the OAF model to try to reduce the computation turn-around time for the ETMs and mitigate timing overages with the slower front ends.

Full List (In Chronological Order):
1st Wave (as soon as Peter arrives in the morning)
- Adjust temperature / alignment of PMC remotely from control room

2nd Wave (~8:00 am)
- Monitor PMC temperature / alignment of PMC, if remote change doesn't have desired affect, then PSL incursion.
- Commissioning HEPI HAM1 inertial isolation
- Remove violin mode damping from QUADs, Install into OAF
- Updates to ODC MASTER model to include/update TCS
- New power supply on corner-station TCS Hartmann WFS
- New power supply on corner-station PEM electronics

3rd Wave (~9:00a)
- Add all newly requested channels to GDS broadcaster, low-latency frames
- Total GDS package upgrade
- Charge measurements on ETMs
- Watch for Beckhoff crash at EX

The hope is to begin recovery by 10a, such that we have a full IFO back up and running by 12p PT! This looks to be a pretty light maintenance day (finally!), but we'll see how it goes.

I wrote python scripts that copy the ETMY suspension digital filters and turn the right combination of the filters in the simulated ETMY in CAL CS.

The scripts are now in the following svn locations. It would be ideal if anyone who makes a modification run the following scripts to update the CAL CS suspension filters.

/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/CALCS/copySus2Cal.py

/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/CALCS/setCalcsEtmy.py

At the time being, they are written in a hardcoded way so that they only do the copy and paste operations only for the SUS ETMY filters and it does not even look at the other quad suspensions. The 1st script uses the python foton bindings to copy the foton filters from H1SUSETMY.txt over to H1CALCS.txt. The 2nd script uses nds2 to check the filter settings at a past point and copy the settings for the CAL CS filters. Right now the code does not look beautiful at all because I use the sfm option of the command line cdsutils. I am hoping that I can replace them by fancier functions at some point. For now, they should be good enough. I have tested each code several times in this evening and they seemed to be functional as intended.

JeffreyK, SudarshanK, DarkhanT

Today we took DARM open-loop transfer function measurement using twice stronger drive at higher frequencies, > 1000 Hz, compared to measurement taken last night (see LHO alog #20585). We will need to bump up drive level a little bit more at higher frequencies to get a better coherence in the future measurements.

We also took PCALY sweep with the same frequency vector that started 10-15 minutes after starting DARM sweep.

Since these two measurements were taken close in time and within the same lock stretch we assume that IFO parameters were the same in both measurements. We will try to use both sweeps to estimate DARM parameters.

DARM measurement XML file was committed to SVN:

CalSVN/aligocalibration/trunk/Runs/ER8/H1/Measurements/DARMOLGTFs/2015-08-17_H1_DARM_OLGTF.xml

PCALX and PCALY measurements' XML files were committed to SVN:

CalSVN/aligocalibration/trunk/Runs/ER8/H1/Measurements/PCAL/2015-08-17_PCALX2DARMTF_logscale.xml

CalSVN/aligocalibration/trunk/Runs/ER8/H1/Measurements/PCAL/2015-08-17_PCALY2DARMTF_logscale.xml

H1 now in OBSERVATION MODE

As per LIGO-M1500250, H1 has now been put into "OBSERVATION MODE".  This means:

The ODC-OPERATOR_OBSERVATION_READY bit, which is set to "Undisturbed" by the operator on the GUARD_OVERVIEW screen, will now be automatically unset by the Guardian IFO top node if any of the monitored guardian nodes drop from OK==True.

The most likely ways that this can happen are:

• Lock loss
• SUS or SEI watchdog trip
• TCS laser trip
• Change of state or operational mode of any Guardian node (e.g. OP, MODE, REQUEST, or ERROR change)

Operators must be aware that the INTENT bit must be manually reset after any of these changes.

The list of Guardian nodes being monitored is stored in:

USERAPPS/sys/h1/guardian/IFO_NODE_LIST.py

The current monitored node list is:

ALIGN_IFO
ALS_COMM
ALS_DIFF
ALS_XARM
ALS_YARM
HPI_BS
HPI_ETMX
HPI_ETMY
HPI_HAM1
HPI_HAM2
HPI_HAM3
HPI_HAM4
HPI_HAM5
HPI_HAM6
HPI_ITMX
HPI_ITMY
IMC_LOCK
ISC_DRMI
ISC_LOCK
ISI_BS_ST1
ISI_BS_ST2
ISI_ETMX_ST1
ISI_ETMX_ST2
ISI_ETMY_ST1
ISI_ETMY_ST2
ISI_HAM2
ISI_HAM3
ISI_HAM4
ISI_HAM5
ISI_HAM6
ISI_ITMX_ST1
ISI_ITMX_ST2
ISI_ITMY_ST1
ISI_ITMY_ST2
OMC_LOCK
SEI_BS
SEI_ETMX
SEI_ETMY
SEI_HAM2
SEI_HAM3
SEI_HAM4
SEI_HAM5
SEI_HAM6
SEI_ITMX
SEI_ITMY
SR3_CAGE_SERVO
SUS_BS
SUS_ETMX
SUS_ETMY
SUS_IM1
SUS_IM2
SUS_IM3
SUS_IM4
SUS_ITMX
SUS_ITMY
SUS_MC1
SUS_MC2
SUS_MC3
SUS_OM1
SUS_OM2
SUS_OM3
SUS_OMC
SUS_PR2
SUS_PR3
SUS_PRM
SUS_RM1
SUS_RM2
SUS_SR2
SUS_SR3
SUS_SRM
SUS_TMSX
SUS_TMSY
TCS_ITMX

Sheila, Kiwamu

We newly put an elliptic low pass in ETMY_L3 in order to reduce the number of saturation events.

Today, we noticed that ETMY frequently saturated at the bottom stage. Looking at the spectra and time series, we found that high frequency components above 1 kHz was a culprit. This is related to the activity that Evan et al worked on last night (alog 20585) trying to get a better phase margin. We decided to install a roll off in order to mitigate the issue. We put an elliptic whose cut off is at 950 Hz with a 20 dB rejection in the stop band and 3 dB ripple in the pass band. Since the filter banks in ETMY_L3_LOCK was full, we put it in DRIVEALIGN instead. It is now in FM7 and the ISC_LOCK turns it on in the ETMY_TRANSITION state. This costed a 2.6 deg phase loss at 40 Hz which should be OK according to Evan's open loop measurement.

As a result, it reduced the DAC counts in RMS by a factor of between 2 and 3. This seemed to help reducing the saturation events so far. See the attached. We could have lower the cut off frequency more, but since it is going to be limited by frequency components below a couple of Hz anyway, we leave it as it is.

Conlog stopped running at 10am PDT Saturday 15th:

Aug 15 10:01:21 h1conlog1-master conlog: ../conlog.cpp: 301: process_cac_messages: MySQL Exception: Error: Out of range value for column 'value' at row 1: Error code: 1264: SQLState: 22003: Exiting.

The restart reporter for Sunday shows a h1fw1 restart:

model restarts logged for Sun 16/Aug/2015
2015_08_16 17:28 h1fw1

This is the first unexpected restart of either frame writer following the DAQ reconfiguration made several weeks ago. This restart is likely attributed to LDAS starting the disk archiver at that time, which intensively scanned the frame writers' disk systems. We would expect h1fw1 to be more susceptible to crashing when this is happening since it is writing the larger commissioning frame.

Remote logins to LHO CDS work again.  It  was an issue with a security patch.  A long term solution will be worked on.  FRS3467 has been closed.

There were quite a few partially loaded filter banks which were listed on the CDS Filter DAQ Status screen.  After lookign through the alog, and verifying what filters are in play compared to the archived files, I hit the LOAD button and cleared the "MODIFIED" state on:

H1SUSSRM

H1SUSETMY

H1LSC

H1CALCS

Most were just a filter or 2 that were in developement over the weekend. 

As well, we can hit LOAD COEFF on the H1SUSITMX, H1SUSITMY, and H1OMC, where I verified that the modifications logged in the CDS filter DAQ STat screen are in-deed the only filters which have been commissioned and loaded locally at the bank.

15:10 Lockloss due to EQ in Mariana Islands

16:05 Fil to CER to investigate cosmic ray detector noise

16:06 Fred down both arms with BBC crew

16:07 Peter to diode room to take pics

16:10 Peter out

16:21 JimW taking HAM1 HEPI down for filter install

16:27 Richard to CER

17:00 Kyle and Gerardo to Y-2-8 for ion pump work

17:07 Fil and Richard done

19:03 Kyle and Gerardo back

19:15 started initial alignment

19:35 Fil and Richard to MX PEM vault

19:54 locked low noise

20:05 Richard and Fil done

21:02 Kyle and Gerardo back to Y-2-8

Locking summary:  Several short lock stretches today after an initial alignment post-earthquakes.  Most of the locklosses can be attributed to local seismic activity.  Others are being investigated by commissioners.

Craig, Jenne, Kiwamu,

We did a VCO calibration for the ALS diff vco in the last Friday when the interferometer was suffering from high wind. The new result suggests that the VCO calibration stays the same as the one for ER7 by 0.6%.

The new coefficient is 0.11847  +/- 9.3e-05  [cnts @ DIFF_PLL_CTRL_INMON / Hz @ DIFF VCO frequency read by the timing comparator ].

The attached below shows the fitting result:

The measurement method is the same as the previous one which is described in this alog (alog 18711). One difference is that we used the INMON as opposed to OUTPUT because we did not want to rely on the filter shape of the chain which contains the zpk([[40], [1.6]) which is supposed to cancel the low noise VCO circuit. The sweep was as slow as 255 [Hz/ sec]. The fitting uses a frequency region where the linearity was found to be good (as indicated in the plot as yellow shaded region).

The analysis code, data and figure are saved in SVN as follows:

/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/ALSDiff/vco_calibration_fitting.py

/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Measurements/ALSDiff/2015-08-14_vco_sweep_v2.txt

/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Results/ALSDiff/2015-08-14_VCO_calibration_v2.pdf

/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Results/ALSDiff/2015-08-14_VCO_calibration_v2.png

Laser Status:
SysStat is good
Front End power is 33.48W (should be around 30 W)
Frontend Watch is GREEN
HPO Watch is RED

PMC:
It has been locked 6.0 days, 7.0 hr 35.0 minutes (should be days/weeks)
Reflected power is 3.071Watts and PowerSum = 26.12Watts.

FSS:
It has been locked for 0.0 days 0.0 h and 2.0 min (should be days/weeks)
TPD[V] = 1.472V (min 0.9V)

ISS:
The diffracted power is around 10.11% (should be 5-9%)
Last saturation event was 0.0 days 0.0 hours and 2.0 minutes ago (should be days/weeks)
 

Posted by Travis S.

One of the locklosses Travis has had this morning was due to an oscillation in DHARD.  This lockloss was typical of ones we've been having in the last few weeks a few minutes after powering up.  If we get a chance I will try to measure these loops at full power, especially pit where we added a resG. 

We are aware of problems with the remote 2FA Yubikey login system, we are investigating. It has an opened ticket FRS3467.

Darkhan, Jenne, Stefan, Evan

Upon revisiting the shape of the DARM loop, we found that we had 4 dB of gain peaking from 10 Hz to 50 Hz.

That seems a bit too strong for our taste, so we took a look at the DARM suscomp filter that we've been using for the past few months. Aside from the low-frequency (<2 Hz) suspension resonance compensation and the high-frequency (>900 Hz) inversion rolloff, there is a single pole at 200 Hz.

This serves us well when locking DIFF, but in full, low-noise lock it costs us phase (not least because we also have the effect of the 350 Hz RSE pole).

Anyway, there is now a filter installed in EY L3 lock with a zero at 200 Hz and a pole at 1000 Hz in order to push this pole up to higher frequency. This gives us a modest improvement in phase margin at the DARM ugf (34° to 46°). The DARM ugf is 40 Hz or so (more or less the same as before). The EY ESD has about 16000 ct rms, mostly accumulated around 1 kHz (magenta/orange in the attached spectra show the before/after).

[In the longer term, we should just roll p/z pair into the suscomp filter, so that it rises like f essentially from 1 Hz to 1 kHz. Then put a 200 Hz pole / 1000 Hz zero pair in EX L3 lock, so as not to disturb the DIFF loop shape. However, I've put what we have so far into the guardian and it works fine.]

Darkhan and I retuned the DARM OLTF template in order to not saturate with this new loop shape. The attachment shows the before (blue) and after (red), along with the closed-loop gain and loop suppression. There's still moderate gain peaking below 20 Hz.