Rich A., Lisa, Sheila, Jeff, Evan

The EY ESD driver has tripped twice today, apparently in the same failure mode as described previously. Each time we have had to drive down to the end station to reactivate it.

The supplies are set at ±430 V, each with a 75 mA current limit. When the driver is active, it draws 23 mA dc from the positive supply and 20 mA dc from the negative supply. When tripped, it draws 3 mA dc from each supply.

J. Kissel
Completes WP #5118

I've compiled, installed, restarted, and restored the changes to the h1calcs.mdl and h1omc.mdl front end models that were described in LHO aLOG 17472. This required a DAQ / h1dc0 / frame-builder restart, which I performed at 09:02 PDT (16:02 UTC).

I attach screenshots of further improved MEDM screens for the CAL CS and LSC that employ the new channels.

I've also turned on the OMC SDF system, and committed the new SDF file "safe.snap" to the userapps repo, under
/opt/rtcds/userapps/release/omc/h1/burtfiles/h1omc_safe.snap. It currently has 28 channels not monitored, the only one of which I delibrately turned off was H1:OMC-PZT2_SW2S, which had its output flickering on and off because of the front-end / guardian triggering which toggles the output. The rest appear to be ODC strings that I guess, by default, aren't monitored. 

Note that this SDF system covers a lot of the LSC stuff too, and it looks like the majority of channels that are different from this morning (when I turned on the SDF, when the ISC_LOCK and OMC_LOCK manager were DOWN) are Guardian changed variables. When the IFO was fully locked in the DC readout, and the vanguard was in the process of transitioning to low-noise ETMY control, there were only 20 channels in the DIFF list. I leave it to the commissioning vanguard to remove these channels from the list at leisure. This will be much easier / convenient to do once we get the RCG 2.9.1 upgrade to the SDF system.

= Morning Meeting =

SEI: BRS rung up

SUS: Sorry I couldn't hear

Richard: Shutter control needs work

Facility: Sorry, I couldn't hear again

3IFO: Moving things next Tuesday. Will be noisy.

Jeff: Rebotting calibration front end model

Sudarshan: PCal calibration

Lots of work permits...

= Activities =

Some times early in the morning: Karen and Cris to LVEA

7:00 Richard to EY

7:37 Richard back

8:19 Karen opening exit doors (LVEA)

8:41 Jim B to EY. Installing TCT

8:45 Fil to EY replace shutter box

        Sudarshan and Rick to EX - PCal work

        Jim to EX (stopped BRS damp)

8:52 Matt and Jason to H1 PSL ante room

5:56 Kyle to EY

9:00 both Jim came back

9:06 Jeff done with model changing

9:12 Matt and Jason back

9:19 Betsy and Travisto West Bay (3IFO stuff)

9:34 Kyle out of EY, to EX

9:39 Fil back

10:02 Kyle back

10:05 Andreas drop a bag at LVEA

10:18 Andreas back

10:30 Richard to LVEA 2k area (3IFO)

          Jodi to LVEA 

11:29 Jodi let Gary in, supervised.

12:00 Daniel and Sheila to EY

12:40 Corey to squeezer bay

13:46 Jodi back

13:51 Dan and Eli to EY

13:54 Rick and Sudarshan leaving EX

14:40 Dan and Eli back

          Corey back

Happy 3IFO day

I took out ISC_DOF since it is no longer being used. Since this left a hole, I slid up the SYS_DIAG mini.

Screenshot attached.

A couple of plots from last night. We had an incredibly stable ~4h lock with 20 Mpc range. All the "standard" alignment loops were enaged (the only uncontrolled DOFs are comm/diff ITMs).

Recall that in this lock DARM was controlled by actuating on high noise ESD on ETMX.

Dan, Elli.   EX ring heater power supply is turned off.  Alll other ring heater power supplies are still on  There is a coherence measured between EX magnetometer and DARM at 55Hz (see Gabriele's alog 17423), so we are checking whether turning off the ring heater power supply removes this line.

I've continued my work on improving BSC St2 blends, I think I have something fairly Hippocratic.  I moved the St2 ellips down to 1.5 Hz and added a pole at 10 Hz. This affects the St2 CPS phase (~24 degrees at the blend frequency) but this doesn't seem to negatively affect the performance. I've talked to the commissioners about trying this during a lock, I think I can install and turn the new filters on without breaking anything.

First plot shows my modifications. Red is stock, green is the "old" again, blue is my current.

Second plot shows the performance. Pink is ground before, green is ground after, blue is before, dashed and solid are CPS and GS13 respectively, red is after. Does pretty good between 1 and 20 hz, doesn't seem to do any worse below 1 hz. I think all of the differences below 1  hz can be attributed to the ground.

Sheila, Hugh

IFO lost lock ~1426utc, Richard suspects it was his testing of the ESD.  Almost 45 minutes later, the ISCINF_LONG stepped to its rail, LIMIT set to 250000nm.  And I mean stepped, ~1 second or less.  This of course put a bit much demand on the HEPI loop.  It spiked its output and pretty much instantly tripped the watchdog on the Actuator followed by the ISI shortly thereafter.  The HEPI and ISI reisolated normally.

The first attached plot shows the lock loss.  This 30 minute trend has the DARM OUT with the ISCMON_X_INMON quickly going to zero and the HEPI Outputs moving in response.  No problems here, HEPI does fine.  You also see the arm or something grab again as you can see the Tidal drive to the HEPI start again but this only lasted for a couple minutes.

The second plot attached zooms out to 2 hours, with the IFO lock loss seen on the ISCMON on the left.  ~45 minutes later the DARM steps, this time stepping the ISCMON and tripping HEPI and about 8 seconds later, the ISI (not shown.)

In the third attachment, Sheila dug around in LSC/ALS space and put together some trends showing the spike into HEPI originating when the ALS-X_LOCK_STATE steps from level 2 to 6 (PLL Lock to Red Lock.)  This is unrealistic as the Lock state dropped from Red Lock just ~90 seconds earlier.  Notice the TIDAL_CTRL_INMON steadily growing (DARM not Cleared.)

Sheila is looking deeper into the Guardian feeling it may have failed to do something it possibly should have to prevent the DARM OUT from going to the rail and the unreasonable rapid transitions of the ALS LOCK STATE.

In the continuing saga of the Ring Up of the BRS, I've turned off the damper. I turned it on yesterday thinking that 2 days was sufficient rest for the BRS to settle down, but Krisha reports that it probably needs more like 4-5 days, given current levels of excitement. It would be best if people avoided the end station, as the BRS is now completely uncontrolled.

The damper is controlled by a line of code that sets the variable TestData (or some variation of capitalization thereof). The normal line sets this equal to 2.5 - outvelocity. To disable there is a line immediately below that is commented out that sets TestData to just a constant (2.5). Uncomment this line and restart the code. Done.

Replaced Shutter Controller S1203610 with S1203612. Daniel reported CH1 on S1203610 is not functioning, thus we were controlling the shutter with CH2. I reconnected the shutter to CH1 of the new unit and changed the jumper settings to VS35 on both channels.

model restarts logged for Wed 25/Mar/2015
2015_03_25 14:33 h1nds1
2015_03_25 14:43 h1fw0

both unexpected restarts.

Loss of lock probably due to my work at EY on ESD drive.  

Following up on last nights report I went down to EY to investigate problems with the ESD.
Much to my surprise aside from the unit being in the off state I could not find a problem.  The HV supplies were on and at nominal current draw for no drive (3.2mA).  Went in to check the ESD Chassis and all the lights were green. The only indication of a problem was the main indicating light for the unit showed it to be in the off state.  I had disabled all of the DAC inputs to the driver so I turned it on.  Went over to computer and enabled outputs and immediately saw a response on the OpLev.  I am not sure what interlock could have shut the unit down but further investigation is warranted.  I have left the unit on for now.  please keep me Richard McCarthy informed if this or more problems arise.

Evan, Lisa

So, we leave the interferometer with:

• DARM on ETMX (high noise, ETMY ESD not working, please check it tomorrow), DARM boost on (engaed by hand, not in the Guardian)
• PRM, PR2, PR3, SR2, SR3, ITMX, ITMY in their final low noise state for all stages (robust, in the Guardian)
• ETMX and ETMY L2 in low noise (low noise transition not included the Guardian, mode switched back to acq in the downstate)
• SRCL gain reduced from -25 to -17 to give a UGF of 20 Hz, plus ELP100 cut off added (not included in the Guardian, so this would go back to nominal tomorrow, we just want to test if it is stable)

After all of this we saw that the range was around 20 Mpc, even without the low noise ETMY ESD, and without any MICH/SRCL feedforward, so we might indeed have made some improvement.

As far as we can tell right now, the locking sequence tomorrow should work as usual.

Evan, Dan, Lisa

Dan and I wanted to leave, but Evan forced us to stay...so, here we are.

Dan and Evan, the experts in roll and bounce damping science present in the Hanford control room at mid-night, eventually succeeded in damping.

So we relocked to DC readout, right before the transition to low noise ETMY, and switched the coil drivers for PRM-PR2-SR2 (tested at least twice, now in the Guardian) and BS in their final low noise state (for BS we used the script which switches one coil at the time, so this transition is not included in the Guardian yet). We also switched ETMX, ETMY, ITMX, ITMY L2 to their final low noise state (state 3), but we are not including this step in the Guardian yet as we tested it only once (we modified the down state to go back to the acquisition mode for the quads).

We also investigated why the transition to ETMY stopped working by measuring the DARM open loop while actuating on both ETMX and ETMY at the same time.

As far as we can tell, the transition is not working simply because the ETMY ESD is not doing anything at all.

Please go to EY tomorrow morning and check the ESD driver. 

Evan, Kiwamu, Sheila, Dan, Lisa

Summary

We still haven't quite finished our list of "low hanging fruits", but we are getting close. The OMC is not shaking anymore after engaging the DARM boost and reducing the RIN in transmission of the OMC by nearly a factor of 10. Also, we switched the vertex control from POP to POPAIR, which has ~5 times more light, and in doing so we reduced the noise coupling from MICH to DARM. We could reach ~ 28 Mpc again without engaging the MICH feed-forward.

Sadly, we don't have any improvement in Mpc units because a few "technical problems" slowed us down, and we didn't have a chance to finish the coil driver transitions to low noise, improve the vertex length DOF loop cut-offs, and tune the MICH/SRCL feed-forward subtraction while on POPAIR.

In particular, a typo in the ISC_LOCK Guardian (switching of a filter in the "DAMP_M0" filter bank, instead of "DARM_DAMP_MO") badly tripped ETMY (thanks to Jeff who helped us recovering by phone), and caused both roll and bounce mode to run up insanely, and we can't lock. Our damping techniques with ifo unlocked are very (very) slow, so we are giving up.

Also, the transition to ETMY with low noise ESD worked well once today, but later attempts failed (not totally clear why, the only known difference in later attempts is that we engaged the DARM boost before making the transition).

The full list of things we did/tried is below.

Fixed the "shaking" OMC (in the Guardian)

We reduced the DARM motion between 1 and 6 Hz, as Dan reported here, and now the OMC is happy (and we are happy with him).

Transitioned to POPAIR (in the Guardian)

POPAIR has ~5 times more light than POP, so we decided to transition from the in vac diode to the in air one, to reduce the noise coupling of the vertex length DOF to DARM.  Indeed, we could reach the same sensitivity as the other day (~28 Mpc) without any MICH feed-forward.

Close ALS shutters on ISCTEX/EY (in the Guardian)

Not clear improvement, but this is the beginning of the "closing beam diverter/shutter" campaign.

Test of SRCL cut-off  (not in the Guardian yet)

The SRCL loop had 60 degrees of phase at 30 Hz, and a very modest cut off filter. We tested a more aggressive cut-off, which removed all the coherence we see between DARM and SRCL above 100 Hz. We tested this filter only once, so we are not including it yet in the locking sequence.

Test of BS coil driver switch in DRMI  (not in the Guardian yet)

Kiwamu and Evan imported and adapted the script that Den wrote to do the BS coil driver switching one by one. It was tested in DRMI - 1f, and it worked. We haven't tested it in full lock yet.

Laura, Josh, Andy, Joe, Detchar

We took a look at the 28 Mpc lock on 24th March to see if there was any evidence of arches that we have seen previously at LLO (for example LLO alog 17090).

The first attachement shows a histogram of the rate of glitches (omicron triggers) binned by the value of IMC-F at the same time. The plot looks pretty Gaussian, except for a few bins specifically between IMC-F values of -37.5 to -41.7 and -45.8 and -50. Between these values we see evidence of arches in DARM. The second attachment is a pdf file showing the arches in DARM lining up with values of IMC-F in the second bin. Here are links to many spectograms confirming the arches in the IMC-F bins outlined (spectrograms for IMC-F bin -37.5 to -41.7 / spectrograms for IMC-F bin -45.8 to -50). Each spectrogram time corresponds to a glitch as seen by omicron.

LLO has dramatically reduced the effect of these arches, see LLO alog 17191.

Also the value IMC-F has when these arches appear in DARM seems to drift, i.e. there does not appear to be a specific value of IMC-F which produces the arches (this is evident in the second attachement). This is not what we have seen at LLO, does anyone know what could be causing this?

Time Code Translator work at end stations, WP5116

Dave, Jim:

We racked up the missing TCT in EY, then connected it to one of the unused fiber pairs running between the VEA rack and the computer rack. We verified it was using ports 19 and 20 in the MSR (same as EX). Once connected the TCT synced to the time and the error LED on the MSR sender was extinguished.

We then ran 30 foot 50-Ohm coax, BNC terminated, between the TCT in the computer rack, through the wall, to the first port on the CER comparator. This was done at both EX and EY. The 1PPS port on the back of the TCT is a TNC connector, so we used a TNC-BNC adapter on that port for EX. We need to do the same for EY.

At EX, the comparator is now displaying a time of -22.2uS on the TCT port, roughly the time of flight between the buildings. We will revisit the TCT internal settings, they are most probably what was left from iLIGO.

Rebuild of ASC model

Daniel, Dave:

Daniel though h1asc would benefit from a clean build, install and restart. This was done, I'll see if this helped in the IPC errors.

PCAL filter modules, port from PEM file

Sudarshan, Dave:

Sudarshan discovered that I had forgotten to migrate the PCAL filters from when the CAL system shared a core with the PEM. He copied them from H1PEME[X,Y].txt to H1CALE[X,Y].txt, making a CAL_PCAL to PCAL name change in the process. The filters were then loaded into the PCAL models at both end stations.