Matt, Lisa, Hang, Evan

Tonight we went to full power, turned on the new boost and cutoff in dHard (along with a small lead filter around 3 Hz), and turned off the oplev damping on the ITMs. Then we took an OLTF. Blue shows the new loop with the lead off, and red is the loop with the lead on. So far we've been at full power for more than 3 hours without any sign of instability.

There is some new, untested code sitting in the ISC_LOCK guardian:

• In COIL_DRIVERS, the ISC_LOCK guardian requests the IMC_LOCK guardian to go to PREPARE_ISS.
• In LSC_FF, the new dHard filters are engaged, the ITM oplevs are ramped off, and the IMC_LOCK is requested to go to ISS_ON.

However, this new, untested code is commented out (search ISC_LOCK.py for 'guardbomb' to find it). We can uncomment it the next time there is someone in the control room to supervise the lock acquisition.

Additionally, I got impatient with damping the roll modes during the acquisition sequence, and so I have set the quad coil drivers to be high range until the COIL_DRIVERS state, at which point they are switched to low noise. This seems to work fine (i.e., I didn't notice any glitching on the cameras).

The IMC_LOCK guardian now has a state PREPARE_ISS which tunes the offset slider to bring the second loop servo board out of saturation before engaging the second loop.

This work was previously being done by the CLOSE_ISS state, but since it can take a few minutes and the offset tuning does not disturb the IFO, it can be done in parallel with other changes as soon as the operating power level is reached.  The ISC_LOCK guardian will be changed accordingly.

A secondary advantage is that the IMC_LOCK guardian can return from PREPARE_ISS to LOCKED without going through ISS_ON, which can be useful for testing.

Drift monitor thresholds updated with 120 seconds averages during lock at 1122488064 GPS,
Aug 01 2015 18:14:07 UTC.

I didn't quite fully anticipate all of the affects of separating DOWN from the rest of the graph.  In particular, one really bad unanticipated effect was that after lockloss, when the ISC_LOCK jumps to the LOCKLOSS state, it doesn't find any paths from LOCKLOSS to the last requested state, which causes it to just stall out in LOCKLOSS, and not proceed to DOWN.  In other words, DOWN was not run after the lockloss this morning after last night's 10 hour lock.

When I came in this morning I therefore found a bit of a poo show that I then had to clean up.  None of the control signals had been shut off, multiple SUS and SEI systems were tripped, and bouce roll modes were rung up.  Evan and I eventually wrangled everything back under control, and we're now back to locking.

I have reconnected DOWN to the rest of the graph.  NOTE, however, that this problem is not inherent in the fact that DOWN was disconnected.  It's just that once you do something like that you remove the ability of guardian to find the right path for you, so you have to be careful to make sure you have all the appropriate jumps to get you where you need to be.  I'll rethink things.

Some notable issues:

• HAM3 had tripped, which somehow send IMC_LOCK into "christmas mode", where it was continually jumping between DOWN and MOVE_TO_OFFLINE.  This was due to some questionable logic in the IMC_LOCK guardian that sends it to MOVE_TO_OFFLINE when HAM2 or HAM3 SEI are not in their nominal states.  It's doing this even from the DOWN state, which makes no sense and was causing this problem.  I'll try to clean up the logic here.
• The "slow" LSC IMC feedback (IMC_MCL) was not shut off, even after we had recovered ISC_LOCK and IMC_LOCK into their respective DOWN states. This was causing signals to continually be sent to MC2, even though it was misaligned, which was causing HAM3 to continual trip during isolation, which therefore led to "christmas mode" above.

Lesson's learned:

• We need to be very careful when working on things that might affect the prompt running of the ISC_LOCK DOWN state.   DOWN needs to be run promptly after lockloss, to prevent ringing up the bounce/roll modes that take a while to damp out.  Both yesterday and today we had different issues that resulted in DOWN not being run after lockloss, both of which resulted in extra hours of recovery.
• We should consider adding triggering to the front end to shut off all the control signals after a lockloss, and not rely on something external to do it.  This is a natural extension of the loop engage triggering, but we hadn't quite gotten there yet.
• Seeminly inocuous guardian changes can sometime have unanticipated far reaching affects.  We should be particularly careful when adjust the graph logic, as I did yesterday.

It seems like the rate of epics freezes has increased today, I have seen more than 5 in the last 2 hours. 

Sheila, Evan, Jeff B, Corey

Both yesterday and this morning, we had extremly rung up bounce and roll modes (both times because the IFO lost lock and DOWN was not run, yesterday for the reasons explained in comments to alog 20103, today because of a different snafu).

When this happens, we need to damp bounce on ETMY while locked on ALS.  To do this, it seems that we need to use a phase that is +150 degrees compared to the phase we use in full lock.  This phase shift comes from the difference between the DARM loop here and in full lock.  When locked on RF DARM, we need to use +120 degrees compared to the normal settings.

We also had difficulty yesterday with rung up roll modes.  To damp roll we use AS WFS, so we need to get to RF DARM before we try to damp them this way. One difficulty we had was that the roll mode notches in the PUMs were not wide enough (Evan adds that the notch needs to be wide because of the Shapiro effect), so that DHARD could saturate because of the roll mode.  

Bringing these things down when they are verry rung up is very slow, because the actuation authority is small compared to the amount of energy in the mode.  Fortunately, we are normally in the regime where the mode is small and it only takes a few minutes to damp them. 

Evan, Lisa

This entry is to clarify the fact that the impact of  this excess of high frequency noise  is actually bigger than the coherence with the ASC channels suggests, as it can clearly be seen by comparing OMC NULL and SUM.

For example, around 2 kHz, the discrepancy in the noise floor between OMC SUM (total noise) and OMC NULL (shot + dark noise) is about 15%, so corresponding to a noise which is 0.6 times shot + dark.

The attachment shows OMC SUM/NULL in H1 at low noise (left) compared to L1 (right). 

So, the message is that we are looking for something quite big here..

After 17:30 UTC the interferometer was not undisturbed: I was making PEM injections.

 Measured temps of heated areas of RGA -> 95C < temps < 120C -> Made slight changes to variac settings -> Aux. cart @ 2.5 x 10-5 torr (seems high for this configuration)

Its been a week since the DAQ reconfiguration which reduced the NFS/QFS disk loading and both framewriters continue to be 100% stable. The attached plot shows the restarts of h1fw0 (red circles), h1fw1 (green circles) and the DAQ system as a whole (blue squares) for the month of July. The Magenta lines show when h1fw0 and h1fw1 were modified. In the past 7 days, the only restarts of the framewriters are associated with complete DAQ restarts.

Matt, Lisa, Evan

Tonight we looked at the coherences between the OMC DCPD channels and ASC AS C, this time at several different interferometer powers. In the attached plots, green is at 11 W, violet is at 17 W, and apricot is at 24 W.

Evidently, the appearance of excess high-frequency noise in OMC DCPD sum (and the coherence of OMC DCPD sum with ASC AS C) grows as the power is increased. We believe that this behavior rules out the possibility that this is excess noise is caused by RIN in the AS port carrier, assuming that any such RIN is independent of the DARM offset and of the PSL power. Since the DARM offset is adjusted during power-up to maintain a constant dc current on the DCPDs, RIN in the AS carrier should result in an optical power fluctuation whose ASD (in W/rtHz) does not vary during the power-up. This is the behavior that we see in the null stream, where the constant DCPD dc currents ensure that the shot-noise-induced power fluctuation is independent of the PSL power.

I modified the ISC_LOCK guardian to revert the DOWN state back to being a 'goto'.  This allows you to select the state directly, without having to go to MANUAL.

The reason it had been removed as a 'goto' was because occaissionally someone would accidentally request a lower state while the IFO is locked, which would cause the IFO to go back through DOWN to get to the errantly requested state.  To avoid this I implemented some graph shenanigans:  I disconnected DOWN from the rest of the graph, but told it to jump to a new READY state at the bottom of the main connected part of the graph once it's done:

This allows DOWN to be a goto, so it's always directly requestable, but prevents guardian from seeing a path through it to the rest of the graph.  Once DOWN is done, though, it jumps into the main part of the graph at which point guardian will pick up with the last request and move on up as expected.

J. Kissel
WP 5395
ECR E1500230
II 1054

I've removed all redundant IPC Error EPICs channels from the top-lvel models of all SUS this evening. This is in accordance with ECR E1500230. The models compile, and have been committed to the SVN. They will be installed this coming Tuesday.

Once installed, this closes out the ECR and Integration Issue for LHO.

Something happened at lock loss and the IFO did not reach the defined DOWN state.

Symptoms:

• SUS: ITMX, ITMY, ETMY - driven and rung up, watchdogs tripped, I had to adjust the RMS limit on ETMY from 8000 to 10000 to reset watchdog
• SEI: ITMY ISI tripped - Jim rest
• IO: MC WFS left pushing optics to a bad place and needed to be cleared by hand
• PSL: ISS and FSS in oscillation - outer loop on while IMC was unlock, throwing inner loop into oscillation
• GRD: ISC_LOCK - red for a while, cleared after INIT

Dave, Jaime, Sheila, operators, and others are investigating.