Upon my arrival (at 7:45 am PT) in the control room I found the IFO locked and in "Observation" mode with the intent bit set.  The Lock Clock showed 8:50 hours and counting.  The input power was 24.3 W.  Camera images looked stable and as expected. (Because the video5 computer had frozen! argh).  DARM ASD looked terrible, but not unreasonable for the operating condition of the IFO.  ISC_Lock Guardian was in Nominal Low Noise.

This is a better start of the day then we have had for awhile. This should allow the ER9 testing of the hwinj, cal, and data pipelines and, with the intent bit set, allow the analysis streams to test their readiness.

15:13 UTC (8:13 PT) - just lost lock. Morning activities?

Patrick, Sheila, Carl and Ross damping PIs

After Keita's ISS fixes we had a few hours of bad weather and earthquakes that contributed to dificulty locking.  Once things calmed down we locked and sat at several stages to see if the lock was stable, DC readout, 25 Watts, ISS 2nd loop engaged, and now we have made it to some kind of "nominal low noise" state.  Patrick hit the intent bit.  

There are several things wrong with this spectrum, some of which could be solved by going back to 40 Watts in the morning:

• We skipped the coil drivers state, since we don't expect it to help our noise and we lost lock in this state earlier. We don't see any reason not to try doing this tomorow though. 
• There is a lot of intensity noise in DARM.  (1st attached screenshot shows 2nd loop PDs before ISS was on 6:40 UTC July 8th, and after the 2nd loop is engaged at 7:30 UTC. The 2nd attachement shows that there is still a lot of coherence of intensity noise in DARM.)
• The CO2 guardians were in preheating mode for the first bit of time we had the intent bit set (this is a result of not thinking through all the consequences of changing the nominal power down to 25 Watts).  We set the CO2 guardians to NOMINAL_LOCKED_CO2 level, which means that they are now at the settings that are appropriate for 40 Watts I think, which is probably closer to the right settings for 25 Watts than preheating at least. 
• Our low noise hard ASC loops were designed for 40 Watts, so the decision to go to 25 Watts means that we would need to do some measurements and loop design work to have a low noise state for 25 Watts. 
• We probably have different spot positions on our optics now than we would at 40Watts, so we would need to run a2l for 25 Watts if we want low noise.  
• DIAG main is complaining that the POP beam diverter is not closed, this is because we are still locking on the POPX WFS, this will be open throughout ER9.
• The OMC alingment is clearly off, on the camera it is on the left half of the screen. 
• The DCPD whitening is off, which makes our high frequency noise worse.

The last attached screenshot is just to show how many locking attempts everyone made in the last 14 hours. The range displayed by the DMT viewer in the control room seems to have a problem, it should be stable at some low range but on the DMT viewer it looks like we lost lock in the last half hour.  

There has been something wrong with the ETMY oplev for quite a while now, it says that the optic is moving a lot more than the other test masses, swinging around by more than half a micro radian.  This must be false, and needs some investigation.  

after 8 hours of writing a complete set of framed data with no errors, h1fw0 went unstable again at 10pm PDT. I have reconfigured it to stop writing commissioning frames, and it has been running for 20 mins. We will leave it in this configuration overnight.

The lockloss from the state REFL_POP_WFS which happened at 23:56:29  UTC July 07 2016 is very similar to what is described in alog 26840 and comments.  I've attached the guardian logs for both ISC_DRMI and ISC_LOCK, and a plot of the lockloss.  

The first thing that happens is that the ISC_LOCK gaurdian starts to transition our front end triggering for the LSC from POP18 to POPLF, by lowering the thresholds, sleeping 0.1 seconds, and then setting the trigger matrix elements to 0.  

The ext_alert process on h1fescript0 which alerts the control room of Gamma Ray Burst and Supernovae events has stopped running because its robot certificate (needed to query GraceDB) expired on 27 June 2016. We are in the process of renewing this certificate.

Yesterday at 13:40 and at 22:40 PDT the CW stopped exciting the EX PCAL because its testpoint was cleared from awgtpman. The excitation slot in the awg remained even though the excitation was not active, so the CW sender (psinject process on h1hwinj1) was unaware that its excitation was no longer operational and did not try to restart. When this happened the CDS overview screen did show a red excitation error for h1calex (meaning that an excitation should be present but is missing) and the operator medm overview should have shown a missing CW error as well.

The problem was found to be a "feature" in diag which allows the user to clear every test point on every model with one command. The feature was accidentally acitvated yesterday when the dcu-id and the testpoint were interchanged in the tp clear command.

Jim is working on removing this feature from the current diag code to prevent this accidentally happening again during ER9.

The reason why the IFO second loop didn't like the ISS 2nd loop is because the ISS 2nd loop offset is now hard coded (28076) while the power and diffraction both changed because of the diode swap and subsequent tune-up.

I remeasured it at 40W (H1:PSL-ISS_SECONDLOOP_REF_SIGNAL_ANA=-0.027689945 for 16-17% diffraction) but Vern told me that it was decided that we'll run IFO at 25W during ER9, so I measured it yet again at 25W (-0.027651985).

I'm puzzled that the number changed this much (previously it was -0.9826934814453125 at 40W).

It's also odd that there's not much difference between 40W and 25W. The diodes are definitely connected, the readback of analog sum (H1:PSL-ISS_SECONDLOOP_PD_14_SUM_OUT) is almost 100% coherent with the digital sum of individual PDs (H1:PSL-ISS_SECONDLOOP_SUM14_AC_OUT and H1:PSL-ISS_SECONDLOOP_SUM58_AC_OUTPUT).

Anyway IMC_LOCK guardian was modified for 25W (H1:PSL-ISS_SECONDLOOP_REF_SIGNAL_ANA, iss_diffracted_power_target=16.5).

Update:  I was likely tricked by MEDM screen (graphics of switch states sometimes don't agree with reality) when I was doing the above measurement with only MC locked.

With full IFO the above offset was found totally off, and at 25W H1:PSL-ISS_SECONDLOOP_REF_SIGNAL_ANA=-0.5885 or so for 15%-ish diffraction. Didn't have time to remeasure at 40W.

We were showing Dan the h1fw0 problems (after turning on all frame writing again) and after less than an hour h1fw0 became stable again and has been running for 2+ hours. We will leave it in this state for now, but if it becomes very unstable again we have several options:

Turn off LDAS frame comparison jobs (reads every frame on both writers)

Remove all LDAS archiving from h1fw0 over to h1fw1

Reduce h1fw0 back to only writing science frames

Fingers crossed we don't have to do any of these.

Dan confirmed how the frame files written by CDS are currently accessed by LDAS for archival and checking, summarized below.

h1fw0 writes to the ldas-h1-frames file system (located on SATABOYs in the warehouse)

h1fw1 writes to the cds-h1-frames file system (located on SATABOYs in the MSR)

h1fw0/ldas-h1-frames

science frames are copied over to LDAS every 64 secconds

science frames are read every 64 seconds to compare them with h1fw1's frame

commissioning frames are read every 64 seconds to compare them with h1fw1's frame

second trend files are copied over to LDAS archive area every 10 minutes

second trend files are read every 10 minutes to compare them with h1fw1's frame

minute trend files are copied over to LDAS archive area every 10 minutes

minute trend files are read every 10 minutes to compare them with h1fw1's frame

Started and ran the purge-air skid, Turbo, QDP80 and newly added scroll pump.  Tested the functionality of the Turbo's Safety Valve with the control cable connected to the scroll pump's relay box - demonstrating that the Safety Valve closes upon the loss of scroll pump motor AC and thus mimicking the "QDP80 Running" 
signal.

Note: The purge-air skid has developed some problems from lack of use over these past few years, namely the radiator fan never came on while running the compressors, the drying tower never cycled and the low pressure alarm never sounded.  These features worked fine when last this unit was run.  Now they don't.  Also, the Turbo spun-up normally even with the locally mounted scroll pump running (new vibration source) but tripped into EMERGENCY OPERATION upon BRAKING and when at ~1/3 NORMAL rpm.  This behavior is seen on other Turbos (XBM for example) and might be an age related issue?  

I am leaving the Turbo energized overnight to ensure that the rotor is completely stopped.  I will de-energize it tomorrow.  

State of H1: locked at 25W, but no ISS 2nd loop

Activities:

• Peter King - PSL - FSS tuning
• Kyle - MX - vacuume work
• Robert - electronics room
• JimW - HAM2 blend filter update

Glitches:

• PSL AC left on - now off
• Confusion over a guardian state - lock loss
• ISS 2nd loop is not working - currently being worked on

Special Instructions:

• SRM human servo - found on ASC medm tab, SRM POP90 - use to keep the signal low and smooth (offsets in guardian were for 40W, but Sheila fixed by taking them out)

[Everyone]

We have determined (a few hours ago) that flipping the ESD bias sign on ETMX does not allow us to lock ALS DIFF.  Jeff has looked through the settings, and everything is flipped to match the bias sign as appropriate, but we're still not able to lock.  It looks like a crossover is unstable, or something like that.  For now, we have reverted the ETMX ESD bias to its pre-Tuesday state, which has facilitated locking.  Team SUS will look into this later.

The ISS is much better behaved now that the alignment work was done.  However the FSS gain change was making the IMC loop very nearly unstable, and we lost lock during the power-up twice due to this instability.  Sheila and Keita will post their measurements that discovered and fixed this issue.

We have so far been able to power up to 40W once, but it looks like a PI rang up pretty quickly.  Carl will post details, but this is a new mode that hasn't been a problem previously, so it does not yet have damping settings.

At this time, it looks like there is no problem in getting to 40W, and other than PI damping we should be able to get all the way to low noise.

If Carl is unable to find damping settings relatively quickly, we may choose to instead only go to 20W for tonight, so that we can get to low noise and set the intent bit.  Stay tuned for more updates...