TITLE: 12/02 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 73.9893Mpc
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.41 μm/s
QUICK SUMMARY:

• Somewhat quiet.  The useism has been slowly increasing over the last 10hrs.
• H1's range is hovering around 75Mpc & currently going on a 16.5+hr lock.  
• A2L was last run around 16hrs ago (~0:13utc).
• Quick Look at DARM Spectra:  Looks good.  Only places it's noisy by glancing at it (i.e. where I see reference) are:  
  • 10-70Hz (perhaps warranting an A2L?  Or just happen to see a glitch?  Or related to increased usesim.
  • A couple bumps between 300-350Hz

As reported in logs 32076 and 32079, HAM3 ISI Corner2 CPSs started glitching around 8am yesterday.  Looking this morning, given that they had glitched several times in just a few hours yesterday once they started and now have not reoccurred since power cycling the satellite rack and seating the probe gauge boards a few times, I'd say the issue has been suppressed for now.

Given that saturations are automatically removed after 1 hour, just looking at the HAM3 watchdog periodically is not sufficient to know if saturations have occurred.  If one wants to know if the platform has seen sensor saturations, trend:

H1:ISI-HAM3_WD_CPS_SAT_COUNT and

H1:ISI-HAM3_WD_CPS_SAT_SINCE_RESTART

The former channel will be reset upon WD trip, the latter channel will accumulate the saturation.

TITLE: 12/02 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC

STATE of H1: Observing at 75.0267Mpc

INCOMING OPERATOR: Corey

SHIFT SUMMARY: Quiet shift. Been locked all night. TCS hasn't been a problem. BRS thing also didn't happen again during the shift.

TITLE: 12/02 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 73.5385Mpc
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY: Mostly a quiet shift, but there were a couple SDF issues
LOG:
Corey had just relocked after an earthquake when I arrived. Shortly after going to OBSERVE the TCS guardians knocked us out, as Kiwamu logged. Then quiet until just a couple minutes ago. SDF_DIAG kick us out of OBSERVE, looking at the log I find:

2016-12-02T07:51:08.18971 DIAG_SDF [RUN_TESTS.run] USERMSG 0: DIFFS: ngn: 1
2016-12-02T07:51:10.70676 DIAG_SDF [RUN_TESTS.run] USERMSG 0: DIFFS: ngn: 3
2016-12-02T07:51:17.18839 DIAG_SDF [RUN_TESTS.run] USERMSG 0: DIFFS: ngn: 1

I can't find an (the? I didn't know we had one) NGN guardian, but I know where the CS_BRS guardian lives. When I looked at that guardian, it had just started a recenter cycle at the same time:

2016-12-02T07:51:07.99681 CS_BRS JUMP target: TURN_OFF_DAMPING
2016-12-02T07:51:07.99694 CS_BRS [RUN.exit]
2016-12-02T07:51:08.05901 CS_BRS JUMP: RUN->TURN_OFF_DAMPING
2016-12-02T07:51:08.05920 CS_BRS calculating path: TURN_OFF_DAMPING->RUN
2016-12-02T07:51:08.05959 CS_BRS new target: TURN_OFF_PZT_CTRL

...

Did the CS_BRS guardian throw an SDF difference in NGN that dropped us out of OBSERVE?
 

Sheila and Jenne informed me that the TSC ITMY CO2 guardian suddenly had changed its state this evening, kicking us out of the observing mode

I trended some relevant data and checked the guardian logs. It seems that since the upgrade of the guardian machine this morning (32072), the TCS CO2 guardians for both X and Y arms unlocked multiple times for some reason. The first attachment is the trend of some relevant channels for this past 12 hours. As shown in the upper panels, the guardians changed their states multiple times in the past hours due to the lasers unlocked. Reading the guardian logs (which are attached) and codes, these events were indentified to be due to the PZT output signals exceeding some thresholds that are set in the guardian codes. This had not happened in the past four or five days until this morning. The number of incidents seems to decrease as a function of time. I am hoping that the CO2 lasers will eventually settle to some kind of stable point in which the guardians don't need to change the state any more.

For operators,

If this happens again (which should be obvious as it flips the intent bit back to commissioning), please follow the following procedure.

• Determine whether if the TCS ITMX(Y) CO2 guardian changed the state.
  • The easiest would be to look at the log of the IFO guardian (accessible at the very top of GUARD_OVERVIWE.adl)
  • If it's not TCS. The below doesn't apply.
• Record the time of the incident so that the time stamp can be noted in the operator's summary log.
• Wait for the TCS guardian to come back to the nominal state.
• Once the TCS guardian is back to nominal, hit the intent bit and go back to observing.

TJ investigated why the LOCKLOSS_SHUTTER_CHECK guardian was sometimes mistakenly identifying locklosses when there had not been any lockloss while ISC_LOCK was in the DRMI_ON_POP state.

As a reminder, the only purpose of LOCKLOSS_SHUTTER_CHECK is to check that the shutter triggers after locklosses in which we had more than 25kW circulating power in the arms.  The lockloss checking for this guardian is independent of any other guardian.  

The problem TJ found was a consequence of the work described in 31980 .  Since that work when we switch the whitening gain on the transmon QPDs, there is a large spike in the arm transmission channels which the LOCKLOSS_SHUTTER_CHECK guardian recognizes as a lockloss (TJ will attach a plot).  

We edited the ISC_LOCK guardian to hold the output of the TR_A,B_NSUM filter before making the switch, and turn the hold off after the switch is done.  We loaded this when we were knocked out of observe by TCS.  This is a simple change but if operators have any trouble with DRMI_ON_POP tonight you can call TJ or I.  

TITLE: 12/01 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Jim
SHIFT SUMMARY:

Much_Of_Morning Main Issues:

1. ETMy 4735Hz Violin Mode Harmonic damping!  Upgraded Filter Banks for ETMy & damped out the mode(!)
2. HAM3 ISI Saturation & Trips.  Hugh noted, addressed, & appear to be better so far.

LOG:

• 16:25(?) HAM3 ISI tripped while in PRMI; this is unusual.
• 16:35-16:55 Down for Guardian Memory Upgrade (WP6366)
• 19:03 Mid-station visit (Karen)
• 21:57 OBSERVING!
  • NOTE:  Have been getting DIAG_MAIN message about the FAST Shutter
  • 22:08 Lockloss, nothing obvious from Seismic or StripTools.  It was NOT HAM3ISI.
  • Ran Lockloss select which brought up mainly ASC Channels, but I did not see anything obvious.
• 22:39 OBSERVING
  • 23:22 Incoming Peruvian EQ, Transitioned EX & EY ISIs to "250_SC_EQ" state
  • 23:25 Lockloss
• 0:15 OBSERVING
  • Jim ran another A2L with his new medm/script interface before going to Observe.

Locking Notes:

After hand-off this morning, held at VIOLIN_MODE_DAMPING.  Kiwamu came in to take a look at H1 & wanted to note a few items he checked.

• ETMy:  SUS L3 outputs saturated when trying LOWNOISE_ESD_ETMY:  The Suspect was:  higher order violin mode harmonic at ~4753Hz.
• For next lockloss attempt, Enabled FM3 for LSC DARM2 & Made it to NLN, then made first damping attempt of 4753 Mode.
• A2L run, but later received "OMC DCPD Saturdations"
  • Jenne:  OMC_Lock:  REMOVE_WHITENING state & then back to READY_TO_HANDOFF
• Lockloss when trying to take out phase of MODE9....I should have ramped gain down, changed phase, & ramped gain back up. 
• HAM3 trip while going up to REDUCE CARM OFFSET
• Spent a few hours damping 4753Hz in DC READOUT (see other's alogs)

USGS Link

• Magnitude of EQ (according to Terramon):  6.3
• Location of EQ (according to Terramon):  43km NE of Huarichancara, Peru
• Starting time of Event:  22:49UTC (started seeing 0.03-0.1 BLRMS increase
• Lock Status:  H1 Lockloss at 23:25UTC

J. Kissel, J. Driggers, C. Gray 

We've successfully damped the 4735.09 [Hz] violin mode harmonic. The key: the +/- 60deg phase filters in the ETMY_L2_DAMP_MODE9 bank, which we thought were moving the damping control signal phase around the unit circle, was actually not adjusting the phase at this 4.7 [kHz] since they were tuned for the 2nd harmonics at 1 [kHz] (thanks to Jenne who dug into foton to check that these filter made sense). This left us with essentially only 0 [deg] (+ gain) or 180 [deg] (- gain) as options, and neither worked well. 

After rearranging some filters between MODE9 and MODE10 filter banks, Jenne was able to create new phase adjustment filters with 30 [deg] increments for 5 [kHz]. +60 [deg] with a positive gain worked well for the first two orders of magnitude, but we eventually needed to nudge by -30 [deg] once other modes around these frequencies began to be resolved, confusing the error signal. Thus, the final settings that we think will work well:
MODE10: 
gain = +0.02   (we were able to use up to +0.1 while we were trying hard)
FM4       ("100dB")        gain(100,"dB")
FM9       ("+30deg5k")     zpk([0],[28.5238+i*4744.41;28.5238-i*4744.41],1,"n")gain(2.07488e-06)
FM10      ("4735")         gain(100,"dB")*butter("BandPass",4,4734.5,4735.5)gain(120,"dB")

I think all the success I claimed yesterday () was merely by turning on the notch filter in the DARM path and waiting.

I've updated the LHO Violin Mode Wiki, and I've also updated the ISC_LOCK guardian code to ensure this continues to get turned on (specifically I edited the gen_VIOLIN_MODE_DAMPING functional state in the ISC_GEN_STATES.py subfunction that creates VIOLIN_MODE_DAMPING_1 and VIOLIN_MODE_DAMPING_2 in the acquisition sequence).

Unclear if related to Cheryl's reported problems with the fundamentals over night (LHO aLOG 32059). Note the Violin Mode Wiki or the ISC_LOCK guardian have been updated with her changes.

See the first attached, showing the ISC lock state (Ch16) dropping as the HAM3 ISI WD (Ch1) trips to State3 (Damping only.)

All the CPSs show a twitch and shift as the DC coupled ISO loops disengage but note the scale of the glitch on H2 and V2 CPSs.  What is curious to me is how the C2 sensors clearly exceed saturation level (20000) but the SAT_COUNT (Ch2) does not count them.  Ahhh, I think that any 'trip' of the watchdog, even if only to damping only, will clear the saturations.  I think we should change this in the code although I think someone coded it explicitly this way (BTL/HRAP.)  I think this hinders us though and maybe we can rework it.

I decided to Power Cycle the CPS and while off reseated the corner2 Gauge Boards in the satellite rack.  Did this several times.  ISI reisolated without difficulty.

Attached as well is another view a few hours ago of this glitching dropping the IFO out of some state better than zero.  Prior to the ISI trip is another glitch on the Corner2 CPS that was not enough to rile things up.

Finally, a 7 hour trend capturing all the recent occurrances of this.  Prior to this, 22 Nov saw a CPS saturation and trip but that was a Tuesday.

All BSC spectra looks pretty normal. Noise below 20 Hz is due to different platforms behaving differently.

HAM spectra looks fine above 20Hz except for some non-consequential bumps between 20 and 40Hz in HAMs 2&5.

Jenne, Sheila Keita

We had another instance of a jump in POP90, in which both the I and Q phases increased. We think this is a problem with the readback, similar to what is described in 31181.  

We were acquiring lock, and had no ASC running on the SRC.  We looked at witness sensors for all the interferometer optics, and it looks like none of them moved at the time.  We also don't see changes in other RF sensors, like AS36, AS90, or POP18. We looked at both quadratures of POP90 before rotation and it seems to have both a phase shift and a 3.5 dB increase in the sum of I+Q. The RFmon and LO mon on the demod don't have any jumps nearly that large, so if it is a problem in the demod it is probably downstream of the directional coupler for the RFmon. 

This seems not to be the same as the jumps in SRC alingment that started after last tuesday's maintence, (31804 31865 and other alogs), but since the symptom is very similar it would make debugging the other problem easier if this issue could be fixed. Since we use POP90 for a dither lock of the SRM angle durring lock acquisition, this can cause a lockloss if it happens while we are trying to lock. 

I hit load and cleared the error. But I have no idea about the state of the injection since it didn't happen the first time. Looks like it will try again at GPS 1163938117?