Wednesday 12jun2019 - Sunday 16jun2019: No restarts

Monday 17jun2019

2019_06_17 19:00 h1alsey
2019_06_17 19:00 h1caley
2019_06_17 19:00 h1iopiscey
2019_06_17 19:00 h1iscey
2019_06_17 19:00 h1pemey

Kernel crash of h1iscey

Tuesday 18jun2019 - Monday 24jun2019: No retstarts

Tuesday 25jun2019:

2019_06_25 15:06 h1sysecatc1plc2sdf
2019_06_25 15:09 h1sysecatc1plc4sdf
2019_06_25 16:53 h1sysecatx1plc2sdf
2019_06_25 17:00 h1sysecatc1plc1sdf
2019_06_25 17:00 h1sysecatc1plc3sdf
2019_06_25 17:00 h1sysecatx1plc2sdf
2019_06_25 17:00 h1sysecaty1plc1sdf
2019_06_25 17:00 h1sysecaty1plc3sdf
2019_06_25 17:01 h1sysecatx1plc2sdf

Tuesday maintenance. Most Beckhoff systems were restarted, which in turn crashed some of their respective SDF systems.

TITLE: 06/27 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 9mph Gusts, 7mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.04 μm/s
QUICK SUMMARY: 8 hour lock, EQ has subsided.

TITLE: 06/27 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
LOG:

There was even a second and third "uptick" in the ITMY Z-axis eq BLRMS AFTER Seismon reported the R3.5 waves had come and gone. X reached 1micron; Y reached 2; Z reached 3+ microns. Ride 'em Cowboy! We sure got them "8 seconds"!

12:13 UTC manually ended GRB Standown - INJ_TRANS to INJECT_SUCCESS

This one is a FERMI - Latency 7.06 - Trig Dur .512 . Extending Stand Down. EQ creeping in.

Swift alert - Trig duration is exactly 2.048

Initiated INJECT_KILL and Standing down for at least 15 minutes. EQ may get us. We'll see (uSeiesm is really low)

11:12 received 2 alerts. Sesmon says 6.6 and 6.4 magnitude.

09:31 Intention bit commissioning - squeezer must have re-locked itself while I was in the lbr.

09:34 Intention bit set back to Observing

TITLE: 06/26 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Calibration
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

First couple hours we were down for Calibration.  Then we were down for another couple hours for a big wind storm + ALSx issue. 

4th day of earthquake activity & it's getting a little painful.  Tonight, the EARTHQUAKE state helped us ride through the EQ, but tonight we had Tidal Errors which drove multiple signals and prevented me on correcting matters by taking us back to the nominal WINDY state.  Not sure what is different tonight, because this my first time seeing these signals rail in O3.

Crossing fingers for no EQs!!
LOG:

• Starting off with Calibration work (cont.)
• 0:50 incoming storm!  Lightning, thunder, & gusts up to 40mph
  • 1:12:00 Thunder (preceded by lighnight
  • 1:12:37 lightning
  • 1:15:03 lightning
  • 1:18:04 lightning
  • 1:20 Winds up to 60mph!  H
• 0:57 OBSERVATORY Mode switched from Calibration to Lock Acquistion momentarily
• 1:01 Back to OBSERVING!
• 1:19 lockloss
• 1:23 MORE_WINDY transition!!  (gusts over 70mph!)
• 1:25 While H1 down, Laser vibrometer moved from BSC2 to BSC3 (Robert and crew)
• 1:28 Noticed ALSx spot has disappeared!
• 3:34-4:15 Initial Alignment started
  • Align BS failed the first time and ALIGN_IFO went back to SET_SUS_FOR_ALS_FMPI, so I reselected MICH_BRIGHT_OFFLOAD for ALIGN_IFO.  But it froze at ACQUIRE_MICH_BRIGHT.  Went to DOWN & tried it again, but it froze again.  Went for "usual" MICH_DARK_LOCKED.
  • BS was way off in Pitch because ALIGN_IFO was stuck at ACQUIRE, but pitched it enought to get to LOCKED & did the usual tweaking to optimize BS.
  • Returned to the new INIT_ALIGN node to see if I could take it to SRC_ALIGN_OFFLOADED, but it was still stuck with MICH_BRIGHT_ALIGNING....
  • So, ignored INIT_ALIGN, and went back to using ALIGN_IFO.
  • and went to SRC_ALIGN_OFFLOADED
• 4:15 Locking begins.  First lockloss was in DRMI acquiring.
  • 4:57 LSC tidal X&Y Error
• 4:29 5.4 Russsia EQ on VERBAL with EARTHQUAKE mode recommendation...since waiting for drmi, I went to EARTHQUAKE mode.  This seemed to increase the CPS Feedforward & also be related to the Tidal Errors noted above.
• 5:51 LSC tidal X&Y Error again.  Eventually lose lock due to this.
• 5:56 Lockloss---I immediately switched us back from EARTHQUAKE back to WINDY!!
• 6:50 Back to OBSERVING (in under an hour no less)!  This time all of our signals on the Tidal striptool look much flatter!  (compared to that last lock where I had the incoming EQ!)
• 7:00 Handing off to Ed.

TITLE: 06/27 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:

After the storm and ALSx issue, ran an alignment, went for locking & finally back to Observing.

NOTES: 

• 4:29 During locking (of course) there was an incoming earthquake.  Took SEI_CONF to EARTHQUAKE state while waiting for DRMI.
• 4:57 LSC tidal X&Y Errors received via Verbal (could see ETMy Drive Align hitting rails on the Tidal strip tool...see 1st attachment)
• Made it to NLN, but had the same DIFFS we've been having since Tues Maintenance.

Commissioners & Jason:  Should We UNMONITOR these? (see 2nd attachment)

• ASC changes during locking every time (not sure what is different here...did we start monitoring this all of a sudden?
• PSL ISS diff is a 10^-14 difference Jason mentioned we'd get after Maintenance.
• Have been ACCEPTING these since Tuesday

Went to Observing

• 5:51  LSC tidal X&Y Errors received via Verbal again.  Once again ETMy Drive Align was flat-lining at a rail.  This caused the lockloss.  
  • Before lockloss, I was watching Jim's new signals for transitioning SEI_CONF from EARTHQUAKE to WINDY, but they remained in the red the whole time, so I worried we'd lose lock.  We did lose lock before I could transition.  
  • NOTE:  LSC-CPSFF_INMON was showing a large oscillation while in the EARTHQUAKE state.  As soon as I transitioned to WINDY, this signal became small. (see 3rd attachment).

After the storm was passing through, ALS x & y were obviously flashing all over the place.  Unfortunately, within a few minutes the green spot for ALSx disappeared (and we had the error:  "ReflPD A; Green input align").

1)  Looking at ETMx, TMSx, & ITMx

Saw shifts for all of these guys, so I returned them to where they were during the lock, but no change for the spot on video.

2) ENDx Overview:  No spot for Green WFS (& offcenter spots for Green QPDs)

3) PDH Locking screen: 

• XALS Laser Green Mon is OK with a power of 1.2mW
• Xarm Green ReflA has an ERROR note & 0mW----When I click this the Error says "Power too low"

Attached is a screenshot of the issue.  Since the power is low I'm wondering if this is still an alignment issue & I just need to sweep optics to get something on the camera.  But I don't really know where to start and where to go.  This is something I can try next.

For the GREEN WFS, I'd prefer to not touch picomotors to get a spot on them, but I'm wondering if ETMX/ITMx/TMSx alignment sweeps could help this?

...........Oh Darn........

As I was going through the Troubleshooting wiki, there is mention of:  If one has symptoms as above (i.e. no power for XARM Green ReflA), one should check the shutters.  I just saw that the ISCTEX green shutter was CLOSED.  I opened it and I atleast now see light (ugly light, but light).  Why would this shutter close on me???

OK, now that I'm not blind, going to try and find the spot for Green X-arm!