TITLE: 09/11 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 3mph Gusts, 1mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

Sheila is here to asist wit the re-locking efforts following the computer crash.

TITLE: 09/11 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

• Whew!  What a night!  A bit of a rough night after the first hour!!  Thank you to Jamie & Jenne for helping from afar, Dave for taking a middle-of-the-night phone call, and Richard helping out with some final work  at EY to get us through the computer problem. 
• From 9:30-13:25 discovered and addressed issue related to h1boot1 crashing on us (see plethora of other logs from overnight).
• Unfortunately, during the last hour of the shift, after a nice Initial Alignment, H1 is having issues getting past ASC setps (it's locking in DRMI quick and fine though).  For the ENGAGE ASC locklosses it was at a time before the "saturation period" we get as we walk the spot through the ITMy point absorber.  :-/

LOG:

• 8:06 Lockloss (this was preceded by 17min with about 5-7 glitches.)
  • 8:06-8:32 LOCK#1 
    • DRMI flashing didn't look good, so went to CHECK MICH FRINGES (BS was off...mainly yaw) & then PRMI (PRM need a little yaw which I did, but then let ASC optimize).
    • Lockloss at RESONANCE
  • 8:33-8:59 LOCK#2
    • DRMI didn't lock & guardian locked PRMI to get DRMI to lock
    • Lockloss at ENGAGE ASC FOR FULL IFO

• 9:00-XXXX LOCK#3:  THIS IS WHEN h1boot1 issue began...in the middle of running an Initial Alignment
  • This is when the h1boot1 issues first started as we lost EPICS read/write capbabilities, now we transition to Observatory Mode to CORRECTIVE MAINTENANCE to address h1boot1.  After talking with Jamie/Jenne, there was nothing we could do, so now reached out to Dave.
  • Saw suggestion of taking OP to STOP & then EXEC (this was listed as a suggestion in ALIGN_IFO guardian log)...but this didn't appear to do anything.
  • Ran a LOAD do clear the fault.  then ran a DOWN again, and this brought back the error.  :-/
  • ~ 9:30utc Issues Begin:  For MICH BRIGHT, ALIGN IFO went into FAULT/red with a EZCACONNECTION ERROR (something due to not observeing effect of writing value to switch channel PRC1_P within a time out)
  • Initial Alignment run
• h1boot1 Recovery Begins  (see specifics from David's alogs)
  • 10:22(3:22amPDT): Left Message with David for h1boot1 recovery
  • 11:20-11:55 Rebooting front ends (sus, sei, isc) at EY (Corey)
  • 13:08 Richard off to EY to address communication issue between Corner & EY
  • 13:25 Dave calls to say we look all recovered as far as from the CDS side.  (my first order of business is Green EY....I don't see any flashes in that arm....Green EX has been locked).
  • Observatory Mode is remaining in CORRECTIVE MAINTENANCE until we get back to NLN.
• Restored sliders to 6:00 utc. via Patrick's script:  This immediately brought back flashing to Green ETMy!

• 13:40-13:59 Ran Initial Alignment (ran through all steps with no issues this time!)
• 13:59-14:10 LOCK#4
  • 14:10 Lockloss at  CARM TO TR (NOTE:  This is 1st lock after an Initial Alignment!)
• 14:10-14:23 LOCK#5
  • 14:23 Lockloss at ENAGE ASC FOR FULL IFO as LSC signals like PR GAIN all began to quickly nosedive.
• 14:25-14:41 LOCK#6
  • Paused at PREP_ASC for a minute or three.
  • 14:41 Lockloss at ENAGE ASC FOR FULL IFO as LSC signals slowly drifted down over the order of 30sec (did not see a growing Control Signal as an obvious culprit to address, via Jeff K's Control Signal Reduction script...but maybe I was not looking hard enough).
• 14:41 - 15:03 LOCK#6
  • Holding at PREP ASC FOR FULL IFO for about 7 minutes.
  • 15:03 Lockloss at ENAGE ASC FOR FULL IFO for a 3rd time!
  • Tried looking at ASC signals and maybe  both DHARD pit & yaw could be culprits, but PR Gain (& other LSC signals) all dropped very fast that using the Control Reduction Script would probably not help.  At a loss for what to try next...maybe another alignment?  This will be the gift I hand over to Ed.

FRS13571

Richard, Corey, Dave:

I disabled Dolphin EY port 4, Richard moved the RFM cable from port 4 to port 5 and we now have IPC traffic in both directions to EY. 

CDS overview is now all GREEN, looks like our recovery from h1boot1 crash is complete.

DQ Shifter: Vaishali Adya

Email: vaishali.adya@anu.edu.au

Mentor : Paul Altin

Fellow(s): Kara Merfeld

Summary (highlights) of the DQ Shift for an aLog:

• Hveto has picked out H1:LSC-REFL_A_LF_OUT_DQ from Wednesday to Sunday
• 511 Hz violin mode present prominently after all locklosses
• Very odd range drops that do not coincide with any PEM or BLRMS channels (that I have noticed). These range drops are sharp and turn on and off suddenly. On Thursday : 5 MPc (02:25 UTC), Friday : 5 MPc (01:20 UTC) and 06:30 UTC, Saturday : 5 MPc (19:48 UTC) and at 22:30 UTC.
• Overall duty cycle : 75.24 %
• 48 Hz line present prominently throughout the shift
• Lockloss study suggests : ADS glitches to be the reason of some locklosses during the shift (alog)
• VCO glitches strongly correlated to ground motion
• BRS - X temperature increase (alog)
• Hardware injections successful (alog)

Full report: https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20190902

h1cdsrfm is back up and running, long range dolphin is working except from EY to CS. I suspect a left over from July's problems, I'm working on this.

Also h1edc appears to have lost connection to 90 digital video channels?

Corey, Dave:

Corey power cycled h1susey, h1seiey and h1iscey. This cleared the DAQ errors with EX and corner station ISC. h1susey is running, both SEI and ISC IOP models have a negative IRIG-B excursion which should clear soon. In the mean time their DAQ and IPC status is bad.

After h1boot1 came back (took a few minutes, and FSCK was needed) the overview MEDM came back to life.

It looks like h1iscey is the only casualty, and it has taken the DAQ data down for all the other front ends which share its h1dc0 port (both ends, corner station ISC and SUSAUXB123).

I'm starting the h1iscey reboot process.

Looking around I can ping the front end machines but cannot ping the boot machine, Corey is taking a look at h1boot1's console to see if it needs rebooting.

[Edit:  Originally thought this was a Guardian Issue, but we later found out it was due to the h1boot1 computer.]

New behavior to me tonight occurred when attempting an INITIAL ALIGNMENT.  Everything was fine up until INIT ALIGN wanted to run the MICH BRIGHT step.  This step runs a DOWN command for ALIGN_IFO, but then in an early step of the DOWN code it repeatedly gets an connection error (for PRC1_P).  After this, ALIGN_IFO goes into a FAULT state & a continuous loop.  Below is the log:

2019-09-11_09:24:03.821637Z ALIGN_IFO REQUEST: MICH_BRIGHT_ALIGN
2019-09-11_09:24:03.822289Z ALIGN_IFO calculating path: DOWN->MICH_BRIGHT_ALIGN
2019-09-11_09:24:03.822719Z ALIGN_IFO new target: PREP_FOR_MICH
2019-09-11_09:24:06.469708Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:24:06.824697Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:24:06.830620Z ALIGN_IFO REQUEST: DOWN
2019-09-11_09:24:06.831255Z ALIGN_IFO calculating path: DOWN->DOWN
2019-09-11_09:24:06.831255Z ALIGN_IFO new target: DOWN
2019-09-11_09:24:08.924272Z ALIGN_IFO [DOWN.main] USERMSG 0: EZCA CONNECTION ERROR: Did not observe effect of writing value to switch channel ASC-PRC1_P_SW1R within EZCA_TIMEOUT (2.0s).
2019-09-11_09:24:08.949899Z ALIGN_IFO EZCA CONNECTION ERROR. attempting to reestablish...
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: State method raised an EzcaConnectionError exception.
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: Current state method will be rerun until the connection error clears.
2019-09-11_09:24:08.950576Z ALIGN_IFO CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume.
2019-09-11_09:26:16.909062Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:17.160830Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:26:29.469452Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:29.721182Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT
2019-09-11_09:26:42.038308Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP1_P => OFF: INPUT
2019-09-11_09:26:42.290377Z ALIGN_IFO [DOWN.main] ezca: H1:ASC-INP2_P => OFF: INPUT

....

I then gave up on ALIGN_IFO (I took INIT ALIGN to DOWN a while ago), and moved back to ISC_LOCK and ran a DOWN, but now it looks like it's having the same issue.  For isc lock it's listing a connection error...with a different channel & can't perform a down and is stuck in a loop.

Not sure what I can do at this point.......will keep investigating Guardian Land.  :-/

Marking this DOWN TIME as CORRECTIVE MAINTENANCE since I can no longer run an alignment...or even return to locking apparently.

Fil and I went to EX to look inside the BRSX enclosure in prep for the work next month. While we were there I touched up the centering of the BRS. While we had the box open, we must have disturbed the cable into the interface box inside, because when we got back to the corner station the BRS wasn't damping down. I went back down to look at it and it took a little to realize there is not retention on the ethercat cable into the interface box, so it's really easy to knock loose. It immediately started damping down when I pushed the ethernet cable into the interface chassis.

The BRS is still calming down, so we shouldn't use it for a little while yet. The operator should watch the the BRS health ndscope and wait for the BRSX RY OUT to look more like the BRSY RX OUT signal, i.e. the blue trace on the middle timeseries looks like the yellow(?) trace on that same plot.