H1 back to Observing (as well as L1 & V1). All is quiet seismically (although winds have been picking up slightly since shift change this morning).
Tuesday afternoon at 3pm PDT I gave an overview of the programs I had written to help managing those SDF setpoints which are not being monitored and have not been controlled by Guardian ( wiki-page ). The det-eng and comm teams reviewed some subsystems after the meeting. I regenerated the unmonitored list again at 10:37 PDT today and compared with yesterday's list (at 11:09). Shown below are the summary statisics for those systems which have been changed (listed in order YESTERDAY then TODAY). I've colour coded changes as GREEN = less unmonitored, YELLOW = more unmonitored.
h1isibs std-mon 1658 std-umon 48 flt-mon 3826 flt-umon 270 std_pc 2.81 % flt_pc 6.6 %
h1isibs std-mon 1656 std-umon 50 flt-mon 3826 flt-umon 270 std_pc 2.93 % flt_pc 6.6 %
h1isietmy std-mon 1727 std-umon 67 flt-mon 3506 flt-umon 798 std_pc 3.73 % flt_pc 18.5 %
h1isietmy std-mon 1725 std-umon 69 flt-mon 3590 flt-umon 714 std_pc 3.85 % flt_pc 16.6 %
h1isiitmx std-mon 1686 std-umon 20 flt-mon 3808 flt-umon 288 std_pc 1.17 % flt_pc 7.0 %
h1isiitmx std-mon 1684 std-umon 22 flt-mon 3808 flt-umon 288 std_pc 1.29 % flt_pc 7.0 %
h1pemcs std-mon 194 std-umon 12 flt-mon 768 flt-umon 48 std_pc 5.83 % flt_pc 5.9 %
h1pemcs std-mon 170 std-umon 0 flt-mon 672 flt-umon 0 std_pc 0.00 % flt_pc 0.0 %
h1pemey std-mon 126 std-umon 12 flt-mon 496 flt-umon 48 std_pc 8.70 % flt_pc 8.8 %
h1pemey std-mon 138 std-umon 0 flt-mon 544 flt-umon 0 std_pc 0.00 % flt_pc 0.0 %
h1psliss std-mon 391 std-umon 0 flt-mon 1043 flt-umon 13 std_pc 0.00 % flt_pc 1.2 %
h1psliss std-mon 391 std-umon 0 flt-mon 1056 flt-umon 0 std_pc 0.00 % flt_pc 0.0 %
h1susprocpi std-mon 3811 std-umon 47 flt-mon 5558 flt-umon 330 std_pc 1.22 % flt_pc 5.6 %
h1susprocpi std-mon 3858 std-umon 0 flt-mon 5574 flt-umon 314 std_pc 0.00 % flt_pc 5.3 %
h1sussr3 std-mon 801 std-umon 2 flt-mon 2016 flt-umon 0 std_pc 0.25 % flt_pc 0.0 %
h1sussr3 std-mon 802 std-umon 1 flt-mon 2016 flt-umon 0 std_pc 0.12 % flt_pc 0.0 %
h1sussrm std-mon 749 std-umon 19 flt-mon 1840 flt-umon 0 std_pc 2.47 % flt_pc 0.0 %
h1sussrm std-mon 766 std-umon 2 flt-mon 1840 flt-umon 0 std_pc 0.26 % flt_pc 0.0 %
h1tcscs std-mon 801 std-umon 60 flt-mon 2538 flt-umon 102 std_pc 6.97 % flt_pc 3.9 %
h1tcscs std-mon 818 std-umon 43 flt-mon 2538 flt-umon 102 std_pc 4.99 % flt_pc 3.9 %
Attempt #1:
These are notes from first locking attempt (thought I'd give it a go before trying an alignment).
Attempt #2:
Giving it a try on 2nd lock and going straight for ACQUIRE_DRMI_1F, and made it here with no issues. At this point, I was wary if I could tweak anything to help prevent POP18 being driven off by ASC. Decided to go for it.
Once again, ASC scarily drove POP18 down (& POP90 up), but this time ASC turned them around and we survived.
Continuing with LOCKING! (Stay Tuned!)
Topped off crystal chiller with 50mL of water. Diode chiller and filters were OK.
Just to give a status of where we currently are.
IMC_LOCK log:
2019-06-12_15:22:54.206925Z IMC_LOCK [ACQUIRE.run] waiting for lock...
2019-06-12_15:22:54.207585Z IMC_LOCK [ACQUIRE.run] ezca: H1:SYS-MOTION_C_SHUTTER_A_OPEN => 1
2019-06-12_15:22:54.208201Z IMC_LOCK [ACQUIRE.run] ezca: H1:IMC-REFL_SERVO_IN1EN => 0
2019-06-12_15:22:54.583894Z IMC_LOCK [ACQUIRE.run] timer['IMC_offset'] done
2019-06-12_15:22:55.211545Z IMC_LOCK [ACQUIRE.run] ezca: H1:IMC-REFL_SERVO_IN1EN => 1
2019-06-12_15:22:55.212340Z IMC_LOCK [ACQUIRE.run] timer['IMC_buttkick'] = 5.0
2019-06-12_15:22:55.213292Z IMC_LOCK [ACQUIRE.run] ezca: H1:SUS-MC2_M1_DRIVEALIGN_L2L_SW1 => 8
2019-06-12_15:22:55.464426Z IMC_LOCK [ACQUIRE.run] ezca: H1:SUS-MC2_M1_DRIVEALIGN_L2L => PRESS: OFFSET
2019-06-12_15:22:55.464968Z IMC_LOCK [ACQUIRE.run] timer['IMC_offset'] = 15.0
2019-06-12_15:22:55.517140Z IMC_LOCK [ACQUIRE.run] waiting for lock...
h1ecatc1 C1_PLC1_Device3 is under reporting its slave count (actual = 156, configured = 291).
We lost the units around 13:20 UTC (06:20 PDT)
Connection between corner chassis 4 L0 and M0 has failed. The first red block in the topology screenshot is corner chassis 4 M0 (EK1100).
Patrick and Corey are working the problem.
The time for the error Dave mentions above coincides with the H1 Lockloss (and issues with IMC) at 13:21utc (6:21amPDT).
Patrick has tracked this down to the Corner Chasis 4 in the CER and says this may need to get pulled. Fil has been notified and will work with Patrick.
For something like this which might have taken H1 down, should this be a Verbal Alarm? (only way an operator would know about this issue is to happen to catch the red box on the CDS Overview (which we both didn't notice).
[Attached is the RED box on the CDS Overview.]
FRS Ticket submitted: #13049.
https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=13049
So far down time is 3.5hrs, but now I need to try locking.
Side Note: Alog would not let me link the FRS ticket to the text of "13049" above. 
EK1100 EtherCAT Coupler failed and replaced. Unit was scanned and communication errors have cleared. We did noticed a buzzing sound coming from one of the terminals. Tried various combinations of powering off different components to isolate noisy terminal(s), but decided to reinstall unit to get IFO back up. Corner 4 Slow Controls Chassis S1107450 F. Clara, P. Thomas
On the CDS Overview MEDM, if there is a Beckhoff hardware error the slow controls banner will turn RED for better visibility. I simulated a C1PLC1 error to demonstrate (see attachment).
All in all this cost us about 5 hours. See e.g. https://ldas-jobs.ligo.caltech.edu/~detchar/summary/day/20190612/ .
This FRS ticket (13049) is now CLOSED/RESOLVED.
TITLE: 06/12 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
Wind: 7mph Gusts, 4mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY:
IMC is not locked. Got a rundown of what Jim attempted to the issue which happened during the last 90min of his shift. Switched OBSERVATORY_MODE from Locking to CORRECTIVE MAINTENANCE, since this looks like a lockloss/downtime which is out of the norm.
OK, going to get settled and start looking into this and also wait for reinforcements!
TITLE: 06/12 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Busted, IMC isn't locking
LOG:
9:30 TCS knocks us out of observing, but resolved itself so fast I couldn't see what the change was
13:20 Lockloss, IMC is not locking, I tried our troubleshooting guide, but no luck so far
Work will begin shortly on the LHO aLOG upgrade. Please save any open drafts so that they are transferred over to the new server. WP#8237
Testing L-mail on new server.
Testing uploads.
Upgrade work is complete.
One brief note on the post editor: if you are having trouble seeing the rich-text editor, please force-reload the page (one of: Ctrl-R, Shift-Ctrl-R, Command[Apple]-R, Shift-Command-R, or similar depending on your operating system and browser) to retrieve a new copy. This is caused by the browser aggressively caching a copy of the old version of the editor.
To report problems, please use the FRS system and categorize the report under LLO Operations, LLO GC, aLOG. We are sharing this area with LLO as problems with the aLOG codebase are common between the sites.
Summary: H1 back to Observing after Beckhoff crash. No Initial Alignment needed (but arms, BS, & PRM needed tweaking). ASC still has issues in DRMI (sometimes).
(Attempt #2 [which started at 17:28utc] Continuing from DRMI...)
Thanks Corey for the detailed information.
I had changed around which states were requestable (show up in darker blue and in drop down menu) to be just the states that we expect to be normally in use to help clear up some confusion that arose after we added TMS WFS to the normal sequence. I had left things inconsistent between the two arms, as Corey noted, so he and I fixed that just before going to observe.
The issue with the DRMI ASC needs more work, we could think about setting an offset for the 36WFS when we are aligned using AS45, then ramping that offset off in full lock.