attached is a 24 hour minute trend plot of the Beckhoff MSR temperature channelss, starting at 7am Friday morning PDT. Current situation is that the doors between the control room and MSR are closed, the doors between MSR and hallway are open and a fan is directing air behind the racks. The temperature variation changed around 18:45 PDT yesterday and is now more constant and slightly elevated. Reminder that if either channel exceeds 30C (86F) cell phone alarms will be generated.

looks like upgrading scipy will take a bit more work. An FRS was opened yesterday for this issue FRS8048

TITLE: 05/06 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 65Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    Wind: 7mph Gusts, 5mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.12 μm/s
QUICK SUMMARY: Locked for 4.5hrs with calm environment. DIAG_MAIN is reporting that the IM1/2 Yaw is out of norm, Cheryl gave me some values to put in if we lose lock.

• changed IM alignment
• initial alignment
• some issues with FSS going into oscillation, multiple time
• relocked H1
• couldn't run a2l, gave errors
• more SDF changes accepted, mostly small Xe-14 differences
• IM changes not quite as advantageous as yesterdays, gave TJ numbers to revert at lock loss
• also couldn't get timemachine to work - didn't open a new window

TITLE: 05/06 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 68Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
LOG: 

23:30 Apollo on site to work on MSR cooling, I never saw when he left

23:30 Lockloss and MikeL in CR with a tour

6:50 ITMY TCS guardian takes us out of OBSERVE for 20minutes, no other obvious effect on the IFO
 

A TCS guardian just knocked us out of observe again.It has knocked us out of OBSERVE for about ten minutes, while it fiddles with some PZT offsets. I also couldn't find the link to the SDF screen for the ITMY guardian to see what the differences are. It looks like the guardian is having a hard time staying locked.

2017-05-06T07:03:50.60179 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 45.5
2017-05-06T07:03:50.60248 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] = 3
2017-05-06T07:03:53.60372 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] done
2017-05-06T07:03:53.72259 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 49.0
2017-05-06T07:03:53.72295 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] = 3
2017-05-06T07:03:56.72339 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] done
2017-05-06T07:03:56.85062 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 52.5
2017-05-06T07:03:56.85110 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] = 3
2017-05-06T07:03:59.86033 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] done
2017-05-06T07:03:59.95678 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 56.0
2017-05-06T07:03:59.95728 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] = 3
2017-05-06T07:04:02.95752 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] done
2017-05-06T07:04:03.08470 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 59.5
2017-05-06T07:04:03.08496 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] = 3
2017-05-06T07:04:06.08646 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] done
2017-05-06T07:04:06.19308 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 63.0
2017-05-06T07:04:06.19765 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] = 3
2017-05-06T07:04:09.19820 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] done
2017-05-06T07:04:09.30854 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 66.5
2017-05-06T07:04:09.30885 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] = 3
2017-05-06T07:04:12.31090 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] done
2017-05-06T07:04:12.41767 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] ezca: H1:TCS-ITMY_CO2_PZT_SET_POINT_OFFSET => 70.0
2017-05-06T07:04:12.42090 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] = 3
2017-05-06T07:04:15.42433 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['wait'] done
2017-05-06T07:04:15.51860 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] 3.5, 7.0, 10.5, 14.0, 17.5, 21.0, 24.5, 28.0, 31.5, 35.0, 38.5, 42.0, 45.5, 49.0, 52.5, 56.0, 59.5, 63.0, 66.5, 70.0
2017-05-06T07:04:15.51863 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] 54.8988848698, 54.8350498672, 54.891019514, 55.1177868928, 55.3451070593, 55.5405800837, 55.7137985201, 55.8408840514, 55.6183281204, 55.3447073749, 55.1125390519, 55.0134612237, 55.0977940265, 55.192294462, 55.2613368151, 55.3127282154, 55.3339794561, 55.3309874091, 55.3280875704, 55.2947084872
2017-05-06T07:04:15.61037 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] slopecheckfail 0
2017-05-06T07:04:15.61039 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] slope [-0.01823857  0.01599133  0.06479068  0.06494862  0.05584944  0.04949098
2017-05-06T07:04:15.61040   0.03631015 -0.06358741 -0.07817736 -0.06633381 -0.02830795  0.02409509
2017-05-06T07:04:15.61041   0.02700012  0.01972639  0.01468326  0.00607178 -0.00085487 -0.00082853
2017-05-06T07:04:15.61041  -0.00953688]
2017-05-06T07:04:15.61042 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] gain setting for servo is -54.8288534565
2017-05-06T07:04:15.61042 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] 0
2017-05-06T07:04:15.61043 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] 3.5
2017-05-06T07:04:15.61043 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] 54.8988848698
2017-05-06T07:04:15.61044 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] current chiller setpoint 20.9
2017-05-06T07:04:15.62412 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] ezca: H1:TCS-ITMY_CO2_CHILLER_SET_POINT_OFFSET => 21.0
2017-05-06T07:04:15.62907 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] ezca: H1:TCS-ITMY_CO2_CHILLER_TSTEP => 1178089473.0
2017-05-06T07:04:15.62921 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] new chiller setpoint 21.0
2017-05-06T07:04:15.63284 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] timer['chillerstep'] = 240
2017-05-06T07:04:15.63295 TCS_ITMY_CO2 [FIND_LOCK_POINT.run] no setpoint found:stepping chiller temp

model restarts logged for Thu 04/May/2017

Many restarts due to site power glitches from lightening storm and recovery thereof. Full report attached as text file.

model restarts logged for Wed 03/May/2017 No restarts reported

model restarts logged for Tue 02/May/2017
2017_05_02 11:50 h1dc0
2017_05_02 11:55 h1broadcast0
2017_05_02 11:55 h1dc0
2017_05_02 11:55 h1fw0
2017_05_02 11:55 h1fw1
2017_05_02 11:55 h1fw2
2017_05_02 11:55 h1nds0
2017_05_02 11:55 h1nds1
2017_05_02 11:55 h1tw1
2017_05_02 11:57 h1broadcast0
2017_05_02 11:57 h1dc0
2017_05_02 11:57 h1fw0
2017_05_02 11:57 h1fw1
2017_05_02 11:57 h1fw2
2017_05_02 11:57 h1nds0
2017_05_02 11:57 h1nds1
2017_05_02 11:57 h1tw1

Maintenance day. Reboot of h1dc0 for day208bug, added fire pump chans to DAQ. Required several restarts (duplicate fmcs chans).

TITLE: 05/05 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 67Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY:  Locked for the entire shift and for ~13 hours total so far.  No issues to report.
LOG: 

Hugh requests that the HAM2 and 3 GS13 gains be set to correct values at the next Out of Observe opportunity.  Contact Hugh for more info.

Jason requests that the ISS diffracted power be lowered to ~3% at the next Out of Observe opportunity.

Both of these items are on the Operator white board.

Locked for 9.5 hours.  LLO is still down due to weather which seems to be both site's biggest issue lately.

Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 2896 seconds. TC B did not register fill. LLCV set back to 18.0% open.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill not completed after 3600 seconds. LLCV set back to 42.0% open.

MSR UPS reports going onto battery power three times during the storm:

Both BRS machines at the end stations did not reboot or require code restarts.

The corner station Beckhoff computer (h1ecatc1) did not reboot (on ups power), the units at the end stations did reboot around 20:48. h1ecatx1 required a manual reboot at 22:25 to complete its code startup.

None of the three HEPI pump controller computers rebooted or needed code restarts.

MY weather station stopped operating 21:34 PDT, was restarted this morning.

All Front End Computers rebooted around 20:46, some multiple times. Report generated from h1fs0's NFS mount requests for /opt/rtcds:

20:45:11    h1pemmy
20:45:12    h1pemmx
20:45:15    h1sush2b
20:45:18    h1lsc0
20:45:20    h1oaf0
20:45:22    h1seib3
20:46:42    h1pemmy
20:46:44    h1seiey
20:46:44    h1seiex
20:46:45    h1susauxey
20:46:46    h1pemmx
20:46:46    h1iscex
20:46:46    h1sush2b
20:46:47    h1iscey
20:46:48    h1seib1
20:46:49    h1susauxex
20:46:49    h1seih23
20:46:50    h1susauxh56
20:46:50    h1lsc0
20:46:51    h1susb123
20:46:51    h1susauxh2
20:46:52    h1susauxh34
20:46:52    h1asc0
20:46:52    h1susauxb123
20:46:52    h1oaf0
20:46:53    h1seib3
20:46:53    h1seih16
20:46:53    h1seib2
20:46:54    h1psl0
20:46:54    h1sush56
20:46:55    h1seih45
20:46:57    h1sush34
20:46:57    h1sush2a
20:47:07    h1susex
20:47:09    h1susey
 

It looks like the storm brought down part of the new FMCS - possibly communications to the LSB? All other buildings appear to be working fine with both the old and new systems.