TITLE: 11/27 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Travis
CURRENT ENVIRONMENT:
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.36 μm/s
QUICK SUMMARY: On our way back up after some issues described in the log below mine.
TITLE: 11/26 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
LOG:
17:50 Kyle on site checking pumps.
18:01 Patrick to CER
18:05 Robert to EY
19:50 Robert and AnnaMaria to put beam dump in MC Refl, Patrick to CER taking top off of Beckhoff module.
20:30 ECat chassis is fixed. Start relocking.
23:50 PCal X 5001.3 Hz line turned off.
At the request of Adam Mullavey (via a phone call from Rick), I have turned off the 5001.3Hz PCal line at EndX for some injection work. I did this by zeroing the amplitude of the line, leaving all other parameters unchanged. See attached screenshot for setting before I changed them.
In a later email forwarded by Rick, Shivaraj mentions having to change Guardian code to keep these lines off at LLO. I'm unaware of any such PCal Guardian code at LHO (my guess is that this is due to the fact that LHO doesn't use PCal for locking like LLO does), so made no changes to Guardian. If someone reading this knows otherwise, please contact myself and the current operator.
We are just starting to do initial alignment after the Beckhoff issue was resolved by Patrick and Richard. So far, so good.
~0945 - 0955 hrs local -> Kyle R. and Travis S. in and out of LVEA Suspected that the backing pump cart's pirani gauge might have increased from its steady, bottomed-out, value of 1.7 x 10-3 torr at the start of the bake to, a now steady, 1.9 x 10-3 torr value -> Closed the O-ring valve at the "hung" turbo's exhaust for several tens of seconds while Travis monitored the pirani reading -> Displayed value remained unchanged while valve closed and after valve opened (should have taken notes - initial value was probably "1.9" all along). Temperature survey a little high after yesterday's modest variac increase -> As found today, more like 95C +/- 10C -> Decreased variacs slightly. The plan now is to start the temperature step down beginning tomorrow morning such that pump hardware can be removed and PT180 be re-exposed to the site vacuum volume (i.e. return to nominal state) on Tuesday
It appears the problem is either with ISC Common L20, ISC Common M0, or the link between (see attached screenshots). I will go out to the CER and take a look.
I tried slowly power cycling the ISC Common chassis in the CER twice. Each time it came back in the same state. Since the problem is reported as being between two terminals internal to the chassis, I'm not sure what else can be done without pulling the chassis out of the rack.
I've left a voice message with Richard. I want to touch base with him before I go further.
Richard came in and we opened the top of the chassis in situ. We swapped the internal cables around and determined that the problem was with the ISC Common M0 module. This is the EK1100 EtherCAT module in the middle row of the ISC Common chassis. We replaced that module and it is back up and running. The only cable removed during this was one of the Ethernet cables in the back of the chassis. This cable should be replaced with a longer one. The serial number of the module that was replaced is 04110621.
Everything looks fine to me, but I'm no expert. See attached screenshots.
TITLE: 11/26 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Down due to Beckhoff issues
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
Wind: 2mph Gusts, 1mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.49 μm/s
QUICK SUMMARY: Down due to Beckhoff issues per Nutsinee's aLog. Have contacted Keita and Patrick. Keita is looking at things remotely and Patrick is on his way to the site.
TITLE: 11/26 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Unknown
INCOMING OPERATOR: Travis
SHIFT SUMMARY: H1 stayed locked most of the shift. Now I'm running into an issue I've never dealt with before. See alog31847 for details. This alog has disappeared magically. See alog31850 (comment below) for details.
LOG:
9:05 Noticed range slowly dropping, out of Observe to adjust SRM P
9:08 Back to Observe
9:12 Noticed that I made it worse, back to commissioning to move SRM P the other way round.
9:16 back to Observe.
13:39 Lockloss
14:24 THE confusing lockloss. Below I copied some of the Verbal Alarm message that might be useful in giving people insight of what's going on. It would be useful to know how long does the code take to loop though all the status of things.
14:23:16 VIOLIN_MODE_DAMPING2
14:23:28 TCS CO2 lasers tripped
14:23:24 ISI HAM6 WD tripped
14:23:48 ETMX ISI ST1 WD tripped
14:23:48 ETMX ISI ST2 WD tripped.
14:24:01 DOWN
My alog describing Beckhoff issues seems to have dissapeared (HOW IS THIS EVEN POSSIBLE?) so I'm copying the content here.
----------------------------------------------------------------------------------------------------
After 7 hours H1 lost lock for not so obvious reason (as always, I just haven't had a chance to investigate). This confusing lockloss occured as I tried to recover from the 7-hour lockloss. I stepped out after got through DC READOUT and came back to find that the lock broke at Violin Mode Damping 2 according to the Verbal alarm.
Couple of strange things happened (and still happening)
1) PSL power request suddenly went all the way up to 156W but the actual output got stuck at 30W. I couldn't use LASER_PWR guardian to move it back to 2W. The log repeatedly output the message:
2016-11-26T14:47:54.58953 LASER_PWR [GOTO_POWER_2W_FAST.run] ezca: H1:PSL-POWER_SCALE_OFFSET => 28.95
Seems like LASER_PWR guardian keeps outputing the offset to the channel. I can't manually put it in. I can't caput it. This offset is probably based on the input power to IMC so it make sense that it agrees with the input power. I also couldn't turn the rotation state manually on the rotation state screen.
2) Both TCSX and Y mysteriously tripped during prior to this lockloss. The medm status is READY so I went to check the interlock box. GATE and ENABLE lights are on but Remote Enable light is off. I've never seen this before. CDS overview doesn't have anything red on it. If the IR sensors saw a flash of carrier light as the lock broke, the GATE light wouldn't have been on and it shouldn't do anything to the Remote Enable light.
I'm really really confused right now. Still investigating and hopefully fixing seomthing. If somebody wake up and see this log and have some idea of what might be going on, please call the control room. Otherwise I (and Day shift op) will be calling people after 8am.
----------------------------------------------------------------------------------------------------
**UPDATE**
The rotation state medm screen seems to be stuck. It won't calculate angle requested when you put in new power, and true the other way round. The state is stuck at "busy"
According to ECAT1PLC3 SDF differences. It seems to me that TCS (and likely PSL) rotation states got stuck some how. How do I verify if Beckhoff computer controling RS is working properly?
**UPDATE 15:30**
Found CDS overview medm shows ERROR status for ECAT C1PLC1 and 2
**UPDATE 16:08**
Beckhoff error may have caused the most recent lockloss. C1PLC1 error flag came up prior to ISC_LOCK guardian went down. This is also true for the 7-hour lockloss. I don't know what's going on with C1PLC2.
The screen in alog 30842 can be used to diagnose hardware errors. It can be found from the menu SYS under EtherCAT. Each computer has one somewhere under PLC1.
Slavecountactual: The number of EtherCAT modules seen by the software
Slaveccountconfig: The number of configured modules
These two numbers should agree, indicated by slavecounterror.
Lostframes: Number of lost Ethernet frames since last reset. Any large number, or one that increases steadily is a problem.
Same goes for Lostqueuedframes.
The subscreen under workingcounters keeps track of the different types of EtherCAT frames, but is maybe more for the experts.
TITLE: 11/26 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 70.3054Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT: Wind: 4mph Gusts, 3mph 5min avg Primary useism: 0.04 μm/s Secondary useism: 0.62 μm/s
QUICK SUMMARY: TJ had it locked for me.
TITLE: 11/26 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY: PEM work for the whole shift, broke lock a few times, but recovery was pretty usual. POP90 jumped on me once, but I missed it and didn't notice till Anamaria pointed out the loss of range. I turned off the SCR1 loops and thenmoved SRM back to where it was.
The PEM crew is running the lines from alog31845
LOG:
Today we shook HEPIs, all chambers, and shook the input and output arm beam tubes at the corner station. A few of the lines in DARM were excited by the BS injections. We left low frequency lines In the cal line of ETMX L1 (0.3 Hz lenght) and ITMX L1 (0.2 Hz yaw) to produce side bands on certain lines. These lines are considerably smaller than the microseismic peak, so we didnt want them to keep us from going to observation mode.
TITLE: 11/26 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Calibration
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.46 μm/s
QUICK SUMMARY: Robert and Anamaria are doing their injections, we are currently locked at 72Mpc.
20:40 LockLoss OMC PD saturation. Robert is possibly the cause, doing low frequency acoustic injections, and suggests that maybe the ITMY oplev is sensitive to the freq range he's using. Maybe we'll try turning the OpLev damping off for next lock.
21:14 Lockloss at SWITCH_TO_QPDS
21:19 Wind has been picking up a little and blowing moderatly from the NE. X arm has been shaky. I switched ISI config to WINDY_MOD_WIND_BRSX and that seems to be holding rock steady for now, once all the nodes arrived. TRX and TRY don't look 'raggedy' any more.
21:30 Lockloss at DRMI_ON_POP
22:02 NLN. Beam Diverters closed 68Mpc. OpLev damping off experiment proved to be not good to try.
22:08 Robert out to LVEA to do PEM injections
22:07 PI mode 28 ringing up. Changed from -60ø to -100ø
22:17 PI mode 27 ringing up. Changed to 120ø
22:21 GRB alert. PEM injections happening at LHO. Not in Observing .
22:39 Lockloss. HAM6 ISI WD trip. May have been my fault playing with bp filters for roll mode while trying to damp.
23:19 back to NLN. Diverters closed 75Mpc
23:31 Manual overfill of CP3
25:51 Lockloss. HAM6 ISI WD trip
23:50 Tour(s) in e control room
00:35 Last tour out of control room. Handing off to TJ.