TITLE: 11/26 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Unknown
INCOMING OPERATOR: Travis
SHIFT SUMMARY: H1 stayed locked most of the shift. Now I'm running into an issue I've never dealt with before. See alog31847 for details. This alog has disappeared magically. See alog31850 (comment below) for details.
LOG:
9:05 Noticed range slowly dropping, out of Observe to adjust SRM P
9:08 Back to Observe
9:12 Noticed that I made it worse, back to commissioning to move SRM P the other way round.
9:16 back to Observe.
13:39 Lockloss
14:24 THE confusing lockloss. Below I copied some of the Verbal Alarm message that might be useful in giving people insight of what's going on. It would be useful to know how long does the code take to loop though all the status of things.
14:23:16 VIOLIN_MODE_DAMPING2
14:23:28 TCS CO2 lasers tripped
14:23:24 ISI HAM6 WD tripped
14:23:48 ETMX ISI ST1 WD tripped
14:23:48 ETMX ISI ST2 WD tripped.
14:24:01 DOWN
My alog describing Beckhoff issues seems to have dissapeared (HOW IS THIS EVEN POSSIBLE?) so I'm copying the content here.
----------------------------------------------------------------------------------------------------
After 7 hours H1 lost lock for not so obvious reason (as always, I just haven't had a chance to investigate). This confusing lockloss occured as I tried to recover from the 7-hour lockloss. I stepped out after got through DC READOUT and came back to find that the lock broke at Violin Mode Damping 2 according to the Verbal alarm.
Couple of strange things happened (and still happening)
1) PSL power request suddenly went all the way up to 156W but the actual output got stuck at 30W. I couldn't use LASER_PWR guardian to move it back to 2W. The log repeatedly output the message:
2016-11-26T14:47:54.58953 LASER_PWR [GOTO_POWER_2W_FAST.run] ezca: H1:PSL-POWER_SCALE_OFFSET => 28.95
Seems like LASER_PWR guardian keeps outputing the offset to the channel. I can't manually put it in. I can't caput it.This offset is probably based on the input power to IMC so it make sense that it agrees with the input power. I also couldn't turn the rotation state manually on the rotation state screen.2) Both TCSX and Y mysteriously tripped
duringprior to this lockloss. The medm status is READY so I went to check the interlock box. GATE and ENABLE lights are on but Remote Enable light is off. I've never seen this before. CDS overview doesn't have anything red on it. If the IR sensors saw a flash of carrier light as the lock broke, the GATE light wouldn't have been on and it shouldn't do anything to the Remote Enable light.I'm really really confused right now. Still investigating and hopefully fixing seomthing. If somebody wake up and see this log and have some idea of what might be going on, please call the control room. Otherwise I (and Day shift op) will be calling people after 8am.
----------------------------------------------------------------------------------------------------
**UPDATE**
The rotation state medm screen seems to be stuck. It won't calculate angle requested when you put in new power, and true the other way round. The state is stuck at "busy"
According to ECAT1PLC3 SDF differences. It seems to me that TCS (and likely PSL) rotation states got stuck some how. How do I verify if Beckhoff computer controling RS is working properly?
**UPDATE 15:30**
Found CDS overview medm shows ERROR status for ECAT C1PLC1 and 2
**UPDATE 16:08**
Beckhoff error may have caused the most recent lockloss. C1PLC1 error flag came up prior to ISC_LOCK guardian went down. This is also true for the 7-hour lockloss. I don't know what's going on with C1PLC2.
The screen in alog 30842 can be used to diagnose hardware errors. It can be found from the menu SYS under EtherCAT. Each computer has one somewhere under PLC1.
Slavecountactual: The number of EtherCAT modules seen by the software
Slaveccountconfig: The number of configured modules
These two numbers should agree, indicated by slavecounterror.
Lostframes: Number of lost Ethernet frames since last reset. Any large number, or one that increases steadily is a problem.
Same goes for Lostqueuedframes.
The subscreen under workingcounters keeps track of the different types of EtherCAT frames, but is maybe more for the experts.