State of H1: unlocked, recovering
Activities:
6:09 Bubba on his way to the site to investigate instrument air.
16:25 Initial alignment: Input align not showing any IR. Trended IM4 and PR2 after successfull ALS alignment. Still no luck. Going to full lock to further assess the need(s) for continuing IA.
16:35 Bubba on site
16:40 Johnathan called to inform me of a tour that he's meeting on site at 10:00PST
16:45 Both TCS Lasers tripped off.
16:49 Set H1 Observatory Mode to Unknown due to investigation of TCS issue.
16:57 John W on site going to assist Bubba
18:30 Checked on and cycled the TCS lasers
18:38 contacted Aiden and then Alistair
19:00 Granted remote access to Alastair
19:26 Carlos granted remote access
19:45 I spoke to Fil on the phone. It has been determined that there's a hardware error. He's heading to the site to troubleshoot.
23:43 Having an unusually hard time re-aligning arms
16:25 Handing off to Cheryl
1215 hrs. local -> Kyle on site to check on 80K pumps Local temperature are dipping into the "single digits" which means now is an opportune time to investigate if our old problem of over-filling or under-filling of 80K pumps will result from frozen LLCV valve stems is going to be a problem. Recall that in the past this had occurred but could not be confidently attributed to water freezing in the old pneumatic actuators or from water freezing in the valve stem from having got past too loose packing nuts. Or, both for that matter. Chandra R. and John W. had replaced the pneumatic actuators with electric (water tight) ones last spring so now is the first real world test. Initially, I'm on site to verify that the recently elevated exhaust pressures on CP3 and CP4 weren't the result of continuous overfilling -> Inspection of the Y-mid's CP3 and CP4 showed neither was overfilling and both are fine for now (remnant standing LN2 in lines from yesterday's overfill?). By dumb luck, I found that CP6 was indicating a frozen valve stem while I was on site - "Go figure!" -> As found, the local actuator pointer was showing ~30% open and the actuator motor was making noise. The PID output was "railed" at its programmed minimum of 20% open. I had John W. (he and Bubba also on site for separate instrument air issue) switch CP6's level control to manual 50% open -> the motor then went quiet and it stroked open to 50% as per the pointer. Next, I had him change the value to 0% open and the stem stroked closed smoothly until stopping at about the 30% open value (as indicated by the local pointer). I then tested the valve stem packing nut and found that it was completely loose, i.e. no thread engagement. So, even though the local heat tape is wrapped around most of the exposed valve stem, was energized and working, water which made it past the ineffective packing washer was able to freeze nonetheless. This is because this area is about 2" away from the closest heat tape wrap -> I heated up the frozen area with a heat gun and lightly engaged the packing nut. I then had John W. alternate between 100% open and 0% open to demonstrate that the valve could stroke through its whole range smoothly etc.. I setup and am leaving a heat lamp to prevent this unit from refreezing (which it would do, now that water is past the packing washer) and had John W. reduce the Beckhoff PID minimum to 15% open down from its as found value of 20%. 1535 hrs. local -> Kyle leaving site
Man this is hard to read - a consequence of cutting and pasting edits. I think this could be a new record for "run on sentences!"
Mon 12th, Tue 13th, Wed 14th: only restarts of h1tw0 reported (due to failed RAID)
Thu 15th, Fri 16th: No restarts reported
Ed, Dave B., Fil
The PSL FE Computer tripped off again but this time it was due to the repair efforts on the Rotation Stage Beckhoff chassis. Dave is preparing it for a re-start.
h1psl0 is out of the Dolphin fabric and is powered down, Ed is doing the power cycle of the IO Chassis and powering h1psl0 back up.
h1psl0 models running again, IOP has negative IRIG-B excursion, waiting for it to return to nominal. In the mean time Peter and Ed will recover the PSL via MEDM.
h1ioppsl0 has returned from its IRIG-B meandering. I've cleared all IPC and CRC errors. The board is green again.
John and I have been on site investigating the instrument air alarm at E Y, we suspect a small ice block on the blow down used for eliminating the moisture from the compressor tank. We reset the blow down after applying some heat and all seems to be working correctly now. We also addressed some temperature issues at E X, we turned on more heat at HC-7. Tuesday I will verify all heating elements are functioning on the main heater. Temperature is stable now.
apologies, it's the Beckhoff Rotation Stage Chassis.
Alastair asked me to check if the corner station Beckhoff slow-controls system was running correctly. The system is reporting an error on the C1DEVICE0 error flag channel. I've attached the current medm screen shot.
The error started at 08:43 PST this morning, at the same time as the h1psl0 IO-Chassis glitched. I suspect a hardware error at the Beckhoff chassis, which in turn possibly glitched the PSL IO-Chassis via a shared 24VDC power supply.
caget H1:SYS-ETHERCAT_C1DEVICE0_ERROR_FLAG
H1:SYS-ETHERCAT_C1DEVICE0_ERROR_FLAG Error
Please post the device0 medm, and check the number of configured vs. actual slave modules. If the later is smaller, the following module is in error or broken.
I spoke to Filiberto on the telephone. He's heading in to the site to have a look.
attached is the ecatc1 device0 medm, it show actual_slave_count = 290, config_slave_count = 345 which Daniel says indicates hardware error.
The Beckhoff slow controls chassis for PSL/TCS was found tripped off. Found the D13001631 Rotation Stage board for TCS Y was current limiting. Removed D6, D3, D7, and D4. Both TCS X and Y boards now have same configuration. Unit was reinstalled. PSL IO chassis will most likely need to be restarted. Past experience shows that power cycling the beckhoff chassis will cause errors with IO chassis.
Update: as of 2:08UTC, ALS X arm is locked at 0.92, COMMA_DEMOD is at 3.7 which is low, X arm in IR is flashing at 0.7, I'm declaring relative victory and moving on to the Y arm
X arm recovery:
Update: as of 2:41UTC, ALS Y arm locked at 1.05, X arm needs some attention again
Update: 2:59UTC - heading into LVEA to reset Noise Eater, second time my shift
Update: 3:14UTC - noise eater reset - X arm IR locked and WFS offloaded
Update: as of 3:33UTC, Initial Alignment is complete
3:44UTC - locking full IFO