Ed, Dave B., Fil
The PSL FE Computer tripped off again but this time it was due to the repair efforts on the Rotation Stage Beckhoff chassis. Dave is preparing it for a re-start.
John and I have been on site investigating the instrument air alarm at E Y, we suspect a small ice block on the blow down used for eliminating the moisture from the compressor tank. We reset the blow down after applying some heat and all seems to be working correctly now. We also addressed some temperature issues at E X, we turned on more heat at HC-7. Tuesday I will verify all heating elements are functioning on the main heater. Temperature is stable now.
apologies, it's the Beckhoff Rotation Stage Chassis.
Alastair asked me to check if the corner station Beckhoff slow-controls system was running correctly. The system is reporting an error on the C1DEVICE0 error flag channel. I've attached the current medm screen shot.
The error started at 08:43 PST this morning, at the same time as the h1psl0 IO-Chassis glitched. I suspect a hardware error at the Beckhoff chassis, which in turn possibly glitched the PSL IO-Chassis via a shared 24VDC power supply.
caget H1:SYS-ETHERCAT_C1DEVICE0_ERROR_FLAG
H1:SYS-ETHERCAT_C1DEVICE0_ERROR_FLAG Error
Please post the device0 medm, and check the number of configured vs. actual slave modules. If the later is smaller, the following module is in error or broken.
I spoke to Filiberto on the telephone. He's heading in to the site to have a look.
attached is the ecatc1 device0 medm, it show actual_slave_count = 290, config_slave_count = 345 which Daniel says indicates hardware error.
The Beckhoff slow controls chassis for PSL/TCS was found tripped off. Found the D13001631 Rotation Stage board for TCS Y was current limiting. Removed D6, D3, D7, and D4. Both TCS X and Y boards now have same configuration. Unit was reinstalled. PSL IO chassis will most likely need to be restarted. Past experience shows that power cycling the beckhoff chassis will cause errors with IO chassis.
Ed, Jason
The LASER is back up. I'm waiting on a timing error in the FE computer to clear.
h1ioppsl0 IRIG-B timing has recovered. I've cleared the IPC errors and accumulated CRC counters.
Timing error cleared @ 18:23UTC. Going to the LVEA to check on the statues of the TCS LASERs.
Seems like this is the bogger issue at the moment. I noticed this after the Verbal Alarm about the TCS issue. The LASERs are still showing power. I have no QUAD FOM view. THe attached screenshot shows the current CDS FE status.
Actually all the front-end models are on a single computer. Either the power for the I/O chassis has tripped, or the communications was dropped for a few cycles. If you reboot the PSL front-end computer (h1psl0), that would fix the latter. If that fails, you need to power down the PSL front-end computer, then power-cycle the I/O chassis (resetting DC power supply if needed), then booting the PSL front-end computer. If that fails, then we need more investigation.
I have Jim Batch on the phone to troubleshoot
The IOP chassis was not visible from the computer, so I remotely removed the h1psl0 computer from the dolphin network and powered down the computer. Ed power cycled the I/O chassis and powered the computer back on. The cards in the I/O chassis are now visible. Timing on the h1ioppsl0 model has gone "negative", meaning the DAQ data is bad until it works it's way back positive. But the PSL computer show be able to control the PSL OK. There is no known cause for this event.
trending h1ioppsl0's state_word shows that at 08:44 PST the FE and ADC bits faulted. It looks like a timing issue which glitched the IOP into a non-recoverable state.
IRIG-B is still in a slow recovery, DAQ data will be invalid until it completes.
at 16:44UTC both TCS lasers tripped off. The Ops overview screen shows that the Flow alarm status is green. It seems to be only the Laser Power that's off. I'm currently scouring the Wiki to see what I need to do to get these back up.
Just spoke to Ed and Jonathan. The flow rate now looks green. It's possible that this has been triggered by other computers going down. I can't get remote read-only MEDM access to check the status of the system. Ed will call me if they can't key the lasers back on at the controller.
Ed and Dave got my remote access turned on so I could log in and take a look. It appears both TCS lasers are in a good state but are not outputing power. Power supply voltage is good. Flow rate is good. Temperature is good. Current is around 0.7A indicating that they are getting RF and are ready to laze. The Beckhoff screen shows red indicators on several read back channels, and it looks like this may be the cause of the lasers not coming on.
I've spoken to Dave and Ed again and they are going to take it from here since it looks like a Beckhoff issue. I'll be available by phone if needed. The laser chassis will probably need to be re-keyed off and on once this has been fixed.
16:22 Also noticed that tw0 is "white". Not certain at this time how long it has been that way
h1tw0 is currently down while we are rebuilding its raid.
It's pretty cold on site this morning, but is it possible that the temperature sensors are past their limits? See screenshot.
Actually this looks more like a conversion error between signed and unsigned words for the Fahrenheit readbacks. If you subtract 2^16/10 from these numbers, it makes sense! Filed FRS 6963.
h1psl0 is out of the Dolphin fabric and is powered down, Ed is doing the power cycle of the IO Chassis and powering h1psl0 back up.
h1psl0 models running again, IOP has negative IRIG-B excursion, waiting for it to return to nominal. In the mean time Peter and Ed will recover the PSL via MEDM.
h1ioppsl0 has returned from its IRIG-B meandering. I've cleared all IPC and CRC errors. The board is green again.