Ed, Jason
The LASER is back up. I'm waiting on a timing error in the FE computer to clear.
Seems like this is the bogger issue at the moment. I noticed this after the Verbal Alarm about the TCS issue. The LASERs are still showing power. I have no QUAD FOM view. THe attached screenshot shows the current CDS FE status.
Actually all the front-end models are on a single computer. Either the power for the I/O chassis has tripped, or the communications was dropped for a few cycles. If you reboot the PSL front-end computer (h1psl0), that would fix the latter. If that fails, you need to power down the PSL front-end computer, then power-cycle the I/O chassis (resetting DC power supply if needed), then booting the PSL front-end computer. If that fails, then we need more investigation.
I have Jim Batch on the phone to troubleshoot
The IOP chassis was not visible from the computer, so I remotely removed the h1psl0 computer from the dolphin network and powered down the computer. Ed power cycled the I/O chassis and powered the computer back on. The cards in the I/O chassis are now visible. Timing on the h1ioppsl0 model has gone "negative", meaning the DAQ data is bad until it works it's way back positive. But the PSL computer show be able to control the PSL OK. There is no known cause for this event.
trending h1ioppsl0's state_word shows that at 08:44 PST the FE and ADC bits faulted. It looks like a timing issue which glitched the IOP into a non-recoverable state.
IRIG-B is still in a slow recovery, DAQ data will be invalid until it completes.
at 16:44UTC both TCS lasers tripped off. The Ops overview screen shows that the Flow alarm status is green. It seems to be only the Laser Power that's off. I'm currently scouring the Wiki to see what I need to do to get these back up.
Just spoke to Ed and Jonathan. The flow rate now looks green. It's possible that this has been triggered by other computers going down. I can't get remote read-only MEDM access to check the status of the system. Ed will call me if they can't key the lasers back on at the controller.
Ed and Dave got my remote access turned on so I could log in and take a look. It appears both TCS lasers are in a good state but are not outputing power. Power supply voltage is good. Flow rate is good. Temperature is good. Current is around 0.7A indicating that they are getting RF and are ready to laze. The Beckhoff screen shows red indicators on several read back channels, and it looks like this may be the cause of the lasers not coming on.
I've spoken to Dave and Ed again and they are going to take it from here since it looks like a Beckhoff issue. I'll be available by phone if needed. The laser chassis will probably need to be re-keyed off and on once this has been fixed.
16:22 Also noticed that tw0 is "white". Not certain at this time how long it has been that way
h1tw0 is currently down while we are rebuilding its raid.
TITLE: 12/17 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Ed
SHIFT SUMMARY: Microseism is up and EQ is finally ringing down. Having difficulty locking arms green, possibly due to a combination of ground motion factors. Last 5 hours of shift were waiting for EQ to ring down to reasonable levels.
LOG:
See previous aLogs for details.
15:53 Starting IA.
It's pretty cold on site this morning, but is it possible that the temperature sensors are past their limits? See screenshot.
Actually this looks more like a conversion error between signed and unsigned words for the Fahrenheit readbacks. If you subtract 2^16/10 from these numbers, it makes sense! Filed FRS 6963.
H0:VAC-EY_INSTAIR_PT499_PRESS_PSIG went into alarm. See attached screenshot of the trend for the past 2 hours. I'll continue to monitor this and make some calls to VAC folks if it persists.
Updated plot for the past 2 hours. Continuing to "trend" downward. Looks a little suspicious how the trend smoothed out around 37.
H1 was most of the way through relocking after an unrelated(?) lockloss. IMC started losing lock during relocking attempts when I noticed a vertical line starting to appear on the BLRMS. L1 has been down due to microseism for several hours.
This EQ was large enough (10^2 um/s in the EQ band according to the BLRMS) that it tripped every ISI platform and several SUSes. I switched ISI_CONF to LARGE_EQ_NOBRSXY, but it didn't seem to help the end station ISIs.
h1ioppsl0 IRIG-B timing has recovered. I've cleared the IPC errors and accumulated CRC counters.
Timing error cleared @ 18:23UTC. Going to the LVEA to check on the statues of the TCS LASERs.