TITLE: 12/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:
After CDS & H1-Locking work of yesterday, today we transitioned to starting prep work for the upcoming vent of HAM7.
However, there was more CDS work today which was related to ISC Slow Controls....but in the middle of this...
There was a 90min power outage on site!!
LOG:
- early-1615 Randy was at the FCES putting stickers on items
- 1539-1900 pre-vent technical cleaning begins (kim.nellie)
- 1716 SR3 + HAM5 tripped; gave it some time, untripped/redamped @1805
- 1545 SEI_ENV transitioned to MAINTENANCE for vent
- 1609 TCSx Laser off (TJ contacted; prob due to cleaning near the rack)
- 1624-1745 BSC2 platform & then sorting parts (randy)
- Tripped BS, ISI, HEPI---leaving tripped until randy finishes in about an hr
- Once he was done, BS was taken to ALIGNED & SEI_BS taken to DAMPED_HEPI_OFFLINE
- 1745-1845 Sorting through rail parts in North Bay
- 1626 quick assistance for cleaning (chris)
- 1628-1635 lvea walkthru (richard)
- 1630 cer to pull slow control chassis for ISC (fil)
- This affects a PZT for the IMC, so it lost lock. Keeping IMC_LOCK at OFFLINE during this beckhoff work.
- 1718-1810 Unboxing the JAC in the optics lab (jennie, jason)
- 1728 Taking Integrating sphere (had been at KAGRA) into PCal Lab (tony)
- 1730-1855 Grabbing parts at offsite vendor (mitch)
- 1751-1900 Building 2nd CHETA enclsure in the CHETA/JAC Lab (matt)
- 1803-2012 OPO Testing cont (KarMeng.Eric)
- 1812 Shiela joining but will be in/out with them + looking for parts
- 1813-1820 chamber measurements for JAC (jason)
- 1814 Kobelco (purge air compressor) being turned on (travis,jordan)
- 1848-2000 Inventory work at EX + EY (randy)
- 2005 Grabbing wipes in Cleaning Room (ibrahim)
- 2023 CHETA Viewport part search (gerardo)
- 2025 Power Outage ON Site!
- Ended up down for over 90min
- See oli's alog for a summary of recovery.
Recovery milestones:
22:02 Power back-up
22:04 CDS / GC seeing what's on/functional, then bring infrastructure back up
22:10 VAC team starts Kebelco to support the ar pressure that's keeping the cornerstation gate valves for closing
22:13 Phones are back
22:17 LHO GC Internet's back
22:25 GC wifi back up, alog and CDS back up
22:40 RyanS, Jason, Patick got PSL beckoff back up
22:55 VAC in LVEA/FCES back up
22:57 CDS back up (only controls)
23:27 PSL computer back
00:14 Safety interlock is back
00:14 HV and ALSY on at EY
00:35 opslogin0 back up
00:37 CS HV back up
00:53 CS HEPI pump station controller back up
01:05 CO2X and CO2Y back up
01:10 HV and ALSY on at EY
01:22 PSL is alive
Note: EY 24MHz oscillator had to be power cycled to resync
Casualties:
- lost ADC on seiH16
- 18V power supply failure at EY in SUS rack
Log:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 22:32 | VAC | Gerardo | MX, EX, MY, EY | n | Bringing back VAC computers | 23:41 |
| 23:32 | VAC | Jordan | LVEA, FCES | n | Bringing back VAC computers | 22:55 |
| 22:54 | PSL | RyanS, Jason | LVEA | n | Turning PSL chiller back on | 23:27 |
| 23:03 | VAC | Jordan | LVEA | n | Checking out VAC computer | 23:41 |
| 23:56 | SAF | Richard | EY | n | Looking at safety system | 00:25 |
| 00:02 | JAC | Jennie | JOAT, OpticsLab | n | Putting components on table | 01:24 |
| 00:19 | SAF | TJ | EX | n | Turning on HV and ALSX | 00:49 |
| 00:38 | TCS | RyanC | LVEA | n | Turning CO2 lasers back on | 01:06 |
| 01:03 | EE | Marc | CER | n | Power cycling io chassis | 01:14 |
| 01:06 | PSL | RyanS, Jason | LVEA | n | Checking makeup air for PSL | 01:10 |
| 01:07 | JAC | Jennie | LVEA | n | Grabbing parts | 01:14 |
| 01:08 | beckhoff | patrick | cr | - | BRSx recovery | 02:20 |
| 01:08 | sei | jim | EX.EY | - | HEPI pump station recovery | 01:58 |
| 01:25 | SEI | Patrick | EX | n | BRSX troubleshooting | 01:54 |
| 01:27 | EE | Marc | EX | n | Looking at RF sources | 02:09 |
| 02:00 | EE | Fil | EY | n | Power cycling SUSEY | 02:19 |
| 02:09 | VAC | Gerardo | MX | N | Troubleshooting VAC computer | 02:27 |
| 02:12 | EE | Marc | CER | N | Checking power supplies | 02:16 |
Since Oli's alog, I tried to keep a rough outline of the goings-on:
Marc and Fil went down to EY to replace the failed power supply, which brought life back to the EY frondends.
Dave noticed several models across site had timing errors, so he restarted them.
Gerardo continued to torubleshoot VAC computers at the mid-stations.
Once CDS boots were finished, I brought all suspension Guardians to either ALIGNED or MISALIGNED so that they're damped overnight.
I started to recover some of the Guardian nodes that didn't come up initially. When TJ started the Guardian service earlier, it took a very long time, but most of the nodes came up and he put them into good states for the night. The ones that didn't come up (still white on the GRD overview screen) I've been able to revive with a 'guardctrl restart' command, but I can only do a couple at a time or else the process times out. Even this way, the nodes take several minutes to come online. I got through many of the dead nodes, but I did not finish as I am very tired.
Main things still to do for recovery: (off the top of my head)
- Finish reviving dead Guardian nodes
- Maybe restart the main file server (Dave suspects this is the cause of general slowness of CDS machines)
- Ensure seismic is in a good place
- Finish Beckhoff software updates that were in-progress at the time of the outage
- Clear up remaining IPC errors across CDS land
- Turn lasers back on
- Check suspension positions
- Lots of other things probably
- Thank everyone for their tireless efforts