We had what appears to be a site-wide power glitch at 08:12 PDT this morning. Both CDS and GC UPS systems report being on battery power for about 6 seconds. Systems on UPS power rode through this, all other systems are down.
Vacuum controls is OK, MEDM overview screen is attached. Also DAQ is OK.
Systems down includes end station Beckhoff, all front ends, FMCS. We are getting multiple text alarms due to the systems being INVALID.
Here is the GC-UPS system's report:
Subject: lookout UPS gc-osb Power Failure !!!
lookout UPS gc-osb Power Failure !!!
APC : 001,035,0858
DATE : 2017-09-16 08:12:37 -0700
HOSTNAME : lookout
VERSION : 3.14.12 (29 March 2014) debian
UPSNAME : gc-osb
CABLE : Ethernet Link
DRIVER : SNMP UPS Driver
UPSMODE : Stand Alone
STARTTIME: 2017-08-22 08:29:07 -0700
STATUS : ONBATT
LINEV : 120.0 Volts
LOADPCT : 26.0 Percent
BCHARGE : 100.0 Percent
TIMELEFT : 209.0 Minutes
MBATTCHG : 15 Percent
MINTIMEL : 5 Minutes
MAXTIME : 0 Seconds
MAXLINEV : 121.0 Volts
MINLINEV : 120.0 Volts
OUTPUTV : 120.0 Volts
SENSE : Unknown
ITEMP : 22.0 C
BATTV : 211.0 Volts
LINEFREQ : 60.0 Hz
LASTXFER : Unacceptable line voltage changes
NUMXFERS : 4
XONBATT : 2017-09-16 08:12:31 -0700
TONBATT : 6 Seconds
CUMONBATT: 6556 Seconds
XOFFBATT : 2017-09-10 14:13:57 -0700
SELFTEST : OK
STESTI : 168
STATFLAG : 0x05060010
EXTBATTS : 20
BADBATTS : 0
END APC : 2017-09-16 08:12:37 -0700
Here are the CDS UPS reports (20 seconds on battery power):
Name : ups-msr-0
Location : LHO MSR
Contact : CDS Administrators
http://ups-msr-0.cds.ligo-wa.caltech.edu
http://10.99.3.10
Serial # : ZA0522010987
Device Ser #: PD0615240264
Date: 09/16/2017
Time: 08:12:29
Code: 0x0109
Warning - UPS: On battery power in response to an input power problem.
---------------------------------------------------------
Name : ups-msr-0
Location : LHO MSR
Contact : CDS Administrators
http://ups-msr-0.cds.ligo-wa.caltech.edu
http://10.99.3.10
Serial # : ZA0522010987
Device Ser #: PD0615240264
Date: 09/16/2017
Time: 08:12:49
Code: 0x010A
Informational - UPS: No longer on battery power.
For reasons unknown, we are unable to initiate new remote login sessons to lhocds and cdslogin. Bubba is currently logged in, presumably he started his session prior to the power glitch. Jonathan suspects the cdsadminctrl machine.
The good news is Bubba is able to remotely view the FMCS status.
The bad news about the remote access issus is that I am currently unable to turn off the cell phone texter, and we are getting multiple messages. This will quiet down as the alarms age.