Wed Jan 03 10:06:32 2024 INFO: Fill completed in 6min 29secs
Gerardo confirmed a good fill curbside
TITLE: 01/03 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 1mph Gusts, 0mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.43 μm/s
QUICK SUMMARY:
- H1 currently on a 16 hour lock
- Seismic activity is low, CDS/DMs ok
Yesterday John Z restarted the BLRMS GDS monitor, the control room SEIS BLRMS FOM on nuc5 started displaying data from 9pm onwards.
Verbal Alarms, which had been reporting leaps.py issues, started working normally yesterday afternoon. We currently don't understand the original error or why it cleared up.
TITLE: 01/03 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:
Other than the one hitch related to the Leap Second, this was a fairly nice and smooth shift post-Maintenance.
LOG:
Rainy night continues. H1 has been locked for 4.25hrs (with about 45min out of observing due to the CW injection Leap Second issue). Rode through a M4.9 EQ from Japan.
At 0037utc, H1 was bumped out of OBSERVING due to an SDF change. The change has been this channel:
I chatted with Louis (phoned him because it is a CAL channel), but he mentions this is not an "online calibration issue", and this has to do with the Continuous Wave Injection. It also does not appear to be guardian-related. After chatting with Louis, since Jonathan and Erik are on-site, I just notified them about the issue.
We have currently been OUT of OBSERVING for about 40+min.
Attached is another screenshot with:
Currently, Dave, Erik, & Jonathan are investigating.
0128: Back to OBSERVING! (thanks to Dave, Erik, Jonathan, and also Louis for my first chat with him).
Dave, Erik, Jonathan, This was caused by an expired leap second file on the h1hwinj1 machine. It looked like the CW injection process was restarting. After investigation we noticed that the psinject service was failing, with an error loading the leap second database file. The solution was to update the tzdata package. On SL7 this is "yum update tzdata". After we did that the injection process was able to startup and run.
TITLE: 01/03 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 6mph Gusts, 4mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.46 μm/s
QUICK SUMMARY:
Austin handed off an H1 nearly at NLN. I eventually took H1 to Observe, at 0011utc, but at 0037, have been getting bumped out of OBSEVERING due to a gain change (i.e. H1:CAL-INJ_CW_GAIN) for CALINJ SDF.
Attached is the last 20+ min of these gain changes from 1.0 to 0.0 every ~1min. Have Louis on the phone now.
Forgot to mention that Jim chatted with me regarding the HAM3 ISI, and that if the glitching issues return, I should phone him if it is before ~8pm. Luckily, we have not had to deal with this for our current 24+hr lock!
After the glitches last night, I went to HAM3 to do the usual fixes. I powered off the corner 2 CPS, unplugged the boards in the satellite chassis at the chamber, put it all back together and powered everything back on. The CPS haven't glitched since then, so seems like maybe things are fine now.
The 65-100 hz blrms are a good witness glitches for a couple hours before they started tripping the HAM3 ISI. Attached trend shows about 5 hours before the first HAM3 trip on Monday. The top row are the raw H2 and V2 CPS in counts, the middle row is the watchdog state, and the bottom row are the 65-100hz blrms for the corner 2 and corner 3 blrms. The H2 and V2 cps start seeing glitches that don't trip the ISI about 3 hours before the first trip, these glitches don't really show up in the in the corner 3 CPS either. These glitches also don't coincide with locklosses, if the ISI doesn't trip. Under normal circumstances, these blrms are well below 10 nm, the first few glitches are up to 600 nm, but a glitch of ~1000nm causes the ISI to trip. There haven't been any glitches since I touched the CPS yesterday, so I think we are in the clear for now.
I'm still not sure of the right way to alarm on this, but some sort of days-ish timeseries trend when ISI trips on CPS would probably be a good place to start.
TITLE: 01/02 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
- Light maintanence day, mainly VAC turbo/ion pump work and a HWS card swap from the CDS team
- Ran an initial alignment after ISC went through CHECK MICH FRINGES thrice - ran without issue
Note that there was a large tow truck on site towing a car from 22:16 - 22:25 UTC. Tagging DetChar in case this shows up on their end as noise.
- Lockloss @ 23:10 - cause unknown
- H1 is currently relocking at TRANSITION FROM ETMX
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 16:06 | FAC | Tyler | LVEA/Mids | N | Check 3IFO items | 16:47 |
| 16:06 | FAC | Ken | EX | N | Lighting | 19:54 |
| 16:24 | VAC | Janos/Travis | MY/EY | N | Turbo test | 19:30 |
| 16:24 | FAC | Karen | EY | N | Tech clean | 17:30 |
| 16:25 | Kim | EX | EX | N | Tech clean | 17:44 |
| 16:27 | FAC | Randy | LVEA | N | Checks | 16:45 |
| 16:35 | FAC | Chris | LVEA | N | Pest control | 18:03 |
| 17:26 | CDS | Erik | MSR | N | Replace HWS sensor card | 21:18 |
| 17:39 | PEM | Mitch | EX/Y | N | DM FAMIS task | 18:19 |
| 18:16 | FAC | Karen/Kim | LVEA | N | Tech clean | 19:41 |
| 18:18 | CDS | Tony/Randy | EX/Y | N | Deploy end station laptops | 19:20 |
| 18:24 | VAC | Janos | LVEA | N | Ion pump work | 20:07 |
| 18:59 | EPO | Oli + tour | LVEA | N | Tour | 19:48 |
| 19:09 | SEI | Jim | CR | N | HAM 5 TFs | 19:31 |
| 19:57 | CDS | Tony | EY/EX | N | Set up workstations | 21:05 |
| 20:14 | ISC | TJ | LVEA | N | LVEA sweep | 20:30 |
| 20:19 | FAC | Ken/Richard | FCES | N | Checks | 20:39 |
| 23:11 | CDS | Tony | CER | N | Take pictures | 23:13 |
Both EY and EX now have Laptop charging stations in the garb rooms which contains 2 CDS laptops. These Laptops are actively being charged while they are plugged into the charging station.
Note: these laptops will not let you login if the wifi is turned off at the End station so be sure to let the opperator know that you need wifi at the end station.
The Laptops at EX are:
cdsdell425
cdsdell437
The laptops are EY are :
cdsdell431
cdsdell285
Lockloss @ 23:10, looks like DARM IN1 LSC saw the motion first - cause unknown.
I swept the LVEA after maintenance activities completed. Everything was nominal except for the CER lights were left on.
As part of maintenance I updated the x509 certificate for lhocds, it is good for another year.
H1 is still down for maintanence work, but should begin relocking in the next 15 minutes.
Dust monitors running smoothly.
Tue Jan 02 10:06:20 2024 INFO: Fill completed in 6min 17secs
Travis confirmed a good fill curbside.
Jonathan, Erik, Dave:
Erik is replacing the failed h1hwsmsr (ITMX) V1 computer with a spare unit this morning. I have stopped the temporary HWS-ITMX IOC (started on cdsioc0 yesterday) to prevent duplication of EPICS channels. This is the cause of the EDC and SDF disconnect channel counts.
TITLE: 01/02 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 2mph Gusts, 1mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.28 μm/s
QUICK SUMMARY:
- H1 down for maintanence day, looks like a relatively light day today
- CDS/SEI/DMs ok
