Lockloss @ 22:58, looks like ASC-AS_A_DC saw motion first. Also noticed that the IMC unlocked right before the lockloss (does this usually happen?)
H1 has just back it back to NLN (@ 19:47 UTC), after 18 hours of being down due to issues relocking. Problem was diagnosed to be a culmination of high microseism and the ISI SC guardians not being in the right state for all 4 quads after the master timing was swapped on Friday. The ISI SC issue has since been fixed and microseism is finally starting to trend down.
Sun Nov 05 10:10:16 2023 INFO: Fill completed in 10min 13secs
Last night, while LLO was down and before we lost lock, I used 1/2 hour to inject 13.1 Hz ground currents onto the chasis of AC1 in the CER, and also imitated the 13.1 Hz sound AC1 made using the large speaker. Neither the acoustic nor the electronic ground injections produced noise in DARM at the same level as AC1 had.
TITLE: 11/05 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: SEISMON_ALERT
Wind: 5mph Gusts, 4mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.43 μm/s
QUICK SUMMARY:
- Issues still ongoing from last night with a bad COMM/locklosses at TR_CARM, will start trying to troubelshoot
- Microseism is on the rise, making ALS hard to lock
- CDS/DMs ok
Lots of issues keeping GREEN_ARMS locked (maybe from microseism) and finding IR (keeps missing DIFF), we're unable to survive past CARM_TO_TR. The common mode servo board output signals don't look abnormal, there was some glitching but trending back it usually does this. I tried holding at START_TR_CARM and PREP_TR_CARM and we can survive, I looked at a bunch of the signals and they all seemed normal, MICH, ADS, IMC/LCS REFL_SERVO. When I tried to move on I got the NO IR in arms and we lost lock. Other times in the same state we lost lock at various stages of the CARM gain reduction.
Both the arms have been pretty unstable the whole shift (Y more so), but the wfs has managed to keep it locked for a decent amount of the time, it got worse towards the end. While the microseism is high we've been able to lock with it at the same levels before so it's probably not the sole issue but it certainly isn't helping.
All of the test mass BSC ST2 sensor correction guardians were complaining of being in the wrong state. The gains were all zero for these, so all of the test masses were moving more than intended at 1/2hz. Init-ing these guardians recovered the gains, then I init'd the SEI_ENV guardian to get everything back to managed. This allowed Austin to get past DRMI and finally to NLN. I should have glanced at this after hearing about the front-end crashes on Friday, all of the ST2 SC guardians had yellow notifications, so it was easy to spot. When we restart an ISI front-end we should probably cycle SEI_ENV through maintenance then back to CALM to make sure everything is in the right state.
I can't get past FIND_IR consistently, the last attempt COMM was doing nothing so I found it by hand then it was stuck in an endless cycle of "found seomthing, sleeping for 5 seconds". Next attempt was successful but there were no flashes on DRMI, went to CHECK_MICH which offloaded, still no flashes. I brought PR3 back to where it was and still didn't get any flashes and we went back into CHECK_MICH which again finished, then we were able to lock PRMI but not DRMI, got PRMI again then lost lock. I'm going to send out some calls for help now. Keita said he's going to check it out.
Made it up to CARM_TO_TR then lost lock, looks like greens arm wfs oscillated then killed it? Guardian was stepping up the LSC_REFL_SERVO_IN2GAIN as we lost it. Happened a 2nd time in the next attempt, hard to tell whats losing lock first, DRMI or green arms.
We made it to CARM_150_PICOMETERS and the IN2GAIN was fully stepped down to its -30 value then we lost lock as we were about to go to DARM_TO_RF
Weird PRCL oscillation/glitches that shows up during these CARM locklosses, which seem to be from the CARM gain reduction.
TITLE: 11/05 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:
Tagging in Ryan C to continue troubleshooting.
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 23:09 | PEM | Robert | CR/CER | - | PEM commissioning | 23:40 |
After our most recent LL from NLN we haven't been able to relock, microseism has increased quite a lot over the past 12 hours, yarms pretty unstable. The COMM beatnote isnt great -16, PR3 looks to have drifted a tiny bit in yaw? Not sure if its a real culprit. I think im going to try another IA and try to get the COMM beatnote better by following Jennes alog 73892 then call for help if I can't make any progress.
Almost a half hour into IA and Yarm still can't lock or even get above 75%
Now NoMachines giving me trouble and my ndscopes keep freezing and guardian logs are taking a full 2 minutes to open and I'm stuck at AQUIRE_XARM_IR in IA and nothing seems to be happening. I ended up bailing out of the IA and once green arms lock I tapped PR3 in yaw to bring it back to where it was based on the OPLEVS which brought the beatnote to -7 and then I ran into the ndscope issues.
None of my ndscopes are working anymore, it loads for 5 seconds then freezes, I've never had this issue before on NoMachine. After about 10 minutes they started working again
Lockloss @ 04:12 UTC - no obvious cause.
ETMX saturation right before; looks like LSC DARM saw first motion. Secondary microseism has been on the rise over the past 12 hours as well.
State of H1: Observing at 152Mpc
H1 has been locked for just over 24 hours. Quiet evening so far since PEM commissioning wrapped up.
TITLE: 11/04 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 4mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.28 μm/s
QUICK SUMMARY: H1 has been locked for over 20 hours. We've just started opportunistic PEM commissioning while L1 is down (no more than 30 minutes).
H1 is back to observing as of 23:32 UTC.
I've done some injections today but I'll alog about that later.
After I was done with the injections, the breakout board was removed from the back of the OM2 heater driver and the Beckhoff cable was fully connected again. This happened at around 21:40 UTC today.
Yesterday during the maintenance, Fernando and Daniel changed the Beckhoff configuration such that the Beckhoff terminal won't switch between thermistor 1 and thermistor 2 (alog 73886). Now it's only reading thermistor 1 (cold one) w/o switching. Thermistor 2 (hot one) is not read out. Detchar, please see if you can see the comb.
Good news- no sign of the 1.66 Hz comb in Fscan daily spectra after Nov 1 (checked the 2nd-4th).
Further change: Beckhoff was rewired to use 2 separate terminals to measure the 2 thermistors.
