ryan.crouch@LIGO.ORG - posted 01:01, Monday 10 July 2023 (71188)
Lockloss @ 07:57 UTC
DCPD saturation then lockloss
DCPD saturation then lockloss
TITLE: 07/10 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 137Mpc
SHIFT SUMMARY:
IFO is in NLN and OBSERVING. Locking was tough to achieve today. As reported during the midshift update, initial alignment just couldn't work and as such, we skipped it when manually aligning its failing parts (MICH and SRC) and normal locking went smoothly. DRMI was initially acquired without resistance but then lock was lost thereafter. On the second trial, PRMI had to be locked first but then DRMI locked easily. We paused briefly at Engage DRMI ASC in order to ensure that the RF18 and RF90 lines were stable. At this stage, I touched the beamsplitter ever so slightly in order to bring the beams to a circular shape on the AS_AIR cameras. Locking went smoothly until OMC_Whitening, which took the longest time to progress from due to terrible violin modes (this week's culprit I'm told). While waiting, learned that sometimes, guardian might change the gain values for the violin mode filter banks but then not change them back, resulting in slower progress. After learning from Camilla and Austin, I bypassed this by changing all the gain values to their nominal values (slowly and incrementally). This still took a while, but finally, NLN OBSERVING was reached at 5:28 UTC.
Other:
- I tweaked the SQZ OPO Temp per Vicky's request during the long OMC Whitening phase. The values of the temperature were changed to maximize the green trace as per step 3 of the instructions in alog 70050. Both screenshots demonstrate the change in the green trace. The first (1) is right after the change and the second (2) is while in NLN Observing. The overall improvement was from ~0.0658 to 0.0666. The SDF difference this temperature change induced was accepted in order to move to observing.
- Since the mid-shift update, Daniel has posted more details concerning the potential issues with initial alignment. We did NOT fix it and this issue will likely occur in the future as this is the third time in a row MICH failed to lock during initial alignment and not even the first time this step was skipped. Work will lilkely need to be done this week to see what this issue is.
- Ryan C had a similar issue during initial alignment with locking SRC where he could not get the differential to be remotely near a delta of 5 (highest I could get is 3.5) in the "high highs and low lows" value for H1:ASC-AS_A_DC_NSUM_OUT16*0.001. He also just bypassed the issue by moving to normal locking, where it worked. This is documented in alog 71026. Again, that issue was not fixed.
- Rahul will be working on violin modes remotely from India (and will be in communication with Ryan C, the next operator).
- Special thanks to Camilla, Austin and Daniel who worked with me (and I, them) to get this thing locked!
LOG:
TITLE: 07/10 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 137Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 18mph Gusts, 15mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.07 μm/s
QUICK SUMMARY:
IFO is still not locked. Daniel and Austin (during and post-shift) and thankfully now, Camilla have all tried helping but so far we don't know what the problem is. Here are some of the leads.
Initial Alignment is not, and has not been working since 12PM today (and from last week's initial alignment, this seems to be a known issue). Firstly, MICH_BRIGHT and MICH_DARK do not lock according to guardian (but they seem to lock according to Austin and Daniel at the start of the shift). The behavior of the MICH trace is oscillating between the threshold (4000 cts) and 0, which is odd. After getting MICH "as locked as possible", we (Daniel, Austin and I) manualled over to SRC locking, which also failed in a similar way. Initial alignment did not work at all. According to Daniel, the gain had to be bumped 10x to lock MICH but then the init_alignment just failed completely afterwards (everything swinging). I tried touching the various documents' manual alignment sliders (BS, SRM, SR2) to no avail. Even during SRC manual locking, the differential in the "high highs" and "low lows" was nowhere near what it trended as. With zero indications of the problem after parsing through the code and attempting to call other people, Camilla offered help.
With this help, we could not get locked still so we thought to try normal locking without any confirmation that initial alignment works. Somehow, this worked. IFO is currently powering up. DRMI locked easily while bumping the BS during Engage_DRMI to better the usual signal traces.
In diagnosing the issues with the help of past alogs and trends, the last time MICH offloaded successfuly was June 26th. It failed on the 1st, 3rd and today. While the locking issue seems to have been fixed, we still have not fixed the initial (incl. manual) alignment locking problems. Expecting to be locked within the next 30 minutes now.
Special thanks to Austin, Daniel and Camilla for volunteering their time to assist with this eventful eve.
Summary:
ASC-AS_A seems to have been badly clipping, but the LSC signal came from the two quadrants on which only a tiny fraction of the beam power was falling. A bad recipe as alignment-length sensing coupling is maximized.
Details:
In the 1st attachment (same time window as Daniel's plot, but more channels), the right-most column shows the AS_A_DC_NSUM was about half of AS_B_DC_NSUM. (FYI this is never the case in e.g. full lock.)
The second column from the right shows AS_A_DC_PIT_OUT16 was basically 1 except some transient responses (circled in red), which means that all of the power was on quadrants 1 and 2. If you look at AS_B_DC_PIT and YAW on the same column you can see that the beam was not clipping on AS_B.
2nd and 3rd column from show that most of the signal was in quadrants 1 and 2 (unsurprising, but they were in I phase not Q as was pointed out by Daniel). However, for whatever reason, quadrants 3 and 4 seem to have somewhat better (but not much better) demod phase, therefore Q3 and Q4 were actually larger than Q1 and Q2.
This resulted in AS_A_RF45_Q_SUM signal mostly coming from Q3 and Q4 (circled in cyan).
I cannot remember if any kind of DC centering is done during initial alignment, but if not, we need one.
Also measure AS_A and B RF45 phase in full lock (maybe we can check this using calline or violin?).
Here some traces when we tried to lock the bright Michelson. A couple observations:
Turns out that there's no DC centering on AS WFS during BRIGHT MICH in the initial alignment. See the first attachment.
45MHz demod phase was OK for full lock with 60W power for both AS_A and AS_B. 2nd and 3rd attachment. I don't know if that's the case for lower power. But the demod phase is clearly off for initial alignment, and I don't want to bump up the LSC gain by a factor of 10 to compensate.
What we need is:
TITLE: 07/09 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 14mph Gusts, 10mph 5min avg
Primary useism: 0.06 μm/s
Secondary useism: 0.07 μm/s
QUICK SUMMARY:
IFO is down - getting debrief from Daniel and Austin as to what the locking problems are. Currently having issues with initial alignment.
TITLE: 07/09 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY:
- Began shift with looking into a potential electronics failure on ITMX, causing ISI WDs to trip and breaking the lock
- 17:30 - inc 5.6 EQ from Chile
- After recovery, I began an initial alignment, which again got stuck trying to align MICH BRIGHT (an issue independent of today's events) - even after touching the BS and getting the peaks/trough to have a delta of 3.5
- 22:34 - inc 4.6 EQ from Taiwan, 22:54 - EQ mode activated
- Unfortunately have to pass H1 to Ibrahim with the issue still ongoing, Daniel is currently on TS helping, and I have sent an additional SOS out to MM
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 17:50 | SEI | Jim | CER | N | Power cycle ITMX chassis to diagnose glitching issue | 19:53 |
I came in to investigate the ITMX ISI troubles. Seems like the problem was (is?) an issue with the ISI coil drivers, which is a new one for me.
What I tried so far this morning:
1. Killing all requested drives in the model, turned off masterswitch.
2. Checked rack, all of the chassis seemed to be in a normal state, no missing lights.
3. Power cycled ISI interface chassis, T240 chassis. No change.
4. Turned off all the coil drivers. Boom, oscillation goes away. See attached trend. L4Cs go to 0 when I turn off ISI interface chassis, but come back in oscillating state. The ring down is when I powered off the coil drivers.
5. Turned each coil driver one at a time to see if it comes back. All quiet, but there is a kick each time a chassis gets turned on. Must be some offset somewhere in the electronics? Model is still requesting 0. I haven't seen that before, worth doing some more testing.
6. Turned on damping loops, everything is still quiet.
7-? Start bringing ISI back. Get sidetracked by blend settings not being INIT'd in SDF....
Sunday morning ISI ITMX problem started around 06:00:50 PDT. I checked the MAINS_DQ channels around that time, did not see any AC power issues.
Sun Jul 09 10:07:40 2023 INFO: Fill completed in 7min 40secs
WP11288
root@h1daqframes-1:~# zpool status
pool: frames-1
state: ONLINE
scan: scrub repaired 0B in 3 days 22:44:28 with 0 errors on Sun Jul 9 07:49:57 2023
Jim, Ryan, Austin, Dave:
We restarted the h1isiitmx model as part of the lock-loss investigation.
Sun09Jul2023
LOC TIME HOSTNAME MODEL/REBOOT
08:13:50 h1seib3 h1isiitmx
model restart did not fix anything, we are now power cycling the IO Chassis
IO Chassis power cycle did not fix anything. Austin reports all the B3 chassis externally look good (green leds).
Sun09Jul2023
LOC TIME HOSTNAME MODEL/REBOOT
08:13:50 h1seib3 h1isiitmx
08:56:51 h1seib3 ***REBOOT***
08:58:24 h1seib3 h1iopseib3
08:58:37 h1seib3 h1isiitmx
08:58:50 h1seib3 h1hpiitmx
SEI is heading to the site to investigate
The psinject memory leak had almost used up all the memory on h1hwinj1. During the lock-loss I restarted the process and usage is now 7%, which should get us through to Tuesday maintenance.
TITLE: 07/09 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Unknown
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 10mph Gusts, 7mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY:
- IFO is currently DOWN due to potential electronics failure during Ryan C. shift, around 13:00 UTC - (1 / 2), Jim/Dave are currently on TS trying to diagnose
- DMs/ground motion low
TITLE: 07/09 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Unknown
SHIFT SUMMARY:
Superevent S230709bi
Lockloss at 13:01 from the ITMX ISIs tripping, RIP to the new longest lock of O4 of 54 hours
Went through the call list, got ahold of Jenne and Keita, who came on to TS and helped diagnose. We're seeing glitches in the T240 channels, the glitches only stopped once we went into OFFLINE (We later noticed that this was just due to the channels being saturated), we tried turning off the master switch and glitches persisted so this might be an electronics issue. Called Jim again and he answered and is getting online soon as of 14:52UTC.
Jim is currently on TS helping diagnose and I'm handing off to Austin as of 15:00
LOG:
No log for this shift
ITMX ISI stage 1 & 2 watchdogs tripped which caused a Lockloss, Seismon was showing an increase in motion but not by that much. The T240s saw a large increase in motion, the values are going crazy. LSC_CPS_FF is now a lot noisier than it was before the trip
I tried to untrip the WDs after the T240 values had calmed down but it retripped. The BLRMs are also all very high
I called Jim and left a message
JoeB and I picked a time, and we're running /opt/rtcds/userapps/release/pem/h1/scripts$ python inject_mag_10to40.py (attached here with the current gps time we used).
I also made sure that our amplifiers were on (I think / hope) via the medm screen, which I've also attached.
In case it's helpful, here's the state of the filter bank.
Injection was succesfull.
Fig1 &2: Hx magnetometer - Lx magnetometer: Coherence/CSD, before, during and after injection
Fig3 &4: H strain - Lstrain: Coherence/CSD, before, during and after injection
Channels used:
Hx mag = H1:PEM-CS_MAG_LVEA_VERTEX_X_DQ
Lx mag = L1:PEM-CS_MAG_LVEA_VERTEX_X_DQ
H strain =H1:GDS-CALIB_STRAIN
L strain = L1:GDS-CALIB_STRAIN
Times used:
Before: start: 900sec before injection - duration: 300 sec, 10 sec fft, 50% overlap
Injection: start gps = 1372010416 (June 28 - 17:59:58 UTC) - duration: 300 sec, 10 sec fft, 50% overlap
After: start: 600sec after injection - duration: 300 sec, 10 sec fft, 50% overlap
