WP12186

Richard, Fil, Erik, Dave:

We performed a complete power cycle of the h1psl0. Note, this is not on the Dolphin fabric so no fencing was needed. Procedure was

• Stop all models and power down h1psl0 computer
• Power down IO Chassis, all AA chassis and all AI chassis.
• Wait for a minute
• Power up AA chassis and IO Chassis (keep AI's off)
• Power up h1psl0 computer, wait for models to start
• Finally, power up AI chassis.

The system was power cycled at 10:11 PDT. When the iop model started, it reported a timing error. The duotone signal (ADC0_31) was a flat line signal of about 8000 counts with a noise of a few counts.

Erik thought the timing card had not powered up correctly, so we did a second round of power cycles at 10:30 and this time the duotone was correct.

NOTE: the second ADC failed its AUTOCAL on both restarts. This is the PSL FSS ADC.

If we continue to have FSS issues, the next step is to replace the h1pslfss model's ADC and 16bit DAC cards.

[   45.517590] h1ioppsl0: INFO - GSC_16AI64SSA : devNum 0 : Took 181 ms : ADC AUTOCAL PASS
[   45.705599] h1ioppsl0: ERROR - GSC_16AI64SSA : devNum 1 : Took 181 ms : ADC AUTOCAL FAIL
[   45.889643] h1ioppsl0: INFO - GSC_16AI64SSA : devNum 2 : Took 181 ms : ADC AUTOCAL PASS
[   46.076046] h1ioppsl0: INFO - GSC_16AI64SSA : devNum 3 : Took 181 ms : ADC AUTOCAL PASS

C. Compton, R. Short

I've updated the PSL_FSS Guardian so that it won't jump into the FSS_OSCILLATING state whenever there's just a quick glitch in the FSS_FAST_MON_OUTPUT channel. Whenever the FSS fastmon channel is over its threshold of 0.4, the FSS_OSCILLATION channel jumps to 1, and the PSL_FSS Guardian responds by dropping the common and fast gains to -10 dB then stepping them back up to what they were before. This is meant to catch when there's a servo oscillation in the FSS, but it's also been happening when the FSS glitches. Since moving the gains around makes it difficult for the IMC to lock, the Guardian should now only do that when there's an actual oscillation and should save time relocking the IMC.

FAMIS 31058

Trends are all over the place in the last 10 days due to several incursions, but I motsly tried to focus on how things have looked since we brought the PSL back up fully last Tuesday (alog80929). Generally things have been fairly stable, and at least for now, I dn't see the PMC reflected power slowly increasing anymore.

TITLE: 11/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 28mph Gusts, 19mph 3min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.56 μm/s
QUICK SUMMARY: H1 has been sitting in PREP_FOR_LOCKING since 08:05 UTC. Since the secondary microseism has been slowly decreasing, I'll try locking H1 and we'll see how it goes.

• Wind gusts moderate, microseism around 90th percentile
• All other systems nominal

TITLE: 11/04 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Microseism, LOCK_ACQUISITION
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

We spent most of the day with the secondary microseism mostly above the 90th percentile, it started to decrease ~8 hours ago but still remains about half above the 90th percentile. The elevated microseism today is from both the storms by Greenland and the Aleutians as seen by the high phase difference for both the arms with the corner station (bottom plot).

We keep losing it at the beginning of LOWNOISE_ESD_ETMX, I see some ASC ringups on the top of NUC29 (CHARD_P I think?), no FSS glitches before the locklosses, no tags on the lockloss tool besides WINDY. It's always a few seconds into the run method, I was able to avoid this by slowly stepping through some of the higher states (PREP_ASC, ENGAGE_ASC, MAX_POWER, LOWNOISE_ASC). I'm not sure which of these is the most relevent to not see the ringup, probably not the ASC states as when I paused there but forgot MAX_POWER I still saw the ringup.

LOG: No log.

• 00:53 UTC LOWNOISE_ESD_ETMX lockloss
• 01:57 UTC LOWNOISE_ESD_ETMX lockloss
  • These both had similar behavior to what Corey saw here alog80720 and TJ here alog80954
• 02:44 UTC MAX_POWER lockloss (did not ring up the violins luckily)
  • IMC lost it at the same time as ASC, FSS_FAST channel starting growing ~7 seconds before the LL
  • Over a dozen IMC locklosses trying to get to GREEN_ARMS
• 03:30 UTC LOWNOISE_ESD_ETMX lockloss
  • I tried stepping a little slower through some of the states leading up to it (PREP and engage ASC), I was going to also pause in MAX POWER but I accidentally moved it on
• 04:00 UTC PREP_DC_READOUT_TRANSITION lockloss
  • IMC and ASC lost it at the same time, some small FSS glitches starting ~350ms before the LL
  • PSL-FSS_AUTOLOCK_STATE moved ~20 ms before ASC
• 04:41 UTC lockloss TRANSITION_FROM_ETMX
  • Things were looking great (The ASC signals mainly), I had paused for 30-45 seconds after PREP_ASC and ENGAGE_ASC, then I waited 60 seconds after MAX_POWER and another 60 seconds after LOWNOISE_ASC finished
  • ASC lost it 150ms before IMC, I did not see any ASC ringups this time (IMC_REFL_SERVO_SPLITMON ran away?)
• 05:38 UTC TRANSITION_FROM_ETMX lockloss, lost it at a different spot in the run method than before
  • No ASC ringup, ASC lost it ~100ms before IMC

TITLE: 11/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Microseism
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently still trying to relock, and at POWER_25W. All day we've been working on getting back up, but the secondary microseism is so high. It is slowly coming down though. At one point, we were in OMC_WHITENING for over an hour trying to damp violins so we could go into NLN,  but we lost lock before we could get there.
LOG:

15:30 In DOWN due to very high secondary useism
15:38 Started relocking with an initial alignment
    - When we got to PRC, I had to do the Pausing-PSL_FSS-Until-IMC-Is-Locked thing again (see 81022)
    - 16:04 Initial alignment done, relocking
    - Lockloss from ACQUIRE_DRMI_1F, OFFLOAD_DRMI_ASC, ENGAGE_ASC_FOR_FULL_IFO, TRANSITION_FROM_ETMX
    - 18:07 Sitting in DOWN for a bit
    - 18:45 Started relocking
    - More locklosses
    - 21:48 Lost lock after sitting in OMC_WHITENING damping violins for over an hour
    - More trying to lock and losing lock                                                                                                                                               

 Most of the locklosses over this weekend have the IMC tag and those do show the IMC losing lock at the same time as AS_A, but since I don't have any further insight into those, I wanted to point out a few locklosses where the causes are different from what we've been seeing lately.

2024-11-02 04:17UTC

• No IMC tag, does have a DCPD tag
• Everything looking normal, then 100ms before the lockloss is first seen by anyone, PSL-ISS_AOM_DRIVER_MON_OUT shows a steep drop in value, getting larger in amplitude until the lockloss (ndscope1, ndscope2)

2024-11-02 17:13 UTC

• No IMC tag, does have a DCPD tag
• Everything looking normal, then 1s before the lockloss is first seen by anyone, PSL-ISS_AOM_DRIVER_MON_OUT shows a steep drop in value, getting larger in amplitude until the lockloss (ndscope3, ndscope4)
• Very similar to the above example, except the glitch isn't as steep. There is another one of these glitches that happens a few seconds before, but that one is even less steep and it is able to catch itself

2024-11-02 18:38:30.844238 UTC

• No IMC tag, does have a DCPD tag
• Small glitch in the FSS FASTMON channel caused NPRO temperature to spike and NPRO power jumps down, then ISS AOM DRIVERMON sees glitch up to the lockloss start (ndscope5, ndscope6-zoomed into FSS drop)

TITLE: 11/03 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Microseism
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 22mph Gusts, 18mph 3min avg
    Primary useism: 0.13 μm/s
    Secondary useism: 0.60 μm/s
QUICK SUMMARY:

• High microseism has made locking a struggle, in addition to the FSS issues
  • Violins have been rung up and we haven't stayed locked long enough to fully damp them for OMC_WHITENING
• Microseism seems to been slowly coming down
• Wind is increasing a little bit
• Relocking at CARM_TO_REFL

Closes FAMIS#27801

Last checked a week ago by Camilla, but since the water swap was done recently, I am doing the next check only a week later.

CO2X

• Before: 30.2
• After: 30.2
• Did not add any water

CO2Y

• Before: 10.2
• After: 10.2
• Did not add any water

There was no water in the leak cup.

Sun Nov 03 10:13:13 2024 INFO: Fill completed in 13min 10secs

TITLE: 11/03 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.09 μm/s
    Secondary useism: 0.72 μm/s
QUICK SUMMARY:

Looks like we have been sitting in PREP_FOR_LOCKING since a relocking lockloss at 10:52UTC(81032). I just started an initial alignment. The secondary microseism is really high so not sure how far we'll get

H1 has been having a hard time locking tonight due to very high microseismic motion. Unlocked at 05:43 UTC (lockloss tool) and has been down since; highest state I've seen H1 reach is TRANSITION_FROM_ETMX, alignment doesn't seem to be the problem.

TITLE: 11/03 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:Short locks, the IMC kept losing lock during SDF_REVERT/READY/LOCKING_GREEN as it did yesterday. Primary useism has been increasing over the last few hours (there's been more small US EQs today than normal... Missouri, Idaho, Nevada all in the 3s).
LOG: No log

• 22:15 UTC Observing, low range check
• 00:44 UTC lockloss (2:29 lock)
  • 01:09 DRMI_TO_3F lockloss
  • 02:00 UTC Observing
• 02:28 UTC lockloss (0:31 lock)
  • 03:45 UTC Observing
    • Range isn't so good
  • 03:43-04:45: UTC dropped observing from the SQZer losing lock
    • The FC keeps losing IR lock trying to get GR locked, it says "LO SERVO RAILED" alog81027

The FC keeps losing IR lock, it says "LO SERVO RAILED" pretty much as soon as the LO_LR grd startings working after the FC locks IR.

Building on work last week, we installed a 2nd PI AI chassis (S1500301) in order to keep the PI signals separate from the ESD driver signals.  Original PI AI chassis S1500299.

We routed the LD32 Bank 0 thorugh the first PI AI chassis to the ESD drive L3, while keeping the old ESD driver signal driving the PI through the new PI AI chassis.

We routed the LD32 Bank 1 to the L2 & L1 suspension drive.

We did not route LD32 Bank 2 or Bank 3 to any suspensions.  The M0 and R0 signals are still being driven by the 18 bit DACs.

The testing did not go as smoothly as planned, a watchdog on DAC slot 5 (the L1&L2 drive 20 bit DAC) continousouly tripped the ESD reset line.  We solved this by attaching that open DAC port (slot 5) to the PI AI chassis to clear the WD error.

Looks like we made it to observing.

F. Clara, R. McCarthy, F. Mera, M. Pirello, D. Sigg