The EY Slow controls issue escalated overnight, DEV5 now has no communication with EY.
The link deteriorated over the course of 5 hours, starting at 23:24 Sat 19sep2026 and was completely lost by 04:23 Sun 20sep2026.
Trending the slave_count_actual for DEV5 shows it erratically drops from 127 to 1 over this time period.
I turned off the JAC heater servo at 23:45:00 UTC today to do a longer step response test over the weekend.
TITLE: 09/18 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Keita worked more on PRMI and ASC, we now have working ASC MICH loops. We also locked DRMI around 23:00 UTC at ~68 and ~12. The pump down continues over the weekend on both ETMX and the X/Y BMs.
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 15:17 | VAC | Jordan | LVEA | N | Restart X/YBM pump downs | 15:43 |
| 16:29 | VAC | Jordan | LVEA | N | Check on pump down | 16:52 |
| 20:09 | FAC | Mitch | LVEA | N | FAMIS tasks, HEPI checks... | 20:53 |
| 21:01 | VAC | Jordan | EndX | N | Check out pump down, turn off compressor | 21:34 |
| 21:34 | VAC' | Jordan | LVEA | N | Check on pump down | 22:34 |
Short shift tonight.
ASC-MICH loop works after ASC-AS_B rephasing (it looked nonsensical before). I have no idea why they changed. AS_A phase was OK though not great, no adjustment was made.
Before making any adjustment:
I refined PRMI alignment manually, locked PRMI and measured the response of ASC-AS_B_RF45 used for MICH. I also measured AS_A just because.
In the first attachment (AS_B_RF45_VS_AS_A_before_phase_adjust.png), I'm misaligning BS in PIT both ways without enabling MICH nor PRC1 ASC feedback (but did enable DC3 and DC4 to center the beam on AS_A and AS_B) to test how AS_A and AS_B RF45 output respond.
MICH ASC error signal (AS_B_RF45_Q_PIT, yellow) is useless as it has a very small response to BS misalignment and it never crosses zero. This is what Elenna, Louis and I observed yesterday, too. In comparison AS_A_RF45_Q_PIT looks more promising (but should not).
In ASC-AS_B_RF45_phase.png, I'm injecting PRCL (not angle) dither at 80Hz to PRM M3 stage TEST_L and see the line in on each quadrant of each WFS. The idea is:
As you can see, AS_B doesn't make sense. For all AS_B quadrants, PRCL line shows stronger in Q than in I, and the phase of the third quadrant is flipped.
ASC-MICH works after rephasing.
I made the following changes to roughly minimize PRCL line in Q AND to make the phase relation of all I signals the same relative to PRCL error:
| before | after | |
| H1:ASC-AS_B_RF45_SEG1_PHASE_R | -59.698 | 10 |
| H1:ASC-AS_B_RF45_SEG2_PHASE_R | -31.5 | 40 |
| H1:ASC-AS_B_RF45_SEG3_PHASE_R | 146 | 20 |
| H1:ASC-AS_B_RF45_SEG4_PHASE_R | -80 | 10 |
The phase numbers themselves look more similar to each other now, PRCL line in Q phase is much smaller in all quadrants (ASC-AS_B_RF45_phase_after_adjustment.png), and most importantly, ASC-AS_B_RF45_Q_ERR responds to the MICH misalignment sensibly with good amplitude and it crosses zero (AS_B_RF45_VS_AS_A_after_B_phase_adjust.png).
Dark offset adjustment might be needed, though.
Anyway, after this adjustment ASC MICH loop worked fine with ASC-MICH_P_GAIN and Y_GAIN of 0.5. See ASC-MICH_works.png.
Demod phase values were accepted in sdf.
ASC PRC1 is not working yet.
According to Elenna, she's having a hard time finding a good PRC1 sensor in PRMI these days. She's been trying to use POP_X (because REFL sensors stopped working after the last vent, and because we haven't phased REFL WFSs for which we want to use CARM for phasing).
Since the POPX signal level is fairly small in PRMI, I set the dark offset (and accepted in SDF), locked PRMI, enabled DC6 loop to keep the POP_X centered, and moved PRM in PIT and YAW manually. Turns out that PIT never crosses zero, it comes close but then stops changing there. YAW looked OK in that at least it crosses zero.
When turning the servo ON with an integrator, YAW doesn't run away but PIT does, see POPX_P_not_working.png. In the same screen shot, you can see that the 80Hz PRCL dither appears in Q phase not I for all quadrants. I temporarily changed the phase so the dither shows up in I not Q, but PIT still didn't cross zero so I reverted them back.
| Original (current value) | Minimize PRCL in Q (reverted to original value) | |
| H1:ASC-POP_X_RF_SEG1_PHASE_R | -21 | -70 |
| H1:ASC-POP_X_RF_SEG2_PHASE_R | -15 | -90 |
| H1:ASC-POP_X_RF_SEG3_PHASE_R | 18 | -70 |
| H1:ASC-POP_X_RF_SEG4_PHASE_R | 10 | -90 |
Ryan C, Dave:
The Beckhoff CS-AUX-DEV5 slave_count_actual dropped from 127 to 0 five times between 08:01 and 08:07 this morning. As far as we know there is no Beckhoff work ongoing at this time.
Now DEV5 (End Y) is continuously showing an "at least one device not in OP state" error. The FRAMEWORKINGCNT MEDM shows Error_2 is in error. Trend shows how this error relates to the slave count drops.
Erik and I investigated this a bit more. We found the recent error appered on the EY ISC PLC. It is reporting ALS and ASC errors. The attached MEDMs show the tree for ALS -> QPDA -> Whitening which ends with Invalid Data.
We think that if this fault only impacts ALS and ASC at EY, it most probably can be left until Monday for the fix. We are contacting Daniel and Fil to confirm.
TITLE: 09/17 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Corey
SHIFT SUMMARY: *I forgot to click post for this last night*
The LVEA was transitioned to SAFE, the CHETA viewports were installed, EndX pump down continues, and the HAM2 CPS cables were rerouted. Commissioners spent the afternoon touching up some corner alignment then they worked on PRMI and some phasing. A few optics lab and PSL anteroom dust alarms in the late afternoon, corresponding roughly with wind increase but the wind increase 3 hours previously did not increase the dust counts.
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 15:26 | OPS | Betsy | LVEA | Y -> N | Transition to laser SAFE | 15:42 |
| 15:30 | VAC | Jordan | EndX | N | Check on pump down | 16:22 |
| 15:37 | FAC | Kim | FCES | N | Tech clean | 16:49 |
| 15:40 | VAC | Gerardo | LVEA | Y -> N | Vent BSC7 volume XARM | 16:43 |
| 15:42 | ISC | Betsy | LVEA | N | Checks | 15:44 |
| 15:48 | CRS | Jim | LVEA, H2 | N | H2 PSL room, wrap and bag parts | 18:16 |
| 15:49 | EE | Fil | LVEA | N | HAM2 cps cabling, L4C chassis work | 18:26 |
| 15:49 | VAC | Camilla | LVEA | N | Bring out viewport parts | 15:59 |
| 15:51 | FAC | Mitch | LVEA | N | Checks | 16:00 |
| 16:25 | FAC | Chris + contractor | LVEA | N | North bay crane work | 18:13 |
| 16:34 | VAC | Jordan, Travis | EndX, High bay | N | Bring pumps from EndX to high bay | 17:08 |
| 16:43 | VAC | Camilla, Gerardo | LVEA | N | Yarm viewport swap with Gerardo | 20:00 |
| 16:47 | FAC | Richard | LVEA | N | Safety checks | 17:37 |
| 16:47 | TCS | TJ | LVEA | N | Cleaning up | 17:54 |
| 17:08 | VAC | Jordan | LVEA | N | Viewport swap | 20:17 |
| 17:30 | CAL | Tony | EndX | N | Turn on PCAL laser | 17:59 |
| 17:31 | VAC | Travis | LVEA | N | Check with VAC crew on viewports | 18:50 |
| 17:56 | FAC | Kim | MY | N | Technical cleaning | 18:46 |
| 18:00 | FAC | Richard | LVEA | N | Give crane documents to Travis | 18:07 |
| 18:16 | SEI | Jim | LVEA | N | Join Fil to finish CPS cabling | 18:26 |
| 18:32 | EE | Fil | LVEA | N | Remove camera used for cartridge removal, by Yarm | 20:29 |
| 20:43 | VAC | Jordan | LVEA | N | Start pump down for BSC8/7 volumes | 22:12 |
| 20:46 | CAL | Tony | PCAL lab | LOCAL | New laser inspection | 22:16 |
| 20:47 | TCS | TJ | High bay | N | Water access | 22:09 |
| 21:06 | VAC | Gerardo | LVEA | N | Join Jordan with the pump down | 22:11 |
| 21:07 | VAC | Richard | LVEA | N | Help VAC team with the pump down | 21:31 |
| 21:17 | VAC | Travis | LVEA | N | Join VAC team | 22:06 |
| 22:11 | TCS | TJ | LVEA | N | Put buckets under new tcs lines just in case | 22:21 |
| 22:44 | TCS | TJ | LVEA | N | Cover outlet hole | 23:00 |
| 23:16 | TCS | Camilla | Vac prep, Optics, and LVEA | N | Clean up | 00:16 |
| 23:22 | CRS | Jim, Mitch | LVEA | N | H2 PSL wrap and bag | 00:22 |
TITLE: 09/18 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 5mph Gusts, 2mph 3min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.07 μm/s
QUICK SUMMARY:
Whoops, I forgot to post EVE summary last night (left site around 730pmPDT; here it is:
TITLE: 09/17 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY:
Elenna/Louis/Keita were looking at MICH Pitch ASC Error signal with PRMI at beginning of the shift (see Elenna's alog).
Using PRM to put a line on AS_A_B and look at all quadrants' I & Q. Want to see the line only on Q, so adjust the Rotation Angle of AS_A_B while watching I's & Q's
Once commissioning work was done, handed h1 over to Jim for HAM2+3 measurements.
LOG:
(Travis S., Jordan V., Richard M., Gerardo M.)
After the installation of CHeTA viewports, we connected an ISP-1000 scroll pump and a SS-500 cart to each of the main turbo pumps at the X beam manifold and Y beam manifold. After getting power sorted out to the ISP-1000s, thanks Richard, we started the pumpdown for both of the manifolds.
Something that we did noted was the new gauge for PT-180 shows almost 1000 Torr for pressure at atmosphere, this is not correct, then the other gauge for PT-170 shows 750 Torr for pressure at atmosphere, also not right.
Pumpdown has been stopped at the end of the day, and will be restarted tomorrow morning by Jordan.
Attached is a plot of both manifolds initial pumpdown.
Continued pumpdown of the XBM and YBM this morning, using 1x ISP1000 and 1x ISP500 for each volume.
| Starting Pressure: 45 torr | Start Time: 8:25 am |
| Turbo Crossover Pressure: 500 mtorr |
Turbo On: 3:10 pm (XBM) 3:25 pm (YBM) |
Total roughing time was ~ 10 hours, 3 hours and 15 minutes yesterday + 6 hours and 45 minutes today.
Once the turbopump reached full speed and chamber pressure was ~5E-5 Torr, I transitioned from the roughing pumps to the dedicated ISP250 and 80 l/s turbopump for the foreline.
Cooling water was adjusted so that pump and controller temps were ~40C. Setpoints adjusted to 1E-5 Torr.
We will need to leak test the 3 newly installed viewports before opening gate valves.
[Elenna, Louis]
After the HAM2 work (91959) finished today, we decided to double check AS port powers in a couple states. I trended back about a week to September 10, long before the RCG upgrade (91936). We are seeing a ~20% drop in counts on AS_A and AS_C in PRX_LOCKED and SRY. Also, our PRMI buildup right now is about 56. Last time we locked PRMI without ALS it was 52.
|
ALIGN_IFO State |
Reference Time | AS_A (ref) | AS_A (now) |
AS_A |
AS_C (ref) | AS_C (now) | AS_C now/ref (%) |
POP_A_LF (ref) | POP_A_LF (now) | POP_A_LF now/ref (%) |
|---|---|---|---|---|---|---|---|---|---|---|
| PRX_LOCKED (N=32) | 2026/09/10 15:35:29 UTC | 1240 ct | 992 ct | 80 % | 0.01465 W | 0.0118 W | 81 % | --- | --- | --- |
| MICH_DARK_LOCKED (N=43) | 2026/09/10 15:41:13 UTC | --- | --- | --- | --- | --- | --- | 4.18 | 2.25 | 54 % * |
| SRY (N=62) | 2026/09/10 15:31:56 UTC | 5147 | 4127 | 80 % | 0.061 W | 0.05 W | 82 % | --- | --- | --- |
| SR2_ALIGN (N=59) | 2026/09/10 15:35:35 UTC | 1868.5 | 1797 | 96 % | 0.0224 W | 0.0216 W | 96 % | --- | --- | --- |
* Not sure how comparable this number is since we don't know much about the centering on the LSC POP diode at the reference time. POPX pit & yaw are ~ -1 during this reference time stretch.
After we locked PRMI today, I tried to engage the MICH ASC. The first try failed, so I tried engaging by hand. I had previously phased the AS B RF45 signal to maximize MICH length in Q, so I thought the error signal was fine.
Then, I checked the MICH P error signal. I first moved PRM and BS to try to maximize the POP18 buildup. Then, I tried moving BS pitch to check if the error signal crossed zero when the signal was maximized. That did not happen.
So I checked the phasing again- three segments needed to be adjusted to improve the phase.
Then, I tried my error signal test again. It still does not cross zero.
I attached my phasing attempts and the SDF, but maybe I am not phasing this correctly- maybe it is better to check in a MICH fringe.
Over the last few days, Oli and I removed four unused chiller lines at the Yarm termination slab. Two were old TCS lines that were already cut off and mostly empty, the other two were valved out around the spool area but unused after. We cut, vacuumed, and removed all the copper pipes and their support hardware up to that spool area. Still to do is solder an end cap to the two cut lines.
WP 13630
The PTZ camera used to document the BSC2 dome and cartridge install/removal was uninstalled. Camera will be deleted from the access system camera log.
(Travis S., Jordan V., Gerardo M.)
Today we were only able to vent the YBM (manifold on the Y-Arm), to do so, we hard closed GV1, GV2, GV5 and GV7. GV5 and GV7 are pneumatic valves, they closed with pressure of 25 psi both, very good pressure. Then we measured the purge air dew point, -42.1 oC. One issue that we encountered was that it took a while to vent to atmosphere, the wide range gauge at PT180 is way off indicating 998 torr for a vented to atmosphere volume, should be more like 762 Torr today, we will check on how to reset that value.
Vacuum envelope for the YBM is vented and ready for the viewport work tomorrow. However, the manifold for the X-Arm still under vacuum, no work should take place there until fully vented.
After closing up BSC9 yesterday we started pumpdown of the EX volume this morning using the two mobile ISP1000 scroll pumps.
There was minimal overpressure in the system, so I could not get a good dewpoint measurement of the blow down air.
It took ~6 hours to rough the system down from atmosphere to ~500 mtorr, where we start the turbopump.
The cooling water was adjusted on the turbopump stand so that the pump and controller pressures were <= 40C. Once the turbo was at full speed and the inlet pressure ~5e-5 Torr, I switched the backing pump to the dedicated ISP250 on the turbo stand.
Setpoints were adjusted to 5E-2 Torr on Channel 1 (Foreline) and 5E-5 Torr on Channel 2 (Turbo inlet).
We will need to leak check the TMDS port gate valve since it was removed to verify the gate o-ring was in place.
Also, the EX cleanroom was powered off at ~3:10 pm local time.