TITLE: 05/12 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 1mph Gusts, 0mph 3min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.17 μm/s
QUICK SUMMARY: Systems appear to be fully recovered after the h1susb123 crash and output arm turbopump shutoff this weekend. More vent work planned for today includes wrapping up HAM1 alignment and more wind fence work at EX.
I noticed that the pressure in the LVEA spiked up several hours ago, see attachment, typical behavior of a turbo pump turning off. Drove to the site and found the output tube main turbo pump off, this system was isolated from the main volume first, then the turbo system was restored; scroll pump running, small turbo up to speed, and main turbo pump up to speed, only then the turbo system was valved in to the main volume. Nothing out of the ordinary was found or noted. As of now no power glitches have been reported.
At 15:36:55 Sun 11may2025 PDT all models on h1susb123 stopped due to an ADC/Timing error.
h1susb123 DMESG output:
[Sun May 11 15:36:49 2025] rts_cpu_isolator: LIGO code is done, calling regular shutdown code
[Sun May 11 15:36:49 2025] h1iopsusb123: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Sun May 11 15:36:49 2025] h1susitmpi: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Sun May 11 15:36:49 2025] h1susitmy: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Sun May 11 15:36:49 2025] h1susbs: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Sun May 11 15:36:49 2025] h1susitmx: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
(diskless)controls@h1susb123:~$
If I do a "sudo lscpi" this command freezes for about 30 seconds, then eventually returns. While it is frozen, the EDC disconnects from h1susb123 EPICS channels.
This is a 8-core W2245 machines with 5 models, so 3 general purpose cores. The IOC disconnect suggests the lscpi command is taking over all 3 cores.
Interestingly h1iopsusb123 is not in a DACKILL state, its DAC drives are non-zero, frozen at their last values (see attached).
The IPC senders have stopped updating, causing IPC receive errors downstream and SEI SWWD trips.
System has been recovered by power cycling the front-end computer, no IO Chassis restart was needed.
Procedure:
"rtcds showcards" sees all the ADC/DAC/BIO cards in the IO Chassis.
"sudo lscpi" completes immediately, no 30 second freeze.
I have untripped the SWWDs (SUS and SEI) for BSC1,2,3.
I have not ununtripped any user model watchdogs, will hand over to control room for that.
I've brought the BS, ITMX, ITMY suspensions back to Aligned after untripping the user WD. I also untripped the sei for those and watched them go back to nominal. All looks good.
Sun11May2025
LOC TIME HOSTNAME MODEL/REBOOT
16:52:48 h1susb123 ***REBOOT***
16:55:12 h1susb123 h1iopsusb123
16:55:25 h1susb123 h1susitmy
16:55:38 h1susb123 h1susbs
16:55:51 h1susb123 h1susitmx
16:56:04 h1susb123 h1susitmpi
Sun May 11 10:11:51 2025 INFO: Fill completed in 11min 47secs
Sat May 10 10:10:22 2025 INFO: Fill completed in 10min 18secs
After GV7 was closed, the HAM3 table optical lever transceiver components were removed, with the exception of the viewport guard assembly. The items were on port VP7 of MCA1 module.
All items removed were stored in the optical lever cabinet, the cabinet is located at the south bay of the LVEA.
TITLE: 05/09 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:
Bees:
There is a box potentially full of bees sat right next to a spool of wire and a chair at EX as the Bee keeper is trying to remove the beehive from the wire spool.
There is still a bee swarm near the gate as well.
HAM1 work:
The IST1 table has been moved back into place.
GV7 has been soft closed.
Rahul took some SUS transfer functions to check for rubbing.
Laser power was turned up at the Request of Camilla & Sheila to aid in finding the IR beam spot alignment on to the ISCT1 table.
Wind fence:
Wind fence team got another set of cables strung up for the WindX work down at End X
GRB-Short E570982 @ 22:34:04 UTC
GRB-Short 570981 @ 22:35:15 UTC
Oli & Camilla took the Laser power back down to 2W for trhe night at 23:39 UTC.
LVEA is still LASER hazard.
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
15:07 | SAF | LVEA is Laser HAZARD | LVEA | YES | LVEA is Laser HAZARD | 07:07 |
14:58 | Bee | Tyler & the BGs | End X | N | Tyler taking the Bee Guy to EX to potentially remove bees | 17:36 |
15:21 | FAC | Jim, Randy | EX | N | Working on Wind fence & Bees | 21:08 |
15:30 | FAC | Kim & Nellie | LVEA | - | Technical Cleaning | 16:10 |
15:38 | ISC | Sheila & Camilla | LVEA HAM1 | Yes | Aligning optics on HAM1 | 19:30 |
15:49 | SUS | Rahul | Ham1 | N | Checking for rubbing on RM1 | 16:10 |
15:54 | ISC | Keita | LVEA HAM1 | - | talking wwith Sheila | 19:30 |
16:25 | PEM | Robert, carlos , Kiet | The Vault in the Desert | N | PEM install work. | 19:58 |
17:15 | HAM1 | Betsy | LVEA HAM1 | n | Checking on HAM1 team | 18:43 |
17:32 | FCA | Tyler | LVEA Ham1 | N | Moving the ISCT1 table back into place. | 18:11 |
19:32 | VAC | Gerardo Y Travis | LVEA +X Arm | - | Soft closing GV7 | 20:08 |
19:47 | PCAL | Francisco | PCAL Lab | Yes | Finishing up a measurement | 19:52 |
20:19 | SUS | Rahul | Remote | N | Taking RM SUS TF | 21:39 |
20:32 | ISC | Keita & Sheila | LVEA HAM1 | YES | Working on IR beam path inside HAM1 | 00:32 |
20:42 | ISC | Jennie W | LVEA HAM1 | Yes | Helping with Beam alignment | 22:44 |
21:39 | VAC | Gererdo | LVEA Ham2 +X | N | Decommisioning Optical laver housing | 22:20 |
21:56 | PSL | Oli | LVEA | Y | Turn up PSL to 20Ws | 22:05 |
21:59 | EE | Marc | MidY | N | Dropping off parts | 22:59 |
22:05 | ISC | Camilla | LVEA HAM1 | Yes | Helping with HAM1 work | 23:05 |
23:34 | PSL | Oli | LVEA HAM1 | Yes | Turning the power back down to 2W | 23:54 |
Betsy, Camilla, Keita
Name | Path | Beam being dumped | Checked? |
BD1
|
POP
|
LSC POP_A Reflection
|
|
BD2
|
POP
|
ASC POP_A WFS Reflection
|
|
BD3
|
POP
|
RM3 Transmission
|
|
BD4
|
POP
|
BDV2 Reflection
|
|
BD5
|
REFL
|
BDV1 Reflection
|
|
BD6a
|
REFL
|
M7 Ghost
|
|
BD6b
|
REFL
|
M7 Ghost
|
|
BD7
|
REFL
|
Unknown, beam from upstream of M2
|
|
BD8
|
REFL
|
M2 Transmission
|
YES: photo
|
BD9
|
REFL
|
Unknown, beam from upstream of M2
|
|
BD10
|
REFL
|
ASC REFL_A Reflection
|
YES: photo
|
BD11
|
REFL
|
ASC REFL_B Reflection
|
YES: photo
|
BD12
|
REFL
|
RM2 Transmission
|
YES: photo
|
BD13
|
REFL
|
LSC REFL_A Reflection
|
YES: photo
|
BD14
|
REFL
|
LSC REFL_B Reflection
|
YES: photo
|
BD15
|
REFL
|
M2 Ghost (photo showing distance from main beam)
|
YES: photo
|
BD16
|
REFL
|
RM1 Transmission
|
YES: photo
|
BD17
|
PSL
|
Nozzle Baffle on PSL-HAM1 VP
|
YES: photo
|
BD18
|
PSL
|
Nozzle Baffle on PSL-HAM1 VP
|
YES: photo
|
Name
|
Path
|
Beam being dumped
|
Checked?
|
BD1
|
POP
|
LSC POP_A Reflection
|
YES: photo
|
BD2
|
POP
|
ASC POP_A WFS Reflection, beam clears by ~15mm, photo
|
YES: photo
|
BD3
|
POP
|
RM3 Transmission (Beam too weak, estimated placement)
|
Estimated: photo
|
BD4
|
POP
|
BDV2 Reflection MOVED LOCATIONS
|
YES: photo
|
BD5
|
REFL
|
BDV1 Reflection, alos catching 2 ghost beams, photo
|
YES: photo
|
BD6a
|
REFL
|
M14 Ghost MOVED LOCATIONS
|
YES, see BD6b.
|
BD6b
|
REFL
|
M14 Ghost
|
YES: photo
|
BD7
|
REFL
|
Unknown, beam from upstream of M2, MOVED LOCATIONS
|
YES: photo
|
BD8
|
REFL
|
M2 Transmission
|
YES: photo
|
|
|
|
Not using, dumped by BD8
|
NewBD
(BD9)
|
REFL
|
REFL ASC A low reflection, beam bounces off ISI, photo
Call this new beamdump BD9 as now not using BD9. |
YES: photo
|
BD10
|
REFL
|
ASC REFL_A Reflection
|
YES: photo
|
BD11
|
REFL
|
ASC REFL_B Reflection
|
YES: photo
|
BD12
|
REFL
|
RM2 Transmission
|
YES: photo
|
BD13
|
REFL
|
LSC REFL_A Reflection
|
YES: photo
|
BD14
|
REFL
|
LSC REFL_B Reflection
|
YES: photo
|
BD15
|
REFL
|
M2 Ghost (photo showing distance from main beam)
|
YES: photo
|
BD16
|
REFL
|
RM1 Transmission
|
YES: photo
|
BD17
|
PSL
|
Nozzle Baffle on PSL-HAM1 VP
|
YES: photo
|
BD18
|
PSL
|
Nozzle Baffle on PSL-HAM1 VP
|
YES: photo
|
HiPWR
|
REFL
|
M14 and M17 Reflection
|
YES: photo
|
EPO tag for pics of laser safety in practice.
I've written an EPICS IOC which reads the smart Geist rack power strips and records their current draw in Amps.
CDS racks in the MSR and end stations have two verically mounted, full height, power strips. Most MSR racks have a UPS power strip and a facilities power strip. Rack mount servers with dual redundant power supplies are connect to each power strip for fault tolerance. The MSR exceptions are racks 01 (fiber rack, has a single APC power strip) and the front end racks 03, 04 which have 3-phase power strips with no network connection. Front end computers are not UPS powered because their IO Chassis are also not UPS powered.
The end station racks have two power strips, but they are both on technical power, there are no FE UPS units in the outbuidings.
I have created an MEDM called H1CDS_PWR_STRIP.adl, it can be opened from the CDS pulldown on the SITEMAP.
The IOC calculates the total rack current draw, which is displayed in the circle. UPS power strips are denoted by their orange background.
Attached photograph shows a typical power strip configuration at the rear of a CDS rack, with the UPS strip using keyed outlets to prevent accidental connection to non-UPS outlets.
I forgot to credit Cyrus Reed who installed these smart power strips and put them onto the network many moons ago.
The MSR atomic clock has drifted sufficiently in the past 6 months to reach the edge of its 1uS tolerance, causing 1PPS Timing errors.
Note this is a gradual drift as opposed to the major jumps we had seen in 2023/2024. Those jumps were large fractions of a second each, requiring rsyncing the atomic clock back to the timing master. Daniel says this is not needed in this case of a gradual few microseconds per year.
I have adjusted the nominal time difference for the atomic clock from 0ns to 1000ns, keeping the tolerance at 1000ns.
Attached trend shows that the timing master itself is not the cause of the drift, as verified by the end station CNS-II GPS independent clocks.
Following the soft close of GV7 at 12:46 PDT I reconfigured the alarms system to expect this GV to be soft-closed.
We ended up using pico in HAM3 to bring down green ALS beam as well as IR POP rather than raising the top periscope mirror after confirming that it was safe to do so.
We first confirmed that the X arm and corner alignment were still the same as yesterday. X arm was still locked to 00 green, and when I misaligned PRM the IR flash in H1:LSC-TR_X_IN1 was reaching 0.8 or more.
We convinced ourselves that the alignmernt was reasonable and probably similar to the alignment before the vent, that the ALS beams were pretty high but was still hitting the top periscope mirror before just like ALSX beam is as of now. We discussed the use of the pico in HAM3 (instead of moving the top periscope mirror up) and decided that it's OK to do so as far as the IR beam is far from clipping somewhere in HAM2.
We moved the HAM3 pico for PIT in "Down" direction by 700 counts and the IR beam position looked about right. We moved the pico further down by 1000 counts (that's the total of 1700 counts down from the initial state) and the IR beam missed the bottom of the top periscope mirror but no sign of clipping in HAM2, so we were comfortable to go 1000 counts "UP" and search for an optimum positision. In the end we ended up going back to 700 counts "Down" relative to where we started.
Sheila and Camilla adjusted top and bottom periscope mirror. M10 (dichroich) was also touched up. Steering mirrors for in-air beams were also roughly adjusted, These are M7 (for REFL air), M11 (for green ALS), M13 (for IR ALS beam from the PSL) and M16 (for POP air).
Now we're ready to move ISCT1L in position to confirm that all beams hit the periscope mirrors on the table.
Photos attached of the Green X-arm and POP beams on the HAM1 top periscope mirror before the HAM2 pico-ing and after the HAM2 pico-ing in the final position.
Tyler fork-lifted ISCT1 into position and we correctly set the height. There was some confusion due to a typo by me on the -X-Y corner height measurement that we've now corrected.
PSL ALS and Green ALS Beams:
To get the PSL ALS and green ALS beams through the VP simulator and onto the top periscope mirror, Sheila needed to move both final steering mirrors (M11 and M13) to have the beams more centered on the VP simulator:
Photos of the final positions attached, on the VP simulator (inner red circle is the location of the actual window), on the upper periscope mirror, on the lower periscope mirror and on the ALS shutter, we expect to need to do some steering of the UPM to get the final beams better on the LPMs, this will be done once ISCT1 is installed and realigned.
POP air and REFL air alignment
(Note: At this point, HAM1 drawing https://dcc.ligo.org/D1000313-v19 was not useful for exact placement of the mirrors because, as was reported earlier, POP and REFL beam position coming from HAM2 into HAM1 are different from what the drawing suggests by an inch or so. )
POP air steering mirror (M16) had to be relocated closer to the periscope, otherwise the POP air beam hit the -X edge of the viewport simulator when we tried to center POP air on the periscope mirror on ISCT1. But then the REFL air beam became close to the M16 mirror mount, so we ended up moving M16 MUCH closer to the periscope and the REFL air steering mirror (M7) got further down in -X direction. As a result, two beams now crosses with each other inside the chamber. See the first attachment where the new POP air beam is in orange and the new REFL air path is in cyan. This is for illustration purpose only.
2nd attachment: Overview of the POP and REFL air path in chamber.
3rd and 4th attachment: Detailed location of M16 and M7, respectively.
5th attachment: REFL air beam is comfortably away from the M16 mirror holder.
6th attachment: Beam position on the viewport simulator. Dimmer beam is POP air.
7th and 8th: REFL and POP beam on the top periscope mirror of ISCT1.
This means that the beam diverters WILL have to be relocated and reconfigured.
Just swapping the positions might not work.
I wonder if using one diverter at the edge of the table will catch both beams. (That's not a good idea as we might want to use in-air POP WFS if in-vac one doesn't work well.)
Distance from POP air steering mirror (M16) to the top POP periscope mirror in ISCT1 was: 34 1/4 + 30 1/4 = 64.5"
Distance from REFL air steering mirror (M7) to the top REFL periscope mirror in ISCT1 was: 13 3/4 + 30 + 31.5 = 75.25"
Measurement was done using long-ish ruler, but it wasn't long enough so we had to measure the distance in multiple segments. I'd say that the error bar of this is roughly +-3/4".
Gerardo, Jordan, Janos GV7 (X-arm manifold) was opened today, to make possible HAM1 optical alignment. 13:06 - all the turbos were valved out 13:12 - the gate valve started to move up After ~4 hrs, PT-144 (vacuum gauge next to CP2 on the beamtube side) is at 3.7E-8 Torr, the pressure in the corner is <1E-7 Torr. GV7 stays open overnight, then tomorrow there will be some more alignment work (following aLog 84323), and then GV7 will be closed again. The turbo pumps are still running, and after GV7 is closed, they will be valved back to the corner. The RGA (p-t) scans of the main peaks will be posted - in particular during the closure and the opening of GV7
Oli, Keita, Elenna, Jennie, Sheila, Ryan S, Rahul, Betsy, Camilla
Day 1: 84193, Day 2: 84228, Day 3: 84230 and 84239, Day 4: 84274, Day 5: 84292
In the morning Keita and Jennie aligned the POP flashes on the periscope mirrors and M10 dichroic. The height of this beam is 3.75" on these mirrors to account for the ALS green beam being 13mm above the POP beam. Keita was a little concerned about the height of the top periscope mirror so we'll need to check the green ALS beam carefully on this and if it 's near clipping we may need some in-vac pico-ing.
Rahul then moved PM1 to it's final position. Elenna and Rahul checked that the signed of the PIT and YAW sliders were correct.
Shiela, Rahul and I then worked on centering the beam on PM1 and L2 without the 90/10 M12 in place, so that we could later have enough power to beam profile the POP single bounce beam. To get the beam height back up to 4" at PM1, Rahul needed to pitch PM1 mechanically.
Oli then increased PSL input power to 20W, we aligned to POP single bounce and Sheila and I took beam profile measurements before PM1 and after L2, details in 84307.
Elenna, Rahul and I then replaced the 90/10 M12 and recentered the beam on PM1 and L2. We started aligning the POP air path but aren't happy with it, dumped the beam and will continue to work on.
Betsy let in the PSL ALS beam into HAM1 by opening the light pipe and I dumped it straight away. See 84312.
Today we plan to: align the POP, REFL and PSL ALS beams out of the chamber, open the X-arm check the green ALS beam alignment on the shared optics and off the table, including rolling up ISCT1 and adding the VP simulator. The POP LSC/ASC diodes and all beam dumps still need to be aligned and checked.
Keita and Jennie's log on the work done in the morning is 84308.
Just adding a clarification. "Elenna and Rahul checked that the signed of the PIT and YAW sliders were correct. "
We confirmed that adding a positive pitch offset in the alignment slider of PM1 results in PM1 pitching downward, as expect in a right handed coordinate system, and verified by watching the beam reflecting off PM1 shifting down. We also confirmed that a positive yaw offset results in the optic rotating left (as seen from behind the optic), again as expected from the right handed coordinate system, and verified by watching the beam reflected off PM1 shifting left.
I went back and forth a few times with the offsets while Rahul held an indicator card in front of the optic to watch the beam move to see the response of PM1.