TITLE: 12/08 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering // Earthquake
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
SEI_ENV state: LARGE_EQ
QUICK SUMMARY:
M. Todd, J. Driggers
I wanted to turn on SR3 and watch the HWS while it heats up to compare to models of the defocus we expect. Due to the earthquake yesterday however most of the suspension watch dogs had tripped, and would not let me re-align the ITMs, BS, and SR3 for making sure the HWS were aligned.
With Jenne's help, we untripped and took ISI's to isolated for ITMs, BS and SR3. I then turned on the SR3 heater to 4W, and I will watch to make sure it responds well. It should be relatively well thermalized in about 5 hours, at which point I'll turn it off.
M. Todd
I went back in to turn off the SR3 heater, and discovered that the watch dogs for the BS and ITMY had again tripped. I reset the watch dogs but was not successful in trying to put their ISIs to fully isolated because of an error 'WATCHDOG TRIP : SUBORDINATE'.
Regardless, the suspensions report that theyr'e aligned and SR3 is cooling down so the HWS should be able to get the cool down data as well.
Sun Dec 07 10:07:10 2025 Fill completed in 7min 7secs
Another SWWD trip of ETMY
ETMY Suspension rings-up within 15 minutes of reset causing a SWWD trip of SEI (reset at 08:00, tripped 08:15 this morning).
I'll keep SUS tripped for now and reenable SEI in bypass mode until we can determine what is going on with h1susetmy.
BSC8 annulus ion pump (AIP) railed late yesterday around 10:02 pm local time, see attached plot.
After the pump railed there is no noted effect inside the main vacuum envelope. The annulus system will be diagnosed on next opportunity.
Last failure of this ion pump was mentioned on 2/19/2002.
BSC SWWDs were tripped around 13:00 this afternoon following a 7.0 mag EQ in Alaska.
After Dave texted me about the eq, I logged in, reset all the watchdogs, set all of the ISI to damped and turned off the sensor correction. This should be a safe enough state until people get back in on Monday.
SUSETMY tripped a second time on Saturday after its reset, I untripped it at 08:00 Sunday.
We are still recovering CDS from Thursday's power outage. All critical systems have been recovered and IFO locking was started yesterday.
Both Alarms and Alerts systems are operational and sending cell phone texts, I'm having issues sending alert emails.
Here is a brief summary of we are currently working on:
Timing:
Timing has two issues. The main one is with EY timing fanout chassis, its single mode link to MY (port15) is showing delay issues. Despite this h1pemmy appears to have good timing.
The secondary issue is that the atomic clock in the MSR has time jumped by 0.4S and needs resyncing.
Disk Failures:
The MSR file cluster has lost 4 disks. Resilvering for 2 of them is ongoing.
h1susauxh2 Power Supply Failure:
One of h1susauxh2's power supplies has failed, FRS36264.
EDC Disconnected Channels:
EDC currently has 574 disconnected channels, relating to auxiliary IOCs which still need to be started.
CDS OVERVIEW
Referencing the CDS Overview attached:
Range 7-segment LED service needs to be restarted.
Range of -1 MPc shows an outstanding GDS issue
EDC disconnect count mentioned above
Timing RED because of issues covered above
Picket Fence WHITE, needs restarting (see list below)
CDS Alarm RED due to EDC disconnect count
CDS SDF YELLOW because of remote power control and FCES-WIFI issues
Missing Auxillary EPICS IOCs
List may not be complete:
Picket Fence, End Station HWS, Mains Power (CS, EX), SUS Violin monitor, ncalx, cds load mon h1vmboot1, cal inj exttrig, range led, Observation mode
EY Geist Watchdog1250 actually went good for a few days after the power outage and has only recently failed again, suggesting a possible power supply issue.
Sat Dec 06 10:19:46 2025 Fill completed in 19min 42secs
TITLE: 12/05 Day Shift: 1530-2300 UTC (0730-1500 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY: IFO is currently in IDLE. We were eventually able to get the IMC relocked after a lot of touching up optics and the PZT, but it seems like whenever it's been unlocking, it's having more trouble relocking than usual. We ran the dither align scripts for the TMSs and ITMs, and also ran through several full or partial initial alignments trying to get everything looking better. We had issues with basically every part of initial alignment, and after finishing alignment we would still have issues with flashes looking bad, especially for PRMI. Only green arms and MICH have been consistantly looking good.
Matt has started running an inverse filter on the ITMX ring heater over the weekend.
LOG:
- initial alignments and first few relocking states
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 15:49 | FAC | Eric | FCES | n | Heating issues | 16:58 |
| 15:50 | CDS | Richard | Ends | n | RFM troubleshooting (EX, then EY) | 16:34 |
| 16:12 | FAC | Kim, Nelly | LVEA | n | Mega cleanroom recleaning | 19:17 |
| 16:44 | FAC | Randy | LVEA | n | Moving septum in from high bay and fork lifting around | 19:51 |
| 16:45 | CDS | Richard | LVEA | n | Checking on things | 16:51 |
| 18:05 | JAC | Jennie | LVEA, Prep lab | n | Grab parts in LVEA, then JAC table work | 19:00 |
| 18:06 | CDS | Marc | CER Mezz | n | Checking power supplies | 18:14 |
| 18:10 | VAC | Gerardo | LVEA | n | Checking on purge air | 18:19 |
| 18:17 | CDS | Marc | CER | n | Pulling Beckhoff chassis 3 | 18:40 |
| 18:18 | CDS | Daniel | EX | n | Checking Bechoff chassis | 19:21 |
| 18:51 | SUS | Betsy, Rahul | High bay | n | Cables | 18:59 |
| 19:00 | PCAL | Rick, Volker | PCAL lab | n | Checking on lab | 19:12 |
| 19:02 | CDS | Marc | CER | n | Putting Beckhoff chassis back in | 19:15 |
| 19:16 | SUS | Rahul | LVEA | n | Putting things in sus storage racks | 19:24 |
| 19:27 | CDS | Daniel, Marc | EX | n | Checking on power supply | 19:46 |
| 19:41 | TCS | TJ | LVEA | n | Turning TCS power supply on | 19:51 |
| 21:20 | VAC | Gerardo | LVEA | n | Getting parts for AWC | 21:29 |
| 21:38 | SQZ | Kar Meng, Eric | Opt Lab | n | OPO work | 00:04 |
| 21:38 | PCAL | Oli | EX | n | Turning on PCAL X laser | 22:38 |
| 21:52 | JAC | Jennie | LVEA | n | Looking for parts | 23:52 |
| 21:57 | VAC | Travis | MY | n | Parts transportation | 22:21 |
| 22:17 | - | Matt, Jason | LVEA | n | Check on PSL env and plug back in IMC whitening cable | 22:21 |
| 23:29 | PSL | Jason | LVEA | n | Resetting PSL environmental settings | 23:34 |
I used Tony and I's statecounter.py to take a look at the past year of data using minute trends. Minute and second trends do some rounding that I had to consider in my search, ETMX goes from -8.9 to +6.05425 during a lock aquisition, these average to ~ -1.42. I ended up searching the chan H1:SUS-ETMX_L3_LOCK_BIAS_OFFSET for above and below -2.0 doing the following calls:
This gave me an outfile.txt file full of results in the format "idx (data_idx_start, data_idx_stop) gpsstart gpsstop duration" which I did some brief analysis on yielding:
Percentage of the time the Bias offset was [+]: 43.70 % *Past locked LOWNOISE_ESD_ETMY
Percentage of the time the Bias offset was [-]: 56.25 % *Between PREP_FOR_LOCKING and LOWNOISE_ESD_ETMY
Total duration in [+]: 205.3 [Days], [-]: 159.5 [Days] over 365.0 [Days]
Data timespan missing due to minute rounding and FW restarts 0.1965 [Days] (0.054 % of total time).
Where the BIAS_OFFSET is changed from negative to positive has changed a little over the past year but it's always during one of the final states of ISC_LOCK and it's always reset to negative PREP_FOR_LOCKING.
I haven't done the same search for ETMY but looking through where it's set in ISC_LOCK and ndscope I can say that ETMY spends the large majority (>90%) of time at -4.9. Specifically it stays at this value for 4min 23sec per acquisition, it gets changed to +9.3 during LOWNOISE_ESD_ETMY but it's brought back to -4.9 in the next state LOWNOISE_ESD_ETMX as we switch back.
While TJ was running initial alignment for the green arms, I noticed that the ALS X beam on the camera appeared to be too far to the right side of the screen. The COMM beatnote was at -20 dBm, when it is normally between -5 and -10 dBm. I checked both the PR3 top mass osems and the PR3 oplevs. The top mass osems did not indicate any significant change in position, but the oplev seems to indicate a significant change in the yaw position. PR3 yaw was around -11.8 but then changed to around -11.2 after the power outage. It also appears that the ALS X beam is closer to its usual position on the camera.
I decided to try moving PR3 yaw. I stepped 2 urad and which brought the oplev back to -11.8 in yaw and the COMM beatnote to -5 dBm. Previous slider value: -230.1, new slider value: -232.1.
The DIFF beatnote may not be great yet, but we should wait for beamsplitter alignment before making any other changes.
Actually, this may not have been the right thing to do. I trended the oplevs and top mass osems of ITMX and ETMX and compared their values during today's initial alignment before moving PR3, to the last initial alignment we did before the power outage. They are mostly very similar except for ETMX yaw.
| P then | P now | Y then | Y now | |
| ITMX oplev | -7.7 | -7.8 | 5.8 | 5.6 |
| ETMX oplev | 2.7 | 3.1 | -11.5 | -3.1 |
I put the PR3 yaw slider back to its previous value of -230.1 until we can figure out why ETMX yaw has moved so much.
I moved PR3 yaw back to -231.9 on the slider. This allowed us to see PRMI flashes on POPAIR. We can revist if we really want to keep this alignment of PR3 on Monday.
Related: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=87729
We disconnected everything from the ISS array installation spare unit S1202965 and stored it in the ISS array cabinet in the vac prep area next to the OSB optics lab. See the first 8 pictures.
The incomplete spare ISS array assy originally removed from LLO HAM2 (S1202966) was moved to a shelf under the work table right next to the clean loom in the optics lab (see the 9th picture). Note that one PD was pulled from that and was transplanted to our installation spare S1202965.
Metadata for both 2965 and 2966 were updated.
ISS second array parts inventory https://dcc.ligo.org/E2500191 is being updated.
Rahul and I cleared the optics table so Josh and Jeff can do their SPI work.
Optics mounts and things were put in the blue cabinet. Mirrors, PBS and lenses were put back into labeled containers and in the cabinet in front of the door to the change area.
Butterfly module laser, the LD driver and TEC controller were put back in the gray plastic bin. There was no space in the cabinets/shelves so it's put under the optics table closer to the flow bench area.
Single channel PZT drivers were put back in the cabinet on the northwest wall in the optics lab. Two channel PZT driver, oscilloscopes, a function generator and DC supplies went back to the EE shop.
OnTrack QPD preamp, its dedicated power transformer, LIGO's LCD interface for QPD and its power supply were put in a corner of one of the bottom shelf of the cabinet on the southwest wall.
Thorlabs M2 profiler and a special lens kit for that were given to Tony who stored them in the Pcal lab.
aLIGO PSL ISS PD array spare parts inventory E2500191 was updated.
I was baffled to find that I haven't made an alog about it, so here it is. These as well as other alogs written by Jennie, Rahul or myself in since May-ish 2025 will be added to https://dcc.ligo.org/LIGO-T2500077.
Multiple PDs were moved so that there's no huge outlier in the position of the PDs relative to the beam. When Mayank and Siva were here, we used to do this using an IR camera to see the beam spot position. However, since then we have found that the PD output itself to search for the edge of the active area is easier.
After the adjustments were made, the beam going into the ISS array was scanned vertically as well as horizontally while the PD outputs were recorded. See the first attachment. There are two noteworthy points.
1. PDs "look" much narrower in YAW than in PIT due to 45 degrees AOI only in YAW.
Relative alignment matters more for YAW because of this.
2. YAW scan shows the second peak for most of PDs but only in one direction.
This was observed in Mayank/Siva data too but it wasn't understood back then. This is the design feature. The PDs are behind an array plate like in the second attachment (the plate itself is https://dcc.ligo.org/D1300322). Red lines show the nominal beam lines and they're pretty close to one side of the conical bores on the plate. Pink and blue arrows represent the shifted beam in YAW.
If the beam is shifted too much "to the right" on the figure (i.e. pink), the beam is blocked by the plate, but if the shift is "to the left" (i.e. blue) the beam is not blocked. It turns out that it's possible that the beam grazes along the bore, and when that happens, a part of the broad specular reflection hits the diode.
See the third attachment, this was shot when PD1 (the rightmost in the picture) was showing the second peak while PD2 didn't.
(Note that the v2 plate which we use is an improvement over the v1 that actually blocked the beam when the beam is correctly aligned. However, there's no reason things are designed this way.)
We used a PZT-driven mirror to modulate the beam position, which was measured by the array QPD connected to ON-TRAK OT-301 preamp as explained in this document in T2500077 (though it is misidentified as OT-310).
See the fourth attachment where relatively good (small/acceptable) coupling was obtained. The numbers measured this time VS April 2025 (Mayank/Siva numbers) VS February 2016 (T1600063-V2) are listed below. All in all, horizontal coupling was better in April but vertical is better now. Both now and Apr/2025 are better than Feb/2016.
| PD number |
Horizontal [RIN/m] |
Vertical [RIN/m] |
||||
| Now |
Apr/2025 (phase NA) |
Feb/2016 (phase NA) |
Now |
Apr/2025 (phase NA) |
Feb/2016 (phase NA) |
|
|
1 |
6.9 | 0.8 | 20 | -0.77 | 34.1 | 11 |
| 2 | 7.1 | 2.7 | 83 | 5.1 | 2 | 25 |
| 3 | 8.2 | 5.5 | 59 | 2.2 | 4.4 | 80 |
| 4 | 8.8 | 2.3 | 33 | 0.30 | 1.1 | 21 |
| 5 | -19 | 5.1 | 22 | 11 | 12.3 | 56 |
| 6 | -14 | 12.9 | 67 | 16 | 30.4 | 44 |
| 7 | -18 | 10.2 | 27 | 2.9 | 42.7 | 51 |
| 8 | -19 | 5.3 | 11 | 12 | 52.1 | 54 |
Phase of the jitter coupling: You can mix and match to potentially lower jitter coupling further.
Only in "Now" column, the coupling is expressed as signed numbers as we measured the phase of the array PD output relative to the QPD output. Absolute phase is not that important but relative phase between the array PDs is important. The phase is not uniform across all diodes when the beam is well aligned. This means that you can potentially mix and match PDs to further minimize the jitter coupling.
Using the example of this particular measurement, if you choose PD1/2/3/4 as the in-loop PD, the jitter coupling of the combined signal is roughly mean(6.9,7.1,8.2,8.8)=7.8 RIN/m horizontally and mean(-0.77, 5.1, 2.2, 0.3) = 1.7.
However, if you choose PD1/3/4/7 (in analog land), the coupling is reduced to mean(6.9, 8.2, 8.8, -18)=1.5 horizontally and mean(-0.77, 2.2, 0.3, 2.9)=1.2.
You don't pre-determine the combination now, you should tune the alignment and measure the coupling in chamber to decide if you want a different combination than 1/2/3/4.
Note, when monotonically scanning the beam position in YAW (or PIT) edge to edge of PDs, some PDs showed more than one phase flips. When the beam is apparently clipped at the edge (thus the coupling is huge), all diodes show the same phase as expected. But that's not necessarily the case when the beam is well aligned as you saw above.The reason of the sign flips when the beam is far from the edge of the PD is unknown but there should be something like particulates on the PD surface.
The QPD was physically moved so the beam is very close to the center of the QPD. This can be used as a reference in chamber when aligning the beam to the ISS array.
After this, we manually scanned the beam horizontally and measured the QPD output. See the 5th attachment, vertical axis is directly comparable to the normalized PIT/YAW of the CDS QPD module, assuming that the beam size on the QPD in the lab is close enough to the real beam in chamber (which it should be).
When I try to run an ndscope I get the following error:
qt.qpa.xcb: could not connect to display
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, xcb.
Aborted
*I think I was trying to launch ndscope in a terminal that I was sshed into a different computer/env*
As a follow up note. Ndscope is working for Ryan, we are not entirely sure what the issue was, maybe the console was in a strange conda environment. When we looked at it, ndscope started fine.