We noticed that the range was dropping even lower than our already low ~145Mpc. There was a lot of low frequency noise that was creeping up. Sheila suggested that we look at the PIs and sure enough, PI31 had started to slowly creep up the same time as the range degredation (see attached). The SUS_PI guardian node will turn on the damping at 3 according to the H1:SUS-PI_PROC_COMPUTE_MODE31_RMSMON channel and turn it back off when it gets below 3. For now we just tried changing that threshold to 0 to all for continuous damping. This let the mode damp down completely and it brought our range back up.
We should think more about damping this down lower and having two thresholds, an on an off.
Changed the threshold to 1. At 0 the node would continuously search for a new phase thinking that it needed to damp it further.
The first attached screenshot shows the spectrum around 10.4 kHz (edited to fix typo), the mode which rang up and caused this issue is at 10428.38 Hz, previously identified as being in the y arm (Oli's slides).
The behavoir seemed pretty similar to what we saw a few weeks ago, 82961, with broad nonstationary noise in DARM.
The downconversion doesn't seem to be caused by the damping, it started to have a noticable impact on the range while the damping was off, and when the damping came on and reduced the amplitude of the mode the range improved.
In the ndscope screenshot attached you can see the range that we are using on the DCPDs ADC, this is a 20 bit DAC so it would saturate at 524k counts, when this PI was at it's highest today it was using 10% of the range, the DARM offset takes up about 20% of the ADC range.
The third attachment is the DARM spectrum at the time of this issue, as requested by Peter. As described in 83330 our range is decreasing with thermalization in each lock, the spectrum shows the typical degredation in the spectrum in these last few days since the SQS angle servo has been keeping the squeezing more stable. The excess noise seen when the PI was rung up has a similar spectrum to the excess noise we get after thermalization.
Also, for some information about these 10.4kHz PI modes, see G1901351
Jeff pointed me to 82686 with the suggestion to check the channels that have different digital AA.
The attached plot shows DCPD sum, and two alternatives, in red is H1:OMC-DCPD_16K_SUM1_OUT_DQ which has a high pass to reduce the single precision noise and no digital AA filters (and some extra lines), SUM2 has 1 digital AA filter and the high pass, and DCPD sum is the darm channel and has 2 digital AA filters and no high pass. The broadband noise is the same for all of these, so digital aliasing doesn't seem to be involved in adding this broadband noise.
The attached screenshot shows that turning on the SQZ angle servo 83270, which is shown by the time when CLF_RF6 phase starts to move around, the 350Hz squeezing (yellow BLRMS3) has been better, which is what this servo is intended to do.
You can also see that the higher frequency squeezing has been worse, and now seems correlated with the range dropping. Early in each lock there is a time when the high freq squeezing reaches 4.7dB, which corresponds with the times when the range is highest around 156 Mpc. This is during the thermalization, you can also see "f_s" which is the estimate of the SRC spring frequency based on the calibration lines, and kappa_c are also changing rapidly in this time, so the change in range could be explained by these changes more than the impact on squeezing.
TJ has posted a sensitivity comparison with coherence here: 83327 which shows that we have some A2L coherence to fix.
Wed Mar 12 10:08:11 2025 INFO: Fill completed in 8min 8secs
Gerardo confirmed a good fill curbside.
Late entry from last night. VACSTAT went into single-alarm-mode with a BSC3 sensor glitch 20:29:44 Tue 11mar2025 PDT. I reset the vacstat_ioc.service on cdsioc0 at 20:36 to clear this.
TITLE: 03/12 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 141Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 15mph Gusts, 11mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.31 μm/s
QUICK SUMMARY: Locked for just over and hour, looks like Ryan had to accept some SDFs overnight, and there were some ALS issues yesterday, but we had 3 relocks overnight. No alarms this morning.
Another ALS_Y sdf diff
13:33 UTC Observing
I had to accept two sets of SDF diffs to get us into observing, for HAM1 and ALS_Y.
09:16 UTC Observing
I added HAM1 ISI model to the DIAG_SDF exclude list yesterday, so it shouldn't have been necessary to accept these ones to get back into observing. They will still show up as differences, but aren't looked at by the DIAG_SDF node.
TITLE: 03/12 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently relocking and in POWER_25W. We've been down for a while trying to figure out a weird issue with the ALSY WFS and we at least have a plan for the night, but it might end up really annoying for Ryan C unfortunately.
For Ryan C (and maybe TJ tomorrow morning): tagging OPSINFO
I started an FRS ticket FRS#33562. Wasn't sure what category to put it under though so I moved it to be under operators, but that's definitely not right.
What we saw/what we tried to do to fix it:
After the lockloss earlier, I wanted to run an initial alignment since we had not run one since yesterday. I noticed that once the WFS would turn on for ALSY, the transmission would slowly drift down until it lost lock(ndscope1, ndscope2). I thought the issue might be DOF2 or 3, since both of those seemed to be diverging and DOF3 WFS were much larger than they usually are(ndscope3). I tried turning DOF2, DOF3, and both sets of WFS off, but we would just lose lock. I left IA and tried locking green arms the normal relocking way, which only uses DOF1 and DOF2 aka ETM and TMS, and that worked well, so I knew it had to be an issue with DOF3 specifically. I ended up needing an initial alignment, and either way we didn't want it to need an initial alignment in the middle of the night. Jenne suggested I take the YARM to LOCKED_SLOW_ETM_TMS_WFS and then slowly align ITMY to lower the ITMY camera error. I tried that, but I kept getting to a point where the DOF2 WFS stopped being able to handle it and would just diverge until we lost lock, even if I tried stepping back the other way, all while the camera errors didn't get much better. After trying this multiple times, Jenne came onto TS and we tried this method again with no success. We also tried adjusting ITMY (usually in Pitch) with YARM in LOCKED_SLOW_NO_WFS, again in LOCKED_SLOW_ETM_TMS_WFS but with DOF3 YAW WFS engaged, we tried resetting the slider values for ITMY, ETMY, and TMSY to the end of the last initial alignment, and we tried aligning MICH to get the beamsplitter in a better spot. None of these methods worked.
Eventually we decided that the only thing we could do for tonight was to have an initial alignment run with ALSY aligning only using ETMY and TMSY. Hopefully during the night there isn't a need for another initial alignment, because if there is Ryan C will need to change the states ALSY will need to go through to be ETM_TMS_WFS_OFFLOADED instead of INITIAL_ALIGNMENT_OFFLOADED.
Now, after finishing an initial alignment where we only used the ETMY and TMSY WFS, I've been trying to relock, but it looks like DOF2 diverging is still happening, ALSY goes down, and then ALSY loses lock(ndscope4). This wasn't happening during the relocks earlier in the day right after maintenance. To get around this, once we got to ARMS_OFF_RESONANCE and ALSY transmission started dropping, I turned off the ALSY DOF2 WFS in both pitch and yaw. We were able to get through the rest of the ALS states like this.
LOG:
22:30 Observing
23:45 We dropped out of Observing for a second and then went back in because I accidently clicked something monitored by SDF
01:02 Lockloss
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
22:19 | ISC | Mayank, Siva | Opt Lab | Local | ISS PD array | 00:59 |
23:42 | ISC | Keita | OpticsLab | n | WFS work | 01:09 |
Still unlocked and having trouble with ALSY WFS DOF3, so troubleshooting that with Jenne.
TITLE: 03/11 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: List of maintenance activities below, the most impactful to locking being the camera server change that we ended up having to revert to resume locking (alog83308). We will try to implement these cameras that are used in locking a different day when we can dedicate time to setting the IFO to their new offsets.
Locking notes:
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
14:36 | FAC | Nelly | Opt Lab | n | Tech clean | 14:48 |
15:06 | SUS | Jeff | CER | n | Move PR3 OpLev cable | 16:01 |
15:06 | FAC | Kim | EY | n | Tech clean | 16:53 |
15:06 | FAC | Nelly | EX | n | Tech clean | 16:05 |
15:14 | CDS | Dave, Erik, Jeff | CR | n | Model restarts (PR3, HTTS, IMs, SUSAUX) also SEIH16 | 15:35 |
15:17 | CDS | Fil, Erik, Marc | CER | n | SEI rack work (HAM1) | 18:21 |
15:18 | FAC | Tyler | LVEA, Ends, Mids | n | 3IFO checks | 17:34 |
15:20 | VAC | Ken | EY MR | n | Compressor electrics | 18:45 |
15:25 | FAC | Contractor | EY | n | Well testing - Large van, water truck, SUV | 16:22 |
15:29 | IAS | Jason, Ryan C | LVEA | n | FARO | 18:59 |
15:35 | OPS | Richard | EY | n | Check on panels and Ken | 16:51 |
15:35 | VAC | Janos, Travis, Jordan | EY, EX, mids | n | Compressor cleanup | 18:18 |
15:51 | PCAL | Rick, Francisco, Sivananda | EX | not yet | PCAL meas | 17:35 |
16:17 | VAC | Gerardo | LVEA | n | Vac rack UPSs | 17:07 |
16:18 | ISC | Camilla, job shadow | LVEA | n | Mark table location | 17:17 |
16:29 | FAC | Nelly | FCES | n | Tech clean | 17:14 |
16:32 | VAC | Norco | CP2 (CS) | n | LN2 fill | 18:19 |
16:49 | CDS | Jonathan, Tony | MSR | n | Pulling digivideo0&1 | 18:03 |
16:54 | FAC | Kim, Nelly | LVEA | n | Tech clean | 18:16 |
16:55 | SEI | Jim, Mitchell | Ends | n | Dropping off wind fence materials | 17:58 |
17:10 | VAC | Gerardo, Jackie | Ends, Mids | n | Vac rack UPSs | 18:36 |
17:11 | SYS | Betsy | LVEA | n | Check on HAM1 vent prep | 17:11 |
17:30 | VAC | Janos, job shadow | LVEA | n | Turbo test | 18:45 |
17:31 | SYS | Betsy | Garbing room | n | Check on covers and other | 18:03 |
17:35 | FAC | Tyler, Eric | EY | n | Replacing pump | 18:23 |
17:41 | ISC | Mayank, Sivananda, Matt | Opt Lab | Local | ISS PD array | 20:12 |
17:46 | VAC | Travis, Jordan | LVEA | n | HAM1 feedthrough check | 18:18 |
18:25 | - | Daniel, Fil, Betsy | LVEA | n | HAM1 vent prep | 19:04 |
18:34 | - | Richard | LVEA | n | Look at cleanrooms | 18:42 |
18:38 | VAC | Gerardo, Jackie | LVEA | n | Join on turbo testing | 18:55 |
18:41 | FAC | Eric | EY, MY | n | Glycol pumping | 19:59 |
18:45 | OPS | Tony | LVEA | n | Sweep | 19:09 |
18:47 | VAC | Marc, Fil | LVEA | n | BSC1 temperature sensor | 18:55 |
18:58 | VAC | Janos | LVEA | n | Turbo test end | 19:04 |
19:47 | PCAL | Dripta | PCAL Lab | Yes | Setting PS5//PS12 Measurement | 20:00 |
20:31 | CDS | Fil, Jackie, Marc | CER | n | Replacing power supply for susb123 | 20:33 |
20:34 | FAC | Jim, Mitch | EndY | N | Wind fence investigation | 21:36 |
22:19 | ISC | Mayank, Siva | Opt Lab | Local | ISS PD array | ongoing |
We had a cold cathode gauge trip during yesterday's maintenance period, H0:VAC-EX_X2_PT524B_PRESS_TORR is the channel from the signal for the cold cathode gauge. The gauge pair, cold cathode and the pirani gauges, are located next to the gate valve number 20 (GV20), not far from the P-Cal transmitter pier. These gauges are very sensitive to interference from communication devices, such as two way radios and mobile phones, please try to keep communication devices away from all vacuum gauges, thank you.
Now, we wait for the gauge to comeback on its own.
TITLE: 03/11 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 5mph Gusts, 2mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.28 μm/s
QUICK SUMMARY:
Observing at 152Mpc and have been Locked for 1 hour.
WP12374 WP12370 ASC and SUS model changes.
Daniel, Jeff, Jonathan, Erik, TJ:
New models were installed for h1asc, h1susim, h1sushtts and h1suspr3. A DAQ restart was required.
WP12372 Add ISI HAM1 cards to h1seih16 IO Chassis
Fil, Marc, Erik, Dave, Jim, TJ:
h1seih16 was powered down. In its IO Chassis the bad A2 Adnaco backplane was replaced and the cards which had been moved to A3 were returned. Two additional ADCs and one additional BIO6464 were installed as per drawing (attached) which will be used by the new h1isiham1 model. Currently these cards have no external cables attached.
A new h1iopseih16 model was installed with these cards added. Note an additional 16bitDAC was not required because seismic ham models allocate a DAC per chamber, with the HEPI model using the lower 8 channels and the ISI model using the upper 8 channels. DAC-1 was already installed for h1hpiham1.
Note that this was one of two changes to the h1iopseih16 model (see SWWD section).
DAQ restart was required.
WP12373 Install new h1isiham1 Model
Jonathan, Erik, Jim, Dave:
The first version of the ISI HAM1 model was added to h1seih16. It is currently running with nothing connected to its ADC, DAC and BIO cards.
This is not something we do regularly, so we took the opportunity to document the steps needed. The list we have so far has been included as a spreadsheet.
The model was added to the DAQ. A related change to h1hpiham1 was also made. A DAQ restart was required for both.
Attachment shows the CDS Overvew MEDM h1seih16 section, now with a full complement of models.
WP12382 HAM1 SWWD
Jeff, Dave:
In preparation for HAM1 ISI install, we created a HAM1 Software Watchdog (SWWD) on h1iopsush2b and h1iopseih16. Note this is the second change to h1iopseih16 today.
There are two suspensions on HAM1; RM1 and RM2.
Despite its name, h1iopsush2b also controls ham1 suspensions. A SWWD part on h1iopsush2b reads the UL, LL, UR, LR raw ADC channels and calculates their integrated RMS values. If any value exceeds the trip level, the Dolphin IPC channel sent to h1iopseih16 is set to zero.
The IPC receiver on h1iopseih16, on receipt of a trigger signal, starts a DACKILL count-down timer. If the timer expires, it executes a DACKILL for the second DAC card, which drives HAM1 HEPI+ISI.
New h1iopsush2b and h1iopseih16 models were installed. The h1iopsush2b was done at the same time as the h1susim and h1sushtts models since these are the only models running on this front end. h1iopseih16 restart was done when this front end was powered back up after the IO Chassis work.
The SWWD MEDMs were updated with the new HAM1 system (attached)
Note: one complication is that up until now SUS SWWDs have been for larger suspensions with 6 OSEM channels (F1, F2, F3, LF, RT, SD). RM1,2 only have 4 (UL, LL, UR, LR) and so the two unused OSEM inputs have been grounded in the models. On the MEDM I just show ADC0_30, the channel prior to the duotone which is zero.
WP12358 Move Digital Video Cameras from h1digivideo2 to h1digivideo5
Patrick, Jonathan, Erik, TJ, Sheila, Dave:
We moved the six cameras which were on the last of the old servers (h1digivideo2) onto the new AMD/deb12 h1digivideo5 and powered down the old machine.
All was good until locking started, which is was discovered that for cameras which are copied to front end models the X,Y centroid EPICS channels now had offsets which the models could not handle.
For locking expediency it was decided to move the cameras back to the old server for now until the models could be configured to use to the new centroid values. Of the six models, five were moved back to h1digivideo2 (BS, ITMX_RED, ITMX_GREEN, ITMY_RED andITMY_GREEN). POP-AIR is not copied to FE models and so could stay on h1digivideo5.
At time of writing H1 has locked and been in OBSERVE for 50 minutes.
To "green up" the EDC I am running a dummy IOC on opslogin0 to serve the new channels for these cameras which were lost when we reverted them to the old software.
The camera MEDM shows which ones are still running on VID2 (attached).
DAQ Restarts:
Jonathan, Erik, Dave:
Due to the large number of changes we did two DAQ restarts today. The first was primarily for the ASC and SUS model changes and SWWD. The second was for the addition of the new h1isiham1 model and the camera move.
The EDC restart was due to new:
H1EPICS_DIGVIDEO.ini (at the time, moved 6 cameras to new server)
H1EPICS_DAQ.ini (add h1isiham1)
H1EPICS_FEC.ini (add h1isiham1)
H1EPICS_SDF.ini (add h1isiham1)
Tue11Mar2025
LOC TIME HOSTNAME MODEL/REBOOT
08:14:26 h1asc0 h1asc <<< ASC and SUS model changes
08:15:06 h1sush2a h1suspr3
08:17:36 h1sush2b h1iopsush2b
08:18:12 h1sush2b h1susim
08:18:34 h1sush2b h1sushtts
08:19:08 h1susauxh2 h1susauxh2
08:22:02 h1daqdc0 [DAQ] <<< 1st DAQ Restart (no EDC)
08:22:15 h1daqfw0 [DAQ]
08:22:15 h1daqtw0 [DAQ]
08:22:19 h1daqnds0 [DAQ]
08:22:23 h1daqgds0 [DAQ]
08:22:56 h1daqgds0 [DAQ]
08:26:16 h1daqdc1 [DAQ]
08:26:28 h1daqfw1 [DAQ]
08:26:29 h1daqtw1 [DAQ]
08:26:30 h1daqnds1 [DAQ]
08:26:39 h1daqgds1 [DAQ]
08:27:07 h1daqgds1 [DAQ] <<< GDS1 restart, but too quick
08:27:28 h1daqgds1 [DAQ] <<< seconds GDS1 restart good
08:29:46 h1daqfw0 [DAQ] <<< FW0 spontaneous restart
09:38:05 h1seih16 ***REBOOT*** <<< h1seih16 back from BP fix and card install
09:39:40 h1seih16 h1iopseih16
09:39:53 h1seih16 h1hpiham1
09:40:06 h1seih16 h1isiham1
09:40:19 h1seih16 h1hpiham6
09:40:32 h1seih16 h1isiham6
10:04:19 h1seih16 ***REBOOT*** <<< a second IO Chassis incursion to remove 3rd DAC card which is not needed
10:05:54 h1seih16 h1iopseih16
10:06:07 h1seih16 h1hpiham1
10:06:20 h1seih16 h1isiham1
10:06:33 h1seih16 h1hpiham6
10:06:46 h1seih16 h1isiham6
10:48:15 h1daqdc0 [DAQ] <<< second DAQ restart for new h1isiham1 model and various EDC INI files
10:48:28 h1daqfw0 [DAQ]
10:48:28 h1daqtw0 [DAQ]
10:48:32 h1daqnds0 [DAQ]
10:48:37 h1daqgds0 [DAQ]
10:48:48 h1susauxb123 h1edc[DAQ] <<< EDC restart
10:51:22 h1daqdc1 [DAQ]
10:51:34 h1daqfw1 [DAQ]
10:51:34 h1daqtw1 [DAQ]
10:51:35 h1daqnds1 [DAQ]
10:51:43 h1daqgds1 [DAQ]
10:52:18 h1daqgds1 [DAQ] <<< GDS1 restart
11:51:32 h1seih16 h1isiham1 <<< fix ADC card_nums, no DAQ restart needed
The camera server code on both h1digivideo4 and h1digivideo5 was updated from 0.1.16 to 0.1.17 to fix a bug in the periodic archiving of camera images (https://git.ligo.org/cds/software/cameras/pylon-camera-server/-/issues/6).
In addition Tony and Jonathan removed h1digivideo0 and h1digivideo1 from the racks as a cleanup job as part of the camera update WP.
(Jackie F., Gerardo M.)
Replaced the old batteries on the UPS units at both mid stations, both end stations, and two at the corner station.
The UPS units are dedicated supplies for the vacuum racks, the following racks were updated today:
No issues were encountered. One set of batteries had been already installed previously, see aLOG 83151.
Work done under WP #12377.