I'm going through my old alog drafts, this is from Feb 2023 but the times here may now be useful information, so I'm posting this incomplete draft from a year ago.
Last week I did an additional test to follow up on 66843 66751 where excess noise was seen when locking DARM on ETMX with an ESD bias of -125V. Past checks have shown that this excess noise is not due to a calibration error or due to SRCL or MICH feedforward being mistuned with the reduced bias.
The first attachment shows that indeed this noise is there repeatably in a way that is similar to what was seem in past tests, although the spectrum did change over the time I was doing this test. This time I did the measurements in NLN_CAL_MEAS so that ADS and calibration lines were off.
Although the estimate of the ESD quadratic term in 66843 suggested that the ESD quadratic term shouldn't be large enough to explain this noise, I wanted to repeat the test while driving a line on the ESD strongly enough to see the quadratic coupling to confirm that the model of the quadratic coupling is working well. The second attachment shows three examples of driving a 15.45 Hz line with the same amplitude (10 counts at LOCK_L_EXC), each with slightly different results. The green cursors show the injected line and it's harmonic at 30.9Hz. The first time that I injected the line, it seemed as though the broad noise from 20-35Hz was reduced (grey trace, compare to yellow and blue), but in the next two tests this was not repeated.
The third attachment shows what happened when I tripled the amplitude of the injected line, the 15.45 Hz peak is increased by a factor of 3 while the 30.9Hz peak is increased by a factor of 9; this makes sense.
Thu Jan 18 10:02:40 2024 INFO: Fill completed in 2min 38secs
Quick fill, TCs started around 0C.
TITLE: 01/18 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 10mph Gusts, 8mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.34 μm/s
QUICK SUMMARY:
Today's activities: - Relay tube was vented with Nitrogen; it was also taken off (mitre part stayed, the others came off in 1 single piece) - RGA scans have been made at HAM7 and HAM6 - they were satisfactory, details later; the OMC RGA is leaking, so no scan there - leak hunting later the week, or next week - All the RGAs, Cold Cathode gauges, Ion pumps, NEG-pumps were valved out in the volumes to be vented - The corner was vented first. At the start of the vent, the purge air's dew point was at ~-52 deg C, with 0 particles. The vent was carried out from 2 pm to 6 pm - HAM7 was vented after. The purge air's dew point was at ~-42 deg C, with 0 particles. This vent was carried out during the corner vent, taken ~1.5 hours - After the corner vent, GV5 AIP railed: this suggests a leak from the annulus system to the main volume - further investigation is required (history: in the last 3 years this AIP railed 6 times). This Annulus volume is being pumped with an aux cart now during this vent
Pre-Vent RGA scans collected for HAM7 and the corner/HAM6.
Raw data and scan info collected at T2400018, currently only HAM7 are posted but will add the corner scans once back on site.
Attached is HAM7 faraday and SEM scans:
Chamber Pressure: 7.68e-8 Torr
Pumping state: 1200 l/s Ion pump
Valve state: FC1&2 closed, RV1&2 closed, chamber isolated from both HAM5 and BSC3
Below is the summary of the LHO DQ shift for the 9 days period of 2024-01-08 to 2024-01-16: We would like to express our gratitude to the Instrument Team for having a superb detector! Kudos to Zsuzsa Marka for the patient and expert instructions and Beverly and the DetChar Team for the amazing tool and organization! End of O4a shift In observing for the period ~47%. (78.7, 32.3, 6.2, 100, 27, 0, 56, 59.6, 61.3) Range: generally between 150 and 160 Mpc, quite rugged in the last few days. Some earthquakes contributed to lock losses, however, intentional lock loss also happened. Microseism was an issue for a good fraction of the week. There was one day that was covered by a single continuous lock and another day that had no lock whatsoever. For the first part of the period the glitchiness had been normal, but for the last few days there had been quite a bit of large glitches in the bucket. New HVETO channels kept showing up every day, except the last one as compared to the previous period. Combs, such as 23.8Hz, 29.97Hz, 9.474Hz, 9.475Hz, and 9.48Hz showed up on most days. Line counts below the threshold on all days when data is available. Some strange behavior on some PEM channels, as posted in daily logs. Many pyCBC triggers with new SNR > 10 (medium or short) showed up most days; the large glitch rate in the bucket seemed to correlate with how many were there. The full DQ shift report with day-by-day details is available at: https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20240108
Rick and I went to the PCAL LAB to do the PCAL Tx module maintanence today.
We followed Mostly followed the procedure outlined in DCC doc T1600436-v8 which was printed out and attached to the ALOG.
I have also updated the PCALTxMaintenanceLogBook spreadsheet with the latest information from today's activity. This is information is not currently in the DCC.
I will wait until after EX and EY are completed later this week before updating the DCC with the latest spread sheet. The latest information Is currently attached to this alog.
Attached a picture of the OLTF taken with the SR785, I will update this with the .gif that the SR785 gave us tomorrow.
How about I post a PDF of all the pages from the PCAL Lab Tx Module Maintenace.
TITLE: 01/17 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: VAC work continues today.
I've enabled WIFI across the site which Dave accepted in the CDS SDF, and disabled the picomotors for the chambers being vented.
Visitors from Caltech arrived at 1700UTC and did a safety site walkthrough/tour with Richard, Mike, and Danny.
When launching guardian logs and running the CPS ISI weeklies script I kept getting a notification for Numpy being out of date (Its currently version 1.17.3 and it wants version 1.25.0 to work with the current version of scipy. (Tagging CDS)
Vacuum alarms off and on through the day as activities progress. Verbal reported BSC8 and HAM6 high vacuum pressure at 21:59UTC.
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 16:41 | VAC | Jordan, Travis, Janos | SITE | N | VAC WORK all day everyday; purge air, RGA | 02:41 |
| 16:48 | FAC | Kim & Karen | LVEA | N | HAM6/7 clean | 19:47 |
| 16:57 | CAL | Tony & Rick | PCAL lab | LOCAL | PCAL work | 20:48 |
| 17:27 | FAC | Randy | ARMS | N | Run the snowplow | 19:06 |
| 17:35 | FAC | Richard,Danny+Caltech visitors | Site | Safety site tour | 18:22 | |
| 17:48 | FAV | Mitch | Arms | N | FAMIS tasks | 18:34 |
| 18:04 | PSL | Jenne | PSL anteroom | N | Parts check | 18:30 |
| 18:21 | FAC | Richard,Mike,Danny+Safety team | LVEA | N | Safety walkthrough/tour | 19:21 |
| 18:58 | TCS | Camilla | Optics lab | N | Parts search | 19:35 |
| 19:06 | FAC | Randy | LVEA | N | Checks | 19:31 |
| 19:45 | SQZ | Vicky, Daniel | LVEA | N | SQZ work prep | 20:50 |
| 19:47 | FAC | Karen | MidY | N | Tech clean | 20:10 |
| 19:48 | FAC | Kim | woodshop | N | Tech clean | 20:48 |
| 17:00 | EE | Ken | EndY | N | Lights | 19:50 |
| 20:50 | ISC | Camilla, Betsy | Optics lab, PCAL lab | N | Parts search | 21:20 |
| 20:53 | EE | Fil | LVEA/CER | N | Turn off HV | 21:39 |
| 21:52 | CAL | Tony, Rick | EndX | LOCAL | PCAL checks | 23:52 |
| 21:56 | SQZ | Vicky, Daniel | LVEA | N | SQZ work | 22:56 |
Procedure M1300464
The following high voltage power supplies/electronics were powered off in preparation for the corner vent.
1. CER Mezzanine - ESD HV
2. CER Mezzanine - Fast Shutter
3. CER Mezzanine - OMC PZT
4. CER - SR3 and ITM heaters
5. LVEA - Fast Shutter Driver Chassis - Disabled and powered off
6. MER Mezzanine - HAM7 PSAMS
7. MER Mezzanine - HAM7 Piezo
After Fil finished turning off the HV power, following section 3.6 from M1300464 on the DCC and I confirmed that all of the picomotors for HAM3, HAM6, HAM7, and EX were disabled, they were all already off when I opened their medms. MEDMs screenshot
Closes FAMIS 25974, last check in alog75278. We're not totally in nominal chamber or SEI states at the time of this script.
Script reports that:
ETMY_ST2_CPSINF_H2 high frequency noise is high.
HAMs:
HAM2_CPSINF_H3,V3 looks elevated
HAM3_CPSINF_V1,V3 looks elevated
HAM4_CPSINF_V1looks elevated
HAM5_CPSINF_H3,V1,V3 looks elevated
HAM6_CPSINF_H3,V1,V3 looks elevated
HAM7_CPSINF_H1,H3,V1,V2,V3 looks elevated
HAM8_CPSINF_V3 looks elevated
BSC:
ITMY_ST1_CPSINF_V3 looks elevated
ETMY_ST2_CPSINF_H2 looks elevated
I made a comparison of DARM_IN1_DQ and CAL-DELTAL_EXTERNAL_DQ in nominal VS new DARM offloading scheme (the new scheme itself is explained in alog 74887). Data for the NEW_DARM configuration was taken from Dec/21 (alog 74977) when Louis and Jenne successfully transitioned but with calibration that did not make sense.
The main things you must look at are the bottom left panel red and blue, i.e. the coherence between DARM_IN1 and CAL-DELTAL_EXTERNAL in the NEW (red) VS the old (blue) configuration. Blue trace is almost 1 as it should be, but the red drops sharply between 20Hz and 200Hz.
This does not make any sense because CAL-DELTAL_EXTERNAL is ultimately a linear combination of DARM_IN1 and DARM_OUT (see https://dcc.ligo.org/G1501518). Since DARM_OUT is linear to DARM_IN1, no matter where and how the noise is generated and no matter how you redistribute the signal in the ETM chain, CAL_DELTAL_EXTERNAL should always be linear to DARM_IN1, therefore coherence should be almost 1.
So what's the issue here?
The only straightforward possibility I see is that somehow excessive numerical noise is generated in the calibration model even with the frontend's double precision math. Maybe something is agressively low-passed and then high-passed, or vice versa, that kind of thing.
It is not an artefact of the single precision math of DTT. Both CAL_DELTAL_EXTERNAL and DARM_IN1 is already well whitened, and they're entirely within the dynamic range of single precision. For example, RMS of red CAL-DELTAL_EXTERNAL_DQ trace is ~7E-5 cts. From that number, I'd expect that the noise floor due to single precision is very roughly O(7E-5/10**7 /sqrt(8kHz)) ~ O(1E-13) cts/sqrtHz if it's close to white, give or take some depending on details, but the actual noise floor is ~10E-8 cts/sqrtHz. Same thing can be said for DARM_IN1.
It's not the numerical noise in DARM filter as the coherence between DARM_IN and SUS-ETMX_L3_LSCINF_L_IN1 (which is the same thing as DARM_OUT for coherence purpose) is 1 from 1Hz to 1kHz for both configurations (old -> brown, new -> green). (It looks as if the coherence goes down above 1kHz for the old config, but that's irrelevant for this discussion, and anyway it's an artefact of DTT's single precision math. See e.g. the top left blue (old config DARM_OUT) with RMS of 20k counts, corresponding to O(2E-5)/sqrtHz single noise floor due to single precision, give or take some. See where the actual noise floor is.)
It's not a glitch, noise level of CAL_DELTAL_EXTERNAL spectrum didn't change much from one fft to the other for the entire window (I used N=1 exponential to confirm this).
Note that there's also a possibility that excessive noise is generated in the SUS frontend too, polluting DARM_IN1 for real, not just for calibration model. I cannot tell if that's the case or not for now. The difference between the green (new) and brown (old) DARM_IN1 spectrum in the top left panel could just be a difference in gain peaking due to different DARM loop shape.
I'll see if double precision channels (recorded as double) in calibration model are useful to pinpoint the issue. Erik modified the test version of DTT so it handles the double precision numbers correctly without casting into single, but it's crashing on me at the moment.
some more time windows to look into while we were in the NEW DARM state are listed at LHO:75631.
The purge air compressors continue to run as VAC works on getting ready to vent, VAC team hopes to get doors off tomorrow (thurs 18th) or friday (19th).
After looking at beeps and error lights on h0epics2 it looks like we have migrated all the jobs that were on it off already. We powered off h0epics2 at 11:12am localtime.
We found one medm screen that wanted channels from an ioc on h0epics. * h0video - this controlled some of the analog camera system. We have moved that to h0epics and are inquiring if we can just not run this anymore. Our old wiki pages also point to a h0tidal ioc, however its channels have not been in the frame since 2015 and the ioc itself is no longer in the shared drive. So having migrated h0video to h0epics we will keep h0epics2 off.
We are power cycling the FMCS EPICS computer (fmcs-epics-cds) as a first try to regain stability.
After the first cell phone alarms were sent, I've bypassed them for a couple of hours.
Jonathan, Patrick, Dave:
The FMCS IOC computer is back online. The restart code is running again. The cell phone alarm bypass has been removed.
Louis, Jenne, TJ, Sheila
Today we continued to try to transition to the new Darm configuration, which we had suceeded in doing in December but weren't able to repeat last week (75204).
In our first attempt today we tried a faster ramp time, 0.1 seconds. This caused immediate saturation of ETMX ESD. We struggled to relock because of the environment.
Because Elenna pointed out that the problem at roughly 3 Hz with the earlier transition attempts might have been the soft loops, we thought of trying to do the transition before the soft loops are engaged, after the other ASC is on. We tried this first before transitioning to DC readout which wouldn't work because of the DARM filter changes. Then we made a second attempt at DC readout. We also lost lock due to a 2 Hz oscialltion, even without the soft loops on.
Some gps times of transitions and attempt:
Adding two more times to this list:
The second screenshot here shows the transitions from Dec 13th, 14th, and 19th. These are three slightly different configuration of the UIM filters and variations on which PUM boosts were on when we made the transition. On the 14th the oscillation was particularly small, this was with our new UIM filter (FM 2 + 6) and with both PUM boosts on L2 LOCK FM1,2,10 already during the transition. This is the same configuraition that failed mulitple times in the last two weeks.
Today I went back to three of these transitions, December 14th (1386621508 sucsesful no oscillation) and Jan 4 (1388444283) + Jan 5th (1388520329) which were unsucsesfull attempts. It also seems as though the only change to the filter file since the Dec 14th transition is a change copy the Qprime filter into L1 drivealign, which has not been used in any of these attempts (this can't be used because tidal is routed through drivalign).
In short, it doesn't seem that we made a mistaken change to any of these settings between December and January which caused the transition to stop working.
| L1 DRIVEALIGN L2L | 37888 | no filters on | |
| L1 LOCK L | 37922 | FM2,6 (muBoostm, aL1L2) | |
| L2 DRIVEALIGN L2L | 37968 | FM5,7 (Q prime, vStopA) | |
| L2 LOCK L | 38403 | FM1,2,10 (boost, 3.5, 1.5:0^2, cross) on the 5th FM1+ 2 were ramping while we did the transition | |
| L3 DRIVEALIGN L2L | 37888 | no filters on | |
| L3 LOCK L | 268474240 | FM8, FM9, FM10, gain ramping for 5 seconds (vStops 8+9, 4+5, 6+7) | |
| ETMX L3 ISCINF L | 37888 | no filters on | |
| DARM2 | 38142 | FM2,3,4,5,6,7,8 | |
| DARM1 | 40782 | FM2,3,4,7,9,10 |
I added start and end time windows for the successful transitions in LHO:75631.
[Measurements attached]
FAMIS LINK: 25973
BSC CPS: Received following from CPS script terminal output:
NOTE: If one was to follow the above channel as an example, it would seem several other BSCs had CPS' with spectra similarly high by eye:
HAM CPS: Looks good.
Gabriele, Rahul, Jeff, Oli, Betsy
Find below the initial (no damping loop) transfer functions for the built X1 (Staging Building Test Stand) M1 (Top Mass) BBSS.
These were taken generally following Jeff and Oli's alog 74142, where we used their provided DTT template and tuned the excitation amplitude for each DoF to avoid overflows.
Files are saved on the X1 work station system under ligo/svncommon/SusSVN/sus/trunk/BBSS/X1/BS/SAGM1/Data/
They are dated 01/05/24.
Attached is a pdf of annotated frequency transfer functions for the above measurement.
Find below the Transfer Function resonant peaks frequencies in list form. The second column shows cross coupled freqs i.e. these same frequencies were seen/repeated on different degrees of freedom.
| Number of Resonant Peaks | Frequency (Hz) | |
| 1 | 0.40625 | 0.40625 |
| 2 | 0.414062 | 0.414062 |
| 3 | 0.421875 | |
| 4 | 0.445312 | |
| 5 | 0.453125 | |
| 6 | 0.96875 | |
| 7 | 1.03125 | |
| 8 | 1.15625 | |
| 9 | 1.16406 | 1.16406 |
| 10 | 1.32031 | |
| 11 | 1.4375 | |
| 12 | 1.54906 | |
| 13 | 1.875 | |
| 14 | 2.59375 | |
| 15 | 2.60156 | |
| 16 | 2.69531 | |
| 17 | 2.9919 | 2.99219 |
| 18 | 3.84375 | |
The "UNSURE" table contains some peaks that were somewhat noisy (and not included in the annotation). According to the model, there are supposed to be 2 peaks over 10 Hz (~19Hz and ~31Hz), but due to the noise, I couldn't determine if these were measured - there are definitely peaks in this region at the listed frequencies (none in the "right" spots though).
| UNSURE |
| 21.9375 |
| 26.75781 |
| 37.4375 |
| 37.39062 |
For ease of comparison, below is another table with the modeled peak frequencies. Note that while there are 18 peaks, they do not correspond to the same "18 modes" - this is a pure coincidence.
| Mode Number | Mode Frequency (Hz) |
| 1 | 0.412468 |
| 2 | 0.416083 |
| 3 | 0.425915 |
| 4 | 0.473335 |
| 5 | 1.03738 |
| 6 | 1.1633 |
| 7 | 1.16605 |
| 8 | 1.18585 |
| 9 | 1.45428 |
| 10 | 1.46622 |
| 11 | 1.53717 |
| 12 | 1.90606 |
| 13 | 2.64974 |
| 14 | 2.71041 |
| 15 | 3.03671 |
| 16 | 3.85694 |
| 17 | 19.6021 |
| 18 | 31.9402 |
This is still a WIP.