oli.patane@LIGO.ORG - posted 14:14, Wednesday 18 September 2024 - last comment - 15:41, Wednesday 18 September 2024(80168)
Lockloss
Daniel, Marc, Fil, Richard, Erik, Dave:
Attached drawing shows the SUS ETMX DAC changes made yesterday.
Summary:
The new LIGO 28-bit DAC has replaced the h1susetmx model's 20-bit DAC signals. These comprise five L3 ESD signals, four L2 signals and four L1 signals.
The h1susetmx 18-bit DAC channels were not changed (the M0 and R0 signals).
The h1susetmxpi model's two 20-bit DAC signals were not changed, no h1susetmxpi model change was needed.
The routing of the h1susetmx model's L3, L2 and L1 signals to the 28-bit DAC was done using an existing matrix. Gains were applied using existing filter-modules. No h1susetmx model change was needed, this was a hardware change.
Details:
Please reference attached drawing and h1susetmx model snipet.
The h1susex front end comprises two ADCs, three 18-bit DACs, two 20-bit DACs and the new LIGO DAC. Setting the ADCs aside;
The third 18-bit DAC is only used by the h1sustmsx model, and so can be discounted.
The first two 18-bit DAC cards are used by h1susetmx (driving M0 and R0). These were not touched and are not applicable.
The first 20-bit DAC card is used by h1susetmx to drive L1 and L2 (four channels each).
The second 20-bit DAC card is shared between the h1susetmx and h1susetmxpi models. h1susetmx owns the first six channels, and drives five of them (L3-ESD DC+4QUAD). h1susetmxpi owns the last two channels (ESD left and right).
There are two types of Anti-Imaging (AI) chassis used here: the standard 18-bit/20-bit DAC AI (two inputs, each input is driven by a separate DAC), and the Parametric Instability (PI AI) chassis. The PI-AI has one input (block of eight channels) which internally are split into two blocks of six and two channels. The block of six channels is filtered normally and exits as chan 1-6 on front panel ('OUT 1-6'), the block of two channels is PI filtered and exits on its own front connector ('Band Pass Ch 7 & 8'). See attached photo.
Before:
The first 20-bit DAC is connected to one half of a standard AI chassis. Its output drives the L1, L2 signals.
The second 20-bit DAC is connected to a PI-AI chassis. The first 6 channels are driven by h1susetmx, the last two by h1susetmxpi.
The LIGO DAC is not connected to any AI chassis.
Now:
A second PI-AI chassis was installed in the rack, it is used to drive h1susetmxpi's DAC channels.
The first 20-bit DAC was disconnected from its AI (see note below).
The second 20-bit DAC was disconnected from the original PI-AI and connected to the new PI-AI. The PI driver cable was moved from the original to new AI. This means that other than the AI chassis change, the h1susetmxpi model and its drives were unchanged.
The first 8 channels of the LIGO DAC are connected to the original PI-AI (with its L3/ESD field cabling intact). Therefore the ESD analog filters were not changed in the transition from 20-bit to 28-bit DAC.
The second 8 channels of the LIGO DAC are connected to the standard AI (was connected to first 20-bit DAC). This permits drive of L1 & L2.
The attached MEDM snapshot shows the matrix/gain settings. The first 5 channels are driving the ESD, the second block of 8 the L1/L1.
Complications:
Initially the first 20-bit DAC (h1susetmx L1 & L2) was going to be left disconnected from any AI. However the lack of an AI watchdog signal put this DAC into error. Marc connected this DAC to a second input of a PI-AI chassis. This input does not internally connect to any filter block, but the interface card does supply the missing AI watchdog signal.
The IOP model's software watchdog (SWWD) needs to have the new LIGO DAC added to its DACKILL list, now that the new DAC is part of production.
Looking at data from a couiple of days ago, there is evidence of some transient bumps at multiples of 11.6 Hz. Those are visible in the summary pages too around hour 12 of this plot.
Taking a spectrogram of data starting at GPS 1410607604, one can see at least two times where there is excess noise at low frequency. This is easier to see in a spectrogram whitened to the median. Comparing the DARM spectra in a period with and without this noise, one can identify the bumps at roughly multiples of 11.6 Hz.
Maybe somebody from DetChar can run LASSO on the BLRMS between 20 and 30 Hz to find if this noise is correlated to some environmental of other changes.
I took a look at this noise, and I have some slides attached to this comment. I will try to roughly summarize what I found.
I first started by taking some 20-30 hz BLRMS around the noise. Unfortunately, the noise is pretty quiet, so I don't think lasso will be super useful here. Even taking a BLRMS for a longer period around the noise didn't produce much. I can re-visit this (maybe take a narrower BLRMS?), but as a separate check I looked at spectra of the ISI, HPI, SUS, and PEM channels to see if there was excess noise anywhere in particular. I figured maybe this could at least narrow down a station where there is more noise at these frequencies.
What I found was:
I was able to run lasso on a narrower strain blrms (suggested by Gabriele) which made the noise more obvious. Specifically, I used a 21 Hz - 25 Hz blrms of auxiliary channels (CS/EX/EY HPI,ISI,PEM & SUS channels) to try and model a strain blrms of the same frequency via lasso. In the pdf attached, the first slide shows the fit from running lasso. The r^2 value was pretty low, but the lasso fit does pick up some peaks in the auxiliary channels that do line up with the strain noise. In the following slides, I made time series plots of the channels that lasso found to be contributing the most to the re-creation of the strain. The results are a bit hard to interpret though. There seems to be roughly 5 peaks in the aux channel blrms, but only 2 major ones in the strain blrms. The top contributing aux channels are also not really from one area, so I can't say that this narrowed down a potential location. However, two HAM8 channels were among the top contributors (H1:ISI_HAM8_BLND_GS_X/Y). It is hard to say if that is significant or not, since I am only looking at about an hours worth of data.
I did a rough check on the summary pages to see if this noise happened on more than one day, but at this moment I didn't find other days with this behavior. If I do come across it happening again (or if someone else notices it), I can run lasso again.
I find that the noise bursts are temporally correlated with vibrational transients seen in H1:PEM-CS_ACC_IOT2_IMC_Y_DQ. Attached are some slides which show (1) scattered light noise in H1:GDS-CALIB_STRAIN_CLEAN from 1000-1400 on Septmeber 17, (2) and (3) the scattered light incidents compared to a timeseries of the accelerometer, and (4) a spectrogram of the accelerometer data.
Wed Sep 18 08:10:41 2024 INFO: Fill completed in 10min 37secs
Jordan confirmed a good fill curbside.
TITLE: 09/18 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 5mph Gusts, 3mph 3min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY:
Relocking and at CARM_5_PICOMETERS. Not sure yet why we lost lock, but it might've been an earthquake.
Lockloss from earlier this morning, 2024-09-18 13:39UTC, wasn't caused by an earthquake, but by the FSS saturating. This is the first FSS_OSCILLATION lockloss that we've had from NLN in a year
Back to Observing 15:09UTC.
Had to accept this sdf diff for ETMX_L3_ESDOUTF_LR to get into Observing. This change happened during TRANSITION_FROM_ETMX. Since this is our first relock since the DAC was changed out yesterday at EX, wondering if this was related at all to those changes.
I probably had a stray mouse click that turned that decimation filter off, and then did my SDF-accepting of the state of FMs 9 and 10 of the ESDOUTF filter banks. Thankfully it was just the decimation filter that I had wrong, which doesn't affect our sensitivity in any way.
Anyhow, I have updated the safe.snap file to accept the decimation filter being correctly ON (see attached). This will show up as an SDF diff again (opposite of what Oli posted this morning) in Observe and should be accepted. I don't want to set it in Observe right now, since accepting would require changing other things with the ETM, which I don't want to do right now. So, it'll stay as a diff.
TITLE: 09/17 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Calibration
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:
Tony took H1 to Observing, during our Ops hand-off, ending a long Maintenance. Winds were high during the afternoon peaking at about 2315utc; H1 had lower range during this windy beginning of shift, but either thermalization and/or dying winds helped with H1 range hovering around the "usual" 160Mpc.
Dropped from Observing for a thermalized calibration requested by Jenne (with Louis helping).
LOG:
Since H1 was thermalized after this lock which was after a long/busy Maintenance Day, was asked to run another calibration. Notified LLO & Virgo via chat & TeamSpeak to give them a heads up about H1 dropping out of Triple Coincidence for 30min for calibration work.
Measurement NOTES:
This morning the In-Lock SUS Charge Measurements ran. Attached are the plots for all four Test Masses. Closing FAMIS 28371.
L. Dartez, T. Sanchez, J. Driggers, M. Pirello, J. Rollins, J. Betzwieser
The H1 calibration has been updated to account for the additional 15.2us delay due to the new LIGO DAC card used in the ETMX L3 path.
There was a bit of confusion and imperfect communication over how the new DACs were implemented that contributed to some locking troubles after the maintenance period.
- First, the DACs were only installed for ETMX-L3 for now. With conflicting emails and alogs I walkedin thinking all 3 stages had their DACs swapped.
- The gains that I adjusted in the COILOUTF and ESDOUTF filter banks were not used. On L1 and L2 this is because there was no DAC swap. On L3 it's because the new DACs have a built-in gain that serves the same purpose as the ones I adjusted. as such, the '32bit DAC' FMs are ignored and can be purged.
Updating the calibration pipeline for this change required steps that are outside of the anticipated.
Our last exported report, 20240330T211519Z, represents the state of the DARM model that is used to inform the front end and the GDS pipeline. I copied this report into 20240917T222803Z (this time stamp is not associated with any measurement files; i made it up as the time at which I was doing the copying) and updated the pydarm_H1.ini to include an additional 15.2 us delay in the unknown actuation delay parameter for the X arm, bringing the total 'unknown' actuation delay to 30.2us. I then updated the id file to reflect the new report id and regenerated the new report using pydarm report --regen 20240917T222803Z (I should have used --skip-mcmc also but that's ok). Then I had to fix the id file _again_ after the generation before committing and uploading the report to ldas.
Once the report was committed, I exported it to the front end. The front end value changes are all on the order of a percent or two due to rerunning the MCMC during the generation. Note to pydarm devs: I had to mark this report as 'valid' in order to get pydarm to export it. But I didn't want to mark it as valid since we won't want this report to be considered as a unique measurement in the uncertainty budget. We'll need to remove the valid tag before preparing the unc budget for O4b.
There was some trouble getting GDS restarted but Jamie and Jonathan jumped in to help with that. It was just an auth issue.
To clarify L1 & L2 are driven by the new DAC. Gains don't need to be adjusted in the normal path, since they are adjusted in the new paths.
TITLE: 09/17 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 143Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
This morning's Lockloss for maintenance: 2024-09-17 15:19
Relocking after the ETMX DAC swap:
TMSX was off by a lot, realigned by hand.
Turned off SEI maintenance ENV
Reverted sliders to : 1410631523
Adjutsed sliders to get OSUMs back to a place where we had flashes.
Running Baffle_Align for TMSX
Using measured BPD values to center TMSX.
Old offset pitch -100.25911370237192 and yaw -107.50189039633354
New offset pitch -99.5599642875359 and yaw -108.61504480911252
H1:SUS-TMSX_M1_OPTICALIGN_P_OFFSET => -99.5599642875359
H1:SUS-TMSX_M1_OPTICALIGN_Y_OFFSET => -108.61504480911252
Turning off the test offsets to TMSX. (should be P -33.4, Y 30.5)
H1:SUS-TMSX_M1_TEST_P => OFF: OFFSET
H1:SUS-TMSX_M1_TEST_Y => OFF: OFFSET
TMSX dither alignment finished!
PZT Y TRIGGER NOT ON!
Running Baffle Align for ITMX
Using measured BPD values to center ITMX.
Old offset pitch -93.16362911805888 and yaw 109.68269245998749
New offset pitch -98.52206679818204 and yaw 107.62344936551601
H1:SUS-ITMX_M0_OPTICALIGN_P_OFFSET => -98.52206679818204
H1:SUS-ITMX_M0_OPTICALIGN_Y_OFFSET => 107.62344936551601
Turning off the test offsets to ITMX. (should be P -25.7, Y -19.0)
H1:SUS-ITMX_M0_TEST_P => OFF: OFFSET
H1:SUS-ITMX_M0_TEST_Y => OFF: OFFSET
ITMX dither alignment finished!
PZT Y TRIGGER NOT ON!
Baffle Align ETMX was ran,
Then ETMX and TMSX touched up by hand. Light on ALSX!!!
Locking started , reached PREP_DC_READOUT_TRANSITION
Fire Alarm! & Lockloss during the Fire drill.
Louis set some changes to the ETMX Gains for L1, L2,& L3. while H! was in down after that Lockloss. Please see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=80141
Which caused an ALS X Arm Fault. Once we realized that there was an error in the Gain due to the DAC being 28 and not 32 bit. please see Alog from Marc.
Letting Marc and Louis chat, we all learned that there is already gain being applied to compensate for the extra bits and Louis didn't need the gain changes. Thus he was adding more gain to the existing gain which is good for guitars, but bad for IFO's .
Louis's gains have now been reverted.
This allowed us to get past locking ALS.
Poked the BS a little to catch DRMI.
NLN reached at 22:08 UTC!!!
22:12 UTC pydarm measure --run-headless bb was ran from NLN_CAL_MEAS [700]
@ 23:15 UTC pydarm measure --run-headless bb was ran aqgain to compair with the first one that was taken.
Success!!! We are in Observing with the NEW DAC.
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 23:58 | SAF | H1 | LHO | YES | LVEA is laser HAZARD | 18:24 |
| 15:05 | HWS | Camilla | LVEA | YES | Opening the HWFS table | 15:52 |
| 15:14 | EE | Fil | LVEA & HAM Shaq | Yes | Helping Camilla with Power and working out in the FTCE. | 17:52 |
| 15:15 | VAC | Gerardo | Mech room | n | Ion pump work | 15:48 |
| 15:15 | EE | Marc | EX | N | DAC swap | 16:35 |
| 15:22 | IAS | Jason | LVEA | laser hazard | Changing farrow settings. | 15:31 |
| 15:23 | FAC | Karen | WoodShop | n | Technical cleaning | 16:23 |
| 15:26 | SEI | Jim | His office | N | BRS Damping. | 17:26 |
| 15:28 | FAC | Chris | EY | N | HVAC Filter swap. | 17:25 |
| 15:31 | FAC | Contractor | Arms | N | Outhouse maintenace. | 18:31 |
| 15:38 | VAC | Norco | CP2 (CS) | n | LN2 fill | 17:38 |
| 16:30 | PEM | Genivive | LVEA | Yes | Looking for PEM parts Accelerometer & Mic. | 16:58 |
| 16:31 | FAC | Karen & kim | LVEA | Y | Technical cleaning | 17:11 |
| 16:54 | PEM | Sam & Genivive | LVEA | YES | Looking for PEM parts | 17:09 |
| 17:10 | OPS | TJ | LVEA | Yes | LVEA Sweep | 17:40 |
| 17:12 | Access Laser | Camilla , TJ, +2 | LVEA | yes | Show Access Laser the CO2 lasers | 18:12 |
| 17:54 | PEM | Genivive & Sam | Ham Shaq | N | Checking PEM mic and sensors | 18:36 |
| 18:09 | FAC | Chris | EX | n | hvac fILTER SWAP | 19:04 |
If we plan to use the new LIGO 28AO32 DAC to drive ETMX L1,L2,L3 for a while, we should add this new DAC to the IOP model's software watchdog (SWWD) DACKILL list.
Currently if the h1iopsusex SWWD is triggered for 15mins, it will DACKILL the 18bit and 20bit DACs, but not the 28bit.
This requires are restart of all the models on h1susex, but no DAQ restart is required.
Yes, I think that we can squeeze this in as a target-of-opportunity, if the IFO is unlocked during business hours sometime. We have enough other watchdogs (including the lockloss triggers to stop outputs even if guardian is not working), that I think it should be okay to not break the lock for this, or wake someone up in the middle of the night to do the restart.
EDIT: Dave just noted to me in an email that it can also wait until next Tuesday, if we don't get it in before then.
TITLE: 09/17 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Calibration
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 28mph Gusts, 24mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY:
notification: new test result
notification: end of measurement
notification: end of test
diag> save /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240917T221248Z.xml
/ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240917T221248Z.xml saved
diag> quit
EXIT KERNEL
2024-09-17 15:17:59,370 bb measurement complete.
2024-09-17 15:17:59,370 bb output: /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240917T221248Z.xml
2024-09-17 15:17:59,371 all measurements complete.
Second Broadband measurement for compairison:
notification: new test result
notification: end of measurement
notification: end of test
diag> save /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240917T231505Z.xml
/ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240917T231505Z.xml saved
diag> quit
EXIT KERNEL
2024-09-17 16:20:16,365 bb measurement complete.
2024-09-17 16:20:16,365 bb output: /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240917T231505Z.xml
2024-09-17 16:20:16,365 all measurements complete.
Building on work last week, we installed a 2nd PI AI chassis (S1500301) in order to keep the PI signals separate from the ESD driver signals. Original PI AI chassis S1500299.
We routed the LD32 Bank 0 thorugh the first PI AI chassis to the ESD drive L3, while keeping the old ESD driver signal driving the PI through the new PI AI chassis.
We routed the LD32 Bank 1 to the L2 & L1 suspension drive.
We did not route LD32 Bank 2 or Bank 3 to any suspensions. The M0 and R0 signals are still being driven by the 18 bit DACs.
The testing did not go as smoothly as planned, a watchdog on DAC slot 5 (the L1&L2 drive 20 bit DAC) continousouly tripped the ESD reset line. We solved this by attaching that open DAC port (slot 5) to the PI AI chassis to clear the WD error.
Looks like we made it to observing.
F. Clara, R. McCarthy, F. Mera, M. Pirello, D. Sigg
Part of the implication of this alog is that the new LIGO DAC is currently installed and in use for the DARM actuator suspension (the L3 stage of ETMX). Louis and the calibration team have taken the changes into account (see, eg, alog 80155).
The vision as I understand it is to use this new DAC for at least a few weeks, with the goal of collecting some information on how it affects our data quality. Are there new lines? Fewer lines? A change in glitch rate? I don't know that anyone has reached out to DetChar to flag that this change was coming, but now that it's in place, it would be helpful (after we've had some data collected) for some DetChar studies to take place, to help improve the design of this new DAC (that I believe is a candidate for installation everywhere for O5).
Analysis of glitch rate:
We selected Omicron transients during observing time across all frequencies and divided the analysis into two cases: (1) rates calculated using glitches with SNR>6.5, and (2) rates calculated using glitches with SNR>5. The daily glitch rate for transients with SNR greater than 6.5 is shown in Figure 1, with no significant difference observed before and after September 17th. In contrast, Figure 2, which includes all Omicron transients with SNR>5, shows a higher daily glitch rate after September 17th.
The rate was calculated by dividing the number of glitches per day by the daily observing time in hours.
WP 12080 After turning the HWS lasers back on in 80043, today I reinstalled the masks. To get the right amount of light on the CCD camera, ITMX is set to 1Hz and ITMY to 20Hz. New references taken at 15:50UTC, 30 minutes after we lost lock, so the IFO wouldn't have been 100% cold.
HWS is aligned and working well.
Plots are attached comparing 40 seconds, 120 seconds and 20 minutes after we power to 60W input.
Comparing to March 76385, the ITMX main point absorber looks to be heating up more than it has in the past but the main IFO beam also looks a little lower and offset than March (origin cross is the same), so that could be causing the point absorber to look different. The ITMX P2L is the same as March and Y2L 0.1 higher.
Noticed the ITMY HWS code had stopped, error attached. While we were out of observing today, I restarted it.
22:39 Observing
Accepted Jenne's sdf diff in Observe