Seismon and several other IOCs, including lveatemps, picket fence, and external alerts, were restarted.
Camilla, Ansel
At 17:46UTC, Camilla changed ITMX and ITMY HWS to 'sem 3', which is external trigger mode. This seemed to work, and stopped the code taking photos.
It appears that the HWS-associated combs are gone in the magnetometer witness channel after this change. Pre/post 1-hour spectra attached. (All known HWS-associated combs overlaid, just to check-- the near-7Hz is the one actually present in the "pre" spectrum which corresponds to current HWS sync frequency settings.)
Nice work Camilla and Ansel! Let's hope this solves the problem for good.
At 17:38UTC I tunred on the camera/CLink and restarted the HWS EX code which had been off since 74738.
At 17:46UTC I changed ITMX and ITMY to 'sem 3' and left them in this configuration so that they are not taking HWS data now. Now we need to write a script to read H1:GRD-ISC_LOCK_STATE_N state and adjust the camera mode to be sem 2 when we are in states < 580 (locking) and sem 3 when > 580 (locked). This is not trivial as camera computers are separate from EPICS.
Sheila, Naoki, Vicky - SQZ OMC mode scans with cold OM2
Taking SQZ-OMC mode scans, using DTT template saved in $(userapps)/sqz/h1/Templates/dtt/OMC_SCANS/Dec19_2023_PSAMS_OMC_scan_coldOM2.xml
PSAMS 200/200, cold OM2
PSAMS 120/120, cold OM2
Dark
1.9 W PSL input power, PSL-OMC mode scans, Cold OM2 - Sheila
with sidebands ON:
with sidebands OFF:
Dark: 1387052098 - 1387052350
Total scans from today here with zoom-in on SQZ/PSL scans.
Tue Dec 19 10:06:51 2023 INFO: Fill completed in 6min 47secs
Over the past few weeks we've occasionally been seing the notice on DIAG_MAIN that the IMC WFS need to be centered, so I did that during the maintenance window this morning, The process was as follows:
I'm leaving the IMC offline for SQZ/OMC scans.
TITLE: 12/19 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 12mph Gusts, 8mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.28 μm/s
QUICK SUMMARY:
After a test this morning caused H1 to lose lock at 15:33 (see alog74887), maintenance activities have begun. Camilla is in the process of untripping SEI WDs.
Camilla, Louis We dropped out of Observing at 7am PST test the new DARM loop config: - the new DARM loop config, navigated to by going toNLN_ETMY->NEW_DARM, seems to be stable. We were able to sit there for several minutes without issue. - Camilla took a quick look at MICH FF settings while in 'NEW_DARM'. She will follow up with more info on that. - We lost lock again while trying to cal measurements in the NEW_DARM state. Still trying to figure out what's happening there; clearly reducing the amplitudes by 50% (LHO:74883) wasn't nearly enough. It seems we kicked the IFO hard enough for ISI_ETMX watchdogs to trip. due to the lockloss we did not get to test the DARM_RECOVER guardian state.
When the excitations started, the ETMX HEPI, ISI and SUS WDs tripped and (~5 minutes later) the hardware watchdogs for both SEI and SUS.
With all watchdogs tripped, I cleared history of the L2 LOCK L stage of ETMX as this was outputting very large numbers. M0 DAMP also seemed to be outputting very large numbers, I tried cleaning history but it seems like this was physically ust moving a LOT, plot attached.
Rahul via phone walked me though checking that ETMX is ready to be untripped: output counts changing by hundreds not 10,000's. We then reset SUS SW WD (cds > SWWD > ETMX) and untripped the SUS ETMX WD, Rahul said if it was still ringing very badly we would turn on M0 DAMP damping loops one-by-one but they already looked on so I took the GRD to DAMPED then ALIGNED.
After we got ETMX SUS back to usual, I reset the ETMX SEI SWWD and I followed the procedure to uptrip SEI (from wiki, HEPI then ISI). Once this was done Ryan took us to maintenance mode.
The new DARM configuration is shown in this plot is veery effective in reducing the ESD RMS, as shown in the plot below. The blue curve is one hour of normal configuration, showing the ESD total RMS as a function of time. The orange trace is during the test: the first few hundreds of seconds are still the nominal configuration, while between ~1000 and ~2000 seconds the new DARM configuration is running. The RMS is reduced well below the minimum of the normal configuration.
Unfortunately the was no clean data collected during this time to see the effect on the strain.
The MICHFF FM4 that we fit for cold OM2 74877 was actually very good for tis new DARM configuration green trace, much better than the FM1 we fit for the new DARM configuration 74817.
DARM didn't look great at 30-100Hz in either configuration, maybe this could be as the calibration is wrong.
The simulines measurements injected into the following channels:
H1:LSC-DARM1_EXC
H1:CAL-PCALY_SWEPT_SINE_EXC
H1:SUS-ETMX_L1_CAL_EXC
H1:SUS-ETMX_L2_CAL_EXC
H1:SUS-ETMX_L3_CAL_EXC
Injections began at approximately GPS 1387035241.
Summary of strain coherencce with all channels: https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_1386992886_STRAIN/
It looks like there is no coherence with MICH nor SRCL
At low frequency, the usual coherence with DHARD_Y
The only noticeable difference now with respect to hot OM2 is the incresed coherence with beam jitter
https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_1386992886_STRAIN_CLEAN/
Coherences for the CLEAN channel, with jitter removed
Workstations were updated and rebooted. These were OS package updates. Conda packages, which include the main control room tools, were not updated.
I've retuned a Simulines configuration for testing on Tuesday morning. The frequency vector is the same as we nominally use but I reduced all injection amplitudes by 50% across the board. If we're able to run Simulines without losing lock while in the new DARM state in the morning while, I'll need another round with Simulines at some point to determine the best injection strengths moving forward. The new injection configs that I tuned were placed in /ligo/groups/cal/src/simulines/simulines/FreqAmp_H1_newDARMconfig_20231218/. I then sourced the cal pydarm environment withsource /ligo/groups/cal/local/bin/activateand ran the vector optimization script at /ligo/groups/cal/src/simulines/simulines/amplitudeVectorOptimiser.py after adjusting the input and output directories for H1 on lines 33 & 34 (variables changed areinDirandoutDir) to:inDir = 'FreqAmp_H1_newDARMconfig_20231218' outDir = 'FreqAmp_H1_simuLines_newDARMconfig_20231218'This placed new "optimized" frequency vector files in/ligo/groups/cal/src/simulines/simulines/FreqAmp_H1_simuLines_newDARMconfig_20231218/. Lastly, to actually generate the config file that Simulines processes when it's run, while still in the cal virtual environment, I ranpython simuLines_configparser.py --ifo H1, after changing the H1 output filename insimuLines_configparser.pytooutputFilename = outDir+'settings_h1_newDARMconfig_20231218.ini'. This returned the following output:(local) louis.dartez@cdsws22: python simuLines_configparser.py --IFO H1 Total time = 1252.0 Average time taken to sweep individual frequency: 26.083333333333332 Separations in TIME for each sweep: [233.0, 260.0, 253.0, 253.0] Starting Frequencies for each swept sine: [5.6, 7.97, 14.44, 64.78, 339.4] Starting points in relative time for each swept sine: [0, 233.0, 493.0, 746.0, 999.0]I then commented out my temporary variable name changes in the simulines scripts. At the end of the exercise, the simulines file to test in the new DARM loop configuration is/ligo/groups/cal/src/simulines/simulines/settings_h1_newDARMconfig_20231218.ini. The command to execute simulines using this configuration isgpstime;python /ligo/groups/cal/src/simulines/simulines/simuLines.py -i /ligo/groups/cal/src/simulines/simulines/settings_h1_newDARMconfig_20231218.ini;gpstime. This is same as the instructions in the Operator's wiki, with the modification for the new ini file.
The script I used to adjust the injection amplitudes can be found at /ligo/groups/cal/src/common/scripts/adjust_amp_simulines.py.
the script mentioned above now lives at /ligo/groups/cal/common/scripts/adjust_amp_simulines.py
TITLE: 12/19 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:
- Camilla ran a quick MICH FF test before going into observing, which turned out to be beneficial, so I am accepting these SDFs in the observe and safe snap files, her alog here
- Uneventful night otherwise, nothing else to note
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 00:40 | ISC | Camilla | CR | N | Test new MICH FF | 00:55 |
Following the sudden lockloss from earlier, H1 made it back to observing at 0:55 UTC. The IFO looks stable and seismic activity is low.
As Ryan did last week 74741. I have taken PEM_MAG_INJ and SUS_CHARGE from WAITING to DOWN so that they do not run tomorrow. Instead, tomorrow Louis will try the risky DARM loop swaps and calibration. To re-enable the automated measurements, the nodes should be requested to INJECTIONS_COMPLETE before next Tuesday.
I've requested both the SUS_CHARGE and PEM_MAG_INJ nodes to INJECTIONS_COMPLETE so that the automated injections will run again starting next Tuesday.
Lockloss during commissioning during simulines calibration. Investigation ongoing since we have lost lock 3 times during simulines in the last week (with one successful run) - currently unsure if simulines is a culprit.
I looked into this. The lockloss was likely not caused by Simulines injections. I ran simulines about two minutes before we lost lock but simulines exited almost immediately due to an error in one of its configuration files and I confirmed that it did not start any injections before it exited.
Louis, Sheila, Jenne
Today we tried a new DARM loop configuration. We are not implementing this for observing right now, we would need to retune feedforward and redo calibration before we could do that.
There is a new guardian state in ISC_LOCK, called NLN_ETMY. This state turns off feedforward and transitions DARM control to ETMY (L3,L2, L1), it can be used when we are in nominal low noise (but not necessarily from any states before LOWNOISE_ESD_ETMX). This was tested Tuesday morning and today, so it has worked twice.
We set up a new configuration of filters for ETMX. Louis has been using pyDARM to model the DARM crossovers, and we will add more details about these new filters in a later alog. The first attached screenshot shows the configuration that we first transitioned to, without the new boost. This keeps the overall loop similar to our earlier configuration, the main differences are:
Measurements:
The last attachment shows spectra of the DAC counts for ETMX, and DCPD SUM. The new configuration with the boost on reduces the RMS counts on the ESD by a little more than a factor of 2 compared to the configuration we've been using (most of the improvement is from the additional boost). The new configuration also supresses DCPD sum more below 10Hz.
Camilla looked at the MICH FF with the new configuration and saw that it needs to be retuned, she took measurements that can be used to do that.
We lost lock as I was trying to semi-manually transition DARM control back to ETMY from this new configuration.
Here's the DARM1 and DARM2 banks that were used for the test. DARM1 FM1 is off
Gabriele's boost design string is
zpk([-29.9198+i40.0369;-29.9198-i40.0369],[-18.297+i23.0198;-18.297-i23.0198],1)
It's plotted in boost_filter.png in blue. The red trace is the 4.5kHz boost we want to replace. The new filter is meant to be less aggressive in magnitude and cost less in phase.
Here's a quick plot of the DARM loop model with today's test changes in place. Zoomed out. Zoomed in. The OLG measurements Sheila took match quite well. The PUM measurements still don't. And they suggest we're over estimated our phase margin at the PUM stage in the model. Discussion to follow.
I'm hoping that it was just incidental and won't come back, but I would like for us to watch for increases in the bounce and roll modes (and maybe other peaks). They seemed a bit higher on the DARM FOM yesterday, and they also look a bit elevated in the plots in this alog thread (but that was all from the same time, so just a confirmation of what I thought I was seeing on the wall). If this new configuration (which gives us so many improvements) exacerbates those modes, we may have to remember to turn on some active damping for them.
I've set up a new Guardian state inISC_LOCKcalled DARM_RECOVER. This state is meant to be run from Sheila'sNLN_ETMYto bring the IFO back into the nominal DARM loop configuration. I will use test it after tomorrow morning's set of high-risk tests just before maintenance day.