H1 has been locked and in observing for just over 4 hours. All systems appear stable, wind has died down, just had a 5.6 EQ from Mariana Islands but otherwise all appears well.
TITLE: 07/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 136Mpc
SHIFT SUMMARY:
- Temperature at EY has risen by ~0.5F over the past 6 hours - Bubba contacted
- 2:09 Received warning for FMC-CS_WS_RO_ALARM & called Bubba - will be looked at tomorrow
23:07 Detector trying to relock and failing
23:35 I put the detector into Down for 20 minutes to wait for the wind to die down
23:56 Trying to lock again
- 00:27:46 Eventually got to CARM_150_PICOMETERS before losing lock
- 00:47:21 Lockloss at ENGAGE_ASC_FOR_FULL_IFO
1:00 Starting Initial Alignment (accidentally started with H1_MANAGER off, so I left it off for this)
- Was able to get through to SRC_ALIGN_OFFLOADED!!, but INIT_ALIGN was stuck at ALIGNING_INPUT (71669).
- Reenabled H1_MANAGER (although I had issues with getting it to recapture ISC_LOCK) and let it run alignment (on its terms) and everything went smoothly
3:31 Reached OMC_WHITENING
3:40 Arrived at NOMINAL_LOW_NOISE, needed to wait for ASC-ADSs to lower
3:55 Finally back into Observing!
5:40:50 Lost lock immediately after being pushed out of Observing (71673)
6:38 Entered OMC_WHITENING
6:52 Entered NOMINAL_LOW_NOISE
6:56 Back to Observing
LOG:
no log
TITLE: 07/25 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 143Mpc
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 17mph Gusts, 11mph 5min avg
Primary useism: 0.06 μm/s
Secondary useism: 0.05 μm/s
QUICK SUMMARY:
- IFO has just been locked and in observing upon arrival
- CDS/SEI/DMs ok
Lockloss at 5:40:50 - Detector taken out of Observing by SDF Diff (see below for systems) and pushed into Commissioning, and then we lost lock.
Here are the alerts :
ISC_LOCK
2023-07-25_05:40:49.216400Z ISC_LOCK [NOMINAL_LOW_NOISE.run] USERMSG 0: SQZ_MANAGER: has notification
2023-07-25_05:40:49.592132Z ISC_LOCK [NOMINAL_LOW_NOISE.run] Unstalling IMC_LOCK
2023-07-25_05:40:49.948367Z ISC_LOCK JUMP target: LOCKLOSS
2023-07-25_05:40:49.948367Z ISC_LOCK [NOMINAL_LOW_NOISE.exit]
2023-07-25_05:40:49.948367Z ISC_LOCK STALLED
2023-07-25_05:40:50.012243Z ISC_LOCK JUMP: NOMINAL_LOW_NOISE->LOCKLOSS
Back in Observing as of 3:55!
We were only in OMC_WHITENING for 9 minutes! Here's what the violins looked like when we got to OMC_WHITENING: Attachment1, attachment2, attachment3.
ACCEPTED SDF Diffs for H1:ASC-DHARD_Y_TRAMP and H1:ASC-DHARD_P_TRAMP (see 71655).
Tomorrow during maintenance I plan to epoxy 2 L4Cs under HAM1 to try use for feedforward, with the hope I can get some improvement to the ~10hz motion of the HAM1 table. Each of the 4 pier modules on each HEPI has 1 extra L4C input called "witness", with the appriate pre-amp and wiring to send signal to the AA chassis and into the front end. Additionally, all of the HEPI models already have filter models, sensing matrices and INF filter banks that were intended for a 3d L4C array, but we never ended up installing the sensors. So, in order to attach the spare L4Cs I already have in hand, all I have to do is connect the right ADC channels in the model to the inputs to the vestigial 3d L4C path, and plug a couple of L4Cs into the pier modules. After that I will just need a couple of short measurements to design a feedforward filter.
The only hiccup to this is that the adc channels I need in the model are currently attached to a path called "WITNESS" in the model. But the adc channels go into the master part, go into this block called "WITNESS" with 4 filter banks, then go out of the master part and get terminated. GNDN. I can't even find a link to the witness path on the HEPI overview, but all of the 3d l4c stuff is there. I plan to just clip the adc wires going in to witness and reattach them to the 3dl4c path. First 3 attached screen shots show where the adc channels go in the master part on the witness ports, then come out to get terminated, the witness block and hidden filter modules, and the already existing 3d l4c inputs. Last image shows the medm representation of the 3d l4c path on the HAM1 HEPI overview, circled in red crayon, which follows the model infrastructure exactly.
The 4 adc channels currently attached to the witness path are 2, 7, 12 and 17. I think 2 is on the +x/+y pier (call it pier 1) , 7 is on -x/+y (pier 2) , 12 is -x/-y (pier 3) and 17 +x/-y (pier 4). I plan to put the L4Cs near pier 2 and pier 4, so channels 7 and 17, which I will connect to the 3DL4CA_Z and 3DL4CB_Z paths in the model. I still need to probe out the adc connections, before making the model changes.
ECR for this is https://dcc.ligo.org/LIGO-E2300202. No new channels, no daq restart needed.
Lockloss at 18:07 from NLN - possibly due to high wind affecting EY.
Tried relocking for ~40 minutes, but ALSY won't catch (probably from wind - see TJ's alog and current wind). I put the detector into DOWN at 23:35 and at 23:56 am now trying to lock again.
Wind has reduced a lot.
Still working through getting the detector back up - we got close a couple of times (ENGAGE_ASC_FOR_FULL_IFO), but kept losing lock.
I decided to run an Initial alignment, which was able to go all the way through to SRC_ALIGN_OFFLOADED with no issues, but the INIT_ALIGN manager had gotten stuck at ALIGNING_INPUT and I could not move it through to finish. I put ISC_LOCK to DOWN and tried locking like normal, but it lost lock and so H1_MANAGER decided to run another initial alignment, which went well for ALIGN_IFO and INIT_ALIGN, and now we are in the process of locking again.
We're getting a lot of Verbal saturation warnings for SR/PR/MC's while trying to lock instead of just the usual ITMs/ETMs.
After our first lock loss this morning I loaded I loaded in changes I made last week.
Initial Alignment got through to SRC_ALIGN_OFFLOADED!!
SRY locking trigger thresholds change worked for this afternoon's initial alignment. See attached trend.
TITLE: 07/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 29mph Gusts, 23mph 5min avg
Primary useism: 0.07 μm/s
Secondary useism: 0.07 μm/s
QUICK SUMMARY:
Detector is working on locking. Wind is pretty high.
TITLE: 07/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 150Mpc
SHIFT SUMMARY:
High winds forecast from 18utc July24 to 6utc Jul25 (or 11am-11pm today).
Had (3) locklosses today which made for a rough go, and L1 was down for logging today.
LOG:
There was also an SDF Diff for SUS_PROC. This is related to 2nd harmonic (~1500Hz) violin mode work by Rahul. Attached is the SDF Diff which was ACCEPTED by Rahul.
I also accepted the SDF .Observe for the newly created monitor filter for 1.5kHz violin modes.
Today I noticed a large, sustained step in the 100-300mHz band at EY. I didn't know of work that could affect this band that was going on down there, and at first I didn't notice any environmental changes around that time. I asked Jim and he suggested that the sensor correction is the most sensitive from the wind around that band so I looked further into it. Even though the magnitude of the wind didn't increase appreciably at that time, the direction changed. On my plot the average wind speed at bubble 1 is roughly the same at 2 and 3, but the ground motion is higher after the direction change. I'd guess that this direction change just started getting around our wind fence and perhaps started hitting it head-on and this is the fence coupling. Either way, while obvious in concept, it's nice to see in real life as well.
We have evidence that DARM noise in the 10-50 Hz region seems to be modulated by residual motion of DHARD_Y at 2.6 Hz.
At the same time, now DHARD_Y is not limiting DARM by linear coupling above 10 Hz, due to the improved A2L (-30 db)
So there's room to test a new DHARD_Y controller that could give us 2x more suppression at 2.6 Hz, at the price of about 2x more noise above 10 Hz.
Attached a proposed new controller that could be tested to see if DARM stationarity improves.
I uploaded the new controller in FM1 as a multiplicative change to the previous controller. So one can turn on this 2.6 Hz boost by engaging FM1 (5 second ramp) together with the other FMs while locked.
Tested on and off a couple of times while in NLN, the transition is smooth. The effect is shown in the attached spectrum:
All in all the DHARD_Y RMS is reduced to 0.64 times the value with the old controller.
I left this controller active in Observing and Cory accepted the SDF. We should take a look at DARM in the next few hours to see if 1) there is less non-stationary noise in the 20-50 Hz region 2) the higher DHARD_Y control noise doesn't affect us.
I have updated ISC_LOCK to engage this controller when in LOWNOISE_ASC. I did not reload ISC_LOCK code
Attached is the FM1 diff from Gabriele's change this afternoon.
Gabriele has also made a change for this FM1 change in ISC_LOCK, line 4490. Next time H1 is OUT of OBSERVING, please hit LOAD on ISC_LOCK.
ISC_LOCK has been re-loaded.
The new controller improved a lot the DHARD_Y motion at 1.3 Hz and 2.6 Hz, reducing the RMS.
It looks like CHARD_Y coudl use a similar improvement, since it still has a large 1 Hz and 2.6 Hz peaks. The 1 Hz peak is due to gain peaking in the current design, and the loop has little gain at 2.6 Hz
DARM still shows bicoherence with CHARD_Y
Lockloss at 00:41. Detector moved itself out of Observing and into Commissioning, and then immediately lost lock. Cause not yet known.
Got back to NOMINAL_LOW_NOISE at 2:25, and went back into Observing at 2:26. Damping violins for only 50 minutes!!!!
I wasn't able to find anything about what could have caused this lock loss, but I did notice something that I found interesting, although there might be a very obvious explanation for it.
Looking at the H1:ASC-AS_A_DC_NSUM_OUT16 and H1:SQZ-LO_SERVO_ERR_OUT_DQ channels (Attachment 1), the H1:SQZ-LO_SERVO_ERR_OUT_DQ channel first sees the lockloss only 22ms after it starts. This is still during the initial small change in slope of ASC-AS_A_DC that leads into the large slope and subsequent dropoff.
I've also attached the ISC_LOCK and SQZ_MANAGER logs from when the lockloss occurred. The last four locklosses that we have had (possibly more before that), SQZ_MANAGER has given us the 'SQZ ASC AS42 not on?? Please RESET_SQZ_ASC' message ~0.1 seconds after the lockloss started. At least looking at the ASC and LSC channels that come up in 'lockloss select', none seem to respond to a lock loss that quickly.
Nice alog Oli. If you use the H1:ASC-AS_A_DC_NSUM_OUT_DQ channel (sampled at 2kHz rather than 16Hz) and zoom in on the y-axis you can see that the AS_A channel actually sees a discreet bump and runs away before the SQZ_LO_SERVO shows it's glitch, see attached. The SQZ-LO is locked to the OMC TRANS 3MHz sideband so it probably a sign that the light at the OMC has changed.
The messages in the log about SQZ AS42 is because we have already lost lock but ISC_LOCK hasn't yet realized: see the first "refined time" plot in the lock loss tool where the ISC_LOCK_STATE_N shows the lockloss happened nearly 1 second after H1:ASC-AS_A_DC_NSUM_OUT_DQ sees it happen, is this longer than usual? We expect the SQZ AS42 not to be able to be on when we are unlocked, tagging SQZ.
Daniel, Camilla
Following on from Ryan's 71420, we looked at the movement of ITMY (only has ASC control) during DHARD_WFS and see there is now larger spikes during locking than before 29th, see attached plot where H1:SUS-ITMY_L2_OSEMINF_{U,L}{L,R}_INMON spikes during locking are larger after the t-cursor (see bottom purple plot for when violins rang up).
Zooming in, attached plot, we can see when the Input to the DHARD_Y filter is turned on that the output to DHARD_Y filter has a large transient lasting ~1second, not seen in DHARD_Y inmon. The only filter that is on during that time FM6 and although it has a larger step response, there is a 5s TRAMP so Daniel doesn't see an issue with this.
Nothing obvious happened to DHARD on the 29th, only alog around that date shows a change that was reverted 70928.
During relock on Tuesday we plan to increase this DHARD_Y ramp time (currently 5s) to see if we can reduce this transient that appears to have increased by a factor of 10 around June 29th. See attached plots of relock before and after the violins rang up.
There were some SDF Diffs (71670) for H1:ASC-DHARD_Y_TRAMP and H1:ASC-DHARD_P_TRAMP that I accepted, which bumped the ramp times up to 15 seconds.
