Wed Nov 06 10:04:05 2024 INFO: Fill completed in 4min 2secs
Travis confirmed a good fill curbside.
WP 12139
Entry for work done on 11/5/2024
Two 3T seismometers were installed in the LVEA Biergarten next to the PEM area. Signals are routed through the SUS-R3 PEM patch panel into the CER. Signals are connected to PEM AA chassis 4 and 5.
F. Clara, J. Warner
There are 2 of these plugged in, they are 3 axis seismometers, serial numbers T3611670 and T3611672. The first one is plugged into ports 4,5 & 6 on the PEM patch panel, the second is plugged into ports 7,8 & 9. In the CER, T3611670 is plugged into ports 21,22 & 23 on PEM ADC5 and T3611672 is plugged into ports 27,28 & 29 on PEM ADC4. In the DAQ, these channels are H1:PEM-CS_ADC_5_{20,21,22}_2K_OUT_DQ and H1:PEM-CS_ADC_4_{26,27,28}_2K_OUT_DQ. So far the seismometers look like they are giving pretty good data, similar to the STS and the old PEM Guralp in the biergarten. The seismometers are oriented so that the "north" marking on the their carry handles is pointed down the X-arm, as best as I could with eyeballing it.
I need to figure out the calibrations, but it looks like there is almost exactly -15db difference between these new sensors and the old PEM Guralp, but maybe the signal chain isn't exatly the same.
Attached images compare the 3T's to the ITMY STS and the existing PEM Guralp in the biergarten. First image compares asds for each seismometer. Shapes are pretty similar below 40 hz, but above that they all have very different responses. I don't know what the PEM guralp is calibrated to, if anything, it looks ~10x lower than the STS (which calibrated to nm/s). The 3T's are about 5x lower than the PEM sensor, so ~50x lower than the STS.
Second image are tfs for the X,Y & Z dofs between the 3T's and the STS. These are just passive tfs between the STS and 3T's to see if the have similar response to ground motion These are generally pretty flat between .1 and 10hz. The X & Y dofs seem pretty consistent, the Z tfs are different starting around 10hz. I should go and check that the feet are locked and have similar extension.
Third image are tfs between the 3T's and the exist PEM Guralp. Pretty similar to the tfs with the STS, horizontal dofs all look very similar, flat between .1 and 10hz, but the ADC4 sensor has a different vertical response.
I'll look at noise floors next.
The noise for these seems almost comparable to T240s, above 100 mhz, less certain about noise below 100mhz, these don't have thermal enclosures like the other ground sensors. Using mccs2 in matlab to remove all the coherent noise with the STS and PEM Guralp, the residual noise is pretty close to the T240 spec noise in SEI_sensor_noise. Attached plots are the asds and residuals after finding a scale factor that matches the 3T asds to the calibrated ITMY STS asds. Solid lines are the 3T asd, dashed lines are the residuals after coherent subtraction.
For convenience I've attachd the response of the T240 and the STS-2 from the manuals.
These instruments both have a steep fall-off above 50-60 Hz.
This is not compensated in the realtime filters, as it would just add lots of noise at high frequency, and then we'd have to roll it off again so it doesn't add lots of nasty noise.
T240 user guide - pg 45
https://dcc.ligo.org/LIGO-E1500379
The T240 response is pretty flat up to 10 Hz, has a peak at ~ 50 Hz, then falls off rapidly.
STS-2 manual - pg 7
https://dcc.ligo.org/LIGO-E2300142
Likewise the STS-2 response is pretty flat up to 10 Hz, then there is ripple, and a steep falloff above 60 Hz
WP 12189
Entry for work done on 11/5/2024
Cabling and electronics were installed for a huddle test of HS-1 seismometers. A BNC cable was pulled from the Biergarten to the SUS-R3 PEM patch panel. A L4C Interface chassis S1600256 was installed in TCS-C1 rack, U16. Signal will be routed to a PEM AA chassis. Chassis is powered off. Sensor will be connected next Tuesday.
TITLE: 11/06 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 6mph Gusts, 4mph 3min avg
Primary useism: 0.16 μm/s
Secondary useism: 0.18 μm/s
QUICK SUMMARY:
H1 had a 6.5hr lock overnight (after waking up Ibrahim), and just went down about 30min ago due to a M5.9 EQ in the Solomon Islands (switching Observatory Mode to EARTHQUAKE). ISC LOCK is currently running an automatic alignment.
Microseism continues it's trend downward. Winds are negligible.
As noted by Jason and Ryan S yesterday 81073, the power out of the PSL is again not stable. Fluctuating from 1.2 to 2.7W. There's not a known solution for this so we will sit with ISC_LOCK in IDLE until this stabilizes.
The IMC is saying it's locked but the power out if wildly fluctuating from 2W to 50W so we took it to OFFLINE. Plot.
At 17:10UTC this NPRO behavior calmed down and Jason locked the ISS and FSS. I took LASER_PWR to POWER_2W then IMC_LOCK to LOCKED then started the main locking process.
Attached plot showing comparison to Jason/Oli's plot from 81099. You can see:
IFO is in NLN and OBSERVING as of 08:44 UTC
IFO was in MOVE_SPOTS when I was called, which I assume was due to the timer going over 2 hrs. The stall at this state was due to ADS Y signals not converging. While investigating why this may be, they reached the threshold necessary for the next step. I babysat Guardian until it reached OBSERVING. Essentially, I did nothing.
TITLE: 11/06 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: Currently trying to relock and at ENGAGE_DRMI_ASC after finishing an initial alignment. It's been a rough shift, with issues relocking.
Trouble with locking:
Good news is that we haven't had the new AS AIR PD(81080), PEM-CS_ADC_5_19_2K_OUT_DQ, saturate during the last three locklosses! The highest number of counts we've gotten have been around 16k counts. I have also put in a feature request for the integral of the channel's peak at lockloss to be calculated and put onto the lockloss pages, and assigned it to me (Issue#223). (ISC)
LOG:
00:00UTC In DRMI and trying to relock after a 28 minute lock
00:44 NOMINAL_LOW_NOISE
00:50 Observing
00:58 Lockloss
- 01:21 Lockloss from CARM_5_PICOMETERS, and FSS wouldn't let the IMC lock?
- Started getting giant FSS oscillations while it was trying to lock/was locked(attachment2)
- Went to DOWN, went to IDLE, tried toggling FSS autolock, nothing worked
- Put detector in DOWN for 10mins and then tried again
- Seemed fine??
- DRMI locklosses during ENGAGE_DRMI_ASC x 3
- BS saturations and beams on AS AIR visibly start shaking from end of PREP until DRMI unlocks
- 02:12 Started an initial alignment
- At MICH, started having issues with it not being able to get past MICH_BRIGHT_ALIGN (attachment4)
- PSL table flashing all over the place
- Took ifo to DOWN, waited a while, then tried again with manual IA and no issues
- 02:37 Initial alignment done, relocking
03:43 NOMINAL_LOW_NOISE
03:45 Observing
- 03:57 Out of Observing to adjust OPO temperature
04:08 Back into Observing
04:28 Lockloss
- Took 15 minutes for IMC to relock
- Started relocking with H1 Manager
- ENGAGE_DRMI_ASC had the beam spot shaking and BS saturations again, sometimes leading to LL x 4
- I started an IA
- IA done, relocking
Oli's problems last night were due to the issues with apparent NPRO mode hopping noted here. I've attached 2 trends, one showing the 20 minute range with t0 at 01:36 UTC as shown in Oli's first 3 plots, the other over the 5 hour range from when the Oli first noted the issue until 06:06 UTC. As can be seen, the ISS starts oscillating when PMC Refl begins increasing, which in turn causes PMC Trans to decrease. The ISS is doing what it's supposed to, as it interprets the drop in PMC Trans as a call for less diffraction. Once the ISS runs out of power available in the bank (because PMC Refl has increased so much the ISS has drained the bank to try to follow) it unlocks and relocks; upone relock it still sees PMC Trans being low so drives the AOM diffraction down until it again runs out of range and unlocks. The cycle repeats until the mode competition stops and PMC Refl returns to its normal ~17.5W. This happened 3 times in 5 hours last night, and was happening this morning when I arrived onsite.
Lockloss @ 11/06 04:28 UTC after a 45 minute lock. Looks to be related to the ISS AOM driver again - at least I can't find anything that saw it before that (ndscope, bottom left)
Currently Observing at 159Mpc and we have been locked for 30 minutes.
Relocking from the 00:58UTC lockloss took almost three hours due to issues with the PSL/IMC refusing to lock for many minutes at a time or the FSS locking but glitching/oscillating a ton, as well as issues with MICH not being well aligned. Our range was also very low so once we were relocked, I popped out of Observing to adjust the OPO temperature, which is something that Vicky told me I'd probably have to do. Just got done doing that and the range is looking MUCH better - we've gone from ~136 to almost 160Mpc since adjusting the temp, and DARM is looking much better in the higher frequencies.
This lockloss seems to have the AOM driver monitor glitching right before the lockloss(ndscope), similar to what I had noticed in the two toher locklosses from this past weekend(81037).
03:45UTC Observing
In 81073 (Tuesday 05 November) Jason/Ryan S gave the ISS more range to stop it running out of range and going unstable. But on Tuesday evening, Oli saw this type of lockloss with again 81089. Not sure if we've seen it since then.
The channel to check in the ~second before the lockloss is H1:PSL-ISS_AOM_DRIVER_MON_OUT_DQ, I added this to the lockloss trends shown by the 'lockloss' command line tool via /opt/rtcds/userapps/release/sys/h1/templates/locklossplots/psl_fss_imc.yaml
I checked all the "IMC" tagged locklosses since Wednesday 6th and didn't see any more of these.
This happened before we fixed the NPRO mode hopping problem, which we did on Wednesday, Nov 6th. Not seeing any more of these locklosses since lends credence to our suspicion that the ISS was not responsible for these locklosses, the NPRO mode hopping was (NPRO mode hops cause the power output from the PMC to drop, the ISS sees this drop and does its job by lowering the diffracted power %, once the diffracted power % hits 0 the ISS unlocks and/or goes unstable).
Vicky, Camilla
I've updated instructions in the SQZ wiki: https://cdswiki.ligo-wa.caltech.edu/wiki/Troubleshooting%20SQZ
If the squeezing runs away, you should now be able to take SQZ_MANAGER to '''RESET_SQZ_ASC_FDS''' (turns off and clear the ASC and turns off the angle servo, will also reset the SQZ angle to 190 deg if it's outside 100-250 range) and the request FREQ_DEP_SQZ (will got through SQZ_ASC_FDS) to turn back on SQZ ASC and angle control.
This is to avoid issues as in 81027. We still might need to think about what should happen to the FC ASC.
I've roughly copied the LLO configuration for the AS power monitor (that won't saturate after lock losses) and installed an additional photodiode in the AS_AIR camera enclosure. PD output goes to H1:PEM-CS_ADC_5_19_2K_OUT_DQ for now.
GiGE used to receive ~40ppm of the power coming into HAM6. I replaced the steering mirror in front of GigE with 90:10, the camera now receives ~36ppm and the beam going to the photodiode is ~4ppm. But I installed ND1.0 in front of the PD, so the actual power on PD is ~0.4ppm.
See the attached cartoon (1st attachment) and the picture (2nd attachment).
Details:
This is the first look of the lock loss from 60W. At least the channel didn't saturate but we might need more headroom (it should rail at 32k counts).
The peak power is in this example is something like 670W. (I cannot figure out for now which AA filter and maybe decimation filter are in place for this channel, these things might be impacting the shape of the peak.)
Operators, please check if this channel rails after locklosses. If it does I have to change the ND filter.
Also, it would be nice if the lock loss tool automatically triggers a script to integrate the lock loss peak (which is yet to be written).
Tagging Opsinfo
Also checking out the channel during the last 3 high power locklosses this morning (NLN, OMC_WHITENING, and MOVE_SPOTS). For the NLN lockloss, it peaked at ~16.3k cts 80ms after the IMC lost lock. Dropping from OMC_WHITENING only saw ~11.5k cts 100ms after ASC lost it. Dropping from MOVE_SPOTS saw a much higher reading (at the railing value?) of ~33k cts also ~100 ms after ASC and IMC lost lock.
Camilla taking us down at 10W earlier this morning did not rail the new channel, it saw about ~21k cts.
As for the filtering associated with the 2k DAQ, PEM seems to have a standard ISC AA but the most impactful filter is a 8x decimation filter (16k -> 2k). Erik told me that the same 8x filter is implemented as src/include/drv/daqLib.c: line 187 (bi-quad form) line 280 (not bi-quad) and one is mathematically transformed to the other and vice versa.
In the attached, it takes ~1.3ms for the step response of the decimation filter to reach its first unity point, which is not really great but OK for what we're observing as the lock loss peaks seem to be ~10ms FWHM. For now I say that it's not unreasonable to use this channel as is.
I added ND0.6, which will buy us about a factor of 4.
(I'd have used ND2.0 instead of ND1.0 plus ND0.6, but it turns out that Thorlabs ND2.0 is more transparent at 1um relative to 532nm than ND1.0 and ND0.6 are. Looking at their data, ND2.0 seems to transmit ~4 or 5%. ND 1.0 and ND 0.6 are closer to nominal optical density at 1um.)
New calibration for H1:PEM-CS_ASC_5_19_2K_OUT_DQ using ASC-AS_C_NSUM_OUT (after ND was increased to 1.0+0.4) is ~0.708/4.00~0.177W/counts.
Daniel, Camilla, Vicky
After PSL laser frequency change 81073, we adjusted the SQZ + ALS X + ALS Y laser crystal temperatures again, as before in lho80922.
Again the steps we took:
Squeezer recovery at NLN was complicated after this temp change, but after fixing laser mode hopping and updating OPO temperature, SQZ is working again at NLN.
OPO TEC temperature adjusted from 31.468 C --> 31.290 C, to maximize seed NLG. Change is consistent with last time lho80926 (though not obvious that laser freq changes mean the red/green opo co-resonance should change, but ok).
Squeezer laser was likely modehopping at the new settings above. Symptoms:
- Screenshot of TTFSS beatnote before / after / afterAfter the PSL laser freq change.
- PMC_TRANS was 430 (nominal H1:SQZ-PMC_TRANS_DC_POWERMON~675)
- PMC_REFL was 370 (nominal H1:SQZ-PMC_REFL_DC_POWERMON~44)
- Bad PMC transmission meant not enough SHG power, and while OPO could lock, the OPO PUMP ISS could not reach the correct pump power. So aside from mode hopping being bad, it would have meant bad squeezing.
Camilla and I went to the floor. SQZ Laser controller changes made to avoid mode hopping:
This adjustment seemed quite sensitive to laser current and crystal temperature, with a tunable range kinda less than I expected.
Compared to previous SQZ laser adjustments to avoid mode-hopping (see old laser 51238, 51135, 49500, 49327), this temperature adjustment seems small for the current (amps) adjustment.. but, seemed to work.
PMC cavity scans look clean at these new settings (laser looks single-mode), see screenshot. For +/- 200 MHz steps of crystal frequency, still see a 160 MHz beatnote. Later could double-check a PMC cavity scan at +200 MHz just to be sure. PMC cavity scan template added to medm SQZT0 screen (! PMC) and at $(userapps)/sqz/h1/Templates/ndscope/pmc_cavity_scan.yaml.