1381774439

We had that same feature in LSC-DARM that we've been seeing recently ~100ms before lockloss.

TJ popped us out of observing at 17:51UTC so we could touch up the OPO temperature (sdf attached) as SQZ and range had been reducing. We had to do this yesterday too 73549 but expect we'll need to do this less over time as the OPO crystal settles after Tuesday's 73524 spot change.

Thu Oct 19 10:10:24 2023 INFO: Fill completed in 10min 20secs

This is the first fill using the new zotvac0 workstation (newer hardware, Deb11 compared with Deb10).

Richard verified a good fill via camera.

Last night at GPS=1381715243 (Wed Oct 18 18:47:05 2023 PDT) h1oaf and h1calcs long-range-dolphin receivers from h1susetmx reported a single receive error (1 errror in the 16384 packets received in that second). This is the first error of this type this year. There was no issue with h1susetmx at the time, its cpu max is 20uS. I'll clear the errors on the next TOO.

With an error rate of 1 per year, (1 in 5.2e+11) we will just continue to monitor this for now.

TITLE: 10/19 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 162Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 4mph 5min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.51 μm/s
QUICK SUMMARY: IFO has been locked for 16 hours. While the primary useism is starting to trend down, the secondary is coming up. Peakmon seems to be much calmer now, so I'll bring the SEI_ENV threshold back when (if) we go out of Observing today (alog73569).

TITLE: 10/19 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 162Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY: We are Observing and have now been Locked for 8.5 hours. Everything has been going smoothly and we even reached 162Mpc according to H1:CDS-SENSMON_CLEAN_SNSC_EFFECTIVE_RANGE_MPC (the lower one on sensemon) !!
LOG:

23:00UTC Detector in Commissioning and has been Locked for 21 mins
23:08 Went into Observing

Observing at 160Mpc and have been Locked for 4.5hours now. Primary and secondary seism are still slowly rising.

Naoki, Sheila, Camilla, Vicky

Summary: After yesterday's crystal move LHO:73535, we re-aligned SQZT7, and now see 8 dB SQZ on the homodyne, up to measured NLG=114 without a phase noise turnaround! This fully resolves the homodyne loss budget, there is 0 mystery loss remaining on the homodyne, from which we can infer 0 mystery losses in HAM7. Back to the IFO afterwards, after 1 day at this new crystal spot, squeezing in DARM is about 4.5dB - 4.8dB, reaching almost 5dB at the start of lock.

We first re-aligned the homodyne to the IFO SQZ alignment, which reached 4.8dB SQZ in DARM yesterday, so we are more confident the alignment back through the VOPO is not clipping. In yesterday's measurements, we had a sign error in the FC-ASC offloading script, which brought us to a bad alignment with limited homodyne squeezing, despite high 98% fringe visibilities. Attached is a screenshot of homodyne FC/ZM slider values with FC+SQZ ASC's fully offloaded (correctly), to which the on-table SQZT7 homodyne is now well-aligned. After Sheila re-aligned the homodyne to the screenshotted FC/ZM values, fringe visibilities are PD1 = 98.5% (loss 3.1%), PD2 = 97.8% (loss 4.2%).

We then did an NLG sweep on the homodyne, from NLG=2.4 (opo trans 20uW) to NLG=114 (opo trans 120uW). Measurements below and attached as .txt, DTT is attached, plots to follow.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

unamplified_ir = 0.0014 (H1:SQZ-OPO_IR_PD_LF_OUT_DQ with pump shuttered)

NLG = amplified / unamplified_ir (opo green pump un-shuttered)

@80uW pump trans, amplified = 0.0198 (at start, 0.0196 at end) --> NLG 0.0198/0.0014 ~ 14

@100uW pump trans, amplified = 0.0046 (at start, 0.0458 at end) --> NLG 0.046/0.0014 = 33

@120uW pump trans, amplified = 0.16 --> NLG 0.16/0.0014 = 114

@60uW pump trans, amplified = 0.011 (at start, 0.0107 at end) --> NLG = 7.86

@40uW pump trans, amplified = 0.0059 (at start, 0.0059 at end) --> NLG = 4.2

@20uW pump trans, amplified = 0.0034 (at start, --- at end) --> NLG = 2.4

All measurements had PUMP_ISS engaged throughout; we manually tuned the ISS setpoint for different NLGs. For low NLG (20uW trans) we manually engaged ISS. LO power (shot noise) drifted ~5% over the measurement, see trends.

NLG Sweep Procedure:

• Measure LO shot noise
• Measure amplified seed power, adjust OPO TEC temp to maximize NLG for given OPO GREEN TRANS
• Toggle Seed/CLF Back to CLF on SQZT0, shutter CLF, open HD path (Toggle HD Path Green). Measure Mean SQZ with CLF shuttered
• Unshutter CLF, "LOCKED CLF", "LOCKED HD". Set SQZ demod angle to minimize AHF on HD (aka minimize H1:SQZ-ADF_HD_DIFF_IQSUM_RMS). Measure SQZ dB
• Rotate H1:SQZ-CLF_REFL_RF6_PHASE_PHASEDEG to maximize ADF from same H1:SQZ-ADF_HD_DIFF_IQSUM_RMS. Measure ASQZ dB
• Toggle HD back to OPO IR PD (red). Toggle CLF --> Seed. Re-measure NLG. Change OPO GREEN TRANS by changing PUMP_ISS setpoint & repeat.

DTT saved in $(userapps)/sqz/h1/Templates/dtt/HD_SQZ/HD_SQZ_8dB_101823_NLGsweep.xml

The CDS Vacuum workstation zotvac0 got it's upgrade today.
The workstation hardware was changed from an older model ZOTAC to the newer model ZOTAC ZBOX-EN1060k which is capable of running Debian 11.
The hardware install and Debian11 imaging went fine, and the workstation opperates as a cds workstation.
Dave ran a test of the cp1 fill and he is confident that the new vacuum computer can handle this task tomorrow. Though it may not be able to make the plots it usually does.

Current issues:
H1EDC has 17 channels that are currently disconnected that stem from this swap, Dave and I will be working on this first thing in the morning.
Users such as controls, myself, and most other users are able to log in, except vacuum, which had a local account on the old machine (zotvac0 which is currently in the cumputer users room turned off).
But the vacuum user can log in via tty, and shh to make changes to the VAC system. Unfortunately that is not generally how the vaccum users use their account. I have been tinkering for the last hour and noticed that when I switched from xfwm4 to gnome i was able to log in as vacuum using the gui. I then rebooted to see if it would consistently allow vaccum user to locking via gnome and it does not.
So tomorrow Dave and I will be working on getting this running correctly again.

To preserve information about how the vacuum workstation was configured I'll post my notes with IP addresses and macs partially redacted.
 

ZotVac Replacement

Dhcp :
**********************************************
Current Workstations.pool entry:

host zotvac0 {
  hardware ethernet [redacted]:0c:de;
  fixed-address [cds subnet].72;
  option host-name "zotvac0";
}

This must be removed:
host zotws23 {
  hardware ethernet [redacted]:5b:fa;
  fixed-address [cds subnet ].156;
  option host-name "zotws23";
}
-------------------------------------------------
After:
 
host zotvac0 {
  hardware ethernet [redacted]:5b:fa;
  fixed-address [cds subnet].72;
  option host-name "zotvac0";
}

**********************************************

For DNS I just remove zotws23

**********************************************

Old ZOTVAC0 was changed over to the following for troubleshooting purposes.

host zotvac1 {
  hardware ethernet [redacted]:0c:de;
  fixed-address [cds subnet].156;
  option host-name "zotvac1";
}

Tony, Dave:

The EDC is consistently not connecting to 17 of the 27 channels served by the CP1 overfill IOC. This started soon after the upgraded zotvac0 was installed. We ran the original IOC to see if the new machine was the issue, but the EDC still only connected to 10 of the 27 channels in H1EPICS_CP1.ini. It is the same block of 10 which connect, there is no obvious reason why these 10 (checked position in file, type of PV). So we are running the new zotvac0 overnight.

OPS: we expect the EDC disconnect channel count to remain at 17 or hopefully eventually drop to zero.

1381696994

No obvious cause but there was TCS CO2 steps happening at the time, and our funny LSC-DARM wiggle ~100ms before lockloss.

Starting around 11am PDT the picket fence trend plot on nuc5 started restarting itself. This was because the pnsndata service (providing stations OTR on the Washinton penninsular and LAIR at Eugene Oregon) became intermittent. To test this I took these stations out and re-ran picket fence with no issues. Later it looked like pnsndata was more stable, so I added it back in. This did not last and at 12:40 I removed it again from our client.

Note that with the west coast stations removed, the map has centered itself on Idaho.

I'll continue to monitor psnsdata and add it back when it becomes stable.

Using two periods of quiet time during the last couple of days (1381575618 + 3600s, 1381550418 + 3600s) I computed the usual coherences:

https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1381550418/
https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1381575618/

The most interesting observation is that, for the first time as far as I can remember, there is no coherence above threshold with any channels for wide bands in the low frequency range, notably between 20 and 30 Hz, and also for many bands above 50 Hz. I'll assume for now that most of the noise above ~50 Hz is explained by thermal noise and quantum noise, and focus on the low frequency range (<50 Hz).

Looking at the PSDs for the two hour-long times, the noise belowe 50 Hz seems to be quite repeatable, and follows closely a 1/f^4 slope. Looking at a spectrogram (especially when whitened with the median), one can see that there is still some non-stationary noise, although not very large. So it seems to me that the noise below ~50 Hz is made up o some stationary 1/f^4 unknown noise (not coherent with any of the 4000+ auxiliary channels we record) and some non-stationary noise. This is not hard evidence, but an interesting observation.

Concerning the non-stationary noise, I think there is evidence that it's correlated with the DARM low frequency RMS. I computed the GDS-CALIB RMS between 20 and 50 Hz (whitened to the median to weight equally the frequency bins even though the PSD has a steep slope), and the LSC_DARM_IN1 RMS between 2.5 and 3.5 Hz (I tried a few different bands and this is the best). There is a clear correlation between the two RMS, as shown in a scatter plot, where every dot is the RMS computed over 5 seconds of data, using a spectrogram.

In order to more easily track and notify on rising violin modes, I've started making a DARM BLRMS monitor around 490-520Hz. This is currently in position #8 in place of Camilla 6600-6800Hz that didn't give expected results (alog73272). It looks like its working, but not the best it could be. Jenne gave me some pointers that I'll try at another time to get this to better show us the violin mode total magnitude. Once it's looking acceptable I'll make others for the harmonics as well.

To see if the OM2/beckhoff coupling is a direct electronics coupling or not, we've done A-B-A test while the fast shutter was closed (no meaningful light on the DCPD).

State A (should be quiet): 2023 Oct/10 15:18:30 UTC - 16:48:00 UTC. The same as the last observing mode. No electrical connection from any pin of the Beckhoff cable to the OM2 heater driver chassis. Heater drive voltage is supplied by the portable voltage reference.

State B (might be noisy): 16:50:00 UTC - 18:21:00 UTC. The cable is directly connected to the OM2 heater driver chassis.

State A (should be quiet): 18:23:00- 19:19:30 UTC or so.

DetChar, please directly look at H1:OMC-DCPD_SUM_OUT_DQ to find combs.

It seems that even if the shutter is closed, once in a while very small amount of light reaches DCPDs (green and red arrows in the first attachment). One of them (red arrow) lasted long and we don't know what was going on there. One of the short glitches was caused by BS momentarilly kicked (cyan arrow) and scattered light in HAM6 somehow reached DCPDs, but I couldn't find other glitches that exactly coincided with optics motion or IMC locked/unlocked.

To give you a sense of how bad (or not) these glitches are, 2nd attachment shows the DCPD spectrum of a quiet time in the first State A period (green), strange glitchy period indicated by the red arrow in the first attachment (blue), a quiet time in State B (red) and during the observing time (black, not corrected for the loop).

FYI, right now we're back to State A (should be quiet). Next Tuesday I'll inject something to thermistors in chamber. BTW 785 was moved in front of the HAM6 rack though it's powered off and not connected to anything.