TITLE: 11/29 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.23 μm/s
QUICK SUMMARY:

H1 is relocking following maintenance day, currently up to TRANSITION_FROM_ETMX. The PRG trace is noisier than usual, but otherwise things look okay.

TITLE: 11/28 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

1. 16:35 UTC - Dave noticed an EY GPS timing error on the CDS overview that was flashing. The PPS A GPS clock was flashing an error due to a timing difference past tolerance. Error disappeared after 16:40 UTC.
2. 17:00 UTC SNEWS alert test T456241 caught on both Diag main and verbal
3. 18:11 UTC HAM7 Watchdog trip, likely due to SQZ rack work that Fil is doing (pulling cables).
4. 18:46 UTC IOC server temp change: While Erik was updating the IOC LVEA temperature, he realized that the entire IOC server needed to be rebooted.
   1. IOC server reboot happened at 18:46 UTC and connection was reestablished at 18:49 UTC.
   2. IOC reboot temporarily took down:
      1. Remote power
      2. Check violins
      3. Wi-Fi
      4. Ops GraceDB Standown - did not restart as expected, TJ turned it back on
      5. Picket Fence - did not restart as expected prompting further investigation by Erik - 
   3. Wi-Fi was separately turned on during reboot in order to maintain control room work.
   4. 19:09 UTC Erik updating IOC temp. Done at 19:16 UTC
5. Two GRB Short Alerts
   1. 19:25 UTC
   2. 19:37 UTC
6. 19:00 UTC (ish) Temp excursion in CER and SUP Rack 1 - still within “nominal/tolerance” but visibly on the come-up. 
   1. 20:10 UTC Eric investigated and found that the lead unit had a fault and so he reset the system. Temperatures visibly on the come-down, both in CER and SUP.
7. 19:46 UTC - during Beckhoff work, 
   1. Beckhoff restart caused a connection error with guardian ALSY node. Guardian 1 seems to have lost connection, despite Beckhoff showing everything as normal. There are 7 SPM diffs, all pertaining to H1:ALSY (and nothing else). 
   2. Apparently it fixed itself at 19:46 UTC and then the same issue happened at 20:27 UTC
   3. Error message reads as: “CONNECTION ERRORS, see SPM DIFFS for dead channels". See screenshot.
   4. The workstations have no problems and everything else seems normal.
   5. Dave and Daniel suggested doing a Guardian reboot, though Dave is looking into it and seeing if it will/won’t help.
   6. 20:45 UTC: This is currently stopping us from locking
   7. 20:51 (ish) UTC - TJ fixed the issue by bringing the guardian node to STOP and then EXEC, instead of reloading
      1. Apparently this is an issue where sometimes when connections are lost and reconnected a complete STOP rather than a PAUSE/RELOAD is necessary to bring the connection back.
      2. No Guardian reboot necessary
8. DAQ Restart: NUC23 isn’t loading and is yielding an error message - Ryan C couldn’t get into it during the previous maintenance Tuesday.
   1. Ryan C is power cycling it and then attempting to connect
   2. Worked fine after restart
9. Locking at 20:53 UTC 
   1. Starting with an initial alignment - Initial Alignment went fine
      1. Had to touch ALS_Y Yaw to get it caught
10. Lock acquisition 1 at 21:48 UTC: Lock loss due to DRMI lockloss at Turning_BS_Stage2
    1. Caused by DRMI_ASC issue that Sheila caught and is working on.
11. Lock acquisition 2 at 21:52 UTC: Sheila and Naoki - DRMI_ASC won’t work until they fix something so guardian only taken to DRMI_Locked_Prep_ASC
    1. Sheila said it was potentially fixed -
    2. Lockloss at the same stage again 21:57 UTC
12. Lock acquisition 3 at 21:59 UTC
    1. IMC put to DOWN at 22:00 UTC while Sheila and Naoki investigate the DRMI ASC issues
    2. Back to lock acquisition at 22:17 UTC
    3. Paused at DRMI_LOCKED_PREP_ASC - Sheila and Naoki noticed ASC SRC1 Pitch and Yaw offset(s) were off by orders of magnitude but they seem to have been fixed during their investigation. See Naoki’s alog 74457.
    4. 22:28 UTC Continuing lock - it worked!
    5. NLN Achieved at 23:04 UTC
    6. LOCKLOSS at 23:14 UTC while ASC clearing SDF Diffs.
       1. TJ noticed noisy power recycling gain
13. 14:54 UTC: FMCS Air Handler 3B (Reheat) - FMCS alarm handler came onto red and then 3 mins later went back to normal (tagged FMP)
14. Lock acquisition 4 at 23:15 UTC
    1. Power recycling gain is being noisy (again).

LOG:

Arianna, Camilla

There are many glitches making up the “fuzzy” range time from 07:00UTC 23rd November 2023. We identified glitches using ndscope of the DARM BLRMs, (plot attached showing yellow, green, blue BLRMs getting worse at t-cursor) and then used ldvw.ligo.caltech.edu Q-transform to plot omega scans of the glitches. Original troubleshooting in alog 74377. 

Attached is a pdf containing the glitches seen between 07:00UTC and the lockloss at 10:10UTC. There are many more glitches than usual and lots of different types of glitches.

WP11540 Remove SDF safe->OBSERVE exceptions for h1seiproc and h1brs

TJ, Jim:

TJ made changes to guardian to remove the exception that h1seiproc and h1brs do not transition to OBSERVE.snap. h1brs OBSERVE==safe, h1seiproc has a separate OBSERVE.snap which Jim will verify is correct.

WP11546 New FMCS STAT code base

Erik:

Erik rewrote the FMCS STAT IOC to use this new softIOC python module. The code was restarted.

New Guardian node

Camilla, Dave:

Camilla started a new Guardian nodel called SQZ_ANG_ADJUST. I updated the H1EPICS_GRD.ini file. DAQ+EDC restart was required.

DAQ Restart:

Jonathan, Erik, Dave:

The DAQ was restarted to include the new GRD channels into the EDC. This was a good restart with no issues.

cdsioc0 reboot

Erik, Dave:

Erik updated and rebooted cdsioc0. There were some issues getting Picket Fence running again, which Erik resolved. We were reminded that I was running a temporary HWS ETMY IOC in a tmux session, Erik switched this over to a systemd service via puppet.

Jordan

We ran the functionality test on the main turbopumps in MY and EY during Tuesday Maintenance (11/28/23). The scroll pump is started to take pressure down to low 10^-02 Torr, at which time the turbo pump is started, the system reaches low 10^-08 Torr after a few minutes, then the turbo pump system is left ON for about 1 hour, after the hour the system goes through a shut down sequence.
 

MY Turbo:

Bearing Life:100%

Turbo Hours: 208

Scroll Pump Hours: 74

EY Turbo:

Scroll pump made a grinding sound after getting to ~ 5E-2 Torr. I closed all valves and stopped the test. The scroll pump only has 200 hours on it so it will be disassembled to figure out the source of the noise. I have swapped the scroll pump with a new ISP250, but did not have time to run the turbo test. I will resume next tuesday and add a comment to this alog with the EY results.

Closing WP 11544 and FAMIS 24917

F. Mera, D. Sigg, M. Pirello

Per WP11548:

The Squeezer 4 rack was modified duplicating the EL3692 terminals in the positions M13, M14, M19 and M20. Wiring was modified to have readings in CH1 only.

TwinCAT solution was updated accordingly, the system was restarted and succesfully verified with the new configuration.

SQZ 4 rack serial: S2101205

Clean room over Ham 6 lights were turned off
Wifi was turned off at the corner and both arms, except MSR.

LVEA has been swept.

Tuesday maintenance activities are wrapping up - locking thereafter.

• Tony doing LVEA Sweep
• Gerardo wrapping up FCTE valve work (in contact)

Fil Marc Daniel

Today we pulled the lond d-sub cables going from the CER to the squeezer racks. This completes the electronics installtion for the SQZ PMC with the exception of the on table devices.

Closes FAMIS 25967, last checked in alog74134

BSC:

Script reports the following are elevated: ITMX_ST2_CPSINF_{H3, V1, V3 }

ITMY_ST1_CPSINF_{V2, V3}, ETMX_ST1_CPSINF_V2 are elevated at high frequency

HAM:

The HAMs look fine

Workstations were updated and rebooted at about 1500 UTC.  This was an OS update.  conda packages were not affected.

Tue Nov 28 10:03:40 2023 INFO: Fill completed in 3min 37secs

TC-A min = -90.1C

TITLE: 11/28 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 3mph Gusts, 2mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.19 μm/s
QUICK SUMMARY:

IFO was locked (but not observing due to Tues magnetic injections). Sheila unlocked IFO through planned DARM configuration change.

PSL 102 dustmon is still stuck and giving warning: "dust counts did not change please investigate". See Oli's eve (11/27) summary alog 74421 to follow the story on this.

TITLE: 11/28 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Relocked twice today, both locklosses (74432, 74438) being due to earthquakes. Had some issues with the first relocking, needing to go to GREEN_ARMS_MANUAL before running the initial alignment, and after still going through PRMI. Second relocking I also needed to touch both arms and we ended up going through CHECK_MICH_FRINGES on the way up. We are currently at LOWNOISE_COIL_DRIVERS. Other than that, everything was relatively normal.

We actually just got a verbals call out for an incoming earthquake from Fiji that is in the middle of the green zone on the EQ Response Zones script so hopefully we will be able to stay up!
LOG:

00:00UTC Detector holding in DOWN due to earthquake

00:40 Started Locking
    - Needed to align ALSX manually, then on to an Initial Alignment
    - 01:30 Initial Alignment completed, on to Locking

02:22 NOMINAL_LOW_NOISE
02:25 Observing

06:30 Earthquake mode activated
06:31 Lockloss due to earthquake                                                                                                                                                            

Lockloss 11/28 @ 06:31UTC due to earthquake from Alaska

Sheila, Camilla, Jenne, Austin

We droped out of observe for a few minutes to start running the sqz angle adjustment guardian state using the ADF that Naoki wrote on Wed (74376).  This changed the sqz angle by a few degrees in the first 2 minutes it was on, which improved the sqz blrms slightly.  

Because Vicky put some code in ISC_LOCK that set the SQZ_MANAGER request to it's nominal state, the only change we had to make was to change the nominal state of SQZ_MANAGER, and this should now be used in future locks. The new SQZ_MANAGER state is ADJUST_SQZ_ANG_ADF state number 110

TITLE: 11/23 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

The last hour has shown a drops in range while being in OBSERVING.  In the last 30min the range drops have been less severe, but this is an odd one.

Had a few bumps out of observing due to the squeezer, but it would only be for about ~2min as the squeezer relocked itself.  L1 came back to Observing this shift.
LOG:

• Drops from OBSERVING due to the Squeezer unlocking this shift:
  • 2356 - 2359 utc (end of Ryan C.'s shift)
  • 0004 - 0006 
  • 0624 - 0626
• 0244 Short GRB alert
• 0712 Short GRB alert
• 0700 H1 range begins taking a hit with drops and variations.  Have not received any alarms for this & have been in OBSERVING the whole time. Environmentally we look good at first glance.

WP 11533

Checked the HAM4 ISI Coil Driver Chassis. Issue reported last week of noisy fan. Fan is spinning and noise reported last week has not returned. Will leave WP open another week.

Artem, Gabriele, Sheila, Louis

Plots and Jupyter notebook with these results are also available here.


 ESD quadratic noise 

This is a follow up on investigation of potential non-stationary noise at Handford introduced by the ESD drive, reported in 73913 and previous entries linked from there. The idea is that from the ESD signal, Vs, and bias, Vb, voltages we can reconstruct ESD force applied to the mirror, using equations in LIGO-T1700446. The force component linear in Vs gets subtracted by the control loop and therefore does not affect DARM. But the force component quadratic in Vb and both linear and quadratic in Vb can couple to DARM. Therefore in this report, we are calculating respective force components and projecting them to DARM. Then respective ASDs and spectrograms are compared.

 Selection of input channels 

The ESD drive voltages are recorded by Monitors, H1:SUS-ETMX_L3_LVESDAMON_LL_OUT_DQ for Vs and H1:SUS-ETMX_L3_ESDAMON_DC_OUT_DQ for Vb. Important note here: we initially tried channel H1:SUS-ETMX_L3_ESDAMON_UL_OUT_DQ to get Vb, but it looks like it's not connected to anything and recording ASD noise... The H1:SUS-ETMX_L3_ESDAMON_DC_OUT_DQ seems to be fine. Also, LIGO-T1700446 specifies calibrations for voltage channels (equations 13 and 16), but currently the channels mentioned above are already calibrated to voltage.



 Calculating force terms 

Calculation is based on equation 3 in LIGO-T1700446, but this equation is modified in the following way:

Bias voltage is decomposed into DC and varying part, to avoid propagating DC term through the calculation (DC term will not affect DARM). So the following substitution is applied: Vb = Vb_DC + dVb. The term Vb_DC is simply median of Vb time series, and dVb is Vb time series with median subtracted
Brackets are then opened to gather all the terms linear in Vs, quadratic in Vs, linear in dVb, quadratic in dVb, and mixed dVb*Vs term. Noise term is then quadratic in Vs +  linear and quadratic in dVb + mixed dVb*Vs


Here are the new equations:

# new equations with dVb and rearrange terms
F_quadratic_Vb_term = (alpha+gamma)*(dVb_ts**2)
F_quadratic_Vs_term = (alpha+gamma)*(Vs_ts**2)
F_mixed_term = (gamma-alpha)*2*dVb_ts*Vs_ts
F_linear_Vb_term = (alpha+gamma)*2*Vb_DC*dVb_ts + beta2*(dVb_ts+Vb_DC)  + beta*(dVb_ts+Vb_DC)
F_linear_Vs_term = (gamma-alpha)*2*Vb_DC*Vs_ts  + beta2*(-Vs_ts) + beta*(Vs_ts)
F_total_term = F_quadratic_Vb_term + F_quadratic_Vs_term + F_mixed_term + F_linear_Vb_term + F_linear_Vs_term 

# noise term has everything _linear_ in Vs and both _linear and quadratic_ in Vb
F_noise_term = F_quadratic_Vb_term + F_quadratic_Vs_term + F_mixed_term + F_linear_Vb_term

Coefficients alpha, beta, beta2 and gamma are taken from this measurement: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=56613.



 Results: 95%CL ASD and spectrograms for ESD noise converted to equivalent strain 


Force is converted to acceleration by dividing by the mirror mass (40 kg).
Subsequently acceleration is divided by omega^2 to convert to displacement.
Finally displacement is converted to equivalent strain by dividing by arm length (4 km). Now the spectrum can be compared to DARM channel.





 Replacing Vb with white noise at typical level, to avoid projecting Monitor's ADC noise 

Gabriele made a plot showing ASD of Vb for different levels of bias and when the bias is off. What we can see is a shoulder of additional noise which we believe is coming from ADC of the monitor. To get rid of that, we generate white noise time series for dVb at the level of Vb where it is not dominated by additional noise. The new dVb term is then plugged into the equation for the force term and a new set of plots is produced.





 Results: 95%CL ASD and spectrograms for ESD noise converted to equivalent strain (dVb replaced with white noise - see above)