TITLE: 01/17 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: VAC work continues today.

I've enabled WIFI across the site which Dave accepted in the CDS SDF, and disabled the picomotors for the chambers being vented.

Visitors from Caltech arrived at 1700UTC and did a safety site walkthrough/tour with Richard, Mike, and Danny.

When launching guardian logs and running the CPS ISI weeklies script I kept getting a notification for Numpy being out of date (Its currently version 1.17.3 and it wants version 1.25.0 to work with the current version of scipy. (Tagging CDS)

Vacuum alarms off and on through the day as activities progress. Verbal reported BSC8 and HAM6 high vacuum pressure at 21:59UTC.

LOG:                                                                                                                                                                                                                                                                                                                                           

Procedure M1300464

The following high voltage power supplies/electronics were powered off in preparation for the corner vent.

1. CER Mezzanine - ESD HV
2. CER Mezzanine - Fast Shutter
3. CER Mezzanine - OMC PZT
4. CER - SR3 and ITM heaters
5. LVEA - Fast Shutter Driver Chassis - Disabled and powered off
6. MER Mezzanine - HAM7 PSAMS
7. MER Mezzanine - HAM7 Piezo

After Fil finished turning off the HV power, following section 3.6 from M1300464 on the DCC and I confirmed that all of the picomotors for HAM3, HAM6, HAM7, and EX were disabled, they were all already off when I opened their medms. MEDMs screenshot

Closes FAMIS 25974, last check in alog75278. We're not totally in nominal chamber or SEI states at the time of this script.

Script reports that:

ETMY_ST2_CPSINF_H2 high frequency noise is high.

HAMs:

HAM2_CPSINF_H3,V3 looks elevated

HAM3_CPSINF_V1,V3 looks elevated

HAM4_CPSINF_V1looks elevated

HAM5_CPSINF_H3,V1,V3 looks elevated

HAM6_CPSINF_H3,V1,V3 looks elevated

HAM7_CPSINF_H1,H3,V1,V2,V3 looks elevated

HAM8_CPSINF_V3 looks elevated

BSC:

ITMY_ST1_CPSINF_V3 looks elevated

ETMY_ST2_CPSINF_H2 looks elevated

I made a comparison of DARM_IN1_DQ and CAL-DELTAL_EXTERNAL_DQ in nominal VS new DARM offloading scheme (the new scheme itself is explained in alog 74887). Data for the NEW_DARM configuration was taken from Dec/21 (alog 74977) when Louis and Jenne successfully transitioned but with calibration that did not make sense.

The main things you must look at are the bottom left panel red and blue, i.e. the coherence between DARM_IN1 and CAL-DELTAL_EXTERNAL in the NEW (red) VS the old (blue) configuration. Blue trace is almost 1 as it should be, but the red drops sharply between 20Hz and 200Hz.

This does not make any sense because CAL-DELTAL_EXTERNAL is ultimately a linear combination of DARM_IN1 and DARM_OUT (see https://dcc.ligo.org/G1501518). Since DARM_OUT is linear to DARM_IN1, no matter where and how the noise is generated and no matter how you redistribute the signal in the ETM chain, CAL_DELTAL_EXTERNAL should always be linear to DARM_IN1, therefore coherence should be almost 1.

So what's the issue here?

The only straightforward possibility I see is that somehow excessive numerical noise is generated in the calibration model even with the frontend's double precision math. Maybe something is agressively low-passed and then high-passed, or vice versa, that kind of thing.

It is not an artefact of the single precision math of DTT. Both CAL_DELTAL_EXTERNAL and DARM_IN1 is already well whitened, and they're entirely within the dynamic range of single precision. For example, RMS of red CAL-DELTAL_EXTERNAL_DQ trace is ~7E-5 cts. From that number, I'd expect that the noise floor due to single precision is very roughly O(7E-5/10**7 /sqrt(8kHz)) ~ O(1E-13) cts/sqrtHz if it's close to white, give or take some depending on details, but the actual noise floor is ~10E-8 cts/sqrtHz. Same thing can be said for DARM_IN1.

It's not the numerical noise in DARM filter as the coherence between DARM_IN and SUS-ETMX_L3_LSCINF_L_IN1 (which is the same thing as DARM_OUT for coherence purpose) is 1 from 1Hz to 1kHz for both configurations (old -> brown, new -> green).  (It looks as if the coherence goes down above 1kHz for the old config, but that's irrelevant for this discussion, and anyway it's an artefact of DTT's single precision math.  See e.g. the top left blue (old config DARM_OUT) with RMS of 20k counts, corresponding to O(2E-5)/sqrtHz single noise floor due to single precision, give or take some. See where the actual noise floor is.)

It's not a glitch, noise level of CAL_DELTAL_EXTERNAL spectrum didn't change much from one fft to the other for the entire window (I used N=1 exponential to confirm this).

Note that there's also a possibility that excessive noise is generated in the SUS frontend too, polluting DARM_IN1 for real, not just for calibration model. I cannot tell if that's the case or not for now. The difference between the green (new) and brown (old) DARM_IN1 spectrum in the top left panel could just be a difference in gain peaking due to different DARM loop shape.

I'll see if double precision channels (recorded as double) in calibration model are useful to pinpoint the issue. Erik modified the test version of DTT so it handles the double precision numbers correctly without casting into single, but it's crashing on me at the moment.

The purge air compressors continue to run as VAC works on getting ready to vent, VAC team hopes to get doors off tomorrow (thurs 18th) or friday (19th).

After looking at beeps and error lights on h0epics2 it looks like we have migrated all the jobs that were on it off already.  We powered off h0epics2 at 11:12am localtime.

We are power cycling the FMCS EPICS computer (fmcs-epics-cds) as a first try to regain stability.

After the first cell phone alarms were sent, I've bypassed them for a couple of hours.

Wed Jan 17 10:04:57 2024 INFO: Fill completed in 4min 53secs

TCs started high, trip temps were -60C for this fill.

Late entry.
The activities carried out on 1-16:
- 44" GVs: GV2 and GV7 closed nicely, without issues, however GV5 was a bit stubborn - it needed 50 psi and some time to close. The wiring was also messed up, Fil corrected it
- Other GVs: the GVs of the relay tube (RV1, RV2); the GVs between HAM7 and BSC3 (FCV1, FCV2); the GVs along the FCT, after BSC3 (FCV3, FCV4); and the GVs before HAM8 (FCV7, FCV8) are closed
- The RGAs for the corner (OMC, HAM6, HAM7) are being pumped. On 1-17 RGA scans.
- The leaky HAM7 fiber feedthrough was leak checked, see details here: https://services1.ligo-la.caltech.edu/FRS/show_bug.cgi?id=30141
- The Kobleco is running, seemingly without any issues
So far everything went as planned.

For the VAC team to bag and leak check the HAM7 leaky fiber feedthorugh FRS30141, yesterday afternoon I unplugged the  HAM7 FC 532nm Fiber from the Feedthrough, covered with a plastic endcap and then replugged in when the VAC team was finished. 

Patrick, Jonathan, Erik Dave:

While the fmcs ioc continues to be unstable, I wrote an auto-restart script which restarts the IOC if its EPICS values flatline for more than 9 minutes.

In order to control the IOC code we moved it from a screen environment to a procServ, and converted the code to a systemd service.

The auto-restart code runs as david.barker on cdsmanager. Every minute it gets the value of the EX chiller yard water temperature channel H0:FMC-EX_CY_H2O_SUP_DEGF.

If the value of this channel does not change for 9 successive minutes, the code restarts the fmcs_ioc.service on fmcs-epics-cds

    ssh root@fmcs-epics-cds 'systemctl restart fmcs_ioc.service'

I started the auto-restart code at 23:12 PST Tue night, since that time there have been 3 auto-restarts

    Tue 16 Jan 2024 11:35:41 PM PST
    Wed 17 Jan 2024 02:29:52 AM PST
    Wed 17 Jan 2024 04:32:58 AM PST
 

Full details can be found in the wiki page h0fmcsbacnet

TITLE: 01/17 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.25 μm/s
QUICK SUMMARY:

• Snowy, cold conditions on site
• VAC work continues today

Gerardo found that h0epics2, rack12 in the MSR, was beeping.  The beeping later stopped, but the error light is now blinking at approx. 3 Hz. 

Powersupplies are both green.  I was able to log in.  I couldn't find any errors on the system.  The EDC has no disconnections.

TITLE: 01/16 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY:

The O4 Break officially started at the beginning of the shift!  

Lots of activity today--miainly in prep for upcoming chamber incursions.  No volumes were vented (this could possibly start tomorrow).

H1 taken to PLANNED ENGINEERING via Observatory Mode!
LOG:

• 1604 Buzzed in Caltech visitor into the front gate
• 1608-1732 ETM charge measurements (RyanC)
• 1613 SeiEnv to "MAINTENANCE"---which turns off BRS
  • 1633 ISC LOCK to IDLE & IMC LOCK to OFFLINE
• 1613 Transition LVEA to Full SAFE. CO2 lasers off (red on OPS Overview) (TJ)
• 1616 & 1637 Kustom Disaster recovery & Restoration contractor at gate
• 1617 VAC group begins prep for vent (Travis)
  • 1622 Booting up RGAs (jordan)
  • 1627 Kobelco restarts (gerardo)
  • 1642 Closing Gate Valves (gerardo, janos, malena)
    • 1654 GV7 closing
    • 1711 GV5 closing
    • 2328 GV2 closed
    • Now items can be craned over the beamtubes in the LVEA
  • 2054 Grabbing pump cart from FCES (travis, jordan)
• 1619 thermostat needed resets due to high temperature alarms---100deg on one sensor & felt! (RyanC for Hanks)
• 1620 Forklift moving and general work in lvea (randy)
  • Continuing for the afternoon
• SEI Watchdog trips
  • 1630 BS isi Watchdog trips & untripping at 1651
  • 1707 HAM1 HEPI tripped
• 1640 EY light work (ken)
• 1645 APS contractor working on cameras at Filter Cavity End Station
  • 1705 Left site since their cable already here; due back later for camera work.
• 1654-1806 All buildings for Equipment search (travis, jordan)
• 1701-2029 Technical cleaning 1st round (karen, kim, chris)
• 1650-2030 Faro work in West Bay (jason, RyanC)
  • 2139 afternoon work continues.
• 1728-1744 On the floor (...but betsy will be in/out most of day!) (Betsy, mitch)
• 1747-1848 HAM5/6/7 taken to damped for HEPI locking and cleaning at HAM7
• 1749 Locking HAM5/6 (jim, ibrahim)
• 1753 Removing HAM6 camrea housings (camilla, betsy)
• 1758-1813 Escorting new LExC cleaner (Emma) into the LVEA (richard)
• 1821-1838 Garbing area search (tj)
• 1907-1917 LVEA light bulb checks in lvea (richard)
• 1918-2046 Optics Lab work (shiela, julian, camilla)
• 1929 Rebooting Picket Fence (Erik) 
• 1940 Started smelling a burning smell in the Control Room, hallway, Computer Users Room .  Phoned Richard and Tyler and the latter let me know it was Erik doing HVAC work and it is the heating coils for the Lab/Control Room areas.
• 2058 HAM3 cleanroom light check (richard)
• 2132-2224 Power supply temperature measurements with intern team (Marc, Alan, and ~4 interns)
• 2250 Video servers restarted for an update--Our cameras have a "connection lost" notification on them. (Nyath)
• Applying visibility tape on top steps of stairways around the corner station (Danny S)
• 2338 Checking out a Squeezer table in LVEA (camilla, julian)

[Measurements attached]

FAMIS LINK:  25973

BSC CPS:  Received following from CPS script terminal output:

• ITMX_ST2_CPSINF_H1 high freq noise is high!

NOTE:  If one was to follow the above channel as an example, it would seem several other BSCs had CPS' with spectra similarly high by eye:

• ITMx ST2 V1
• ITMx ST1 V1
• ITMy ST1 V2

HAM CPS:  Looks good.

Gabriele, Rahul, Jeff, Oli, Betsy

Find below the initial (no damping loop) transfer functions for the built X1 (Staging Building Test Stand) M1 (Top Mass) BBSS.

These were taken generally following Jeff and Oli's alog 74142, where we used their provided DTT template and tuned the excitation amplitude for each DoF to avoid overflows.

Files are saved on the X1 work station system under ligo/svncommon/SusSVN/sus/trunk/BBSS/X1/BS/SAGM1/Data/

They are dated 01/05/24.