Closes FAMIS 26227

Laser Status:
    NPRO output power is 1.835W (nominal ~2W)
    AMP1 output power is 68.84W (nominal ~70W)
    AMP2 output power is 140.0W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN

PMC:
    It has been locked 1 days, 20 hr 55 minutes
    Reflected power = 18.8W
    Transmitted power = 108.8W
    PowerSum = 127.6W

FSS:
    It has been locked for 3 days 0 hr and 28 min
    TPD[V] = 0.3952V

ISS:
    The diffracted power is around 1.7%
    Last saturation event was 3 days 1 hours and 0 minutes ago

Possible Issues:
    PMC reflected power is high
    FSS TPD is low
    ISS diffracted power is low

TITLE: 01/19 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 10mph Gusts, 8mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.73 μm/s 
QUICK SUMMARY:

• Snowy icy conditions
• The LVEA is getting a little warm, ZONE4 is just past 70 degrees F and Zone 1A ios just under 70 (Tagging facilities)
• End station temps seem stable now
• High microseism

Today's activities:
- The Annulus volumes of HAM6, HAM7, and GV2 were vented with Nitrogen
- HAM6 X+ and X- doors were taken off
- HAM7 X+ and X- doors were prepared to be taken off tomorrow
- HAM7 RGA electronics was disconnected, the aux cart was removed
- The purge air was checked, its quality was satisfactory, see details in comment
- The staff left early because of the unpleasant weather conditions
So far, everything is going according to plan: https://docs.google.com/spreadsheets/d/1F3Z1cjFrFtaNHayVFVVXruY5JwSiY5-S/edit?usp=sharing&ouid=104962245367404322733&rtpof=true&sd=true

WP 11626

Dave B., Jonathan H., Patrick T.

Jan. 17 - Jan. 18

The IOC reading out the FMCS data from the BACnet devices and putting it into EPICS has been migrated to the code from ORNL: https://controlssoftware.sns.ornl.gov/bacnet/index.html

As cdsadmin in /home/cdsadmin/github on fmcs-epics-cds I checked out https://github.com/ornl-epics/BACnet.git at this commit:

cdsadmin@fmcs-epics-cds:~/github/BACnet$ git log
commit b0ebd50612faae162a7b3b0dc898cd44c30b9ea3 (HEAD -> main, origin/main, origin/HEAD)
Author: kasemir 
Date:   Thu Aug 10 09:13:50 2023 -0400

    import


In /home/cdsadmin/github/BACnet/configure/ I copied RELEASE-template to RELEASE and in RELEASE changed EPICS_BASE=/home/8w4/epics/base/base-3-14-11 to EPICS_BASE=/usr/lib/epics. Jonathan installed the necessary build tools to compile the IOC and I did so by running make in /home/cdsadmin/github/BACnet.

As cdsadmin in /home/cdsadmin/gitlab I checked out https://git.ligo.org/patrick.thomas/fmcs-bacnet.git. This repository contains our custom FMCS EPICS database files and scripts and it should replace https://redoubt.ligo-wa.caltech.edu/viewvc/projects/trunk/epics/iocs/fmcs_bacnet/. In /home/cdsadmin/github/BACnet I created a db directory and made these soft links in it:

chiller_bacnet.db -> /home/cdsadmin/gitlab/fmcs-bacnet/chiller_bacnet.db
fces_bacnet.db -> /home/cdsadmin/gitlab/fmcs-bacnet/fces_bacnet.db
fmcs_bacnet_bi_to_ai.db -> /home/cdsadmin/gitlab/fmcs-bacnet/fmcs_bacnet_bi_to_ai.db
monitor.db -> /home/cdsadmin/gitlab/fmcs-bacnet/monitor.db

In /home/cdsadmin/github/BACnet/iocBoot/ioctestIoc/ I renamed the st.cmd file to st.cmd.original and created the soft link st.cmd -> /home/cdsadmin/gitlab/fmcs-bacnet/ornl_st.cmd.

The documentation for this BACnet driver code states that it only supports ai, ao, and stringin records. In order to continue to use our bi records that have ONAM and ZNAM strings defined, I created an additional ai record for each of these. Then instead of having the bi records read from the driver code I made the ai records read from it and send their values to the bi records.

In addition I added a pair of keep alive channels, H0:FMC-BACNET_COUNTER and H0:FMC-BACNET_HEARTBEAT. The former increments its value by 1 at 1 Hz. The latter toggles its value between 0 and 1 at 1 Hz.

This driver code also provides monitoring of the traffic to and from each BACnet device. Therefore for each device I created three channels, H0:FMC-BACNET_{device id}_TX, H0:FMC-BACNET_{device id}_RX, and H0:FMC-BACNET_{device id}_ER. These scan at 1 Hz.

This morning I manually started the code, and after correcting a couple of mistakes in the database files, it seemed to be running stably. It is interesting to note that although the scan rate of each of the records had not changed (the majority at .1 Hz), the values seemed to be updating much faster than before. I therefore reduced their scan rate to 1 Hz.

I have seen a number of timeout and other error messages on the console, but I am not certain how to troubleshoot them. The ER diagnostic channels that I looked at read 0, and the values seem to be updating. Since the old code also had error messages, I'm not sure if it is a problem.

TITLE: 01/18 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: HAM6 doors are off, and HAM7 doors are being prepped for removal tomorrow. Jim locked HAM6 around 20:40UTC. Snows picking up and isn't predicted to stop until tomorrow morning.

• LVEA temperatures are a little wacky, Eric thinks its from the fans being worked on earlier, the quartely lubing, but will keep his eye on it.
  • MX low temp alarm and CS airhandler alarm for LVEA_REHEAT_{3A, 3B}

LOG:                                                                                                                                                                                                                                                                                                                                                                                                                   

After vac pulled doors, I went in and lock HAM6 ISI. Mitch came out and we then installled the work platforms at each door. I couldn't remember if we secured those with c-clamps or not, there weren't any with the platforms. But, they're pretty stable even without.

The VAC team is currently working on getting the doors off of HAM6&7.

PSL 102 red dust alarm, MX low temp alarm (H0:FMC-MX_VEA_202A_DEGF) and a CS airhandler alarm for LVEA_REHEAT_{3A, 3B}

Quarterly lubrication of all applicable supply fans has been performed and AHU's with redundancy are swapped. This quarterly change will include the re-introduction of MY SF 02 which recently underwent a bearing swap at both the intake and discharge side of the fans (no motor bearing swaps).

C. Soike E. Otterman T. Guidry

Patrick, Erik, Dave:

WP11626 New FMCS IOC

Patrick is starting a new version of the FMCS IOC code.

While the new code is being tested, the cell phone alarms have been turned off and the old code disabled.

I'm going through my old alog drafts, this is from Feb 2023 but the times here may now be useful information, so I'm posting this incomplete draft from a year ago. 

Last week I did an additional test to follow up on 66843 66751 where excess noise was seen when locking DARM on ETMX with an ESD bias of -125V.   Past checks have shown that this excess noise is not due to a calibration error or due to SRCL or MICH feedforward being mistuned with the reduced bias. 

The first attachment shows that indeed this noise is there repeatably in a way that is similar to what was seem in past tests, although the spectrum did change over the time I was doing this test.  This time I did the measurements in NLN_CAL_MEAS so that ADS and calibration lines were off. 

Although the estimate of the ESD quadratic term in 66843 suggested that the ESD quadratic term shouldn't be large enough to explain this noise, I wanted to repeat the test while driving a line on the ESD strongly enough to see the quadratic coupling to confirm that the model of the quadratic coupling is working well.  The second attachment shows three examples of driving a 15.45 Hz line with the same amplitude (10  counts at LOCK_L_EXC), each with slightly different results. The green cursors show the injected line and it's harmonic at 30.9Hz. The first time that I injected the line, it seemed as though the broad noise from 20-35Hz was reduced (grey trace, compare to yellow and blue), but in the next two tests this was not repeated.  

The third attachment shows what happened when I tripled the amplitude of the injected line, the 15.45 Hz peak is increased by a factor of 3 while the 30.9Hz peak is increased by a factor of 9; this makes sense. 

Thu Jan 18 10:02:40 2024 INFO: Fill completed in 2min 38secs

Quick fill, TCs started around 0C.

TITLE: 01/18 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 10mph Gusts, 8mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.34 μm/s
QUICK SUMMARY:

• Lab and computer room temps seem stable
• Drive in wasn't bad
• Doors look to be coming off today

Today's activities:
- Relay tube was vented with Nitrogen; it was also taken off (mitre part stayed, the others came off in 1 single piece)
- RGA scans have been made at HAM7 and HAM6 - they were satisfactory, details later; the OMC RGA is leaking, so no scan there - leak hunting later the week, or next week
- All the RGAs, Cold Cathode gauges, Ion pumps, NEG-pumps were valved out in the volumes to be vented
- The corner was vented first. At the start of the vent, the purge air's dew point was at ~-52 deg C, with 0 particles. The vent was carried out from 2 pm to 6 pm
- HAM7 was vented after. The purge air's dew point was at ~-42 deg C, with 0 particles. This vent was carried out during the corner vent, taken ~1.5 hours
- After the corner vent, GV5 AIP railed: this suggests a leak from the annulus system to the main volume - further investigation is required (history: in the last 3 years this AIP railed 6 times). This Annulus volume is being pumped with an aux cart now during this vent

Below is the summary of the LHO DQ shift for the 9 days period of 2024-01-08 to 2024-01-16:

We would like to express our gratitude to the Instrument Team for having a superb detector!

Kudos to Zsuzsa Marka for the patient and expert instructions and Beverly and the DetChar Team for the amazing tool and organization!

End of O4a shift
In observing for the period ~47%. (78.7, 32.3, 6.2, 100, 27, 0, 56, 59.6, 61.3)
Range: generally between 150 and 160 Mpc, quite rugged in the last few days.
Some earthquakes contributed to lock losses, however, intentional lock loss also happened.
Microseism was an issue for a good fraction of the week.
There was one day that was covered by a single continuous lock and another day that had no lock whatsoever.
For the first part of the period the glitchiness had been normal, but for the last few days there had been quite a bit of large glitches in the bucket.
New HVETO channels kept showing up every day, except the last one as compared to the previous period.
Combs, such as 23.8Hz, 29.97Hz, 9.474Hz, 9.475Hz, and 9.48Hz showed up on most days. Line counts below the threshold on all days when data is available.
Some strange behavior on some PEM channels, as posted in daily logs.
Many pyCBC triggers with new SNR > 10 (medium or short) showed up most days; the large glitch rate in the bucket seemed to correlate with how many were there.
The full DQ shift report with day-by-day details is available at:

https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20240108 

Rick and I went to the PCAL LAB to do the PCAL Tx module maintanence today.
We followed Mostly followed the procedure outlined in DCC doc T1600436-v8 which was printed out and attached to the ALOG.
I have also updated the PCALTxMaintenanceLogBook spreadsheet with the latest information from today's activity. This is information is not currently in the DCC.
I will wait until after EX and EY are completed later this week before updating the DCC with the latest spread sheet. The latest information Is currently attached to this alog.
Attached a picture of the OLTF taken with the SR785, I will update this with the .gif that the SR785 gave us tomorrow.
 

After Fil finished turning off the HV power, following section 3.6 from M1300464 on the DCC and I confirmed that all of the picomotors for HAM3, HAM6, HAM7, and EX were disabled, they were all already off when I opened their medms. MEDMs screenshot

Louis, Jenne, TJ, Sheila

Today we continued to try to transition to the new Darm configuration, which we had suceeded in doing in December but weren't able to repeat last week (75204). 

In our first attempt today we tried a faster ramp time, 0.1 seconds.  This caused immediate saturation of ETMX ESD.  We struggled to relock because of the environment. 

Because Elenna pointed out that the problem at roughly 3 Hz with the earlier transition attempts might have been the soft loops, we thought of trying to do the transition before the soft loops are engaged, after the other ASC is on.  We tried this first before transitioning to DC readout which wouldn't work because of the DARM filter changes.  Then we made a second attempt at DC readout.  We also lost lock due to a 2 Hz oscialltion, even without the soft loops on. 

Some gps times of transitions and attempt:

• 1387034100  succsesful transition from December
• 1388520329 attempt from last week with 5 second ramp
• 1388520329 attempt from last week to transition to the NEW DARM in TRANSITION_FROM_ETMX and LOWNOISE_ESD_ETMX with a 5 second ramp
• 1388952732 attempt from today to transition with a 0.1 second ramp, imediate saturation of ESD
• 1388961745 attempt at 2W with soft loops off (5 second ramp)