The new corner station scroll pump air compressor was valved in at 09:39 PDT this morning. I has gone through 3 compression cycles since then. Here are the stats
I have reset the cell phone alarms with the levels: LOW = 80PSI, HIGH = 120PSI
The new compressor runs at a higher pressure so the alarms needed to move with that. I changed the minor and major high alarms for H0:VAC-MR_INSTAIR_PT199_PRESS_PSIG for 127 and 130 respectively in epics via the vacuum computer in the back of the control room. These alarm levels cannot be changed outside of the vacuum network. I did not see this channel in the VAC SDF.
Patrick will check if these values are hard coded into Beckhoff and will adjust accordingly.
Currently the slow-controls SDF is not able to monitor non-VAL fields (e.g. alarm fields). Future release will have this feature.
I made the following changes to the PLC code generation scripts (not pushed to the running code) to enable setting the alarm levels for each instrument air pressure EPICS channel separately. I set the alarm levels to what they currently are. Should any of these be changed? https://git.ligo.org/cds/ifo/beckhoff/lho-vacuum/-/commit/8d17cfe3284147b4e398262091ec9d9f2ddbb6ab If there are no objections, I will plan to push this change when we make the changes to add and rename filter cavity ion pump controllers.
Conda packages updated. Controls rooom tools are updated to version 4.1.4. The only change is a fix to the Python foton module.
FilterDesign::get_zroots() would throw an exception when called. Thanks to Chris Wipf for finding and fixing the problem.
Thu Aug 15 08:08:59 2024 INFO: Fill completed in 8min 55secs
TITLE: 08/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 10mph Gusts, 7mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY: Pumpdown continues.
TITLE: 08/14 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY: The pump down continues (5e-7).
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 15:24 | FAC | Karen | LVEA | n | Tech clean | 16:32 |
| 16:00 | FAC | Kim | LVEA | n | Tech clean | 16:15 |
| 16:01 | FAC | Nellie | HAM Shack | n | Tech clean | 17:01 |
| 16:17 | FAC | Karen | EY | n | Tech clean | 18:17 |
| 16:33 | FAC | Kim | EX | n | Tech clean | 17:57 |
| 16:40 | VAC | Gerardo | LVEA | n | HAM5/6 work with RGA and pump | 17:52 |
| 17:47 | VAC | Janos | LVEA | n | Walkabout | 17:54 |
| 17:52 | VAC | Gerardo | EX | n | Cart hunt | 18:35 |
| 17:56 | FAC | Karen | MY | n | Tech clean | 18:35 |
| 17:58 | FAC | Kim | H2 WH | n | Tech clean | 18:34 |
| 18:35 | - | Camilla, Detchar (8) | LVEA | n | Tour | 19:18 |
| 20:10 | VAC | Chris | LVEA | n | Grab straps for bake ovens | 20:16 |
| 20:19 | VAC | Janos, Gerardo, Travis | Opt lab | n | Viewport inspection | 21:16 |
| 21:43 | TCS | Camilla | Opt Lab | Local | Cheeta testing |
ongoing |
| 21:49 | CDS | Marc | MY | n | Looking for parts | 22:20 |
| 21:49 | VAC | Chris | LVEA | n | Putting straps back | 22:09 |
Yesterday Sam and I went around the LVEA with a portable accelerometer (H1:PEM-CS_ADC_5_28_2K_OUT_DQ) to investigate a ~35.4 Hz peak. This peak dates back to at least June 2023, possibly longer. We placed the portable accelerometer in seven different spots, each time against a wall except for one spot near BS1. See the attachment 1 for the reference map. The peak appears strongest at REF0 and REF5 as seen in attachment 2. We attempted to test outside near the Mitsubishis (outside the OSB kitchen) and the southeast-facing mechanical room wall as well but had data retrieval issues, so we'll go back to that soon.
We are also investigating the ~36 Hz peak seen in attachment 2, strongest near the X OL. It seems to have been around since at least 2023-05-08, but more investigation is needed to find out when it started and if it shows up consistently every day. The frequency of this peak oscillates on a range of about 1 Hz. Attachment 3 shows the peak in H1:PEM-CS_LVEAFLOOR_XCRYO_Z_DQ and H1:GDS-CALIB_STRAIN_CLEAN. This peak also travels between 30-40 Hz, generally near 40 Hz at the beginning and end of the UTC day and dropping towards 30 Hz in the middle of the day, as seen in attachment 4 from the summary pages.
WP12035
Yesterday thermocouple B (TC-B) failed during the fill. Its wire broke and its temperature railed at +1372C. The fill proceded with TC-A to completion.
If TC-A had similarly failed, the fill would have continued until either the discharge line pressure exceeded its trip pressure; or the fill timed out after 30 minutes.
Today's code modification adds a check when reading the TCs that they are not greater than the nominal_max temps (+40C).
Providing at least one TC is in operational range the fill will continue, but now if no TCs are operational the fill will abort with an error.
The new code was installed at 9am today, it will run for the first time during Thursday's fill.
During this investigation I noted that the discharge_line_pressure trip is 2.0PSI. Operationally this pressure has never exceeded 0.6PSI so I lowered the trip down to 1.0PSI.
Wed Aug 14 08:08:37 2024 INFO: Fill completed in 8min 34secs
New TC-B looking good.
TITLE: 08/14 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 2mph Gusts, 0mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY: The pumpdown continues.
The PCAL team went down to EY with PS4 to do an End station measurement.
We mostly followed the procedure , but I did do something that may be of interest.
Beam spot is looking slightly oblong but not too bad. I did not touch up the beam spots.
The martel calibrated voltage source was set to the "mV range" today instead of the usual "V range" while inputing the "zero votage" into the DAQ.
This may present us with a more consistent value of "Zero Voltage". This auxilary analysis is pending.
The normal ES trends analysis is already done and can be found here: https://git.ligo.org/Calibration/pcal/-/tree/master/O4/ES/measurements/LHO_EndY/tD20240813?ref_type=heads
Beam spot after we left still looked a little long.
Installed a new MKS gauge at cross C-4, to do the installation I closed two gate valves for the filter cavity tube, FCV-6 and FCV-9, these valves isolated the volume I was working on and left a gauge that gave me the vacuum pressure in the area that I was working, the pressure was actively monitored, no changes noted.
To install the gauge a metal valve and a T were added to the configuration, the system was pumped down and leak tested, and no leak was noted above 1.0x10-10 torr*l/sec.
To finish the installation for now, the new volume was pumped down, and remains isolated from the main volume. Then FCV-6 and FCV-9 were open.
TITLE: 08/13 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
SHIFT SUMMARY: Light maintenace today while we continue to pump down. One DAQ restart for CAL model updates, and some other work around the site.
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 14:37 | ISC | Corey | LVEA | n | ISC table enclosure checks | 15:16 |
| 15:15 | PCAL | Tony, Dripta | PCAL lab | local | prep meas | 15:22 |
| 15:16 | ISC | Corey | EY, EX | n | ISC table enclosure checks | 15:49 |
| 15:18 | CDS/PSL | Fil, Jason | LVEA | n | Glitch hunting at PSL racks | 17:51 |
| 15:20 | FAC | Karen, Kim, Nellie | FCTE | n | Tube cleaning | 17:51 |
| 15:23 | PCAL | Tony, Dripta | EY | Yes | PCAL measurement | 18:00 |
| 15:37 | VAC | Gerardo | LVEA | n | Open HAM7 ion pump | 17:33 |
| 15:46 | CAL+DAQ | Dave | remote | n | New h1calcs model, DAQ restart | 19:31 |
| 17:03 | CDS | Marc | CER, EX,MX,MY,EY | n | Power supply checks | 18:08 |
| 17:14 | CDS/PSL | Richard | LVEA | n | Check in with Fil and Jason | 17:33 |
| 17:16 | VAC | Gerardo | FCES,FCTE | n | Leak detector move | 19:04 |
| 18:03 | REC | Christina | OSB Receiving | n | Throwing stuff in the recycling bin w/ forklift | 19:33 |
| 18:08 | - | Mike, tour | LVEA | n | Tour | 19:08 |
| 20:07 | FAC | Chris | LVEA | n | Grabbing barrel | 20:37 |
| 20:41 | PEM | Genevieve, Sam | LVEA | n | Moving an accelerometer around | 22:16 |
| 20:44 | CDS | Dave | CER | n | Check EY HWWD | 21:14 |
| 20:45 | VAC | Janos | LVEA | n | Checking pumps | 21:45 |
| 22:41 | VAC | Gerardo | LVEA | n | HAM6 RGA bake temp increase | 22:53 |
I checked the Kepco power supplies located on site and noted the following per WP12033:
CER Mezanine (Ambient 69F)
C1 - U38 RHS slight vibration noted, 89F front, 115F back, good airflow, supplies SUS_C8 (18V, 5A)
C2 - U29 slight vibration noted, otherwise all temps below 75F
C3 - No vibrations, all temps below 75F
C4 - No vibrations, all temps below 75F
C5 - U5 slight vibration noted, all temps below 75F *same supply had vibration last check in May ALOG77832
C6 - No vibrations, all temps below 75F
MER (Ambient 70F)
VDC-M1 - No Vibrations, all temps below 75F
TCS-M1 - No Vibrations, all temps below 75F
EX (Ambient 69F)
VDD1 - No Vibrations, all temps below 75F
VDD2 - No Vibrations, all temps below 75F
MX (Ambient 71F)
Power Rack - No Vibrations, all temps below 75F
FCAV (Ambient 70F)
H1-VDC-C1 - No Vibrations, all temps below 75F
MX (Ambient 71F)
Power Rack - No Vibrations, all temps below 75F
EY (Ambient 71F)
VDC-C1 U29 (24V) slight vibration, all temps below 75F : U33 (TCS 12V) slight vibration, all temps below 75F
VDC-C2 No Vibrations, all temps below 75F
WP12034 h1calcs mitigation divide-by-zero
Louis, JoeB, Jeff, Dave:
A new h1calcs model was installed. DAQ restart was required:
++: slow channel H1:CAL-CS_TDEP_OPT_GAIN_EXPORT_CNT_PER_M added to the DAQ
--: slow channel H1:CAL-CS_TDEP_OPT_GAIN_SCALE removed from the DAQ
WP12030 Test ITMY HWWD
Erik, Dave:
We tested the ITMY HWWD unit:
Disconnected the top satellite amp cable
Noted that the unit went into LED alarm state
Reconnected the cable
Verify LEDs are all green
One item of note, after the sat amp cable was disconnected the PD went RED for 1 second, then both PD and LED were green again for 1 second, then the LED went RED. From that point onwards LED=RED, PD=GREEN.
DAQ Restart
Dave:
The DAQ was restarted for the h1calcs model change.
Tue13Aug2024
LOC TIME HOSTNAME MODEL/REBOOT
11:01:13 h1oaf0 h1calcs <<< model restart
11:06:17 h1daqdc0 [DAQ] <<< 0-leg restart
11:06:30 h1daqfw0 [DAQ]
11:06:31 h1daqnds0 [DAQ]
11:06:31 h1daqtw0 [DAQ]
11:06:39 h1daqgds0 [DAQ]
11:07:32 h1daqgds0 [DAQ] <<< 2nd GDS0 restart needed
11:14:48 h1daqdc1 [DAQ] <<< 1-leg restart
11:14:59 h1daqfw1 [DAQ]
11:15:00 h1daqnds1 [DAQ]
11:15:00 h1daqtw1 [DAQ]
11:15:08 h1daqgds1 [DAQ]
J. Oberling, F. Clara
Summary: We think the 9.5Hz peak is coming from the new flow meters installed as part of the 2021 PSL laser upgrade.
Today we continued hunting for the 9.5Hz comb triplet that Ansel had tracked to the PSL back in May. Robert subsequently used a temporary magnetometer and found a strong 9.5Hz feature in the neoLASE LVEA Control Box. To try to isolate the source of this 9.5Hz peak we shut down the PSL in a controlled manner and watched the peak on the magnetometer. The order of the shutdown:
There was no change in the peak observed during the PSL shutdown, so we next turned off the neoLASE Control Box; the peak clearly disappeared with the Control Box off. Our next step was to check if the peak was being fed to the Control Box via something plugged into it. We turned the Control Box back on and the peak returned (as expected), then we unplugged the cable that feeds the flow/temperature sensor signals to the Control Box (via the PSL Sensor Interface Chassis (SIC), which feeds the Control Box with several monitor signals (PDs, power meters, flow meters) and the system shutters). Lo and behold, the peak was much smaller, sometimes disappearing entirely (see attached pictures; first is with the PSL off but the flow meters plugged in, second is with the flow meters unplugged). We next took the top off of the SIC and placed the magnetometer near where the flow meter signals both come into and exit the SIC, and did not see the peak. Fil then placed the magnetometer on the flow sensor connectors themselves (the flow sensor cables route to a pig tail on the back of the SIC, which goes from an individual M12 connector for each flow meter into a DB25 connector to plug into the SIC) and the 9.5Hz peak was clearly visible again. If we unplugged the main flow meter cable from the back of the Control Box the peak again disappeard; this makes sense as the Control Box supplies the 24 VDC the flow meters need, and unplugging the cable removes power from the flow meters. We then put the top back on the SIC and placed the magnetometer back in its spot directly underneath the Control Box and plugged the flow meters back in; the peak again returned. Fil disconnected the individual M12 connectors for the flow meters and the peak again disappeared. We then plugged each individual flow meter in to observe how the peak behaved (I did not take pictures of this, unfortunately). With 1 flow meter plugged in the peak was visible but much smaller; with 2 plugged in the peak was even smaller (???); with three plugged in the peak was back to its usual magnitude. With this odd behavior we suspect a grounding issue somewhere. To finish things off we unplugged the other items feeding signals into the Control Box (shutters, PDs/power meters, NPRO, Diode Box A, Diode Box B), one at a time, to observe how/if the peak changed; we observed no change in the peak during this process, the only thing that changed the peak with the Control Box turned on was unplugging the flow meter cable. We think this fairly definitively tags the flow meters as the source for the 9.5Hz peak seen in Robert's magnetometer, and likely the source of the comb triplet seen around this frequency.
We are now looking through documentation to see how we can mitigate this. One option is to tie the grounds of all 3 flow meters together and see if this changes the peak; this requires running long cables from each flow meter to tie them all at the same spot. We also plan on using the spare Control Box to check into how the internal grounding of the Beckhoff terminal is done and see if there's any improvements we can make there. Another option is to isolate power to the flow meters; currently they are powered by the Control Box, but we could power them separtely and see if that changes the peak. Investigation continues, more to come.
The PSL is now back ON, and the PMC and FSS RefCav are locked; I'm letting the system warm up for a bit before locking the ISS. This closes LHO WP 12031.
Edit to add: When unplugging the flow meters individually via their M12 connectors, we did try each of the 3 flow meters by themselves; the peak was the same regardless of which individual flow meter was plugged in. Things got weird when we had 2 plugged in at the same time (the peak was smaller than with any of the individual flow meters plugged in, hence why we suspect a grounding issue somewhere), and the peak returned to its starting magnitude when all 3 were plugged in at the same time.
The bake for the RGA on the end door for HAM6 has started, turned on at 16:00 utc.
RGA system at 150 oC, cal leaks are open, time start 00:01:00 utc 08/14/2024.
During the Tuesday maintenance reboots, zotvac0 did not shutdown properly, meaning the regular 08:00 did not run. Erik got the machine back and I rescheduled today's fill for 08:20
Tue Aug 13 08:29:17 2024 INFO: Fill completed in 9min 13secs
Gerardo confirmed a good fill curbside.
7min 20sec into the fill (1min 48sec before fill completion) TC-B failed. Its temperature reading went from -52C to +1372C
The fill completed using TC-A. In the attachment, the TC-B line disappears due to it being an inverted log plot.
TC-B was replaced following this failure. One of the leads was broken and the other had very little insulation remaining, so it was swapped with a new one. I wrapped TC-A together with TC-B with electrical tape in the most damage-prone area at the bend entering the exhaust line.
I also closed FRS 30032 which requested that this TC be swapped.