The new corner station scroll pump air compressor was valved in at 09:39 PDT this morning. I has gone through 3 compression cycles since then. Here are the stats
I have reset the cell phone alarms with the levels: LOW = 80PSI, HIGH = 120PSI
The new compressor runs at a higher pressure so the alarms needed to move with that. I changed the minor and major high alarms for H0:VAC-MR_INSTAIR_PT199_PRESS_PSIG for 127 and 130 respectively in epics via the vacuum computer in the back of the control room. These alarm levels cannot be changed outside of the vacuum network. I did not see this channel in the VAC SDF.
Patrick will check if these values are hard coded into Beckhoff and will adjust accordingly.
Currently the slow-controls SDF is not able to monitor non-VAL fields (e.g. alarm fields). Future release will have this feature.
I made the following changes to the PLC code generation scripts (not pushed to the running code) to enable setting the alarm levels for each instrument air pressure EPICS channel separately. I set the alarm levels to what they currently are. Should any of these be changed? https://git.ligo.org/cds/ifo/beckhoff/lho-vacuum/-/commit/8d17cfe3284147b4e398262091ec9d9f2ddbb6ab If there are no objections, I will plan to push this change when we make the changes to add and rename filter cavity ion pump controllers.
Conda packages updated. Controls rooom tools are updated to version 4.1.4. The only change is a fix to the Python foton module.
FilterDesign::get_zroots() would throw an exception when called. Thanks to Chris Wipf for finding and fixing the problem.
Thu Aug 15 08:08:59 2024 INFO: Fill completed in 8min 55secs
TITLE: 08/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 10mph Gusts, 7mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY: Pumpdown continues.
TITLE: 08/14 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY: The pump down continues (5e-7).
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
15:24 | FAC | Karen | LVEA | n | Tech clean | 16:32 |
16:00 | FAC | Kim | LVEA | n | Tech clean | 16:15 |
16:01 | FAC | Nellie | HAM Shack | n | Tech clean | 17:01 |
16:17 | FAC | Karen | EY | n | Tech clean | 18:17 |
16:33 | FAC | Kim | EX | n | Tech clean | 17:57 |
16:40 | VAC | Gerardo | LVEA | n | HAM5/6 work with RGA and pump | 17:52 |
17:47 | VAC | Janos | LVEA | n | Walkabout | 17:54 |
17:52 | VAC | Gerardo | EX | n | Cart hunt | 18:35 |
17:56 | FAC | Karen | MY | n | Tech clean | 18:35 |
17:58 | FAC | Kim | H2 WH | n | Tech clean | 18:34 |
18:35 | - | Camilla, Detchar (8) | LVEA | n | Tour | 19:18 |
20:10 | VAC | Chris | LVEA | n | Grab straps for bake ovens | 20:16 |
20:19 | VAC | Janos, Gerardo, Travis | Opt lab | n | Viewport inspection | 21:16 |
21:43 | TCS | Camilla | Opt Lab | Local | Cheeta testing |
ongoing |
21:49 | CDS | Marc | MY | n | Looking for parts | 22:20 |
21:49 | VAC | Chris | LVEA | n | Putting straps back | 22:09 |
Yesterday Sam and I went around the LVEA with a portable accelerometer (H1:PEM-CS_ADC_5_28_2K_OUT_DQ) to investigate a ~35.4 Hz peak. This peak dates back to at least June 2023, possibly longer. We placed the portable accelerometer in seven different spots, each time against a wall except for one spot near BS1. See the attachment 1 for the reference map. The peak appears strongest at REF0 and REF5 as seen in attachment 2. We attempted to test outside near the Mitsubishis (outside the OSB kitchen) and the southeast-facing mechanical room wall as well but had data retrieval issues, so we'll go back to that soon.
We are also investigating the ~36 Hz peak seen in attachment 2, strongest near the X OL. It seems to have been around since at least 2023-05-08, but more investigation is needed to find out when it started and if it shows up consistently every day. The frequency of this peak oscillates on a range of about 1 Hz. Attachment 3 shows the peak in H1:PEM-CS_LVEAFLOOR_XCRYO_Z_DQ and H1:GDS-CALIB_STRAIN_CLEAN. This peak also travels between 30-40 Hz, generally near 40 Hz at the beginning and end of the UTC day and dropping towards 30 Hz in the middle of the day, as seen in attachment 4 from the summary pages.
WP12035
Yesterday thermocouple B (TC-B) failed during the fill. Its wire broke and its temperature railed at +1372C. The fill proceded with TC-A to completion.
If TC-A had similarly failed, the fill would have continued until either the discharge line pressure exceeded its trip pressure; or the fill timed out after 30 minutes.
Today's code modification adds a check when reading the TCs that they are not greater than the nominal_max temps (+40C).
Providing at least one TC is in operational range the fill will continue, but now if no TCs are operational the fill will abort with an error.
The new code was installed at 9am today, it will run for the first time during Thursday's fill.
During this investigation I noted that the discharge_line_pressure trip is 2.0PSI. Operationally this pressure has never exceeded 0.6PSI so I lowered the trip down to 1.0PSI.
Wed Aug 14 08:08:37 2024 INFO: Fill completed in 8min 34secs
New TC-B looking good.
TITLE: 08/14 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 2mph Gusts, 0mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY: The pumpdown continues.
The PCAL team went down to EY with PS4 to do an End station measurement.
We mostly followed the procedure , but I did do something that may be of interest.
Beam spot is looking slightly oblong but not too bad. I did not touch up the beam spots.
The martel calibrated voltage source was set to the "mV range" today instead of the usual "V range" while inputing the "zero votage" into the DAQ.
This may present us with a more consistent value of "Zero Voltage". This auxilary analysis is pending.
The normal ES trends analysis is already done and can be found here: https://git.ligo.org/Calibration/pcal/-/tree/master/O4/ES/measurements/LHO_EndY/tD20240813?ref_type=heads
Beam spot after we left still looked a little long.
Installed a new MKS gauge at cross C-4, to do the installation I closed two gate valves for the filter cavity tube, FCV-6 and FCV-9, these valves isolated the volume I was working on and left a gauge that gave me the vacuum pressure in the area that I was working, the pressure was actively monitored, no changes noted.
To install the gauge a metal valve and a T were added to the configuration, the system was pumped down and leak tested, and no leak was noted above 1.0x10-10 torr*l/sec.
To finish the installation for now, the new volume was pumped down, and remains isolated from the main volume. Then FCV-6 and FCV-9 were open.
TITLE: 08/13 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
SHIFT SUMMARY: Light maintenace today while we continue to pump down. One DAQ restart for CAL model updates, and some other work around the site.
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
14:37 | ISC | Corey | LVEA | n | ISC table enclosure checks | 15:16 |
15:15 | PCAL | Tony, Dripta | PCAL lab | local | prep meas | 15:22 |
15:16 | ISC | Corey | EY, EX | n | ISC table enclosure checks | 15:49 |
15:18 | CDS/PSL | Fil, Jason | LVEA | n | Glitch hunting at PSL racks | 17:51 |
15:20 | FAC | Karen, Kim, Nellie | FCTE | n | Tube cleaning | 17:51 |
15:23 | PCAL | Tony, Dripta | EY | Yes | PCAL measurement | 18:00 |
15:37 | VAC | Gerardo | LVEA | n | Open HAM7 ion pump | 17:33 |
15:46 | CAL+DAQ | Dave | remote | n | New h1calcs model, DAQ restart | 19:31 |
17:03 | CDS | Marc | CER, EX,MX,MY,EY | n | Power supply checks | 18:08 |
17:14 | CDS/PSL | Richard | LVEA | n | Check in with Fil and Jason | 17:33 |
17:16 | VAC | Gerardo | FCES,FCTE | n | Leak detector move | 19:04 |
18:03 | REC | Christina | OSB Receiving | n | Throwing stuff in the recycling bin w/ forklift | 19:33 |
18:08 | - | Mike, tour | LVEA | n | Tour | 19:08 |
20:07 | FAC | Chris | LVEA | n | Grabbing barrel | 20:37 |
20:41 | PEM | Genevieve, Sam | LVEA | n | Moving an accelerometer around | 22:16 |
20:44 | CDS | Dave | CER | n | Check EY HWWD | 21:14 |
20:45 | VAC | Janos | LVEA | n | Checking pumps | 21:45 |
22:41 | VAC | Gerardo | LVEA | n | HAM6 RGA bake temp increase | 22:53 |
I checked the Kepco power supplies located on site and noted the following per WP12033:
CER Mezanine (Ambient 69F)
C1 - U38 RHS slight vibration noted, 89F front, 115F back, good airflow, supplies SUS_C8 (18V, 5A)
C2 - U29 slight vibration noted, otherwise all temps below 75F
C3 - No vibrations, all temps below 75F
C4 - No vibrations, all temps below 75F
C5 - U5 slight vibration noted, all temps below 75F *same supply had vibration last check in May ALOG77832
C6 - No vibrations, all temps below 75F
MER (Ambient 70F)
VDC-M1 - No Vibrations, all temps below 75F
TCS-M1 - No Vibrations, all temps below 75F
EX (Ambient 69F)
VDD1 - No Vibrations, all temps below 75F
VDD2 - No Vibrations, all temps below 75F
MX (Ambient 71F)
Power Rack - No Vibrations, all temps below 75F
FCAV (Ambient 70F)
H1-VDC-C1 - No Vibrations, all temps below 75F
MX (Ambient 71F)
Power Rack - No Vibrations, all temps below 75F
EY (Ambient 71F)
VDC-C1 U29 (24V) slight vibration, all temps below 75F : U33 (TCS 12V) slight vibration, all temps below 75F
VDC-C2 No Vibrations, all temps below 75F
WP12034 h1calcs mitigation divide-by-zero
Louis, JoeB, Jeff, Dave:
A new h1calcs model was installed. DAQ restart was required:
++: slow channel H1:CAL-CS_TDEP_OPT_GAIN_EXPORT_CNT_PER_M added to the DAQ
--: slow channel H1:CAL-CS_TDEP_OPT_GAIN_SCALE removed from the DAQ
WP12030 Test ITMY HWWD
Erik, Dave:
We tested the ITMY HWWD unit:
Disconnected the top satellite amp cable
Noted that the unit went into LED alarm state
Reconnected the cable
Verify LEDs are all green
One item of note, after the sat amp cable was disconnected the PD went RED for 1 second, then both PD and LED were green again for 1 second, then the LED went RED. From that point onwards LED=RED, PD=GREEN.
DAQ Restart
Dave:
The DAQ was restarted for the h1calcs model change.
Tue13Aug2024
LOC TIME HOSTNAME MODEL/REBOOT
11:01:13 h1oaf0 h1calcs <<< model restart
11:06:17 h1daqdc0 [DAQ] <<< 0-leg restart
11:06:30 h1daqfw0 [DAQ]
11:06:31 h1daqnds0 [DAQ]
11:06:31 h1daqtw0 [DAQ]
11:06:39 h1daqgds0 [DAQ]
11:07:32 h1daqgds0 [DAQ] <<< 2nd GDS0 restart needed
11:14:48 h1daqdc1 [DAQ] <<< 1-leg restart
11:14:59 h1daqfw1 [DAQ]
11:15:00 h1daqnds1 [DAQ]
11:15:00 h1daqtw1 [DAQ]
11:15:08 h1daqgds1 [DAQ]
The bake for the RGA on the end door for HAM6 has started, turned on at 16:00 utc.
RGA system at 150 oC, cal leaks are open, time start 00:01:00 utc 08/14/2024.
During the Tuesday maintenance reboots, zotvac0 did not shutdown properly, meaning the regular 08:00 did not run. Erik got the machine back and I rescheduled today's fill for 08:20
Tue Aug 13 08:29:17 2024 INFO: Fill completed in 9min 13secs
Gerardo confirmed a good fill curbside.
7min 20sec into the fill (1min 48sec before fill completion) TC-B failed. Its temperature reading went from -52C to +1372C
The fill completed using TC-A. In the attachment, the TC-B line disappears due to it being an inverted log plot.
TC-B was replaced following this failure. One of the leads was broken and the other had very little insulation remaining, so it was swapped with a new one. I wrapped TC-A together with TC-B with electrical tape in the most damage-prone area at the bend entering the exhaust line.
I also closed FRS 30032 which requested that this TC be swapped.