Jonathan, Dave:
The DAQ detail MEDM has been updated to show FW2's progress within the 64 second cycle for full frames, 600 second cycle for second trends and 3600 second cycle for minute frames.
When each progress bar reaches the end, the next data accumulation phase starts and the frame file writing begins.
A new run_number column has been added, along with a LED stack checking these all agree with each other.
The retransmission column has been removed.
As a test, I've added a second bar for FW2 showing the time it took to write the previous frame as a diamond. If the time to write exceeds the bar's span, e.g. > 64 seconds for a full frame, I have verified that a half diamond is shown on the right margin.
Morning dry air skid checks, water pump, kobelco, drying towers all nominal.
Dew point measurement at HAM1 , approx. -42C
h1daqfw2 is now producing identical raw, second trend, and minute trend frames to h1daqfw[01]. The issues mentioned in https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=84132 have been resolved. There are times where we expect there to be mismatches. These are generally when data is missing, the daqd may replay recent data, where the new frame writer will write zeros. We will continue tests on the frame writer, including software updates. The next test is to watch the behavior of h1daqfw2 when h1daqfw0 & h1daqfw1 have a restart. It is designed to automatically adjust its channel list and have no gaps in the frames and need no restart to accommodate the changes.
Randy, Mitchell, Tyler, TJ, Betsy, Oli, Camilla. WP#12444, WP#12496, moved away from HAM2 in 83686.
We attempted to move IOT2L back into place on Friday but the cleanroom had been moved into the way of the final table position for ISI work and one of the casters of the table was stripping rubber off one side and kept rubbing so we paused.
This morning the cleanroom was moved out of the way (+X by ~1-2 feet) and then Randy finished moving IOT2L into place with the help of the forklift. Once IOT2L's corners were over the markings made in 83296 and the height was correct, we re-attached the bellows, removed the guillotines and replaced the guillotine slot covers. Photos attached. The HAM2 VP furthest to -Y never had a guillotine slot cover, so one was added.
Mon Apr 28 10:08:46 2025 INFO: Fill completed in 8min 42secs
Closes FAMIS 26454. Last checked in alog 83629.
No trends above or blow thresholds.
TITLE: 04/28 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 8mph Gusts, 6mph 3min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.11 μm/s
QUICK SUMMARY:
IFO is in PLANNED ENGINEERING for the vent
Planned activities today:
Sun Apr 27 10:08:59 2025 INFO: Fill completed in 8min 55secs
We had a slight power glitch which was reported by the GC OSB UPS (but not CDS MSR UPS).
Attached trend shows this was seen by h1pemcs in the CER.
Sat Apr 26 10:08:27 2025 INFO: Fill completed in 8min 24secs
Following the SWWD trip of HAM1 earlier this week I realized I had not added HAM1's SWWD to the CDS overview watchdog section. I took this opportunity to also improve this section. New features are:
As before the SWWD section shows the HAMs on the left in approximate relative locations for HAM1-HAM8 and the BSCs on the right.
This ADL file is now generated by the python code /opt/rtcds/userapps/release/cds/h1/scripts/generate_watchdog_medm.py
The attachment shows the current situation with all of HAM1 bypassed.
TITLE: 04/25 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: The HAM1 work continued today, the RMs are back in.
LOG:
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
14:50 | FAC | Kim | LVEA | N | Tech clean | 17:42 |
14:50 | FAC | Nellie | LVEA | N | Tech clean | 17:43 |
16:15 | SUS | Rahul | LVEA HAM1 | N | RM1 Cabling Check | 19:15 |
16:27 | Oli | LVEA | n | Check if Rahul needs help | 16:43 | |
16:55 | SUS | Jim | LVEA | N | HAM2 L4C 3 Inspection | 18:37 |
16:55 | FAC | Richard | LVEA | N | HAM1 Cable Work | 19:36 |
16:55 | SUS | Betsy | LVEA | N | Cable Work | 19:33 |
16:56 | FAC | Randy | LVEA | N | Cable work and misc | 19:56 |
17:32 | ISC | Oli, Camilla | LVEA | N | ISC HAM1, Camillla out @ 19:28 | 19:30 |
17:48 | VAC | Gerardo | LVEA | N | Vacuum Checks | 18:08 |
17:53 | CDS | Dave | FCES | N | Power Cycling HAM8 (FC2, ISI) | 19:54 |
18:04 | PCAL | Francisco | PCAL | N | Integrating sphere cover | 18:14 |
18:05 | EE | Fil | LVEA, MY | N | Inventory | 19:54 |
18:08 | PEM | RyanC | EndY | N | Dust monitor flow check/test | 19:05 |
19:33 | SUS | Rahul | Remote | N | RM1 & 2 measurements | 20:12 |
20:58 | SUS | Rahul | LVEA | N | HAM1, RM checks | 21:41 |
20:58 | SUS | Betsy | LVEA | N | Help with HAM1, checks | 21:35 |
21:00 | ISC | TJ, Oli, Camilla, Raed | LVEA | N | Put IOT2L back in place | 21:41 |
21:03 | ISC | Keita | Optics lab | N | Search for parts/optics | 21:13 |
21:09 | SAF | Richard | LVEA | N | Check with HAM1 crew | 21:35 |
23:22 | VAC | Gerardo | LVEA | N | Preweekend checks | 23:45 |
Attached is a snapshot of the Trello from this last week and into next showing what has been checked off. Major milestones met of ISI in correct location (IAS survey complete), 90% cables installed and mostly tested, nearly all IO/ISC components landed on tables with the exception of diodes which are WIP. A fair amount of troubleshooting, hunting and fixing of ground loop and cable issues in and out of vac. Next week we get serious about letting the PSL beam in and aligning the paths.
Jonathan, Dave:
Jonathan has his new h1daqfw2 frame writer running and writing byte identical frame files compared to production. I have extended the DAQ DETAIL MEDM to perform a check between all three frame writers, warning if their frames differ in size or content.
The attached MEDM shows all three frame writers with identical full and second trend frame files, but with FW2 having written a different minute trend file. This is because FW2 was started during the past hour and therefore wrote an incomplete file.
FW2 is currently not in production, which is shown by its error/warnings being indicated in YELLOW instead of RED.
After another hour and a full minute trend frame being written there are differences between the h1daqfw[01] and h1daqfw2 frames minute trend frames. Looking at H-H1_M-1429650000-3600.gwf there are 767 channels with differences. All of the differences are .mean channels. I randomly sampled a small number of these channels. The difference that I saw in all the channels I looked at was that h1daqfw[01] had set the mean value to -0.0, and h1daqfw2 had computed the mean to be 0.0 (no minus sign). Since having the sign bit set means the underlying bytes are not all zeros the compression isn't as nice and the h1daqfw[01] frame is slightly larger. Seeing differences like this is not unexpected. This is a different implementation and different implementation technology so floating point rounding errors are an issue. My goal is still to be able to get this to be byte for byte identical output. Here is a dump of a minute trend channel that shows the difference (using framecpp_dump_channel). from h1daqfw0: DEBUG: Processing channel: H1:SUS-MC2_M3_MASTER_OUT_LL_DQ.mean DEBUG: Trying as ADC DEBUG: Is a ADC Name: Compress: 257 (Endianness: Little Scheme: 1) Type: 2 NData: 60 NBytes: 22 Data: -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0 from h1daqfw2: DEBUG: Processing channel: H1:SUS-MC2_M3_MASTER_OUT_LL_DQ.mean DEBUG: Trying as ADC DEBUG: Is a ADC Name: Compress: 266 (Endianness: Little Scheme: 10) Type: 2 NData: 60 NBytes: 8 Data: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
Oli, Daniel, Fil, Betsy, Rahul
We have re-installed SUS RM1 and RM2 Tip Tilt suspensions into HAM1 chamber which has the new ISI table - see several pictures attached below. RM1 is on the left (towards PSL side) hand side of this picture and RM2 is on the right (zoomed in). Both the suspensions are positioned roughly as per D1000313_V19, and will be fine tuned (pitch and yaw of the mirror as well) during beam alignment work. The blade springs of both the suspensions have been un-muted - see LHO alog 83984 for details.
Both RM1 and RM2 have been plugged-in to their electronics chain. We had to perform some troubleshooting with RM1 after observing grounding issues. At first Fil and Daniel ruled out any issues with the satellite box and in-air cables. After some investigation, the grounding issue was suspected to be coming from the in-vacuum quadrupus cable (connecting to the Bosem), which is an old cable and looks worn out too.
This morning I replaced the old Bosem cable (D1000228_v3 s/n S1105240) with a 25in long (D2100049 S/N 2200358) cable. Before connecting it to the Bosem, Daniel and I verified that the cable is fine (i.e not grounding) and later on we connected each pin (Bosem) one by one. We had to re-adjust the Kapton tape (installed for additional shielding) since the cable bosem connector was grounding to the chamber. After all these modifications, we can confirm that RM1 and RM2 are free from any grounding issues for now. We will check again before closing the chamber.
Next, I will start taking measurements on these suspensions and will post their health report soon.
Things to do on RM1 (hardware side),
1. Find and replace the new cable (25in long D2100049 S/N 2200358) ) which has just been installed with another one since this one is missing pin no. 1.
2. replace the old style cable table bracket which is currently in HAM1 for RM1 with a new one for the new style of cable. The old style cable brackets don't fit.
Also, adding a picture of the chamber with labels showing RM1 and RM2 suspension.
Electronics testing of both the sus is still ongoing.
Adding a reminder here that the sign of both RM1 and RM2 OSEM readbacks have changed, after the discovery that "more light" = "more negative signal" in LHO:84027.
Tagging EPO with the cool new digs inside HAM1 and on the new ISI.
WP12488 HAM8 ISI binary readback broken following site outage FRS33798
Fil, Erik, Jim, Dave:
Yesterday we power cycled the binary input/output and interface chassis, which did not fix the issue. Today we stopped all the models on h1cdsh8, powered down the computer, then power cycled the IO Chassis.
This also did not fix the problem. Investigation continues.
Fri25Apr2025
LOC TIME HOSTNAME MODEL/REBOOT
11:15:32 h1cdsh8 ***REBOOT***
11:17:25 h1cdsh8 h1iopcdsh8
11:17:38 h1cdsh8 h1isiham8
11:17:51 h1cdsh8 h1susfc2
11:18:04 h1cdsh8 h1sqzfces
11:18:17 h1cdsh8 h1susauxh8
11:18:30 h1cdsh8 h1pemh8
Oli, Matt, Camilla
Late entry, Jim and I worked the last few half days to lace up all of the cabling for SUS, SEI, and IO/ISC in prep for the main reinstall of table components. Fil, some Richard, and I did preliminary ground loop checks on all SUS and IO/ISC cables so we have a handle on the situation before complicating with components. I routed cables on the table top in an attempt to be near where things plug in but also not in the way of landing objects. Further dressing and coiling will be needed after troubleshooting plugins with components as we go. Pics of the pre round of cabling.
Oli, Camilla
This morning we swapped the direction of HW1 so that it is facing the same as before. We also placed L2, M15, LSC POP A and ASC POP X although the diodes are not cabled up yet. They can be next week as Daniel confirmed chassis are powered off.
ASC POP X is new, the box is marked D1102004-v6 S/N 016 S1300637, ICS link. Have added this to the D1000313 BOM googledoc and HAM1 ICS assembly.
Tagging with EPO to document the process.
Richard powered up h1seih23 from the front panel, it is now running.
We power cycled h1psl0, it is back.
Erik reconfigured EX Dolphin front ends to communicate with port 8 of the EY Dolphin switch to get them to get the models going. CDSRFM at EX points to port 7 of EY's switch.
All models are running, only IPC errors are long-range-dolphin.
Should someone from operations start activating the SEI and SUS systems?
After the power outage, I went out to the site ~10pm Pacific to check status of purge air skid. Kobelco, water pump and drying towers were all running upon arrival. I verified the dew point was still ok (-41.8C measured at HAM1) and verified the drying towers switched normally. Note: Purge air was only connected to HAM1 since doors are off, corner volume is vented but not actively purging with no doors off.
While there I also restarted the controller for 3 of the FCT ion pumps and reset solenoid valve on aux cart connected to GV5 annulus. All looks ok. Left site ~10:40pm.
h1guardian1 did not reboot, here is the guardian MEDM
Systems currently down as of 20:49:
FMCS/BACNET
Corner Station HEPI Pump Controller
PSL Diode Room slow controls
CP1 overfill
NCAL
BRS
HWS
DTS test stand
To continue running vac alarms overnight I have bypassed the currently active alarms for the next 24 hours.
Bypass will expire:
Mon Apr 7 09:28:54 PM PDT 2025
For channel(s):
H0:FMC-CS_CY_H2O_PUMPSTAT
H0:FMC-CS_CY_H2O_SUP_DEGF
H0:FMC-CS_FIRE_PUMP_1
H0:FMC-CS_FIRE_PUMP_2
H0:FMC-EX_CY_H2O_PUMPSTAT
H0:FMC-EX_CY_H2O_SUP_DEGF
H0:FMC-EY_CY_H2O_PUMPSTAT
H0:FMC-EY_CY_H2O_SUP_DEGF
H1:CDS-DTS_ENV_TEMPERATURE_DEGF
H1:CDS-STAT_NUMBER_OF_ERRORS
Regarding the 4UHV ion pump controller, the autostart setting is enabled now, thus the unit should power ON after a power glitch. See some of the instructions to access the general menu.