Displaying reports 1981-2000 of 83394.Go to page Start 96 97 98 99 100 101 102 103 104 End
Reports until 08:37, Wednesday 09 April 2025
H1 CDS
david.barker@LIGO.ORG - posted 08:37, Wednesday 09 April 2025 - last comment - 17:10, Wednesday 09 April 2025(83828)
MX weather station wind speed sensor not working

Starting around 22:13 Tue08apr2025 the MX weather station wind speed anemometer stopped recording. The other MX sensors, for example outside temperature, continue to run.

Attachment shows wind speed and temp for EX top row, and for MX bottom row.

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 08:46, Wednesday 09 April 2025 (83829)
david.barker@LIGO.ORG - 17:10, Wednesday 09 April 2025 (83846)

Literally minutes after I made this alog, unbeknownst to me, it fixed itself. I drove to MX this afternoon and visually confirmed the paddles were turning and appared to be unrestricted. The freeze between 10pm Tue and 9am Wed is still a mystery.

Images attached to this comment
H1 General
oli.patane@LIGO.ORG - posted 07:40, Wednesday 09 April 2025 (83827)
Ops Day Shift Start

TITLE: 04/09 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: MAINTENANCE
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.21 μm/s
QUICK SUMMARY:

Plans for today:

LHO VE (VE)
travis.sadecki@LIGO.ORG - posted 18:24, Tuesday 08 April 2025 - last comment - 18:49, Tuesday 08 April 2025(83825)
2025 April vent - VAC diary

Today's activities:

Comments related to this report
jordan.vanosky@LIGO.ORG - 18:49, Tuesday 08 April 2025 (83826)

Purge air dewpoint measured at YBM port of entry during BSC8 door removal. Dew point = -42C

Images attached to this comment
H1 AOS (SEI, SYS)
jason.oberling@LIGO.ORG - posted 18:16, Tuesday 08 April 2025 - last comment - 17:03, Wednesday 09 April 2025(83823)
WHAM1 Passive Stack Optical Table Pre-Deinstall Measurements (WP 12442)

J. Oberling, R. Crouch

Today we took pre-deinstall measurements of the position of the optical table surface of the WHAM1 passive stack.  The plan was to use the FARO to measure the coordinates of several bolt holes, using a threaded nest that locates the Spherically Mounted Retroreflector (SMR) precisely over the bolt hole, on both the +Y and -Y side of the chamber.  This, unfortunately, did not happen in full due to the untimely death of the FARO's climate sensor (or the FARO's ability to read the climate sensor, we're hoping for the former).  The FARO cannot function without this sensor as it relies on accurate measurements of the air temperature, relative humidity, and air pressure to feed into a model of the refractive index of air, which it needs to accurately calculate the SMR distance from the FARO.  We did manage to get a few points measured before the sensor died.  I've reached out to FARO tech support about getting a new climate sensor and should hear back from them tomorrow (they usually replay in 1 business day).

Summary

We were able to get measurements of 3 bolt holes, all in the furthest -Y line of bolt holes, and an old IAS monument from aLIGO install before the FARO's climate sensor died.  The results are listed below under the Results heading.  The most interesting thing here is there appears to be an error in WHAM1 placement in the x-axis, as the bolt holes we measured are all ~37.25 mm too far in the -X direction from nominal.  We also set a scale on the wall across from the -Y door of the WHAM1 chamber that is registered to the current elevation of the optical table; placing an autolevel so it sights 150.0 mm on this scale (sighting the side of the scale with the 0.5 mm tick marks) places that autolevel 150.0 mm above the surface of the passive stack's optical table.

Details

We started on the -Y side of the WHAM1 chamber.  The FARO was set with a good view of its alignment monuments and the passive stack's optical table.  We ran through the startup checks and calibrations without much issue (we did see a return of the odd 'ADM Checks Failing' error, which had been absent for about 1 month, but it immediately went away and didn't come back when we performed a Go Home operation).  FARO monuments F-CS026 through F-CS035, inclusive, were used to align the FARO to the LHO LVEA local coordinate system; the 2 standard deviation device position uncertainty after this alignment was 0.016 mm (PolyWorks does 100 Monte Carlo simulations of the device position).  This complete, we started measuring.

First, as a quick test of the alignment we took a look at old IAS monument LV24.  This monument was used to align the WHAM2 ISI during aLIGO install, and its nominal X,Y coordinates are [-20122.0, -3050.7] mm (there is no z-axis coordinate as we were not setting these in Z back then, a separate set of wall marks was used for z-axis alignment).  The results are shown in the 1st attached picture; again, ignore the z-axis results as I had to enter something for the nominal or PolyWorks wouldn't accept the entry, so I rounded to the closest whole number (this isn't even the surface of the monument, it's the point 2" above it where the SMR was, due to use of the Hubbs Center Punch Nest (which has a 2" vertical offset when using a 1.5" SMR)).  Knowing how we had to set these older monuments, since I'm one of the people that set them, I'm not entirely surprised by the X and Y deviations.  The monuments we set for aLIGO install (the LV monuments) were placed w.r.t. a set of monuments used to align iLIGO, which themselves were placed w.r.t. the monuments used to install the vacuum equipment during facility construction (the PSI monuments), which themselves were placed w.r.t. the BTVE monuments which define the interface between the arm beam tubes and the LVEA vacuum equipment, which we then found errors in their coordinates during our alignment of the FARO during the O4a/b commisioning break in 2024.  Not at all surprised that errors could have stacked up without notice over all of those monuments set off of monuments set off of monuments set off of...  Also, take note of the x-axis coordinate of this monument, this will be important later.

We then set about taking measurements of the passive stack optical table.  To map the bolt holes we measured we used an XY cartesian basis, assuming the bolt hole in the -X/-Y corner was the origin.  We then proceeded to increment the number by the bolt hole (not distance), following the same XY axis layout used for the IFO.  Using this scheme the bolt holes for the table corners were marked as:

We were able to get measurements for bolt holes (0,0), (14,0), and (25,0).  We were in the process of measuring bolt hole (36,0) (the +X/-Y corner bolt hole) when the FARO's climate sensor died.

To get the coordinates for the bolt holes I used the .EASM file for WHAM1 with the passive stack configuration located at D0901821-v4.  From the assembly, using eDrawings, I was able to get coordinates w.r.t. the chamber origin for the bolt holes we measured.  Those were then added to the coordinates for the WHAM1 chamber, in the LVEA local coordinate system, to get nominal coordinates for the bolt holes.  I also had to add 25.4 mm to the z-axis coordinates to account for the 1" offset of the nest we were using for the SMR; the center of the SMR sits 1" above the point being measured, so I needed to manually add that offset to the nominal z-axis coordinate of the bolt hole.  For reference, according to D0901821 the global coordinates for WHAM1 are [-22692.0, 0.0, 0.0] mm; when converted to the LVEA local coordinate system (removing the 619.5 µrad downward tilt of the X-arm) this becomes [-22692.0, 0.0, +14.1].  The measurement results are shown in the 2nd attached picture.  Notice those x-axis deviations?  Remember the measurement we made of LV24?  Clearly the FARO alignment is not 37 mm off, as the measurement of LV24 showed, so something is definitely up with the x-axis coordinate of the WHAM1 chamber (error in chamber placement?  aLIGO WHAM1 is the iLIGO WHAM2 chamber, moved from its old location next to WHAM3).

Results

We can do some analysis of the numbers we have, although limited since we only have 3 points in a line.  This really only applies to the furthest -Y line of bolt holes on the table, since we weren't able to get measurements of the +Y side to get a more full picture of where the table is sitting, but it's something.  Position tolerances at install in 2012 were +/-3.0 mm in all axes.

I do want to note that D0901821-v4 claims the table surface should be -187.8 mm in LVEA local coordinates (-201.9 mm in global), but this is not the number we used when installing the passive stack in 2012.  In 2012 we used -185.9 mm local (-200.0 mm global), as can be seen in D0901821-v2.  To compare our measurements to the install numbers I changed the nominal z-axis coordinate to match that of our install target (-185.9 + 25.4 mm SMR offset = -160.5 mm) and the results are shown in the final attached picture.

Wall Scale Registered to Current Table Surface Elevation

To finish, we set a scale on the -Y wall directly Crane East of the WHAM1 chamber and registered it to the current elevation of the passive stack's optical table.  To do this we used a scale provided by Jim (the scale was in inches, with 0.01" tick marks) and an autolevel.  We set the autolevel at a fixed elevation on the -Y side of the chamber.  The scale was then placed at each corner of the optical table, starting with the -X/+Y corner, and the autolevel was used to sight the scale; only the scale was moved, the autolevel was fixed (rotated only to follow the scale, but not moved otherwise).  We then averaged the 4 scale readings to get the table elevation, set the autolevel to this reading with the scale back at our starting point (we actually didn't have to move it, thankfully), and then set a scale on the wall using the autolevel.  The 4 scale readings:

The average of the 4 readings is 5.9", and since the autolevel was already sighting 5.9" on our starting point at the -X/+Y corner we left it there.  This may seem high, but we had to have the autolevel high enough that we could see over the various components mounted to the table surface.  We then turned the autolevel and set a scale on the wall.  This scale was in mm (since that's what we had), but this worked out OK.  5.9" is ~149.9 mm (149.86 mm to be exact), so we set the wall scale so it read ~149.9 mm when sighted through the autolevel.  So a 150.0 mm reading on this scale (sighting the side with the 0.5 mm tick marks) is ~150.0 mm above the current position of the passive stack's optical table.

This closes LHO WP 12442.

Images attached to this report
Comments related to this report
jason.oberling@LIGO.ORG - 17:03, Wednesday 09 April 2025 (83842)

TJ O'Hanlon informed me via email that there indeed was an error in the x-axis coordinate at both LHO and LLO, due to the thickness of the septum between HAM1 and HAM2 not being taken into account, which had not been propagated to all of the SYS mechanical layout drawings (and some of the CAD files as well).  I had completely forgotten about this, and explains why we had moved the WHAM1 passive stack monument LV25 further in the -X direction some time back in 2012; the first attached picture shows this (the clear cut out next to the existing monument was the old position of LV25 before we moved it).  I went spelunking through my old 2012 emails to find some communication about this, but all I could find was an email chain re: LLO setting the LHAM6 support tubes and not being able to get them in the proper y-axis position.  Dennis replied that this was due to the septum thickness and would apply to HAM1 and HAM6 at both sites, and that he would update E1200625 with the correct coordinates for all involved chambers.  From E1200625 the x-axis coordinate of WHAM1 should be -22726.7 mm, so I have updated the PolyWorks project with this new, correct coordinate; this is shown in the 2nd attached picture.

From this I can now say that the -Y row of holes on the WHAM1 passive stack's optical table are ~2.56 mm too far in the -X direction.  If we were to use the FARO to survey monument LV25 my guess is that would explain the 2.5 mm error, seeing as how nearby LV24 was also ~2.0 mm too far in -X direction.  As stated in the main alog this difference doesn't exactly surprise me given the "monuments placed off of monuments placed off of monuments" situation we have here.  The FARO was aligned to our X and Y axes using monuments PSI-1, PSI-2, PSI-6, and BTVE-1, so any error between these 1st and 2nd generation monuments and the 4th generation LV monuments will be measured by the FARO.

While I was at it I went ahead and applied the required transform for local to global coordinates.  This is done by creating a new coordinate system and applying the requisite tilt of both the X and Y axes.  The tilt must be entered in degrees and for the opposite axis.  This is because our, for example, y-axis tilt angle w.r.t. local gravity is a rotation of the x-axis.  Since PolyWorks works off of axis rotation, we enter the y-axis angle as an x-axis tilt (same for the x-axis angle).  To get PolyWorks to correctly calculate the transform matrix both values should be entered as positive numbers (I'm not entirely sure why).  The values to enter:

  • X-axis rotation: 0.0000125 µrad -> 0.0007162°
  • Y-axis rotation: 0.0006195 µrad -> 0.0354947°

The calculated transform matrix is shown in the 4th attached picture, which properly matches Table 10 in T980044 (note, the numbers in the transform matrix are in radians, even though I had to enter the rotations in degrees).  To confirm this was correct I manually calculated the correct global z-axis coordinate using the formula in Section 2.3 of T0900340 for each bolt hole; the results were the same between my calculation and PolyWorks'.  The final picture shows the bolt hole survey in the LHO global coordinate frame.

Images attached to this comment
H1 SEI
jim.warner@LIGO.ORG - posted 17:01, Tuesday 08 April 2025 - last comment - 11:59, Wednesday 09 April 2025(83824)
Dial indicators added on -X of HAM1, 2/4 HEPI actuators disconnected

This morning, we added dial indicators to the ends of the support tubes to monitor the support tube locations on the -X side. We had to remove a cable tray to get room to do this under the chamber. They are protected under the chamber, but there is a bunch of loose cabling on that side that could bump my set up. Unfortunately, we don't have access to the northside. Not quite sure how to track the support tube ends on the +X side when we float HEPI yet.

While Betsy and crew were pulling parts in chamber, Mitch and I were disconnecting actuators at the piers. Vertical actuators are quite difficult, access is very tight, going to be hard reconnecting them. Horizontal actuators are a lot easier, but we are adding .100" shims and bolts to protect the bellows and IPS, we lost a couple shims inside one of the actuators and it was a struggle to fish them out. We are adding wires to the shims going forward to prevent that mishap. We'll get the other 2 actuators tomorrow. So far dial indicators show we are still in the same spot as before disconnecting the actuators.

Comments related to this report
jim.warner@LIGO.ORG - 11:59, Wednesday 09 April 2025 (83834)

All 4 actuators are disconnected now, we'll move to passive stack de-install after lunch. Dial indicators we have move .002", but that could very well just be drift in my set up.

H1 ISC
elenna.capote@LIGO.ORG - posted 16:45, Tuesday 08 April 2025 - last comment - 16:19, Monday 21 April 2025(83820)
First Half of HAM1 table cleared

[Betsy, Camilla, Elenna, Oli]

We cleared half of the HAM1 table today on the +y side. This cleared optics on the ALS and POP paths, and the first part of the optics on the REFL path. We carefully labeled each component, and Oli logged component and cable names with serial numbers. Betsy laid the components in clean pans lined with cleanroom cloths. The cables are still attached to the feedthroughs and were left lying at the bottom of the chamber.

Attached photos show cleared side of the table.

Before removing components, we reviewed the table layout after Ibrahim and TJ noticed some discrepancies between the solidworks drawing and the optic locations as depicted in Corey's pictures from yesterday. We confirmed that there are some dispcrepancies between the two. TJ has a more detailed report.

Images attached to this report
Comments related to this report
camilla.compton@LIGO.ORG - 16:48, Tuesday 08 April 2025 (83822)

Before we started, Betsy and I replaced the septum plate VP cover we had removed yesterday 83798

Layout before: D1000313-v15

camilla.compton@LIGO.ORG - 16:19, Monday 21 April 2025 (84035)EPO

Added photo of myself, Betsy, Melina and Elenna before the HAM1 ISC removal work started. 

Images attached to this comment
H1 General
oli.patane@LIGO.ORG - posted 16:38, Tuesday 08 April 2025 (83821)
Ops Day Shift End

TITLE: 04/08 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:

More vent work today. I was able to mark some things fully or partially done, but others I'm unaware of their current status

DONE            FAC - 2nd Cleaning BSC8
DONE            Investigate large shifts in trends of PRM and PR2 in YAW (83807, 83808)
DONE            IAS - HAM1 Stack table survey
PARTIALLY   JIM/MITCH - HAM1 Offload HEPI Setup, disconnect actuators
PARTIALLY   ISC - HAM1 - ISC component removal
PARTIALLY   VAC/FAC - BSC8 Door Off - CRANE to Caddy
                       VAC - Start Feedthru Swap
                       EE - HAM4-5 Cable tray work - Ken, some Crane usage
                       IAS - HAM1 support tube ends?


LOG:                                                                                                       

Start Time System Name Location Lazer_Haz Task Time End
14:44 FAC Kim, Nelly LVEA n Tech clean (HAM1, restock) 17:26
14:56 FAC Eric HAM Shack n Annual maintenance 16:38
15:10   Betsy LVEA n Taking to FAC 15:25
15:23 VAC Jordan LVEA N Dew point check 15:36
15:34 IAS Jason, RyanC LVEA n FAROing 18:57
15:51 VAC Jordan, Melina EX n Checking on ion pump 16:22
15:52   Betsy OpticsLab n Prepping to recrimp cables 16:55
15:53 FAC Tyler LVEA, HAM Shack n Checking on things 19:07
16:25 EE Marc CER n Grabbing label maker 16:35
16:58   Corey, Mitchell MY n Grabbing tooling 17:28
17:01 SUS Betsy LVEA N BSC8 Dust Count Check 17:20
17:02 FAC Chris, Pest Control LVEA n Pest control 18:06
17:03   TJ Mech Room n Checking out dust monitors 17:10
17:08 ISI Jim LVEA n Checking if HAM1 is ready 17:45
17:12   TJ LVEA n Checking dust monitor 17:44
17:50 FAC Kim, Nellie EX N Technical cleaning 18:43
17:54 SEI Jim, Mitchell LVEA n HAM1 work 18:01
18:00 VAC Jordan, Melina, Travis LVEA n Start removal of HAM6 turbo pump 19:16
18:01   Mitchell, Jim HAM Shack n Looking for parts 18:15
18:06 FAC Chris LVEA N Equipment check 18:57
18:12 PEM Robert LVEA n Talking to VAC 18:34
18:13   TJ LVEA N Dust monitor test 18:24
18:15 SUS Jim, Mitchell LVEA N HEPI Offload 19:37
18:26 ISC Betsy, Camilla LVEA n Prepping for stuff removal at HAM1 19:15
19:07   Josh, Jeff, +2 LVEA n Tour 19:46
19:42 FAC Travis, Jordan, Randy LVEA N BSC8 Door move 00:19
20:18 ISC Betsy, Elenna, Camilla LVEA n Checking HAM1 layout discrepencies 23:20
20:25   Richard LVEA n Walkabout 22:28
20:52 FAC Tyler, Alpine roofing EY n Looking at roof 21:44
20:58 EE Marc CER n labeling 21:44
20:59   Mitchell, Jim LVEA n HAM1 feedthrough 23:27
21:42   TJ LVEA n Talking about discrepencies 21:58
22:01 VAC Travis MX/MY N Conflat blank 22:29
22:04 FAC Ryan C LVEA N Dust monitor check 22:38
22:29   Oli LVEA N HAM1 deinstall help 23:20
22:30 VAC Travis LVEA N Turbo Swap 00:30
22:30 FAC Richard LVEA N Walkabout + Headcount 22:44
22:49 FAC Tyler, Water contractor Water Tank N Continuing potable water restoration work 00:49
23:35   RyanC LVEA n Checking on dust monitor 00:05
H1 PEM
ryan.crouch@LIGO.ORG - posted 16:19, Tuesday 08 April 2025 (83817)
Dust monitor trend monthly - FAMIS

Closes FAMIS37251

There's some known issues right now with the dust monitors. We can still see LAB1 not writting to the network, EY died for about a day, LVEA5 not writting anything for a few days, TJ swapped this one today with one from staging. The counts were not exactly as expected across both particle sizes, so I'm going to test this one a little more.

I brought LVEA5 over to LVEA10 in the biergarten and tested them side by side and saw similar results, I also did the zero count test on LVEA5 which it passed.

Images attached to this report
H1 SUS
oli.patane@LIGO.ORG - posted 14:23, Tuesday 08 April 2025 - last comment - 15:32, Tuesday 08 April 2025(83818)
SR3 M1 SUS comparison between all DOFs

Jeff asked me to plot a comparison for SR3 M1 between all degrees of freedom comparing it in vacuum versus in air. I've plotted the last two measurements taken for SR3 from last August at the end of the OFI vent. One measurement was taken in air, and the other was taken in vacuum The pressure for the in vacuum measurement wasn't all the way down to our nominal, but as Jeff said in his alog at the time when we were running these measurements: "most of the molecules are out of the chamber that would contribute to buoyancy, so the SUS are at the position they will be in observing-level vacuum" (79513).

Non-image files attached to this report
Comments related to this report
jeffrey.kissel@LIGO.ORG - 15:32, Tuesday 08 April 2025 (83819)CSWG, SEI, SYS
Calling out the "interesting" off-diagonal elements:
                 D R I V E   D O F 
          L     T     V     R     P     Y

     L    --    nc    nc    meh   eand  YI
 
R    T    nc    --    YI    eand  nc    meh
E 
S    V    meh   YI    --    meh   nc    YI
P
     R    VI    esVI  VI    --    YI    VI
D 
O    P    esVI  VI    YI    meh   --    YI
F
     Y    YI    nc    nc    nc    nc   --

Here's the legend to the matrix, in order of "interesting":
  VI = Very Interesting (and unmodeled); very different between vac and air.
esVI = Modeled, but Still Very Interesting; very different between vac and air
  YI = Yes, Interesting. DC response magnitude is a bit different between vac and air, but not by much and all the resonances show up at roughly the same magnitude.
 meh = The resonant structure is different in magnitude, but probably just a difference in measurement coherence
eand = The cross coupling is expected, and not different between air and vac.
  nd = Not Different (and unmodeled). The cross-coupling is there, but it doesn't change from air to vac.
I've bolded everything above "meh" to help guide the eye.

Recapping in a different way, because the plots are merged in a really funky order,
  VI = L to R (pg 14), 
       T to P (pg 22),
       Y to R (pg 33)

esVI = T to R (pg 16)
       L to P (pg 20)

  YI = L to Y (pg 28), Y to L (pg 27),
       T to V (pg 12), V to T (pg 11),  
       V to P (pg 24),
       P to R (pg 25), 
       Y to V (pg 31),
       Y to P (pg 35)


What a mess! 
The matrix of interesting changes is NOT symmetric across the diagonal
The matrix has unmodeled cross-coupling that *changes* between air and vac
For the elements that are supposed to be there, (like L to P / P to L and T to R / R to T), the cross coupling different between air and vacuum.
For some elements, the cross-coupling is *dramatically worse* at *vac* than it is at air.

Why is there yaw to roll coupling, and why is it changing between air and vacuum??

There's clearly more going on here than just OSEM sensor miscalibration that the Stanford team found with PR3 data in LHO:83605. These measurements are a mere 8 days apart!

The plan *was* to use SR3 as a proxy during the vent to test out the OSEM estimator algorithm they were using to improve yaw, but ... with this much different between air and vac, I'm not so sure the in-air SR3 measurements to inform an estimator to be used at vacuum.
H1 General (EPO)
corey.gray@LIGO.ORG - posted 13:04, Tuesday 08 April 2025 - last comment - 13:21, Tuesday 08 April 2025(83790)
HAM1 BEFORE SEI De-install / ISI Install Photos + Contamination Control Tasks

HAM1 Before Photos:  (HAM1 chamber open just under 90min for this activity)

This morning before the deinstall activities begin, took the opportunity to photo document the HAM1 optical layout.  Keita requested I take photos to record the layout wrt to the iLIGO bolt pattern, because rough alignment of optical components on the new SEI ISI for HAM1 will be done utilizing the bolt patterns of the Optics Tables; so I took a few more photos than normal (top view and angled with a focus on the REFL path).  Took large photos with the Canon 60D DSLR camera as well as my camera phone.  

The photos are being populated in this Google Drive folder:  https://drive.google.com/drive/folders/1yDKp7aByA_TYJ12c8j8BnZM_pd1Q2DBZ?usp=sharing

Naming each photo referencing an updated layout Camilla Compton made which labels all beam dumps, but I also had to use an older layout to preserve naming since the layout on HAM1 currently looks like D1000313v16 (which is also referenced for naming the photos).

The above folder has the Canon photos, and I'll be adding the camera phone images next.

Contamination Control Notes:

Images attached to this report
Comments related to this report
jeffrey.kissel@LIGO.ORG - 13:21, Tuesday 08 April 2025 (83816)ISC, SEI, SUS, SYS
Tagging ISC, SUS, SYS, and SEI. Rest in power HAM1 Stack!
LHO FMCS
eric.otterman@LIGO.ORG - posted 12:47, Tuesday 08 April 2025 (83815)
Mitsubishi A/C low ambient hood enabling
The Mitsubishi condensing units at the H2 and MSR have had their low ambient hoods enabled for the season. Due to the large ambient temperature swings that occur in Spring, this may cause less uniform temperature trending in the MSR until the overnight temperatures moderate. 
LHO FMCS
eric.otterman@LIGO.ORG - posted 12:42, Tuesday 08 April 2025 (83814)
FCES annual air handler maintenance
The annual air handler maintenance was done at the FCES this morning. No issues have been observed since. 
H1 CDS
david.barker@LIGO.ORG - posted 11:41, Tuesday 08 April 2025 (83812)
CDS Recovery Status

The DAQ's EDC is green again. Erik and Camilla started the HWS EX and EY IOCs which were the last needed to complete the set.

Daniel and Patrick confirmed that the two Beckhoff terminals lost at EX are the AOS baffle PDs, which are not immediately needed but will be needed to return to O4. For now I have "greened up" the CDS overview by configuring DEV4 to expect 125 terminals of 127 total.

To remind us that DEV4 is in a degraded state, and that DEV1 was degraded in Dec 2024 when it lost its illuminators, these are shown with a darker green block.

Images attached to this report
LHO General
tyler.guidry@LIGO.ORG - posted 08:08, Tuesday 08 April 2025 - last comment - 11:54, Tuesday 08 April 2025(83803)
Water Leak Repair
C&E Trenching arrived yesterday morning and began work on uncovering/locating the potable water leak. Ultimately, the leak was in very close proximity to where I first noticed the persistently wet pavement. It was directly beneath secondary high voltage lines that feed the chiller yard among other things. The cause of the leak ended up being a small crack impregnated with a rock near a smashed/deformed section of pipe. There is no clear explanation as to how the pipe was damaged in the first place. The trenching team suggested that the pipe may have been damaged just before/during installation. Interestingly enough, this section of pipe was found to have two prior repairs made to it as seen in the photo with the two pipe couplers right next to each other. Based on the date printed on the pipe, these repairs were made during construction some time in 96'.

Important note: with the potable supply valved out at the water room, I replenished the holding tank to a level of 57.8" read at FMCS. Once the pipe repair had been made, I reintroduced the potable supply and the tank level fell to 52.5". In conjunction with this, the Magemter gauge prior to line repair read xx6171 gallons. Post repair the gauge read xx6460. However, I don't have much confidence in the Magmeter gauge readout in this scenario as the line turbulence causes some egregious (-250gpm+) reverse flow readings while the line recharges.

After repair, and keeping staff water usage suspended, I held the line pressure at some 80psi for 30 minutes or so and observed no leaks. There were also no drops in system pressure nor was there any flow readout at the magmeter gauge - both important and reassuring improvements.

R. McCarthy T. Guidry
Images attached to this report
Comments related to this report
corey.gray@LIGO.ORG - 11:54, Tuesday 08 April 2025 (83813)EPO

Tagging EPO for the photos!

H1 SUS (CDS, SYS)
jeffrey.kissel@LIGO.ORG - posted 15:03, Monday 07 April 2025 - last comment - 13:20, Tuesday 08 April 2025(83787)
Recovery from 2025-04-06 Power Outage: +18V DC Power Supply to SUS-C5 ITMY/ITMX/BS Rack Trips, ITMY PUM OSEM SatAmp Fails; Replaced Both +/-18 V Power Supplies and Replaced ITMY PUM OSEM SatAmp
J. Kissel, R. McCarthy, M. Pirello, O. Patane, D. Barker, B. Weaver
2025-04-06 Power outage: LHO:83753

Among the things that did not recover nicely from the 2025-04-06 power outage was the +18V DC power supply to the SUS ITMY / ITMX / BS rack, SUS-C5. The power supply lives in VDC-C1 U23-U21 (Left-Hand Side if staring at the rack from the front); see D2300167. More details to come, but we replaced both +/-18V power supplies and SUS ITMY PUM OSEMs satamp did not survive the powerup, so we replaced that too.

Took out 
    +18V Power Supply S1300278
    -18V Power Supply S1300295
    ITMY PUM SatAmp S1100122

Replaced with
    +18V Power Supply S1201919
    -18V Power Supply S1201915
    ITMY PUM SatAmp S1000227
Comments related to this report
jeffrey.kissel@LIGO.ORG - 13:20, Tuesday 08 April 2025 (83810)CDS, SUS
And now... the rest of the story.

Upon recovery of the suspensions yesterday, we noticed that all the top-mass OSEM sensor values for ITMX, ITMY, and BS were low, *all* scattered from +2000 to +6000 [cts]. They typically should be typically sitting at ~half the ADC range, or ~15000 [cts]; see ~5 day trend of the top mass (main chain, M0) OSEMs for H1SUSBS M1,ITMX M0, and H1SUSITMY M0. The trends are labeled with all that has happen in the past 5 days. The corner was vented on Apr 4 / Friday, so that changes the physical position of the suspensions and the OSEMs see it. At the power outage on Apr 6, you can see a much different, much more drastic change. 

Investigations are rapid fire during these power outages, with ideas and guesses for what's wrong are flying everywhere. The one that ended up having fruit was that Dave mentioned that it looked like "they've lost a [+/-18V differential voltage] rail or something," -- where he's thinking about the old 2011 problem LLO:1857 where 
   - There's a SCSI cable that connects the SCSI ports of a given AA chassis to the SCSI port of the corresponding ADC adapter card on the back of any IO chassis
   - The ADC Adapter Card 's port has very small, male pins that can be easy bent if one's not careful during the connection of the cable.
   - Sometimes, these male pins get bent in such a way that the (rather sharp) pin stabs into the plastic of the connecter, rather than into the conductive socket of the cable. Thus, (typically) one leg, of one differential channel is floating, and this manifests digitally in that it creates an *exact*  -4300 ct (negative 4300 ct) offset that is stable and not noisy. 
   - (as a side note, this issue was insidious: once one bent male pin on the ADC adapter card was bent, and mashed into the SCSI cable, that *SCSI* cable was now molded to the *bent* pin, and plugging it in to *other* adapter cards would bend previously unbent pins, *propagating* the problem.) 

Obviously this wasn't happening to *all* the OSEMs on three suspensions without anyone touching any cables, but it gave us enough clue to go out to the racks.
Another major clue -- the signal processing electronics for ITMX, ITMY and BS are all in the same rack -- SUS-C5 in the CER.
Upon visiting the racks, we found, indeed, that all the chassis in SUS-C5 -- the coil drivers, TOP (D1001782), UIM (D0902668) and PUM (D0902668) -- had their "-15 V" power supply indicator light OFF; see FRONT and BACK pictures of SUS-C5.

Remember several quirks of the system that help us realize what's happened (and looking at the last page of ITM/BS wiring diagram, D1100022 as your visual aide):
(1) For aLIGO "UK" suspensions -- the OSEM *sensors'* PD satellite amplifiers (sat amps, located out in the LVEA within the biergarten) that live out in the LVEA field racks are powered by the coil drivers to which their OSEM *coil actuators* are connected.
So, when the SUS-C5 coil drivers lost a differential power rail, that makes both the coils and the sensors of the OSEM behave strangely (as typical with LIGO differential electronics: not "completely off" just "what the heck is that?"). 
(2) Just as an extra fun gotcha, all of the UK coil drivers back panels are *labeled incorrectly* so that the +15V supply voltage indicator LED is labeled "-15" and the -15V supply is labeled "+15".
So, this is why the obviously positive 18V coming from the rack's power rail is off, but the "+15" indicator light is on an happy.  #facepalm
(3) The AA Chassis and Binary IO for these SUS live in the adjacent SUS-C6 rack; it's + and - 18V DC power supply (separate and different from the supplies for the SUS-C5 rack) came up fine without any over-current trip. Similarly the IO chassis, which *do* live in SUS-C5, are powered by a separate single-leg +24V from another DC power supply, also coming up fine without over-current trip.
So, we had a totally normal digital readback of the odd electronics behavior.
(4) Also note, at this point, we had not yet untripped the Independent Software Watch Dog, and the QUAD's Hardware Watchdog had completely tripped. 
So, if you "turn on the damping loops" it looks like nothing's wrong; at first glance, it might *look* like there's drive going out to the suspensions because you see live and moving MASTER_OUT channels and USER MODEL DAC output, missing that there's no IOP MODEL DAC output. and it might *look* like the suspensions are moving as a result because there are some non-zero signals coming into on OSEMINF banks and they're moving around, so that means the damping loops are doing what they do and blindly taking this sensor signal, filtering it per normal, and sending a control signal out.

Oi.

So, anyways, back to the racks -- while *I* got distracted inventorying *all* the racks to see what else failed, and mapping all the blinking lights in *all* the DC power supplies (which, I learned, are a red herring) -- Richard flipped on the +18V power supply in VDC-C1 U23, identifying quickly that it had over-current-tripped when the site regained power.
See the "before" picture of VDC-C1 U23 what it looks like tripped -- the "left" (in this "front of the rack" view) power supply's power switch on the lower left is in the OFF position, and voltage and current read zero.

Turning the +18V power supply on *briefly* restored *all* OSEM readbacks, for a few minutes.
And then the same supply, VDC-C1 U23, over-current tripped again. 
So Richard and I turned off all the coil drivers in SUS-R5 via their rocker switches, turned on the VDC-C1 U23 left +18V power supply again, then one-by-one powered on the coil drivers in SUS-C5 with Richard watching the current draw on the VDC-C1 U23 power supply.

Interesting for later: when we turned on the ITMY PUM driver, he shouted down "whup! Saw that one!"
With this slow turn on, the power supply did not trip and power to the SUS-R5 held, so we left it ...for a while.
Richard and I identified that this rack's +18V and -18V power supplies had *not* yet had their fans upgraded per IIET:33728.
Given that it was functioning again and having other fish to fry, we elected to not *yet* to replace the power supplies.

Then ~10-15 minutes later, the same supply, VDC-C1 U23, over-current tripped again, again . 
So, Marc and I went forward with replacing the power supplies.
Before replacement, with the power to all the SUS-C5 rack's coil drivers off again, we measured the output voltage of both supplies via DVM: +19.35 and -18.7 [V_DC].
Then we turned off both former power supplies and swapped in the replacements (see serial numbers quoted in the main aLOG); see "after" picture.

Not knowing better we set the supplies to output to a symmetric +/-18.71 [V_DC] as measured by DVM. 
Upon initial power turn on with no SUS-R5 coil drivers on, we measured the voltage from an unused 3W3 power spigot of the SUS-R5 +/-18 V power rail, and measured a balanced +/-18.6 [V_DC].
Similar to Richard and I earlier, I individually turned on each coil driver at SUS-C5 while Marc watched the current draw at the VDC-C1 rack.
Again, once we got the ITMY PUM driver we saw a large jump in current draw. (this is one of the "important later")
I remeasured the SUS-R5 power rail, and the voltage on positive leg had dropped to +18.06 [V_DC].
So, we slowly increased the requested voltage from the power supply to achieve +18.5 [V_DC] again at the SUS-R5 power rail. 
This required 19.34 [V_DC] at the power supply.
Welp -- I guess whomever had set the +18V power supply to +19.35 [V_DC] some time in the past had come across this issue before.

Finishing up at the supplies, we restored power / turned to all the remaining coil drivers had watched it for another bit. 
No more over-current trips. 
GOOD! 

... but we're not done!

... upon returning to the ITMY MEDM overview screen on a CDS laptop still standing by the rack, we saw the "ROCKER SWITCH DEATH" or "COIL DRIVER DEATH" warning lights randomly and quickly flashing around *both* the L1 UIM and the L2 PUM COILOUTFs. Oli reported the same thing from the control room. However, both those coil drivers power rail lights looked fine and the rocker switches had not tripped. Reminding myself that these indicator lights are actually watching the OSEM sensor readbacks; if the sensors are some small threshold around zero, then the warning light flashes. This was a crude remote indicator of whether the coil driver itself had over-current tripped because again, the sensors are powered by the coil driver, so if the sensors are zero then there's a good chance the coil driver if off.
But in this case we're staring at the coil driver and it reports good health and no rocker switch over-current trip.
However we see the L2 PUM OSEMs were rapidly glitching between "normal signal" of ~15000 [cts] and a "noisy zero" around 0 [ct] -- hence the red, erratic (and red herring) warning lights.

Richard's instincts were "maybe the sat amp has gone in to oscillations" a la 2015's problem solved by an ECR (see IIET:4628), and suggest power cycling the sat amp. 
Of course, these UK satamps () are another design without a power switch, so a "power cycle" means disconnecting and reconnecting the cabling to/from the coil driver that powers it at the satamp. 
So, Marc and I headed out to SUS-R5 in the biergarten, and found that only ITMY PUM satamp had all 4 channels' fault lights on and red. See FAULT picture.
Powering off / powering on (unplugging, replugging) the sat amp did not resolve the fault lights nor the signal glitching.
We replaced the sat amp with a in-hand spare and fault lights did NOT light up and signals looked excellent. No noise, and the DC values were restored to their pre-power-outage values. See OK picture.

So, we're not sure *really* what the failure mode was for this satamp, but (a) we suspect it was a victim of the current surges and unequal power rails over the course of re-powering the SUS-C5 rack, which contains the ITMY PUM coil driver that drew a lot of current upon power up, which powers this sat-amp (this is the other of the "important later"); and (b) we had a spare and it works, so we've moved on with post-mortem to come later. 

So -- for all that -- the short answer summary is as the main aLOG says:
- The VDC-C1 U23 "left" +18V DC power supply for the SUS-R5 rack (and for specifically the ITMX, ITMY, and BS coil drivers) over-current tripped several times over the course of power restoration, leading us to
- Replace both +18V and -18V power supplies that were already stressed and planned to be swapped in the fullness of time, and 
- We swapped a sat-amp that did not survive the current surges and unequal power rail turn-ons of the power outage recovery and subsequent investigations.

Oi!
Images attached to this comment
Displaying reports 1981-2000 of 83394.Go to page Start 96 97 98 99 100 101 102 103 104 End