TITLE: 08/06 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: n/a
SHIFT SUMMARY:
Big DARPA tour, Relay Tube was installed, and other prep work for pumping down, and then Les Guthman did some filming work.
LOG:
EJ, Marc, Jonathan, Erik, Dave:
EJ noted that the Dolphin PCIe cards located at the endstations and read out by h1cdsrfm in the corner are running much warmer than the rest of the Dolphin cards.
h1cdsrfm has 3 Dophin IX cards on its PCI bus, one is plugged into its local PCIe bank, the other two are in Adnaco expansion chassis which are located at the end stations, and are accessed by a 4km single mode fiber link connected to local adnaco adapter cards.
The local Dolhin card, cooled by h1cdsrfm's fans, runs at 62C. The EY card runs at 74C and the EX card runs at 90C. The end station temps are significantly higher than the manufacturer's recommendation.
The Adnaco expansion chassis were inspected at both end stations:
both chassis side mounted, resting on a shelf
both chassis have their cover installed
both chassis fan is operational
The only difference between the two end stations is the PCIe slot the Dolphin card is installed in. At EY (74C) it is in the slot closest to the fan (closest to center of chassis), at EX (90C) it is the slot furthest from the fan (closest to the edge of chassis).
The chassis has one airflow outlet, the fan itself blowing outwards at the rear. There are several airflow inlets: circular hole pattern on top lid, center of the chassis; rectangular hole pattern on top lid, directly above the four PCIe slots; series of slots on left side, from front of chassis to half way back; ditto on right side.
At 14:06 at EY we put a plate on the top lid which blocks the circular hole pattern. Now air comes in from the holes directly above the PCIe slots, and the left and right slot array. This cooled the Dolphin card down from 74C to 65C in 30mins.
At 15:00 at EX we put a plate on the circular pattern, and blocked the right side slot array with tape. Now input air is directly above the PCI connectors and on the left side array only (the one closest to the cards)
Attached trend shows EX and EY temps over this period, started with 90C,74C and now at a steady 76C,66C
We removed the high voltage bypass which turns off the high voltage power to Ham 6, Ham 7, and the ESD drivers to prepare for pumpdown.
Janos, Marc, Richard
The PCAL End Y Shutter stopped working.
I had Fil check it out and he simply replaced the shutter motor with RS Pro part number 238-9709.
Fil showed me how to replace the motors that go bad in them.
I went out to EY to replace the shutter today, ran a few functionality tests for local operation and remote opperation. It is working well.
Closing FRS ticket 31730
FAMIS Link: 26003
Only CPS channels which looks higher at high frequencies (see attached) would be:
Attached are monthly TCS trends for CO2 laser and HWS. Where one can see where H1 went down roughly just over 3-weeks ago. (FAMIS link)
Tue Aug 06 08:06:55 2024 INFO: Fill completed in 6min 51secs
The ext-alert computer was updated and rebooted.
Workstations were updated and rebooted. This was an OS packages update. Conda packages were not updated.
TITLE: 08/06 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: MAINTENANCE
Wind: 3mph Gusts, 1mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY:
HAM5 was sealed up last night so work to begin pumpdown will be on the docket as well as Maintenance Day today. Also have a big tour for DARPA group this morning & the shift started with prrep for that.
Both +X and -X doors have been installed on HAM5. The inner O-rings at both locations have been replaced, as they were in a substantially bad shape (this was a pre-planned operation). No other issues have been encountered.
DaveB, RickS
Dave changed the image on the contrl room PSL screen (Nuc 21) and at the Laser Room to show a larger image of the PSL table.
Laser Room make-up air fan speed set to 100%. Lights are OFF, but will be switched on in the morning.
I changed the make-up air speed back to its nominal setting of 20% at ~19:15 UTC (12:15 PDT) on 8/7/2024. The make-up air being on did cause the Laser Room temperature to drop by a couple degF, which in turn caused the output power of both Amp1 and Amp2 to drop by ~2W each; will monitor over the next several hours to ensure things return to normal as the enclosure returns to its normal thermal equilibrium.
TITLE: 08/05 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Nobody
SHIFT SUMMARY:
IFO is in MAINTENANCE for VENTING
Today, the main mission was to put the HAM5 doors back on, and this has been accomplished.
Side Missions:
A reminder that the LHO Vent Plan (a living document so alive it's hyperventilating) can be found in the DCC: LIGO-E2400260
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 14:37 | FAC | Kim, Karen | LVEA | N | Technical Cleaning | 15:02 |
| 15:29 | VAC | Travis | LVEA | N | Turbopump Check | 15:37 |
| 15:48 | PCAL | Tony | LVEA | N | FOM Display Check | 15:51 |
| 15:49 | VAC | Travis | LVEA | N | Purge air check | 16:19 |
| 17:23 | PCAL | Richard +2 | LVEA, PCAL | N | Checking on new displays | 17:43 |
| 17:37 | SUS | RyanC | CR | N | ETMX OPLEV charge measurement | 18:37 |
| 19:51 | SUS | Ryan C | EX MR | N | Resetting VFD Panel | 20:45 |
| 20:00 | FAC | Eric | LVEA | N | HAM5 Doors | 22:23 |
| 20:01 | FAC | Chris | LVEA | N | HAM5 Doors | 22:23 |
| 20:01 | FAC | Tyler | LVEA | N | HAM5 Doors | 22:18 |
| 20:07 | VAC | Travis | LVEA | N | HAM5 Doors | 21:07 |
| 20:23 | VAC | Gerardo | LVEA | N | HAM5 Doors | 22:59 |
| 20:44 | PEM | Genevieve, Sam | LVEA | N | Accelerometer Check | 21:29 |
| 20:49 | VAC | Richard | LVEA | N | Gate Valve Air Leakage Check | 21:15 |
| 21:06 | OFI | Betsy | LVEA & Optics lab | N | Working on the OFI optics project. | 21:12 |
| 21:22 | SQZ | Shiela, Eric, Nergis | LVEA | N | SQZ Table Lookaround | 22:01 |
| 21:24 | FAC | Richard | LVEA | N | Y b manifold bangboard check | 22:18 |
| 21:30 | SUS | Betsy | LVEA | N | Looking for things | 23:29 |
| 21:43 | TOUR | Tony, Sabby + 3 | LVEA | N | Family Tour | 22:01 |
| 22:28 | SQZ | Camilla | Optics | Y | SQZ Work | 23:28 |
| 22:47 | OPS | Oli, Jenne | FCTE | N | Tour Tour | 23:08 |
| 22:48 | PEM | Genevieve, Sam | LVEA | N | Accelerometer Search | 23:48 |
ETMX's ISI and HEPI both tripped off at 19:41, seemed like a glitch.
Since this happened, Ryan C went into EX to reset VFD Panel, which allowed HEPI ETMX to stop tripping, but we noticed the T240s railing right after and then switching between extremes - this happened at 13:12 UTC. Screenshot below.
As of 13:31 PT (20:31) UTC, it looks like the T240 counts are coming back down.
A look at the sensors during this trip.
Was epics set point for the hepi pump beckhoff set to zero before resetting the vfd? If not, the t240 saturations are probably caused by the changing pressure in the hepi fluid while the pump servo ramps to operating pressure. Normally just resetting the pump vfd without zeroing out the set point would trip the pump, but maybe we got lucky.
Power glitch Monday 19:41:21 UTC.
Caught by MSR UPS, corner station mains_mon DQ channels for all three phases shown.
H1:HPI-PUMP_EX_CTRL_LOC_SETPT was set to zero but H1:HPI-PUMP_EX_CTRL_RMT_SETPT was still set to 70. I noticed the pressure came back on its own without me having to use the controller, that setpoint must be why. I'll make sure to zero it first next time.
I took the ETMX OPLEV charge measurement, it's been about a week since Rahul updated the L3 HV mon alog79350.
The charge appears to be flatlining now, and even just starting to decrease towards zero on some quads/DOF (UR_Y, LR, LL_P).
A slow motion video of the fast shutter, firing with the high voltage driver in air, is here: slow motion video
Sheila's original video is ~2sec footage of fast shutter, supposedly at 150 fps according to the file metadata.
Attached is an edited video that shows the 50ms of FS action slowed down by a factor of 10. "SRC TC" is the time elapsed since the start of (Sheila's original) video multiplied by a factor of 10, i.e. 1 sec in SRC TC is actually 100ms. This timestamp is not to be trusted at the lowest digit of sub-second (ms level in reality) because there's some frame interpolation going on due to mismatch between the standard output frame rate of 24FPS VS the source frame rate (150 slowed down to 15 FPS). Anyway, the mirror goes down lower than the closed position right after the initial thrust of the HV fast shutter motion, and even a few bounces after that, before the shutter eventually settles to the permanently closed position.
Also attached is an annotated png file of four frames ("Open", "HV thrust", "Bounce to the bottom" and "Close").
In light of recent failure of the replacement FS at LLO where the mirror broke off due to the hard stops that were set too low, and a comment by Rich (LLO alog 73584) I watched the video of the LHO unit again (see my entry above).
Though LHO video was shot from the side where we cannot see the position of the hard stop, the bobbin might be hitting the stops right after the launch even though the height of the stops looks good to my eyes. See TC~00:00:05:06. And the bobbing oscillates several times though it never hits the stops again.
Does this mean that the magnetic braking is too weak?
Noticed that GV5 had hard closed on Saturday at 19:59 utc time, no human intervention. It appears as if the instrument air pressure left for soft close GV5 was enough to hard close it. According to PEM instruments near GV5 made lots of noise, see attached plot, accelerometer and microphone data.
I checked the dial on the gauge to measure the instrument air at GV5, and indeed there was only about 10 psi, this value is within the "usual" pressure used for soft clossing pneumatic gate valves and per procedure.
No interaction is required at this moment from the vacuum team.
Tagging ISC, since sensitive equipment is hanging near this gate valve, like cameras and OpLevs.
I converted this microphone DQ signal to an audio wav file. The microphone saturates at +/-32768 for a fraction of a second, but you can hear the echo fade away after this.
Here is the accelerometer as a wav audio:
(new file added at 13:52, orig file was created using the incorrect frequency of 16kHz, this is actually a 8kHz channel)
The ITMY Oplev (aligned at the time) didn't see this hard close, plot attached.
Remember GV1 is closed, this makes ITMY Oplev blind, Sum on trend above is very small.