J. Kissel, for D. Barker Dave came in this morning to find that the LSC front-end computer had failed. Upon hard booting the computer and IO chassis, even though he tried his darnedest to gracefully take the computer out of the Dolphin Network, upon restart it broke all front-end processes in the corner station, and even killed the h1susham2 computer software entirely. We're now in the process of recovering.
details on this event:
Jeff, Corey, Fil, Dave:
h1lsc0 models were not running, the computer could not see its IO Chassis. In the CER, there was no activity reported near the ISC chassis at the time of crash (09:15 PST), we found the IO Chassis was fully powered.
On h1lsc0 I closed the models and issued the disconnect-from-dolphin command, and then powered down. On power up, h1lsc0 dolphin glitched the corner station, so all models except for sus-aux stopped running. Around the time h1ioplsc0 model started, h1sush2a froze up. Its console was blank and no sshd service. I waited for h1lsc0 to fully start up to verify its problem was transient.
I rebooted h1sush2a using the front panel reset button. As expected, when it came back it had glitched the dolphin fabric and taken down the lsc models.
I then restarted all models on all dolphin'ed corner station front end computers (except for h1sush2a which had just restarted). Procedure was /etc/kill_models.sh followed by /etc/start_models.sh. Restart order in attached restart log.
Once all the models were running again, I cleared the IPC and CRC errors.
Jeff and Corey are recovering the systems.
A small consolation is that the new Dolphin PCIe Gen2 hardware uses a switch-based method to disable ports that seems to be more robust.
Recovery complete -- see LHO aLOG 40580.
TITLE: 02/16 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
Wind: 26mph Gusts, 23mph 5min avg
Primary useism: 0.24 μm/s
Secondary useism: 0.17 μm/s
QUICK SUMMARY:
On this blustery morning a few activities already in action: EY BSC Arm removal, ISCT6 move prep, H1PSL work, cleaning at MY, & film crew around site.
0004 hrs. local -> Arriving on site to start CP4 watch attended "owl" shift. Working alone with Chandra R. as designated "phone buddy". Won't be doing much other than ensuring scroll pump doesn't shut down due to hydraulic lock as the result of the water vapor laden gas load as CP4 warms and off-gasses. Brought the camper-van so look for me to be at Y-mid VEA, my office or in my van (listening to 70's era rock or conservative radio :) Will make a comment to this entry to update CP4 status or when leaving the site.
As the attached graph indicates, the source of gas went away at about the same time the pressure alarm threshold was exceeded. To justify my visit, I did two iterations of 1)closing the turbo exhaust valve. 2)setting the foreline isolation valve setpoint to 950 Torr (maximum allowed value). 3)cracking open the foreline connection at the turbo exhaust valve (allows the scroll pump to pump viscous room air and blows out condensed water from between the two scroll stages). 4)restoring isolation valve setpoint (now 5 x 10-2 Torr). 0500 hrs. local -> Leaving site now.
attached alarm log for this event
Glad to be back in the safe zone! (for now)
Today we decoupled a section of the LN2 fill line inside mid-Y VEA. The male bayonets slid out relatively easy after removing the o-ring clamps. Gerardo is searching for vendor to provide a male bayonet "blank" so we can seal the pump side LN2 pipe extension in the event that we develop an air leak in bibraze joint and need to pump on the reservoir. For now I'm going to pick up a rubber plug from the hardware store for emergency back up.
The power supply for CO2 X laser was fixed (thanks Ed) so I was able to check the operational status of the laser table. The chillers had once again tripped from the PZT control loop sending to high of a value to the chiller setpoints. I disabled integrator, reset the history on the control loop, and restarted the chillers. Everything appeared to be in order on the table, so now both CO2 X and Y are running with beam dumps in place. Tomorrow I'll try and adjust the loops to find a more stable system.
More work on the ETMY suspension in BSC10 - this afternoon, Travis and I:
1) Hosted Barry Barish, Michael, and a film crew chamber-side for a filming project.
2) Added the balance of the sus payload to the cage - Sleeve structure, wedges, cross braces, vibration absorbers.
3) Installed the newest Bounce-Roll Dampers (BRDs) from Kissel (thanks!!). This took a careful hand holding the bitty assembly with needle-nose pliers and inserting them into the mass before tightening with an allen key. We also locked down the UIM and PUM since we were pushing on the UIM blades for this install.
4) unlocked masses, reset ETMY and TMSY watchdogs and turned damping on.
The alignment at first glance looks to still be on centerish. More refinement to finish up tomorrow.
J. Kissel, J. Driggers I continue to search for the at-vacuum violin mode resonances so we can determine the shift in frequency from the detailed in-air measurements (LHO aLOG 39163). This time, instead of looking at the PUM OSEMs and optical lever local sensors (as done in LHO aLOG 40019), I scoured data in the can from last week's arm peaking and DRMI locking, in hopes that an interferometric measurement would be more sensitive. Namely, I found a time when DRMI was locked (starting from the crumbs left in LHO aLOG 40463, and then trending the DRMI guardian lock state, the LSC trigger bits, and AS_C's normalized sum), 2018-02-08 ~21:20 UTC, and measured the ASD of MICH, PRCL, SRCL, and a few of the WFS and QPDs that have been traditionally sensitive to violin modes. Calibration details are below, but the message is that I've still found nothing. The most sensitive length measurement was MICH at ~8e-15 m/rtHz, and the most sensitive angle measurement was (surprisingly) the AS_C QPD at ~6e-12 rad/rtHz. The first four attachments are the calibrated ASDs, at various levels of zoom in frequency. %%%% Details %%%% LSC Signal Calibration Not so tough -- I grabbed the DRMI displacement wall FOM from /opt/rtcds/userapps/release/isc/h1/scripts/DRMI_FOM_displacements.xml (rev. 16689, last changed rev. 14505) and simply converted the pre-calibrated channels from [um] to [m]. ASC Signal Calibration (with guidance from Jenne) - Find a time when the ASC dither signals were on, gather the frequency, amplitude (e.g. H1:ASC-ADS_PIT2_OSC_FREQ, H1:ASC-ADS_PIT2_OSC_CLKGAIN), and which optic was driven. Some ugly combination of dataviewer and time-machine is needed for this search, because which oscillator used pointing to which optic is arbitrary and inconsistent with time. Note, that at this stage, the dither amplitude will be in peak counts of the sine wave at the given dither frequency, [ct_pk] @ f Hz. Here's what I found: DOF f [Hz] Amp [ct_pk] Driven Optic Time On Pitch 21.3 1000 SRM 2018-02-08 (21:31:46 - 21:32:46 UTC) Yaw 21.3 1000 SRM 2018-02-08 (21:33:46 - 21:34:46 UTC) (This was during the DRMI lock stretch from ~21:00 to 21:30 UTC) - Calibrate the dither amplitude into expected displacement ASD [rad_rms/rtHz] at the given optic. - Use the relevant subfunction of the python library of functions, /opt/rtcds/userapps/release/asc/h1/scripts/sensingMatrix/HY_version/DOF_lib.py In this case, SRM was being driven, so I used the PRM sub function, as it was most similar in suspension type. Note that means the attached ASC signal ASDs are calibrated into [rad] of SRM displacement. ]$ cd /opt/rtcds/userapps/release/asc/h1/scripts/sensingMatrix/HY_version/ ]$ ipython In [1]: import DOF_lib In [2]: pitch = DOF_lib.PRM_cts2rads(21.3,'PIT','TFs/H1_PRM.txt') In [3]: yaw = DOF_lib.PRM_cts2rads(21.3,'YAW','TFs/H1_PRM.txt') In [4]: pitch Out[4]: (4.6564401596417816e-14, -2.6756796108295753e-08) In [5]: yaw Out[5]: (2.9158641207661465e-14, -3.8069250094614154e-07) The script uses the EUL2OSEM gain, the DAC gain, the coil driver gain, the magnet strength, the flag-to-CoM distance, and a canned torque to angular displacement transfer function at the specified frequency to calculate the above gain. - The first output of the functions are what we need, a scalar calibration of radians of optic motion per dither drive count, i.e. [rad/ct]. So multiply that by the dither amplitude to get the expected peak displacement, pitch_rad_pk = 1e3 [ct_pk] * 4.66e-14 [rad/ct] = 4.66e-11 [rad_pk] yaw_rad_pk = 1e3 [ct_pk] * 2.92e-14 [rad/ct] = 2.92e-11 [rad_pk] - In converting from peak sine wave amplitude to rms amplitude spectral density, the dither lines were only on for a minute, so I need a short FFT -- I chose 0.1 Hz. With the standard DTT Hanning window, that's an effective noise band width (ENBW) of 0.187499. Thus, to get from peak to rms/rtHz, that's pitch_rad_rms_p_rtHz = pitch_rad_pk * (1/sqrt(2)) [rms/pk] * (1/sqrt(0.187499)) [1/rtHz] = 7.61e-11 [rad_rms / rtHz] yaw_rad_rms_p_rtHz = yaw_rad_pk * (1/sqrt(2)) [rms/pk] * (1/sqrt(0.187499)) [1/rtHz] = 4.77e-11 [rad_rms / rtHz] - Grab a spectrum of all of the respective QPDs and WFS signals for the time when DRMI was locked and the dither excitations were present, and calibrate the signals such that the amplitude of the dither line is as calcualted above. (See 3rd, 4th, and 5th image attached.) Sensor DOF Channel Name Calibration [rad/ct] MICH WFS Pitch H1:ASC-AS_B_RF45_Q_PIT_OUT_DQ 1.8e-08 SRC1 WFS Pitch H1:ASC-AS_B_RF36_I_PIT_OUT_DQ 6.0e-10 AS QPD Pitch H1:ASC-AS_C_PIT_OUT_DQ 8.0e-07 MICH WFS Pitch H1:ASC-AS_B_RF45_Q_YAW_OUT_DQ 1.1e-08 SRC1 WFS Pitch H1:ASC-AS_B_RF36_I_YAW_OUT_DQ 3.0e-10 AS QPD Pitch H1:ASC-AS_C_YAW_OUT_DQ 9.0e-07 I imagine I can't have gotten the accuracy better than a factor of 2, but I wanted the practice in calibrating the WFS & QPD ASDs, and to be able to make a ball-park statement such as "well, we wouldn't expect to see the violin modes, because we can't resolve cavity displacement better than ~5e-12 rad/rtHz at 500 Hz."
Pressure keeps rising in CP4 (PT-245B). I've asked Dave B. to raise the alarm set point one order of magnitude to 5e-4 Torr. I'm staying late again and Kyle is on stand-by to come on site later to babysit pumps, if needed.
alarms were reconfigured and restarted. Here is the difference:
-<Channel name="H0:VAC-MY_Y4_PT245B_PRESS_TORR" low="1.0e-10" high="5.0e-05" description="VE gauge, MY CP4, CC">
+<Channel name="H0:VAC-MY_Y4_PT245B_PRESS_TORR" low="1.0e-10" high="5.0e-04" description="VE gauge, MY CP4, CC">
Weird pressure curve at CP4. Eight hour trend attached. Tried to scan with RGA but pressure is too high for filament to stay on.
Adjacent pressure trends
TITLE: 02/15 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:
Film crew w/ Nobel Laureate, Barry Barish, were around the site today. Lots of prep work going on at EX. EY had afternoon SUS & film crew activity. MY CP4 had moderate activity. The items in BOLD below are still active.
LOG:
This afternoon I centered the OpLev LASER at EX. The optic was misaligned upon my arrival. It was aligned for the procedure and remains that way. The sum out is currently at 18.5K counts. The attached trends show the decline of that number down from the the mid-40Ks as a step function about the end of last June. Upon consulting with Jeff K and Jason, no attempt will be made to increase this number and it should remain "usable" for the next couple of months.
FYI: This trend was 1 year.
PSL: So far, so good.
Commissioning: On Hold mostly, except for SDF clean-up onging
EY/ETMy:
EX/ETMx:
HAM6/Squeezer:
Vacuum:
Maintenance:
Other:
F. Clara, J. Kissel FRS Ticket 9683 In order to convince ourselves once-and-for-all that the problem with H1 SUS ITMY's R0 RT OSEM is because of the in-vacuum short of the cathode signal of the PD (see LHo aLOGs 40134 and 40099 and above cited FRS Ticket 9683), Fil and I have done a quick cable-swap study, where we inter change the cables of the 4-OSEM clusters at the chamber side of the satellite amplifiers. Attached are the corresponding ASDs of the test. Recall that there are three clusters to read out the two top-masses of the quad -- M0 F1F2F3SD, M0LFRT/R0LFRT, and R0 F1F2F3SD. These are cables ITMY_SUS-7, ITMY_SUS-8, and ITMY_SUS-9, respectively as shown in D1100022. With the cables connected as designed, the problem is on the 4th channel of ITMY_SUS-8. The ASD plots attached show what happens when you move the connection of ITMY_SUS-8 to the sat-amp that normally reads out R0 F1F2F3SD, SUS-H2-R6-6 (reference traces, showing badness on R0 SD), or when you move the connection of ITMY_SUS-8 to the sat-amp that normal reads out M0 F1F2F3SD, SUS-H2-R6-4 (live traces, showing badness on M0 SD). One can see that the drastically elevated broadband noise with a ton of frequency combs that is from the R0 RT OSEM follows the cable, regardless of which readout chain into which it's plugged. During both steps of the test, the opposing cable (either ITMY_SUS-7 or ITMY_SUS-9) was plugged into the sat-amp that normally reads out M0RTLF/R0LFRT, SUS-H2-R6-5. In both cases, the M0LFRT/R0LFRT channels show no problems reading out the OSEMs on either SUS_ITMY-7 or SUS_ITMY-9, which exonerates that readout chain of problems. This puts a nail in the coffin, and confirms that the noise arises from the in-chamber short, and we'll need to vent the corner in order to fix this. Whether we *will* vent *just* to fix this is bigger question that requires many team's input. However, there are a few windows of opportunity in the next few months: - 90% of these sorts of problems have been at the chamber feedthrough, so the "vent" may be as simple as bringing the corner up to atmosphere, popping off only that feedthrough, fussing with the in-vac connection at the feedthrough, popping the feedthrough back on, and pumping back down. - Gate-vales are already closed, so we need not worry about the additional cycling on the gate vales. - We will eventually *have* to bring the chambers to *almost* atmosphere when we attempt to discharge the ITMs. - We anticipate the PSL up/downgrade to the 70W amplifier, and the mid-station cryo-trap decommissioning to demand there's little we can do with the IFO until early April-ish. We'll discuss amongst ourselves and see which path we should take.
(Dave B, Gerardo M)
We've borrowed the upper screen of video0 to display a striptool with the following channels, H0:VAC-MY_Y5_PT246B_PRESS_TORR and H0:VAC-MY_Y3_PT210B_PRESS_TORR, this is to keep an eye on pressure next to CP4. This screen is viewable at the CONTROL ROOM (screen shoots), and it carries the title of ASC Error Signals (Pitch, Yaw), located towards the middle of the page. FYI, the screen is capturing 6 hours only.
Launch StripTool then open file CP4_BT_Pressure.stp
File location:
/opt/rtcds/userapps/release/ve/h0/scripts/