Displaying reports 45401-45420 of 84840.Go to page Start 2267 2268 2269 2270 2271 2272 2273 2274 2275 End
Reports until 10:15, Friday 16 February 2018
H1 CDS (CAL, CDS, IOO, ISC, OpsInfo, PEM, PSL, SEI, SUS)
jeffrey.kissel@LIGO.ORG - posted 10:15, Friday 16 February 2018 - last comment - 11:18, Friday 16 February 2018(40574)
H1 Corner Station Dolphin Network Crash
J. Kissel, for D. Barker

Dave came in this morning to find that the LSC front-end computer had failed. Upon hard booting the computer and IO chassis, even though he tried his darnedest to gracefully take the computer out of the Dolphin Network, upon restart it broke all front-end processes in the corner station, and even killed the h1susham2 computer software entirely. We're now in the process of recovering.

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 10:41, Friday 16 February 2018 (40576)

details on this event:

Jeff, Corey, Fil, Dave:

h1lsc0 models were not running, the computer could not see its IO Chassis. In the CER, there was no activity reported near the ISC chassis at the time of crash (09:15 PST), we found the IO Chassis was fully powered.

On h1lsc0 I closed the models and issued the disconnect-from-dolphin command, and then powered down. On power up, h1lsc0 dolphin glitched the corner station, so all models except for sus-aux stopped running. Around the time h1ioplsc0 model started, h1sush2a froze up. Its console was blank and no sshd service. I waited for h1lsc0 to fully start up to verify its problem was transient.

I rebooted h1sush2a using the front panel reset button. As expected, when it came back it had glitched the dolphin fabric and taken down the lsc models.

I then restarted all models on all dolphin'ed corner station front end computers (except for h1sush2a which had just restarted). Procedure was /etc/kill_models.sh followed by /etc/start_models.sh. Restart order in attached restart log.

Once all the models were running again, I cleared the IPC and CRC errors.

Jeff and Corey are recovering the systems.

Non-image files attached to this comment
keith.thorne@LIGO.ORG - 10:58, Friday 16 February 2018 (40578)CDS
A small consolation is that the new Dolphin PCIe Gen2 hardware uses a switch-based method to disable ports that seems to be more robust.
jeffrey.kissel@LIGO.ORG - 11:18, Friday 16 February 2018 (40581)
Recovery complete -- see LHO aLOG 40580.
LHO General
corey.gray@LIGO.ORG - posted 08:11, Friday 16 February 2018 (40572)
Morning Status

TITLE: 02/16 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    Wind: 26mph Gusts, 23mph 5min avg
    Primary useism: 0.24 μm/s
    Secondary useism: 0.17 μm/s
QUICK SUMMARY:

On this blustery morning a few activities already in action:  EY BSC Arm removal, ISCT6 move prep, H1PSL work, cleaning at MY, & film crew around site.

LHO VE
kyle.ryan@LIGO.ORG - posted 23:17, Thursday 15 February 2018 - last comment - 11:40, Friday 16 February 2018(40569)
Pressure in CP4 above agreed upon limit -> Heading for Y-mid to check it out


			
			
Comments related to this report
kyle.ryan@LIGO.ORG - 00:10, Friday 16 February 2018 (40570)
0004 hrs. local -> Arriving on site to start CP4 watch attended "owl" shift.  Working alone with Chandra R. as designated "phone buddy".  Won't be doing much other than ensuring scroll pump doesn't shut down due to hydraulic lock as the result of the water vapor laden gas load as CP4 warms and off-gasses.  Brought the camper-van so look for me to be at Y-mid VEA, my office or in my van (listening to 70's era rock or conservative radio :)  

Will make a comment to this entry to update CP4 status or when leaving the site.  
kyle.ryan@LIGO.ORG - 04:54, Friday 16 February 2018 (40571)
As the attached graph indicates, the source of gas went away at about the same time the pressure alarm threshold was exceeded.  To justify my visit, I did two iterations of 1)closing the turbo exhaust valve. 2)setting the foreline isolation valve setpoint to 950 Torr (maximum allowed value). 3)cracking open the foreline connection at the turbo exhaust valve (allows the scroll pump to pump viscous room air and blows out condensed water from between the two scroll stages). 4)restoring isolation valve setpoint (now 5 x 10-2 Torr).  

0500 hrs. local -> Leaving site now. 
Non-image files attached to this comment
david.barker@LIGO.ORG - 10:59, Friday 16 February 2018 (40579)

attached alarm log for this event

Non-image files attached to this comment
chandra.romel@LIGO.ORG - 11:40, Friday 16 February 2018 (40583)

Glad to be back in the safe zone! (for now)

LHO VE
chandra.romel@LIGO.ORG - posted 20:25, Thursday 15 February 2018 (40568)
LN2 bayonets

Today we decoupled a section of the LN2 fill line inside mid-Y VEA. The male bayonets slid out relatively easy after removing the o-ring clamps. Gerardo is searching for vendor to provide a male bayonet "blank" so we can seal the pump side LN2 pipe extension in the event that we develop an air leak in bibraze joint and need to pump on the reservoir. For now I'm going to pick up a rubber plug from the hardware store for emergency back up.

Images attached to this report
H1 TCS (TCS)
greg.grabeel@LIGO.ORG - posted 18:25, Thursday 15 February 2018 (40565)
CO2X laser running

The power supply for CO2 X laser was fixed (thanks Ed) so I was able to check the operational status of the laser table. The chillers had once again tripped from the PZT control loop sending to high of a value to the chiller setpoints. I disabled integrator, reset the history on the control loop, and restarted the chillers. Everything appeared to be in order on the table, so now both CO2 X and Y are running with beam dumps in place. Tomorrow I'll try and adjust the loops to find a more stable system.

H1 SUS
betsy.weaver@LIGO.ORG - posted 18:07, Thursday 15 February 2018 (40564)
ETMY Suspension rebuild status

More work on the ETMY suspension in BSC10 - this afternoon, Travis and I:

1) Hosted Barry Barish, Michael, and a film crew chamber-side for a filming project.

2) Added the balance of the sus payload to the cage - Sleeve structure, wedges, cross braces, vibration absorbers.

3) Installed the newest Bounce-Roll Dampers (BRDs) from Kissel (thanks!!).  This took a careful hand holding the bitty assembly with needle-nose pliers and inserting them into the mass before tightening with an allen key.  We also locked down the UIM and PUM since we were pushing on the UIM blades for this install.

4) unlocked masses, reset ETMY and TMSY watchdogs and turned damping on.

The alignment at first glance looks to still be on centerish.  More refinement to finish up tomorrow.

H1 SUS (ISC)
jeffrey.kissel@LIGO.ORG - posted 18:02, Thursday 15 February 2018 (40562)
H1 SUS ITMX Violin Mode Search At UHV Remains Fruitless
J. Kissel, J. Driggers

I continue to search for the at-vacuum violin mode resonances so we can determine the shift in frequency from the detailed in-air measurements (LHO aLOG 39163). This time, instead of looking at the PUM OSEMs and optical lever local sensors (as done in LHO aLOG 40019), I scoured data in the can from last week's arm peaking and DRMI locking, in hopes that an interferometric measurement would be more sensitive. 

Namely, I found a time when DRMI was locked (starting from the crumbs left in LHO aLOG 40463, and then trending the DRMI guardian lock state, the LSC trigger bits, and AS_C's normalized sum), 2018-02-08 ~21:20 UTC, and measured the ASD of MICH, PRCL, SRCL, and a few of the WFS and QPDs that have been traditionally sensitive to violin modes. 

Calibration details are below, but the message is that I've still found nothing. 
The most sensitive length measurement was MICH at ~8e-15 m/rtHz, and the most sensitive angle measurement was (surprisingly) the AS_C QPD at ~6e-12 rad/rtHz.
The first four attachments are the calibrated ASDs, at various levels of zoom in frequency.

%%%%
Details
%%%%

LSC Signal Calibration
Not so tough -- I grabbed the DRMI displacement wall FOM from 
    /opt/rtcds/userapps/release/isc/h1/scripts/DRMI_FOM_displacements.xml (rev. 16689, last changed rev. 14505)
and simply converted the pre-calibrated channels from [um] to [m].

ASC Signal Calibration (with guidance from Jenne)
   - Find a time when the ASC dither signals were on, gather the frequency, amplitude (e.g. H1:ASC-ADS_PIT2_OSC_FREQ, H1:ASC-ADS_PIT2_OSC_CLKGAIN), and which optic was driven. Some ugly combination of dataviewer and time-machine is needed for this search, because which oscillator used pointing to which optic is arbitrary and inconsistent with time. Note, that at this stage, the dither amplitude will be in peak counts of the sine wave at the given dither frequency, [ct_pk] @ f Hz. Here's what I found:
       DOF       f [Hz]     Amp [ct_pk]   Driven Optic     Time On
       Pitch      21.3      1000          SRM              2018-02-08 (21:31:46 - 21:32:46 UTC)
       Yaw        21.3      1000          SRM              2018-02-08 (21:33:46 - 21:34:46 UTC)
     (This was during the DRMI lock stretch from ~21:00 to 21:30 UTC)
   - Calibrate the dither amplitude into expected displacement ASD [rad_rms/rtHz] at the given optic.
       - Use the relevant subfunction of the python library of functions,
            /opt/rtcds/userapps/release/asc/h1/scripts/sensingMatrix/HY_version/DOF_lib.py
         In this case, SRM was being driven, so I used the PRM sub function, as it was most similar in suspension type. Note that means the attached ASC signal ASDs are calibrated into [rad] of SRM displacement.
            ]$ cd /opt/rtcds/userapps/release/asc/h1/scripts/sensingMatrix/HY_version/
            ]$ ipython
            In [1]: import DOF_lib
            In [2]: pitch = DOF_lib.PRM_cts2rads(21.3,'PIT','TFs/H1_PRM.txt')
            In [3]: yaw = DOF_lib.PRM_cts2rads(21.3,'YAW','TFs/H1_PRM.txt')
            In [4]: pitch
            Out[4]: (4.6564401596417816e-14, -2.6756796108295753e-08)
            In [5]: yaw
            Out[5]: (2.9158641207661465e-14, -3.8069250094614154e-07)
          The script uses the EUL2OSEM gain, the DAC gain, the coil driver gain, the magnet strength, the flag-to-CoM distance, and a canned torque to angular displacement transfer function at the specified frequency to calculate the above gain.
        - The first output of the functions are what we need, a scalar calibration of radians of optic motion per dither drive count, i.e. [rad/ct]. So multiply that by the dither amplitude to get the expected peak displacement,
           pitch_rad_pk = 1e3 [ct_pk] * 4.66e-14 [rad/ct] = 4.66e-11 [rad_pk]
           yaw_rad_pk = 1e3 [ct_pk] * 2.92e-14 [rad/ct] = 2.92e-11 [rad_pk]
        - In converting from peak sine wave amplitude to rms amplitude spectral density, the dither lines were only on for a minute, so I need a short FFT -- I chose 0.1 Hz. With the standard DTT Hanning window, that's an effective noise band width (ENBW) of 0.187499. Thus, to get from peak to rms/rtHz, that's 
           pitch_rad_rms_p_rtHz = pitch_rad_pk * (1/sqrt(2)) [rms/pk] * (1/sqrt(0.187499)) [1/rtHz] = 7.61e-11 [rad_rms / rtHz]
           yaw_rad_rms_p_rtHz = yaw_rad_pk * (1/sqrt(2)) [rms/pk] * (1/sqrt(0.187499)) [1/rtHz] = 4.77e-11 [rad_rms / rtHz]
    - Grab a spectrum of all of the respective QPDs and WFS signals for the time when DRMI was locked and the dither excitations were present, and calibrate the signals such that the amplitude of the dither line is as calcualted above. (See 3rd, 4th, and 5th image attached.)
        Sensor       DOF      Channel Name                     Calibration [rad/ct]
         MICH WFS    Pitch    H1:ASC-AS_B_RF45_Q_PIT_OUT_DQ    1.8e-08
         SRC1 WFS    Pitch    H1:ASC-AS_B_RF36_I_PIT_OUT_DQ    6.0e-10
         AS QPD      Pitch    H1:ASC-AS_C_PIT_OUT_DQ           8.0e-07

         MICH WFS    Pitch    H1:ASC-AS_B_RF45_Q_YAW_OUT_DQ    1.1e-08
         SRC1 WFS    Pitch    H1:ASC-AS_B_RF36_I_YAW_OUT_DQ    3.0e-10
         AS QPD      Pitch    H1:ASC-AS_C_YAW_OUT_DQ           9.0e-07
I imagine I can't have gotten the accuracy better than a factor of 2, but I wanted the practice in calibrating the WFS & QPD ASDs, and to be able to make a ball-park statement such as "well, we wouldn't expect to see the violin modes, because we can't resolve cavity displacement better than ~5e-12 rad/rtHz at 500 Hz."
Images attached to this report
LHO VE
chandra.romel@LIGO.ORG - posted 17:00, Thursday 15 February 2018 - last comment - 19:58, Thursday 15 February 2018(40561)
CP4 pressure alarm settings changed

Pressure keeps rising in CP4 (PT-245B). I've asked Dave B. to raise the alarm set point one order of magnitude to 5e-4 Torr. I'm staying late again and Kyle is on stand-by to come on site later to babysit pumps, if needed. 

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 17:03, Thursday 15 February 2018 (40563)

alarms were reconfigured and restarted. Here is the difference:

-<Channel name="H0:VAC-MY_Y4_PT245B_PRESS_TORR" low="1.0e-10" high="5.0e-05" description="VE gauge, MY CP4, CC">
+<Channel name="H0:VAC-MY_Y4_PT245B_PRESS_TORR" low="1.0e-10" high="5.0e-04" description="VE gauge, MY CP4, CC">

 

chandra.romel@LIGO.ORG - 19:54, Thursday 15 February 2018 (40566)

Weird pressure curve at CP4. Eight hour trend attached. Tried to scan with RGA but pressure is too high for filament to stay on.

Images attached to this comment
chandra.romel@LIGO.ORG - 19:58, Thursday 15 February 2018 (40567)

Adjacent pressure trends

Images attached to this comment
LHO General
corey.gray@LIGO.ORG - posted 15:59, Thursday 15 February 2018 (40548)
DAY Operator Summary

TITLE: 02/15 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY:

Film crew w/ Nobel Laureate, Barry Barish, were around the site today.  Lots of prep work going on at EX.  EY had afternoon SUS & film crew activity.  MY CP4 had moderate activity.  The items in BOLD below are still active.
LOG:

H1 SUS
edmond.merilh@LIGO.ORG - posted 15:43, Thursday 15 February 2018 - last comment - 15:46, Thursday 15 February 2018(40559)
ETMX OpLev Centering

This afternoon I centered the OpLev LASER at EX. The optic was misaligned upon my arrival. It was aligned for the procedure and remains that way. The sum out is currently at 18.5K counts. The attached trends show the decline of that number down from the the mid-40Ks as a step function about the end of last June. Upon consulting with Jeff K and Jason, no attempt will be made to increase this number and it should remain "usable" for the next couple of months.

Images attached to this report
Comments related to this report
edmond.merilh@LIGO.ORG - 15:46, Thursday 15 February 2018 (40560)

FYI: This trend was 1 year.

LHO General
corey.gray@LIGO.ORG - posted 13:46, Thursday 15 February 2018 (40557)
Thursday Vent Meeting Notes

PSL:  So far, so good.

Commissioning:  On Hold mostly, except for SDF clean-up onging

EY/ETMy:

EX/ETMx:

HAM6/Squeezer:

Vacuum:

Maintenance:

Other:

H1 SUS
jeffrey.kissel@LIGO.ORG - posted 12:07, Thursday 15 February 2018 (40555)
H1SUSITMY R0 RT OSEM Problem is In-Vacuum
F. Clara, J. Kissel
FRS Ticket 9683

In order to convince ourselves once-and-for-all that the problem with H1 SUS ITMY's R0 RT OSEM is because of the in-vacuum short of the cathode signal of the PD (see LHo aLOGs 40134 and 40099 and above cited FRS Ticket 9683), Fil and I have done a quick cable-swap study, where we inter change the cables of the 4-OSEM clusters at the chamber side of the satellite amplifiers. 

Attached are the corresponding ASDs of the test. Recall that there are three clusters to read out the two top-masses of the quad -- M0 F1F2F3SD, M0LFRT/R0LFRT, and R0 F1F2F3SD. These are cables ITMY_SUS-7, ITMY_SUS-8, and ITMY_SUS-9, respectively as shown in D1100022. With the cables connected as designed, the problem is on the 4th channel of ITMY_SUS-8. The ASD plots attached show what happens when you move the connection of ITMY_SUS-8 to the sat-amp that normally reads out R0 F1F2F3SD, SUS-H2-R6-6 (reference traces, showing badness on R0 SD), or when you move the connection of ITMY_SUS-8 to the sat-amp that normal reads out M0 F1F2F3SD, SUS-H2-R6-4 (live traces, showing badness on M0 SD). 
One can see that the drastically elevated broadband noise with a ton of frequency combs that is from the R0 RT OSEM follows the cable, regardless of which readout chain into which it's plugged. During both steps of the test, the opposing cable (either ITMY_SUS-7 or ITMY_SUS-9) was plugged into the sat-amp that normally reads out M0RTLF/R0LFRT, SUS-H2-R6-5. In both cases, the M0LFRT/R0LFRT channels show no problems reading out the OSEMs on either SUS_ITMY-7 or SUS_ITMY-9, which exonerates that readout chain of problems.

This puts a nail in the coffin, and confirms that the noise arises from the in-chamber short, and we'll need to vent the corner in order to fix this. 

Whether we *will* vent *just* to fix this is bigger question that requires many team's input. However, there are a few windows of opportunity in the next few months: 
  - 90% of these sorts of problems have been at the chamber feedthrough, so the "vent" may be as simple as bringing the corner up to atmosphere, popping off only that feedthrough, fussing with the in-vac connection at the feedthrough, popping the feedthrough back on, and pumping back down.
  - Gate-vales are already closed, so we need not worry about the additional cycling on the gate vales.
  - We will eventually *have* to bring the chambers to *almost* atmosphere when we attempt to discharge the ITMs.
  - We anticipate the PSL up/downgrade to the 70W amplifier, and the mid-station cryo-trap decommissioning to demand there's little we can do with the IFO until early April-ish.
We'll discuss amongst ourselves and see which path we should take.
Images attached to this report
LHO VE (VE)
gerardo.moreno@LIGO.ORG - posted 16:07, Wednesday 14 February 2018 - last comment - 14:22, Thursday 15 February 2018(40537)
Borrowing video0 Figure of Merit Display for CP4 Work

(Dave B, Gerardo M)


We've borrowed the upper screen of video0 to display a striptool with the following channels, H0:VAC-MY_Y5_PT246B_PRESS_TORR and H0:VAC-MY_Y3_PT210B_PRESS_TORR, this is to keep an eye on pressure next to CP4.  This screen is viewable at the CONTROL ROOM (screen shoots), and it carries the title of ASC Error Signals (Pitch, Yaw), located towards the middle of the page.  FYI, the screen is capturing 6 hours only.

Images attached to this report
Comments related to this report
gerardo.moreno@LIGO.ORG - 14:22, Thursday 15 February 2018 (40558)VE

Launch StripTool then open file CP4_BT_Pressure.stp

File location:

/opt/rtcds/userapps/release/ve/h0/scripts/

Displaying reports 45401-45420 of 84840.Go to page Start 2267 2268 2269 2270 2271 2272 2273 2274 2275 End