I restarted the h1lsc0 models today.  Dave Barker is trending the timing information and state word and may have more to add.  But at this point we suspect it is likely a glitch in the IRIG-B as seen yesterday.

The system was not responsive via the network via ssh or EPICS CA.  The MEDM screen on the control room wall showed everything green, but trying to view the medm screen from another computer (a new connection) failed with the channels not connecting.

Going to the console showed that the repeated error 'nf_conntrack: table full, dropping packet'.  The system was set to track 64k connections.  I changed the limit (until reboot) to 100,000.  At that point new connections could be made and the medm screens went red with IPC.  I am suprised by this behavior, I would have though the IPC bit would have gone bad on the other machines irregardless of the state of the lsc machine.  At this point I killed all the models and restarted them.  Then TJ and I went through and cleared all the IPC errors through the site after verifying that they where related to h1lsc0.

Reviewing the dmesg output and filtering out nf_conntrack errors showed an ADC TIMEOUT on h1lscaux, h1omc, h1sqz, h1omcpi, h1lsc at 7246965.68s since boot.

Ran the checks for the STS & T240 masses (& will close FAMIS #6094).

STSs:  Fine & within range (same as last month). 

T240s: Now ETMy* HEPI is on its stops in prep for EY vent, and with that said, had the following masses out of range:

Averaging Mass Centering channels for 10 [sec] ...
2018-01-04 10:17:17.353721

There are 13 T240 proof masses out of range ( > 0.3 [V] )!
ETMX T240 1 DOF X/U = 0.42 [V]
ETMX T240 1 DOF Z/W = 0.396 [V]
ETMX T240 2 DOF X/U = -0.445 [V]
ETMX T240 2 DOF Y/V = -0.88 [V]
ETMX T240 3 DOF X/U = 0.318 [V]
ETMX T240 3 DOF Y/V = 0.435 [V]
ETMX T240 3 DOF Z/W = 0.327 [V]
* ETMY T240 3 DOF Z/W = 0.413 [V]
ITMX T240 1 DOF X/U = -0.412 [V]
ITMX T240 2 DOF Z/W = 0.33 [V]
ITMX T240 3 DOF X/U = -0.438 [V]
ITMY T240 3 DOF X/U = -0.648 [V]
ITMY T240 3 DOF Z/W = -1.08 [V]

PDF of pictures attached:

1. 45MHz driver as found
2. 9MHz cable label
3. new (original) 45MHz driver installed
4. 45MHz cable label

Cheryl, TVo, Keita, Jenne

Summary: The IMC is aligned and flashing.  The IMC REFL path on IOT2 is aligned.  The IMC trans path that comes from the transmission through IM1 is coming out of the viewport and light pipe, but not hitting the top periscope mirror by about 2 inches to the right as you're looking at the viewport from the IOT2 door (this is consistent with our having moved this beam farther from clipping on the edge of the viewport as noted in alog 39804). 

To do this:

• We basically followed Cheryl's plan of using the input PZT to align the leakage beam onto MC2 Trans, as a proxy for centering on MC2, while MC3 was misaligned so we weren't getting flashes.  To really see anything, we had to increase the whitening gain on MC2 Trans (H1:IMC-MC2_TRANS_WHITEN_GAIN) from its nominal of 36dB to the max of 45dB, and reset the dark offsets of each segment while the PSL shutter was closed.  We also increased the PSL power injected into the vacuum to ~4W. N.b. that the monitor PD for the amount of light injected seems to be not well aligned, so we just used whatever was in the rotation stage's calibration and requested 4W, but we didn't have an independent monitor of the actual power injected. 
• We then used MC2 to center the beam on the aperture of the baffle that is behind IM1, under the assumption that the baffle was well placed, so it's a proxy for getting the beam where we want on IM1.  Cheryl has a camera and TV set up on the floor for this purpose, as she described in alog 39955.
• MC1 and MC3 were tweaked a bit to maximize the flashes on MC2 Trans, but they didn't need much. 
• (Note to Cheryl: we touched the PZT and MC2 sliders again after you left, so what's in there now are our latest good values.  MC1 and MC3 didn't need any more touching)
• Once MC1 was set, we aligned the beam onto the MC REFL RFPD, IMC WFS A, IMC WFS B, and the IMC Refl camera. 

We're not yet really seeing any signal on H1:IMC-I_OUT, but the LSC computer seems to maybe be bad again, so I'll leave locking as an exercise for tomorrow.  No checking of analog electronics has been done tonight (by me, at least).

Still to do: We need to move the IMC Trans periscope on IOT2 a few inches toward the PSL enclosure, and will have to redo (move) that whole optical path to match.  Also being left as an exercise for the morning.  Until this is done, we won't have the nice trans camera to see the shapes of our flashes.  However, just looking on a card, we're mostly flashing 00, 01, and 10 modes, so everything should be pretty smooth going tomorrow.

GGrabeel, TVo

Topped off the chillers and started them back up this afternoon. They immediately faulted out, due to a set-point error. I set them to run in manual while I checked for leaks, there were no apparent leaks. TVo found the problem in EPICS where an integrator on a control loop was causing the set point to steadily increase (It was up to about 500,000 degrees Celsius I think). After turning off some inputs and filters we were able to get the more reasonable value of 21 degrees Celsius. The chillers will continue to run and be re-checked tomorrow.

J. Kissel, R. McCarthy, P. Fritschel 
FRS Ticket 5063
Prior LHO aLOGs 13556, 14357, and 21481

In perusing the list of closed FRS tickets after the Sep-Dec 2017 vent, Peter found that the above mentioned FRS ticket 5063, "H1 SRM osems (LF RT SD and T3) have excess noise" had been closed without supporting data. Richard clearly states in the most recent comment that the problem was a prominent screw head grounding against the chamber in-vacuo, but just didn't get the chance to post the data.

Attached is the supporting data. Where, in the past, SRM had shown prominent features and combs originating from large peaks around 1700 Hz and 1900 Hz, there are no such features visible in any of the OSEMs on SRM. For random comparison, I show SR3 top mass OSEMs as well.

J. Kissel

I've taken standard rubbing-check, top-mass to top-mass transfer functions for all suspensions in HAM2 and HAM3, now that we're down to about 1e-6 Torr (which is sufficient a vacuum that the suspension positions won't change further from buoyancy). Good news: All suspensions are free of rubbing after pumpdown, including MC1, MC2, MC3, IM1-4, PRM, PR2, and PR3.

Data files are listed below. Detailed results will be processed and posted in due time.
    /ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/MC?/SAGM1/Data/2018-01-03*.xml
    /ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/PR?/SAGM1/Data/2018-01-03*.xml

    /ligo/svncommon/SusSVN/sus/trunk/HLTS/H1/PR3/SAGM1/Data/2018-01-03*.xml

    /ligo/svncommon/SusSVN/sus/trunk/HAUX/H1/IM?/SAGM1/Data/2018-01-04*.xml

As a teaser / proof, I attach screenshots of each M1 Pitch to M1 Pitch transfer functions for the Triples.

Bubba and Bailey are in-process of sweeping the LVEA and transitioning to laser hazard, and opening the main PSL light pipe so we can work on IMC alignment.

We've known for a while that HEPI trips kick the suspensions prett badly. One of the things SamC found looking at earthquakes, is that during very large earthquakes, HEPI is often the first platform to trip. Hugh and I have used the post-vent time to minimize the differences between the free hanging HEPI positions and the target positions, by tweaking the springs. I used ETMY to test if this reduced the kick to the quad and it seems to be pretty good. The first plot shows the ETMY ISI T240s and the IPS cart locations during a trip before (blue traces on all plots) and after we reduced the offsets (red on all plots). Before offloading, the HEPI trip immediately tripped the ISI and kicked everything around quite a bit. After, the ISI actually stayed isolated until the T240s saturated, some 30+ seconds later, and even then the kick to the table wasn't that violent. The quad sees pretty much the same thing. Before, the suspension gets a pretty kick from the HEPI trip, but after the quad barely sees any motion at all.

This may not reflect what we see during an earthquake, but I don't see how this could make things worse. We'll need to monitor this for a little while because HEPI seems to be prone to drifting more that the ISIs, and we don't really know how much it drifts long term. 

All models on h1lsc0 froze up at 15:21 PST due to an IRIG-B timing error on the h1ioplsc0 model. The dmesg logs suggests this is a repeat of the problem seen on Friday 22nd December 2017 which also required a full model restart.

Looking at minute trends, the IOP IRIG-B signal went from 13 uS to 202,710 uS at 15:21:42 PST, later it jumped up to 732,404 uS at 15:30:42 PST. After restarting the model, it returned to 13uS.

On Friday Dec 22 2017 at 09:44:42 PST this signal jumped to 296,598 uS until it was reset 13 minutes later (no subsequent jump into the 0.7 S region).

h1lsc0's dmesg logs for the two events are:

[6123949.214305] h1lscaux: ADC TIMEOUT 0 46035 19 46099

[6123949.214307] h1omc: ADC TIMEOUT 0 46035 19 46099

[6123949.214309] h1lsc: ADC TIMEOUT 0 46035 19 46099

[6123949.214311] h1sqz: ADC TIMEOUT 0 46035 19 46099

[6123949.214312] h1omcpi: ADC TIMEOUT 0 46035 19 46099

           ...

[7179982.197361] h1lscaux: ADC TIMEOUT 0 52188 28 52252

[7179982.197363] h1omc: ADC TIMEOUT 0 52188 28 52252

[7179982.197366] h1lsc: ADC TIMEOUT 0 52188 28 52252

[7179982.197368] h1sqz: ADC TIMEOUT 0 52188 28 52252

[7179982.197370] h1omcpi: ADC TIMEOUT 0 52188 28 52252

This afternoon I removed the lock from and re-enabled the PSL/IOO rotation stage, as stated in LHO WP 7263, for commissioning of the IMC prior to alignment activity in HAM6.  This completes WP 7263.

TITLE: 01/03 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: Starting to hook up the tables and cables from the CS vent, and prep at EY for its vent. SEI crew is has been working on HEPI at HAM4 and EY. Jeff K has ran measurements and cleared all of the large suspensions in HAM2/3 (ie. MC's & PR's). We will be going laser hazard shortly and commissioning work will begin on the IMC.
LOG:

• 15:10 Bubba, Mark, Tyler to LVEA to place IOT2 table and light pipes on
• 16:14 Terry to SQZr Bay, local laser hazard
• 16:28 Richard in and out of LVEA working on EE stops
• 16:31 Bubba out
• 16:42 Marc to LVEA to dress and pull cables.
• 16:46 Fil, ED to LVEA pulling cables near HAM6/SQZr Bay
• 17:02 Jason to LVEA to check on light pipe status before turning on the PSL
• 17:05 Jason out
• 17:08 Mark, Tyler out
• 17:12 TVo to LVEA to HWS table
• 17:18 Mark, Tyler back to HAM2 area
• 17:21 Nutsinee to SQZr Bay
• 17:35 Corey to Optics Lab
• 17:37 Mark, Tyler out and IOT2 is hooked up and in place
• 17:45 Karen to warehouse then MY
• 17:49 Elizabeth to LVEA to HAM6/SQZr Bay area to help with cables
• 17:53 Marc out
• 17:55 Hugh, Jim to LVEA to unlock BSC1&3
• 18:08 Bubba to MX
• 18:23 Christina to MX
• 18:37 Nutsinee out
• 19:10 Hugh, Jim out
• 19:13 Jenne, Keita to LVEA to look for barriers
• 19:14 TVo out, ITMX HWS somewhat aligned
• 19:17 Karen leaving MY
• 19:17 Jason to LVEA to check on PSL racks
• 19:52 Keita, Jenne to LVEA to start setting up laser barrier around HAM1/2
• 19:55 Fil, Ed, Elizabeth out
• 20:08 Keita, Jenne out
• 20:09 Jason out
• 20:49 Fil to LVEA to hook up IOT2L
• 21:02 Ed to LVEA to join Fil
• 21:90 Richard, Mark, Tyler to LVEA to move garbing room.
• 21:10 Hugh, Jim to EY
• 21:15 TVo to LVEA to HWS table
• 21:19 Elizabeth to LVEA to meet with Fil, Ed
• 21:21 Tyler, Mark out
• 21:38 Jenne, Niko to LVE to finish laser barrier
• 21:50 Greg to LVEA to start up chillers and look for leaks
• 21:52 Jason to LVEA to PSL racks
• 22:01 Jenne, Niko out
• 22:19 Keita to optics lab
• 22:42 Keita out
• 23:05 Mark, Tyler to EY to set up cleanroom
• 23:16 Cheryl to PSL to replace some electronics
• 23:21 Jason to LVEA to remove lock from PSL rotation stage (WP#7263)
• 23:23 Jason out
• 23:26 Hugh to HAM4 HEPI inspection
• 23:49 Nutsinee out
• 23:58 Mark, Tyler out
• 00:03 Hugh out

Today I restarted the PSL after it was purposefully shutdown for the duration of the holiday break.  The HPO and 35W FE restarted with no issues, but I did run into an issue while injection locking the two lasers; the injection locking was very unstable.  To ensure the system was at thermal equilibrium I let it warm up for the morning and roughly half of the afternoon before investigating the instability.  By 20:00 UTC (12:00 PST) the injection locking had become more stable, but was still having occasional relocks.  I lowered the ILS gain voltage from 1.0 V to 0.5 V (this is actually a gain increase) and things became more stable.  I also noticed that the SR560 used to boost the low frequency gain of the ILS servo was saturated; I could not get it to clear at the time.  Checking on things again at 22:00 UTC (14:00 PST), the injection locking had stabilized further.  As a double check I looked at the PDH error signal for the injection locking and saw no issues there; this required me to unlock the ILS to see the error signal.  Upon relocking the ILS things looked much more stable, and at about this time I also noticed that the SR560 was no longer saturating.  I turned on the TFIN for the ILS (the signal path we use to boost the ILS low frequency gain) and the signals looked much better.  I left it in that state to continue to warm up for the rest of the afternoon.  As of 23:54 UTC (15:54 PST), the system looked stable and operating normally.  At this point in time I re-enabled the power watchdogs.

I re-enalbed the PMC, FSS, and ISS at ~23:00 UTC (~15:00 PST).  All 3 came back without issue.  I will continue to monitor the system, but as of right now everything appears to be running normally.

I also took this opportunity to do the weekly PSL FAMIS tasks during the morning.

HPO Pump Diode Current Adjustment (FAMIS 8455)

With the ISS still OFF, I adjusted the pump diode operating currents and temperatures this morning as part of the PSL restart.  The changes are summarized in the below tables and a screenshot of the PSL Beckhoff main screen is attached for future reference.

At the time of adjustment the HPO was outputting ~154.5 W; this was at roughly 18:30 UTC (10:30 PST).  This completes FAMIS 8455.

PSL Power Watchdog Reset (FAMIS 4586)

I turned on both PSL power watchdogs at 23:54 UTC (15:54 PST).  This completes FAMIS 4586.

As we prepare to align the IMC and IOT2, we may want to increase the laser power injected into the vacuum from the few hundred mW that we've been at for a few months up to a couple Watts.  There was some concern that we want to ensure that we're not going to harm anything in HAM6, since it is still at atmosphere and the trigger PD for the fast shutter is not currently plugged in.  There shouldn't be any problem however.  I have got the ITMs, PRM and SRM misaligned so that we're not getting any PRMI flashes and no beam is going to HAM6.  Please leave the ITMs, PRM and SRM misaligned. 

Even if the PRMI were to flash, nothing in HAM6 is aligned, so there should be no beam on any PDs or on the OMC.  Also, the PDs would be able to handle that much power, vacuum or atmo.  Given the gross misalignments, the likelihood of the OMC resonating is vanishingly small.

With the recycling mirrors misaligned but the ITMs aligned (so just transmission through optics, no resonant flashes) the max amount of power going to HAM6 is 3mW for every 1W of injected light (3% transmission through PRM, 25% for 2 BS passes, and 35% for the SRM).  We do not intend to go higher than 1W or 2W, which would put about 6mW at HAM6.  However, it is conceivable that the rotation stage could let up to ~50W into the vacuum, in which case we'd have 130mW of light at HAM6.  Not insignificant, but still should be fine. 

So, we are software ensuring that no beam is going to HAM6 while we use about 2W of PSL power, but even if some of the optics get re-aligned it'll all be fine.