• 04:43:24UTC - lockloss
• 04:43:39UTC - ISI BS Stage 2 watchdog trips
• I don't see anything on our seismometers
• 04:44:14UTC - ISI ITMY Stage 2, ETMY Stage 1 and Stage 2 watchdogs trip
• realize it's an earthquake
• 04:44:25UTC - ISI ITMY Stage 1 watchdog trips
• 04:44:50UTC - change SEI_CONF to EARTH_QUAKE_V2
• 04:45UTC - two more ISI watchdogs trip
• 04:48UTC - two more ISI watchdogs trip
• 04:56UTC - all the rest if the ISI watchdogs trip
• ground motion traces are off the top of the 0.03-0.1Hz and 0.1-0.3Hz seismometer plots, i.e. no longer visible in our normal range

Attached is a plot that shows the FSS_TPD_DC has a small drop in power right before lockloss at 23:19UTC (jan21).

The ISS_PDA_REL_OUT also drops, though I don't know what the power is relative to.  The amplitude of the signal is about 0.0001 before the power drop, and the power drops by 0.001 (10x amplitude) before lock loss.

TITLE: 01/22 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 64Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    Wind: 3mph Gusts, 1mph 5min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.87 μm/s
QUICK SUMMARY:

• Noise Eater reset twice before H1 could get to NLN

TITLE: 01/21 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 65Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:

Little frustrating day with and IMC which had an alignment shift and then another lockloss due to ISS saturations.  Other than that, we are battling large useism.
LOG:

• 16:50 PSL Crystal Chiller topped off.
• 17:01  TCSy low chiller flow alarm
• 18:23-21:23 Out due to lockloss (see earlier alog for specifics)
• 23:19 Lockloss (assume this is due to ISS saturations, because didn't see anything seismically, and the IMC couldn't lock because of the ISS.
  • Interesting that the range was shooting up for 10-15min prior to this lockloss---even with the high useism.
• Left H1 in LOCKING for Cheryl, but we also found that the NOISE EATER was off!

At 23:19, H1 had a lockloss (which was unfortunate because it was starting to ramp up in range!).  There were no seismic event,

BUT....

IMC was not relocking.  No other alarms, so delved into PSL's ISS, and found a couple of issues

1)  ISS Was Saturating

Did not really know to go here other than I rememeber having a Diff for the ISS for last lock.  And sure enough, I saw the ISS not looking healthy.  Eventually called Jason and told him the....

SYMPTOMS:  Diffracted Power racing up to 12% & seconds later it would drop.  It kept on cycling like this. 

Jason told me...

THE FIX: 

• Turn AUTOLOCKER OFF
• Open Manual Mode & turn it OFF (diffracted power will stay at saturated value of 12%)
• Then tweak the REFSIGNAL 5-clicks at a time (I went more negative), until you can get the Diffracted Power between 3.5-4.0%
• Then Turn Manual to ON +INT, wait a few seconds.  Then turn the Autolocker back ON.

2) NOISE EATER OFF
When Cheryl arrived she happened to notice that the NOISE EATER was OFF!  (She also found it went off when the OMC broke lock during the lockloss...so maybe this was the cause of the lockloss?).  I went out and turned it on for Cheryl and she relocked the FSS and is now back to locking.

Note:  I did go to a CORRECTIVE MAINTENANCE Observatory Mode for a few minutes while we were down.

No obvious seismic jolt for lockloss, BUT we had been riding some high useism.

Locking Notes:

1) IMC not locking!  (it did look misaligned on Trans when it tried grabbing lock)

• Going through Jenne's troubleshooting doc, the first thing I did was clear WFS history for IMC:  NOTHING
• Check MC1/2/3 witness sensors.  There was a step in most of these channels, so I steered all of the MC's to values when the IMC was locked & this did the trick.  (see attached plot for the step which occurred after the lockloss).

2) H1 Locking Status:  Have had a couple of failed attempts after & before CARM Reduction steps.

• To allow locking of PRMI, I had to adjust the BS & PRM quite a bit.
• I have been able to lock DRMI & made it past CARM reduction steps once (dropped at ANALOG CARM when all signals became noisy & ASAIR spot looked ugly), and also lost lock before  twice CARM reduction at SWITCH TO QPD steps.
• Next going to check Bounce/Roll modes & then try CARM reduction steps by hand.
• (texting Jim about useism and he recommends staying in the WINDY states.

Crystal Chiller topped off with 265mL this morning (Diode Chiller had GREEN light for its level).  This closes FAMIS 6506.

TITLE: 01/21 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 63Mpc
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
    Wind: 5mph Gusts, 4mph 5min avg
    Primary useism: 0.19 μm/s
    Secondary useism: 0.87 μm/s
QUICK SUMMARY:

Not much to hand-off from Patrick other than the Configuration Change note for H1OAF on the CDS Overview. 

useism has inched up in the last 8-10hrs (above the 90percentile and almost clearing it!) & you can see the signal on the Tidal Striptool (nuc1).

Beautiful clear morning with no issues driving in & a crisp 28degF out.

TITLE: 01/21 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 65Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: Remained in observing the entire shift. The filter changed by Sheila that Cheryl reported in her shift summary remains unloaded.
LOG:

09:14 UTC Damped PI mode 28 by changing sign of gain
14:45 UTC GRB alert

Have remained in observing. No issues to report.

TITLE: 01/21 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 65Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    Wind: 4mph Gusts, 3mph 5min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.67 μm/s 
QUICK SUMMARY:

No issues to report.

TITLE: 01/21 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 61Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:

• two GRBs
• one lock loss
• relocking was ok, lost lock after DRMI a couple times and then made it to NLN
• had to accept a -2e-16 change on H1:PSL-ISS_REFSIGNAL to go to Observe
• red dot on CDS overview, filters changed by Sheila, not loaded, no change to IFO, snapshot attached

Background - 
Both CP3 and CP4 are experiencing obstructed "high side" sensing lines which prevent their level transducers from measuring the level of LN2 in the 80K pumps.  We consider that this is due to accumulated ice at the bottom of the pumps inner LN2 vessel where the "high side" sensing line transitions from the relatively large ID aluminum tube to the relatively small ID stainless tube near a 90 degree elbow.  While Chandra R. has developed a procedure for a permanent post-O2 fix, we, nevertheless, would like to pursue more accommodating fix attempts in the meantime.  One of these is to see if by applying a vacuum (for a prolonged period) to the sensing line enough ice could be liberated through reverse sublimation that the obstruction could then be freed by alternating applications of modestly pressurized UHP N2 followed by vacuum pumping.  

Today - 
WP #6442 
Kyle R., Chandra R., Dave B., Gerardo M., Marc P.
I reconnected the "clogged" sensing line (with added parallel valved line used for vacuum pumping and a parallel valved line used for back filling) to the CDS level transducer (Rosemont model 1151DP4E52B404).  I then applied various "on-scale" pressure differentials via the back fill line (as indicated by the parallel connected magnahelic DP gauge) while Chandra R., Dave B. and Gerardo M. monitored the CDS MEDM screen. This confirmed that the transducer was operating and was in rough agreement with the mechanical gauge.  I also switched to the application of vacuum and they noted that the 4-20ma transducer output fell to 0 mA or something less than 4 mA (thought 4 mA output was minimum output?).  I then rearranged the plumbing fittings so as to permit selecting between a vacuum pump or a back fill line (10 psi UHP N2 @ 2 LPM) without the need to undo any tubing fittings as had been the case initially.  During this plumbing rearrangement, the sensing line was open to the room and began to flow GN2 past the obstruction.  This phenomenon had been observed on previous dislodge attempts (see entries from last month?) the difference this time was the significant increase in volume and duration - "Almost got it!"  I experimented with various durations of pumping and back filling etc. and observed on a few occasions that the audible gas load to the diaphragm pump would change spontaneously if let alone for several minutes.  

Anyway, this is all very encouraging and will likely be continued next week.  I halted tonight's attempts in observance of the Vacuum Team's Friday Rule and to allow others (the larger vacuum community) to consider if this approach is thermally cycling the bi-braze joint and poses a risk.  I am leaving the sensing line back filled to 10 psi and isolated from both the vacuum line and back fill line.  

WP6442, alarm on positive LN2 level in CP4 pump

Chandra, Kyle, Gerardo, Dave

CP4 LN2 Pump Level channels were added to the cell phone text alert system ( H0:VAC-MY_CP4_LT250_PUMP_LEVEL_PCT,  H0:VAC-MY_CP4_LT250_PUMP_LEVEL_PCT_ERROR).

Unfortunately the ERROR channel was in Error this afternoon which caused many cell phone alerts. Trending this channel shows that over the past month it has been transitioning between Error and No Error states, which the majority of the time being in Error.

Until this channel's intermittent errors can be resolved, we cannot rely on this as a critical alarm. WP6442 work has been postponed until Monday.