TITLE: 11/22 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

Fairly decent first 2/3's of the shift; then a M7.0 earthquake (+ high microseism) made the last 3hrs of the shift tough---with ALSy being the problem child as usual.  In hindsight, I probably should have just waited out the earthquake longer instead of trying to lock after 2hrs.  

Ended up staying a little extra to confirm the Initial Alignment completed.  Then taking H1 back to locking after tonight's earthquake activity.
LOG:

• 0006 H1 Lockloss from NLN! After only a few seconds of being in this state!
• 0006-0010 Checking foam damper for HAM1 HEPI plumbing to make sure it was not "singing" (Jim)
• 0110 Short GRB on verbal
• 0108 H1 to NLN
  • Adjusted SQZ phase due to cable change by Sheila earlier today & SDF diffs noted/accepted
  • 0131 Back to OBSERVING
• 0500 Earthquake Mode ACTIVATED & 3-seconds later, we were bumped out of OBSERVING, & then a few seconds later---LOCKLOSS from the S&P waves!  This is after reaching 3hrs52min on lock clock.
  • Not clear why we were dropped out of OBSERVING early.
  • The R-wave is due here in ~30min!
  • 0617 Start locking Green Arms...Y-Arm had more issues and went to INCREASE FLASHES several times.
  • 0702 Finally Green Y-arm locks & offloads!!
• 0706 Earthquake motions return! (but no EQ source posted, but seen on Picket Fences, )---this caused locklosses at PRMI & earlier states....so decided to....
• 0717 Initial Alignment run
• 0731-0811 Initial Alignment restarted (still fighting seismic motion)
  • ALSy taking sooooo long to converge.
• 0754 Seismic to CALM
• 0811 Relocking begins

We'll be here for a while!  Have had BSC1's ISI trip and currently waiting for the earth to calm down before returning to locking---staying in the DOWN state until the Picket Fences calm down (it's been almost an hour and they are still in the orange state.

H1's range has been slowly drifting up over the last 3.5hrs (currently, it's just under 155Mpc).  Rode through a M6.0 south pacific earthquake.  Microseism continues its trend down (maybe finally at 90th percentile---after its HUGE level).

WP 11542

Kepco power supply at EX failed, seized fan. Power supply provides -18V rail to SEI-C1 rack. Power supply replaced and a spare unit staged.

Failed Power Supply: S1202026
Replacement Power Supply: S1201925
Spare Power Supply: S1201928

D. Barker, F. Clara, J. Hanks, E. Von Reis, and Jim Warner

H1 made it back to OBSERVING at 0131.

This was after some adjustments were made to the SQZ Phase after (post Sheila's cable change earlier today, alog #74334).  This was my first time adjusting the SQZ Phase while watching the SQZ nuc33 spectra (to get the black trace below the blue trace).  Phase started at 170.8 and was taken up to 188.8.  (see attached)  I did not adjust the OPO Temperature.

For SDF, we had several diffs for SEI, SUS, & ISC subsystems.  I ACCEPTED all of them (see attached screenshots for the diffs before they were accepted).  Oddly, there were no SDF diffs for the TCSx work (atleast for me tonight).

Looks like L1 is also battling microseism and having issues locking DRMI.

WP11542 EX -18VDC Power Supply Replacement

Fil, Erik:

Towards the end of maintenance, coincidentally 2 seconds before the EDC was restarted, the -18V EX SEIS power supply failed. This caused all of the ADC channels to read 0.0 counts.

Fil and Erik replaced the power supply with a spare unit.

WP11534 h1digivideo1 causing flashing-blue-screens

Jonathan, Patrick, Dave:

Since the upgrade of sw-msr-h1aux last week most, if not all, of the cameras on h1digivideo1 flash their gstreamer viewers periodically with a blue screen. Last Friday I restarted MC1's server process, which did not fix that camera's viewers. First thing today I rebooted h1digivideo1, which did not change anything. This was not surprising, all the server machines were rebooted last Tuesday following the switch upgrade.

Looking at the port statistics on sw-msr-h1aux for the h1digivideo[0,1,2] connections we found that h1digivideo1's VLAN106 port (1/1/16) had a 54% utilization and a dropped packet count of about 9000. Comparing this with the other two machines, which had utilizations around 20% and only dozens of dropped packets.

Jonathan unplugged the ethernet cable for the VLAN106 at both h1digivideo1 and the sw-msr-h1aux ports. The cable appeared to have been seated correctly.

The utilization dropped from 54% to 42% and we have not seen any blue flashes on h1digivideo1's cameras. We will continue to monitor.

WP11541 Add Lock Loss Alert IOC channels to DAQ

Dave:

I created a H1EPICS_ALERT.ini file which contains all of the non-string EPICS records in the lock loss alert IOC. A EDC+DAQ restart was required.

DAQ Restart

Dave, Jonathan:

The DAQ and EDC were restarted. The restart process was interrupted by the loss of h1seiex coincident to the EDC restart, which required some investigation to ensure it was a complete coincidence.

Other than that, and gds1 requiring a second restart, it was a unremarkable restart.

EDC channel count increased by 1082 channels from 56469 to 57551

TITLE: 11/22 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 7mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.65 μm/s
QUICK SUMMARY:

Saw H1 make it to NLN & then have a lockloss seconds later as Ryan was handing off!  :(  Now high useism continues (with downward trend for the last 12hrs.  Currently ALSx is in INCREASE FLASHES.

TITLE: 11/21 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Busy maintenance day; SQZ cabling (PMC, OMC), PSL Refcav alignment, ISS 2nd loop, VAC turbopump checks, valves and gauges added to FCT, ALS laser adjustments at EX, HWS work also at EX, cleanroom building at EY, and camera fibers reseeded in the MSR. Back to NLN at 06:00UTC! Lockloss 12 seconds later :(

HAM7 ISI tripped at 17:59 and 18:03 presumably from the SQZ rack work, HAM2 ISI tripped at 18:19 from the measurements being run on HAM2/3. Picket-fence plot died around 18:05 but was able to restart itself after 2 tries and ~5 minutes. Microseism is the highest its been in ~265 days, above the 90th percentile for the past ~12 hours.

Something happened at EndX at 19:29:21, the ISIs state1&2 tripped and so did the HEPI, not sure why, there was no one down there at the time and a DAQ restart was underway. All the sei signals went to zero, the SUS seems fine. After lots of checks and other restarts Erik went down to EX to physically powercycle some chassis, AA, IOP, Fil later joined him, the -18V DC Kepco power supply was found to have failed. FRS ticket 29752, alog74341

21:18 Started to relock

• Green arms went through some rounds of increase flashes (1 for X, 2 for Y), Y arm eventually said "Find by hand" and then I brought it to unlocked and found it by hand (I had to touch ETMY, TMSY, & ITMY) and got flashes to about 95% and then it was able to lock
• Small flashes on DRMI, nothing on PRMI, CHECK_MICH couldn't get MICH_BRIGHT and neither could I by hand so I went to an Initial alignment at 22:26UTC where I had to intervene during SRC/SRY locking and touched SRM. Finished at 22:54
• PRMI was able to easily lock afterwords but DRMI still struggled despite decent flashes, after 4 PRMI locks we were able to lock DRMI
• Violins were a little high

LOG:                                                                                                                                                                                                                                                                                                                                        

FAMIS tasks 24845 and 24869

Procedure checklist for both stations completed.  No issues were identified at this time.

MX: Scroll pump hours: 208.9

       Turbo pump hours: 118

       Crash bearings: 100%

EX: Scroll pump hours: 6374.8

       Turbo pump hours: 274

       Crash bearings: 100%

Today Daniel, Sheila, Dana, and myself went to EX to look further into the ALS system to hopefully fix the locking trouble that we've had (summary in alog74323). We hooked up a spectrum analyzer before the phase frequency discriminator and verified that we saw the 40MHz signal that should be there. We then checked on table to make sure that there was no clipping on the beatnote path (table layout - D1800270). The two beams looked decently co-aligned after ALS-BS3, and while it wasn't perfect it was deemed good enough. The beam on the ALS-M3 mirror was on the very far edge of the optic, but seemed to be fully making it to the BBPD ALS-PD7.

While we were looking at the beam on the table, Daniel tried tuning the temperature and current of the laser and ended up back at the settings we started with (June 30th alog70976).

There was no obvious reason for any of the locking troubles we've seen recently, and now the beatnote power (H1:ALS-X_FIBR_A_DEMOD_RFMON) seems to be back up to where it has been even though we didn't make any changes. I guess we'll revisit this in 6 months when it happens again...

R. Short, J. Oberling

As we've been seeing the FSS RefCav TPD dropping over the past couple of weeks, this morning Jason and I did a proper tune-up of the RefCav alignment. We started by remotely aligning the beam into the RefCav from the control room using the two picomotor-controlled mirrors in the FSS path. With the ISS on and the IMC unlocked, the signal on the TPD started and ended at:

• Start: 0.69 V
• Finish: 0.76 V

Only seeing a slight improvement in the RefCav TPD signal with the picomotors, we then proceeded to head into the PSL enclosure to touch up the FSS path alignment. We started with a power budget on the FSS path (done with the ISS on and both PSL Guardians paused):

• FSS In: 300.0 mW
• AOM In: 295.5 mW
• AOM Out (Single Pass): 210.1 mW
  • Diffraction Efficiency: 71.1%
• AOM Out (Double Pass): 131.9 mW
  • Diffraction Efficiency: 62.8%
• EOM Out: 136.8 mW

The most obvious issue appeared to be the double pass through the AOM, as Jason had left the diffraction efficiency at around 73.8% after the last on-table FSS alignment, but we decided to start from the highest point upstream and make adjustments at each point. We adjusted WP05 to improve AOM IN, the AOM (mostly upwards in height) to improve single pass diffraction, and mirror M21 to improve double pass diffraction. Our results:

• AOM In: 300.0 W
• AOM Out (Single Pass): 223.7 mW
  • Diffraction Efficiency: 74.6%
• AOM Out (Double Pass): 158.1 mW
  • Diffraction Efficiency: 70.7%

Good improvements all around, and although neither diffraction efficiency ended as high as after the last alignment, with an unlimited maintenance window I suspect we would see more gains. We then checked the alignment through the EOM; seeing it was good with no clipping and measuring 158.4 mW out, we moved on to recovering the RefCav using the iris mounted in front of it. With the RefCav now locked, we touched up the alignment using the picomotor-controlled mirrors while watching the signal on the TPD:

• Start: 0.85 V
• Finish: 0.99 V

Excellent. Using a multimeter to watch the DC voltage, we then touched up the alignment onto the RefCav RFPD by adjusting mirror M25:

• Start: 1.210 V
• Finish: 1.316 V

To finish off our activities in the enclosure, we measured the RefCav's visibility:

• Unlocked: 1.316 V
• Locked: 0.220 V
• Visibility: 83.3%

After returning to the control room, we resumed the PSL Guardians, turned the ISS back on, and adjusted the ISS RefSignal to -1.97 V to keep the diffracted power around 2.5%. I have not accepted this value in SDF as it will likely need to be adjusted periodically this evening/tomorrow as the enclosure comes to thermal equilibrium. This closes WP 11537.

WP 11533

Checked the HAM4 ISI Coil Driver Chassis. Issue reported last week of noisy fan. Fan is spinning and noise reported last week has not returned. Will leave WP open another week.

WP 11532

Functionality of the ETMX SUS Hardware Watchdog was tested this morning. System had reported no triggers over the past few months. One of the DB37 field cables was disconnected. Both the software and front panel LEDs went into alarm. Field cable was reconnected and system returned to nominal state.

D. Barker, F. Clara

TJ, Camilla

After swapping to the old ETMX HWS 530nm diode source and seeing that the HWS signal was not good in 73717, TJ and I this morning confirmed that the mode matching does not match the design. We think we need to adjust the position if the initial collimating lenses, but didn't finish doing this. As we misaligned the beam by movng the fiber assembly D1800125, we left the HWS powered off. Accepted in sdf. 

Gabriele refit the 11/15 LSC FF data 74220, allowing a higher magnitude at low frequency to give us a better fit, fit attached. I lowered the Q of the 17.7Hz feature and added this filter to MICHFF FM8 as "11-15-23B".  We'll want to turn this FF slowly or before Tuesday Maintenance, as the excess low frequency could cause instability.

Plot attached, red is current MICHFF FM7,  blue is new FM8 (double the strength of current FF <8Hz), green is Gabirele's original design before adjusting Q of 17.7Hz feature (all including the high pass filter in FM10).

I lowered the 17.7Hz Q's by a factor of 4 from the original design, to minimize effect on KAPPA_TST error 74259:

• Pole @ 17.8247Hz, Q =  194.416 -> reduced to Q = 48.604
• Zero @ 17.7598Hz, Q = 170.142 -> reduced to Q = 42.535

R. Short, with guidance from K. Kawabe and J. Oberling

I've made some updates to the 'CLOSE_ISS' state in the IMC_LOCK Guardian, which handles the closing of the ISS second loop, that should make the closing of the second loop more consistent.

When the second loop is engaged, the DC working point of the ISS is determined by the output of the second loop's digital AC coupling (H1:PSL-ISS_SECONDLOOP_AC_COUPLING_DRIVE). This output is held when the second loop is closed, but it can occasionally be held far from the average of its oscillations before the second loop is engaged, which in the past has caused locklosses and was mitigated earlier this year (see alog 67347). While the second loop isn't causing locklosses anymore (that I can recall), we do still see the digital AC coupling output being held slightly off from the mean, causing the diffracted power to jump. I've expanded upon Georgia's logic by changing the way the output of the AC coupling drive is held to be more consistent. Instead of waiting to hold the output until it's near the mean over the past 10 seconds (a calculation that itself can take several seconds), the process is now as follows:

1. Take a 10-second average on the AC coupling drive output
2. Hold the output
3. Write the 10-second average value to the AC coupling drive's offset
4. Turn off the input and turn on the offset
5. Unhold then rehold the output (to grab the offset value)
6. Perform Georgia's "within 100 counts" check on the output
7. If successful, zero the offset and turn back on the input

I was able to test this logic during the maintenance period today, both with the IMC locked at 2W and 60W, with great success. We'll run with this for a while to see if over several lock acquisitions the second loop is being engaged with a more consistent digital AC coupling drive. The updated IMC_LOCK Guardian code is loaded and committed to SVN.

I added roughly 5 meters of LMR 195 to the cable sending the 3MHz local oscillator to the demod for the OMC 3MHz signal.  This should add roughly 21ns of delay, or roughly 23 degrees of phase for the 3MHz signal. 

We are hoping that this will shift the turn around point of the ADF sqz angle readback so that our operating point is at a better place to use this readback to adjust the sqz angle. (for background see 74256

This means we will need to retune the SQZ angle before we to observing today.

Fil Daniel

PMC install preparation:

• Extended the rack rails on top of SQZT0 to enable mounting the laser locking fiber beat note chassis.
• Installed the auxiliary signals concentrator 11 into the remote squeezer rack.
• Installed the long coax cables going from the remote squeezer racks to SQZT0.