Resetting the IMs to their O1 start and O1 end alignments, I have identified the IMC as the biggest contributer to pitch beam position changes at PRM, and IM1-3 as the biggest contributers to yaw beam position changes at PRM.

Summary and detailed charts attached.

Copper seat has long been bottomed out due to over torque and gazillions of cycles etc.  The resulting air leak into the RGA volume when the chamber is vented necessitates frequent baking of the RGA volume.  This then requires that the electronics module be removed and reinstalled.  These repeated re-installations of the electronics module along with the associated "hit or miss" alignment of the feed-through pins and sockets has caused the sockets to recess into their connector etc. etc. etc. Postponing fixing one problem is soon to result in a 2nd, avoidable, problem.  

First couple of hours of today addressed the TCSy Chiller which went down over the night.  After that, went through an Initial Alignment and had a few locks.  Unfortunately, later locking attempts would break lock during ASC guardian states (this was probably related to changes Sheila notes in her alog last night); specifics below.

Day's Activities:

• 16:03-17:00:  Addressing TCSy (Corey, Gerardo, Vern)
• 16:31-17:46:  CP4 Work (Chandra)
• 16:44:  Inventory at MY throughout day.(Mark)
• 16:53 -17:05:  Mechanical Room trip (Richard, Patrick)
• ~17:15:  EQ observed (most likely 5.1 from Mexico), seismic taken up to 0.9um/s.  Did not adjust SEI_CONFIG.
• 17:27:  Initial Alignment run
• 19:55:  Hartman table work (Nutsinee)
• Commissioning work in the afternoon around the DC Readout step is currently underway.

ISC Locking Note:

• First few locks after initial alignment were fine, but h1 started to break lock at various midpoints through Guardian States---namely when ASC would drive the SRM off (noted by Kiwamu).  This is related to work from last night (see Sheila's alog)
• To ride through the roughness, you will need to tweak the SRM (in 1.0 um steps) to prevent (1) POP90 (green on front StripTool) from increasing & (2) AS over POP90 deceasing.
• Also need to wait before COIL_DRIVERS to allow bounce/roll/violins to damp out.  

OPS INFO Notes:

PRM ALIGN:  If AS_AIR is flashing, you are not locked, and this is most likely due to a misaligned PRM.  Take ALIGN_IFO to DOWN, & then tweak PRM while looking at AS_AIR.  Then go back to PRM_ALIGN for the alignment.

Reminder that two weeks ago we swapped out the 16bit DAC card in h1oaf0, but got a dac-zero'ed error on Sunday (9/25). Yesterday (9/27) we swapped the first and third ADCs and are waiting to see if we get another DAC error. On Keith's suggestion we are running a script on h1oaf0 which will report to a log file if the DAC FIFO status is anything other than OK. We suspect that a FIFO full or empty error is precipitating the zeroing of the DAC channels.

The script is called monitor_iop_dac_status.bsh, running in the background on h1oaf0's general core. It appends text to the log file:

/opt/rtcds/lho/h1/target/h1iopoaf0/logs/h1iopoaf0_16bitDAC_fifo_status.txt

It runs every minute, outputting a date stamp and any errors. The DAC FIFO status is gotten from the /proc/h1iopoaf0/status file.

model restarts logged for Tue 27/Sep/2016
2016_09_27 05:57 h1nds0
2016_09_27 09:37 h1psliss
2016_09_27 09:54 h1psliss
2016_09_27 10:02 h1fw0
2016_09_27 10:02 h1fw1
2016_09_27 10:06 h1dc0
2016_09_27 10:08 h1broadcast0
2016_09_27 10:08 h1fw0
2016_09_27 10:08 h1fw1
2016_09_27 10:08 h1nds0
2016_09_27 10:08 h1nds1
2016_09_27 10:08 h1tw1
2016_09_27 10:24 h1alsex
2016_09_27 10:24 h1calex
2016_09_27 10:24 h1iopiscex
2016_09_27 10:24 h1ioplsc0
2016_09_27 10:24 h1iscex
2016_09_27 10:24 h1lscaux
2016_09_27 10:24 h1lsc
2016_09_27 10:24 h1omc
2016_09_27 10:24 h1omcpi
2016_09_27 10:24 h1pemex
2016_09_27 11:18 h1dc0
2016_09_27 11:18 h1iopoaf0
2016_09_27 11:18 h1pemcs
2016_09_27 11:20 h1broadcast0
2016_09_27 11:20 h1calcs
2016_09_27 11:20 h1fw0
2016_09_27 11:20 h1fw1
2016_09_27 11:20 h1fw2
2016_09_27 11:20 h1iopoaf0
2016_09_27 11:20 h1nds0
2016_09_27 11:20 h1nds1
2016_09_27 11:20 h1ngn
2016_09_27 11:20 h1oaf
2016_09_27 11:20 h1odcmaster
2016_09_27 11:20 h1pemcs
2016_09_27 11:20 h1susprocpi
2016_09_27 11:20 h1tcscs
2016_09_27 11:20 h1tw0
2016_09_27 11:20 h1tw1
2016_09_27 11:25 h1broadcast0
2016_09_27 11:25 h1dc0
2016_09_27 11:25 h1fw0
2016_09_27 11:25 h1fw1
2016_09_27 11:25 h1fw2
2016_09_27 11:25 h1nds0
2016_09_27 11:25 h1nds1
2016_09_27 11:25 h1tw0
2016_09_27 11:25 h1tw1
2016_09_27 11:31 h1broadcast0
2016_09_27 11:31 h1dc0
2016_09_27 11:31 h1fw0
2016_09_27 11:31 h1fw1
2016_09_27 11:31 h1fw2
2016_09_27 11:31 h1nds0
2016_09_27 11:31 h1nds1
2016_09_27 11:31 h1tw0
2016_09_27 11:31 h1tw1
2016_09_27 11:32 h1nds1
2016_09_27 12:04 h1tcscs
2016_09_27 12:35 h1tcscs
2016_09_27 13:48 h1iopiscey
2016_09_27 13:48 h1pemey
2016_09_27 13:49 h1iopiscey
2016_09_27 13:49 h1pemey
2016_09_27 13:50 h1alsey
2016_09_27 13:50 h1caley
2016_09_27 13:50 h1iscey
2016_09_27 14:28 h1nds0
2016_09_27 22:50 h1nds0

maintenance day. h1lsc0, h1isce[x,y] powered for dc-supply install. h1oaf0 powered for adc swap. psl-iss model change. tcs restarts to test alarms. multiple daq restarts. several unexpected nds0 restarts.

model restarts logged for Mon 26/Sep/2016 No restarts reported

There was some discussion about the prospects for pushing up the CARM bandwidth to better suppress frequency noise above several kilohertz. Since we currently run with a 20 kHz UGF or so, adding more gain would require a UGF that approaches (or exceeds) the FSR.

The attachment shows the CARM OLTF taken in the viscinity of the FSR, showing the expected dip and (overcoupled) phase feature in the optical plant. The feature does not have infinite Q, so in principle any secondary UGFs could be eliminated, though it would be tricky. The HWHM is 0.6 Hz, consistent with CARM intensity TFs taken previously.

(Corey, Gerardo, Vern, Betsy on phone)

9am - 10am

As noted earlier, TCSy CO2 Laser was down and we had a chiller flow alarm.

Went out to the Mezanine to assess the TCSy Chiller.  Here's what was noted for the chiller:

• Had power (it was NOT tripped OFF).
• LCD window stated:  WARNING!  LOW LEVEL
• The fluid level indicator was completely empty (!)
• We also went out to the TCSy Table area and noticed lots of air bubbles in the clear tubing...it was also noticeably loud.
• Returned to Chiller.

First thing we did was start filling the Chiller (there was a little "DISTILLED LABORATORY WATER" [1st pic], ended up filling remainder with a jug labeled "Premium Distilled Drinking Water" [2nd pic]).  It took about 3L before we saw the float indicator move from EMPTY.  After that it was filling 0.5L at a time and watching.  We took it up to about 6L and had a level near FULL.  As we were chatting for a minute or two, we noticed water pooling at the base of the front of the TCSy Chiller (!) [3rd pic].  

Investigated by removing the resevoir cover assy (this cover allows you to see resevoir filters and also holds the cap for filling it).  We could clearly see RETURN water splashing all over the place due to the Stainles Steel mesh RETURN filter [4th pic].  Water was easily escaping this resevoir by having some water flow out of the resevoir & down the exerior panels of the Chiller and dripping on the floor.  (In contrast to TCSx Chiller [5th pic] which was a pristine and calm waterfall from the RETURN down into the resevoir.)

Removed the mesh filter & reseated it.  It wasn't trivial.  Our first attempt did not look as nice as TCSx.  Vern reseated again but this time flipped the mesh filter, and this time it looked much better [6th pic] (& more importantly, similar to TCSx).  The OUTPUT "coffee filter" looking filter (which has been noted as being partly green) looked like it had an air bubble under it.  We tried pushing it down, but it didn't help.  But overall, we looked better after reseating the mesh filter.  When reinstalling the Resevoir Cover, we saw a tab under it (most likely to keep the mesh filter in place); our guess is the mesh filter might have been dislodged by this tab when the resevoir was reinstalled yesterday); see [7th pic]  

Since we lost some water due to the leak we witnessed, we topped off the Chiller again with another 0.5L (so total of 6.5L was poured, with some splashing out).

After we were done at the Chiller, we went out to the TCSy table area.  We looked inside the enclosure to make sure there wasn't a puddle of water (there wasn't).  We then turned on the laser controller.  (The water tubes were much quieter than before...no visible air bubbles.).

After an Alignment, I wanted to run an INIT state for ISC_LOCK.  Noticed that it was nowhere to be found.  Mentioned this to TJ, and he ended up running the INIT by hand via Guardian command line.  

While Corey and company were working on the TCS chiller this morning (30037), I took this opportunity to do a quick health-check on OMC with a single-bounce beam. So far things seem good.

• OMC length loop
  • Seems fine. With the nominal locking settings, the UGF was found to be around 6 Hz which is a factor of two higher than it had been in this past February (25605).
• I/Q dmod phase.
  • The current demod phase of 110 deg give us a I/Q separation by two orders of magnitude. Good enough.
  • This was tested by injecting a 10 Hz excitation in pzt2.

The attached is the measured open loop from this morning.

FAMIS#7405

Laser Status:
SysStat is good
Front End Power is 34.68W (should be around 30 W)
Front End Watch is GREEN
HPO Watch is GREEN
PMC:
It has been locked 0.0 days, 1.0 hr 33.0 minutes (should be days/weeks)
Reflected power is 29.41Watts and PowerSum = 138.0Watts.

FSS:
It has been locked for 0.0 days 1.0 hr and 33.0 min (should be days/weeks)
TPD[V] = 3.913V (min 0.9V)

ISS:
The diffracted power is around 3.862% (should be 5-9%)
Last saturation event was 0.0 days 1.0 hours and 33.0 minutes ago (should be days/weeks)

Possible Issues:
PMC reflected power is high

 

Most of the meeting was summarizing Maintenance Day yesterday.

No Safety Meeting (All Hands is tommorrow).

Dicsussion about protocol for transitioning to LASER SAFE for Maintenance.

Since about 2am (about 5min after the lockloss of the night), We have been getting the following TCS Alarm:

"No TCSY chiller flow"

• This corresponds with a RED CO2Y on the Ops Overview, which leads to LASER STATUS Not Ready for the TCSy CO2 Laser.
• Trends show the CO2 Power Supply Current dropping from 20 down to ~0.
• Laser Temp dropped from a steady 24degC down to a sawtooth signal between 20-22degC.

- The DCPDs whitening chassis was found to be switched to Low-Z - transitioned back to high-Z.

- We added notch filters for the 4.01Hz and5.01Hz SRM dither lines into the SRCLFF path. They are a Q of 60, 40dB deep. The high Q was picked to minimize the phase distorsion - it is actually slightly too high for the line width (which is driven by coupling fluctuation.) With this modification, the SRM alignment dither can be running in LOW_NOISE in case we need it.

- We also revived the pr2spotmove.py and pr3spotmove.py scripts (stored in /ligo/home/controls/sballmer/20160927/). They move the spot position on PR2 and PR3 respectively, leaving all other things equal. We were hoping to see a change in the auxiliary loop length noise - aspecially SRCL (they are higher than O1 - see Sheila's attachment.)

- First, we moved the pr2 spot position: (PR3 by 10urad in pit and 5urad in yaw). Next we did the same for the pr3 spot position (10urad on PRM). Unfortunately we did not see any change in the auxiliary loop noise.

This information is based on the cumulative spectrum of recent lock stretches (9/18-9/26, computed from Fscan SFTs with spec_avg_long), plus Fscan magnetometer data.

0.997698 Hz

• Worst comb at present
• Previously noted in ER9
• Higher in DARM 9/24-9/26 than 9/18, 9/23
• Appears in CS mags and is fairly steady there (based on available Fscan data since April).

1.0 Hz

• Present in CS, EX, EY magnetometers

0.5 Hz odd harmonics (AKA 1 Hz with 0.5 Hz offset)

• Looking better! Weaker than ER9 by a factor of ~5-10 (varies across spectrum). Improvement presumably due to timing system LED reprogramming.
• Hoping for more improvements as IO chassis timing slaves are put back on alternate power supplies.

1.0 Hz with 0.998889 offset (new)

• Very close to 1 Hz comb, but distinct peaks are visible

0.987987 Hz (new)

• Present in EX magnetometers
• Increases on 9/11/16 in magnetometers

Attached plots:

• 1: LSC-DARM_OUT_DQ cumulative spectrum with the noted combs highlighted
• 2-4: Representative (SUSRACK) magnetometer channels at CS, EX, EY with combs highlighted
• 5-6: Plots showing 0.5 Hz odd harmonics improvement between ER9 and present

I removed the drivers S1600266 and 267, added capacitor 4.7pF in the C25 position and returned the drivers to service. HV was turned back on and confirmed. See Patrick's aLog entry.

The PSL tripped around 21:37 UTC, it took us about an hour to recover from this . We called Jason and Peter, Jason called back and walked me through restarting it remotely.  I added about 200mL of water to the chiller, there was water on the floor in front of the chiller.  Jason confirmed that this is normal, and that the turbulence and bubbles in the chiller tube are from a known slow leak. 

Sheila, Keita

The 3rd whitening filter does not switch for AS_C segment 3.  The readbacks look OK.