The AC units in the PSL were both run at 24C, and while Jason and I exited the PSL more than 3 hours ago, the temperature in the PSL remains elevated, which has effected the Reference Cavity, and caused us to have to adjust the ALS temperature at EX, and until the PSL/REF Cav temperatures recover, the squeezer is unable to lock.

The history is that we have been running both PSL AC units at 23C, and this cools the PSL to a level that means there is a noticeable recovery 

At some point a few months ago I ran one AC unit at 23C and one unit at 24C, to try and prevent the large excursion in temperature, and this seemed to be better.

Since then there's been a change of running both AC units set to 24C, as they were today, and from trends this appears to have changed November 8th.

PSL persons had the best intentions, and recognize the need to revisit the strategies for minimizing the effect of working in the PSL on the rest of the interferometer.

Attachment 1, 2, and 3: shows TABLE North and TABLE South temperature sensors, over 24 hours, for three different Maintenance Days.

1. both AC units 24C: today, where and one can see that we worked in the PSL, and where the residue heat from our activities has remained in the enclosure, which has added heat to the PSL components.
2. both AC units 23C: October 10th, 2018, where the PSL temperature also had not recovered from the cooling during Maintenance.  This represents that issue we've observed, where the PSL is cooled off and doesn't recover until roughly the next day.
3. both AC units 23C: October 30th, 2018, where the PSL temperature recovered more quickly

Attachment 4: shows 120 days for TABLE North and TABLE South temperature sensors, and on the left one can see that Tuesday Maintenance temperature changes drop in temperature more than the Tuesdays we've worked in the PSL on the right side of the plot.

The update that Jim pushed today required me to update the SEI SC nodes. This allows the HAM nodes to use Brian L's new fader code, just like the stage 1 BSC's. The configuration files needed to be updated and a handful of files needed minor changes, but no overall code changes. The top level SEI_CONF node also reflects these changes.

The Large EQ button was tested and a few of the more popular SEI_CONF states. The IFO has been left in the WINDY configuration.

In the hopes of a single click to initial alignment, Sheila asked if I would write a quick node to manage the other initial alignment nodes and run through the process. This node is just a framework and I imagine that we will add much to it as we test it out. Since this was a long maintenance day, there is no time to test it out tonight.

Terry, Sheila

After the squeezer fiber was replaced in November, the total transmission of the chain was 40%, but it dropped over a couple of days starting Nov 30th 45903.  Terry and I went out this morning, Terry loosened and tightened the connections at the feedthrough, watching the OPO reflection (with the OPO off resonance).  While we were doing this, the transmission to SQZT6 dropped by 30% (from 3.5 mW to 2.6mW)  We then disconnected the fiber and measured the power.   We cleaned the fiber, and the resulting transmission to SQZT6 was the same (2.6mW).  We went to ISCT6 to measure the power after the first periscope mirror, and measured 2.55mW, which agreed with the calibrated channel from the reflected diode.  The OPO was slightly on resonance at this time, when we pull the OPO further off resonance the reflected power went up to 2.6mW.  

With the maximum of 20mW of power injected into the fiber, this allows us to have a nonlinear gain of 2.3. 

TVo and I went on a leak hunt and went through the table, lines, and then opened the chiller. Opening chiller revealed a puddle under the pump (see attached picture). It looks like we have a leaky pump shaft seal. Filed WP8036 and went to swap the chiller with the spare. The spare chiller would give us a "Low Fixed Pressure" warning and would then shut down. The low threshold was set to 3.0psi and when starting the chiller it would only read around 1.4psi. No flow was ever observed while we tried to turn this on. We eventually gave up and swapped the leaky one back in.

Back in April, Greg Grabeel fixed what is currently our spare from an issue that seems to be exactly the same (alog41314). He ran the chiller in a closed loop for a week back then, so I imagine that it should work now as well. Perhaps the issue we had today was just operator error. We will look into getting parts to fix the leaky seal, and how to get the spare running again.

I reenabled the serial communication through the front panel of the TCSX chiller. For some reason the display went blank and became unresponsive after I did so. The serial communication was already running on the TCSY chiller.

WP8030 New Dolphin EEPROM

Jonathan, Dave:

All dolphin'ed machines (except h1psl0) had their EEPROM version changed from 08 to 97. This firmware forces the dolphin IXH611 card to run in PCIe Gen1 mode (max bandwidth 2.5GT/s).

Procedure for this upgrade can be found in my wiki page

After each reboot, we verfied the eeprom version number and that the card reports x8 slot width with reduced bandwidth (excerpt from dis_diag shown)

            PCIe slot state            : x8, Gen1 (2.5 GT/s)
            PCIe slot capabilities     : x8, Gen2 (5 GT/s)

No problems were encountered, the machine reboot order was closely coordinated with detector engineering.

WP8026 Reboot of DAQ computers close to their 208day limit

Dave:

Close to the time for a DAQ restart due to model changes, h1tw1 was rebooted and h1broadcast0 was issued a poweroff command.

At the time of the DAQ restart, h1dc0 was rebooted. It did not automatically start daqd on startup, I had to manually start the process from monit.

Soon afterwards I started h1broadcast0.

In all cases the reboot/poweroff process got stuck and the machine needed its front panel RESET button to be pressed (a known problem with gen1 front ends). Also in all cases an FSCK was needed due to the long uptime, slowing down the recovery time.

WP7996 Removal of EZCAREAD parts from h1odcmaster

Dave:

I modified h1odcmaster to replace EzcaRead input parts with EpicsIn parts. DAQ restart was required. I still need to add the CA data transfer to the CDS_CA_COPY guardian node

WP8027 h1guardian1 removal of swap space

Jonathan, TJ:

h1guardian1 was configured to not have swap space (overflow memory on disk) and was rebooted.

WP7929 ISI Senscor HAM and BSC

Jim W, Jeff K, Dave:

New models for all BSC and HAM ISI plus h1seiproc were installed today. We found a problem with the front end model startup when FIR filters were removed by the fir filter file existed. Details of this upgrade in Jim's alog.

Crash of h1iscey apparently due to running dis_diag

Dave:

After the Dolphin upgrades, I ran my script on h1build which ssh's onto each Dolphined front end and runs dis_diag. It proceeded through the corner station machines but when it got to h1iscey this machine rebooted itself, Dolphin crashing h1susey and h1seiey in the process.

h1susb123 IOP Dackill

Dave:

After recovering the systems, h1iopsusb123 went into a DACKILL state which suggested a possible hardware problem with the 18bit AI chassis. I verified no one was in the CER, and then restarted all the models on h1susb123.

Crash of h1oaf0

Jim W, Dave:

During on of the h1seiproc restarts, h1oaf0 models stopped running. When I ssh'd into the machine and ran "lsmod" the login session froze. I eventually rebooted the computer via the front panel RESET button.

TITLE: 01/08 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY:

I covered this shift for Cheryl as she did IO PSL work during Maintenance today.  

Today was a long Maintenance because of the MX Ion Pump work (which required gate valves be closed...and this is expected to go until/past 4pm).  

Another notable item is the leak for TCSy chiller was discovered by TJ & Thomas (they are currently swapping out the chiller).
LOG:

• Contractors:  Portapotty (16:05), Paradise 17:01, Unifirst (17:30)
• Guardian:  IMC to OFFLINE,  ISI CONFIG to SC_OFF_NOBRSXY,  ISC_LOCK to DOWN,  all SEI to OFFLINE,  all SUS to SAFE
• 16:02-16:25 IM alignment (Cheryl)
• 16:04-16:17 Measurements around HAM2 (Bubba)
• 16:09-16:58 HEPI checks both end stations (Hugh)
• 16:18 MX ion pump work (Kyle, Chandra)
• 16:33-16:50 Work around racks (Richard)
• 16:35 PSL IO camera/other work (Cheryl)
  • will CLOSE psl shutter
• 16:42 SEI HAM3 work (Jim)
• 16:48 FSS electronics at ISCT6 (Ed)
• 16:50-17:40 EX als Power work (fil, richard) ----> ALS laser confirmed to be back ON (TJ)
  • EX als fiber
• 17:30 PSL DBB work (Jason)
• 17:35 EX ground coupling to DARM (robert)
  • 21:30 Fil joining
• Taking ISIs to OFFLINE, SUS to SAFE
• 17:47 EX run a grounding wire (fil, ed)
• 18:00 Bootfest begins!  (dave, jonathan)
• 18:06 LN2 at CP2
• 18:12-18:17 ISCT6 work (Nutsinee)
• 18:16 Pulling fire hydrants--> alarms are being texted to site staff
• 18:25 TCSy checks (tj, thomas)
• 18:25 aip on ham7 (chandra)
• 18:45 snow starting!
• 19:39 external rf measurements at CER & PSL racks (dick, marc)
  • 21:05 Heading out to do measurement
• 20:48 HAM6 work (Terry, Sheila)
• 20:55 TCSy chiller swap (Thomas.Thomas)
• 22:17 EY run a grounding wire (fil)
• 22:25 Measurements out in LVEA for platforms (Bubba)
• Handing off to Cheryl at 22:29pm

For 30 watts of input power:

Arm Power = 127.2 +- 5.4 kW

SRCL Dither to Arm Power Measurement:

Long ago, Peter F came to Hanford and asked me to measure the arm power by moving the SRM.  The idea is to use the SRCL dither to cause the arm power to fluctuate differentially.  When the arm power fluctuates differentially, arm radiation pressure couples to DARM.
I measured the TF from SRCL to arm power transmissions RIN and SRCL to DARM at 30 watts of input power and 43 power recycling gain and SRCL feedforward off.  (Had to divide the DARM calibration by two because our DARM is calibrated such that LDARM = Lx - Ly, but I assumed LDARM = (Lx - Ly)/2.  This was particularly hard to find.)
I then fit some constants α1 and α2 to the TFs: α2/f^2 to the SRCL_ctrl/ArmPowerTransRIN measurements and α1/f^4 to the SRCL_ctrl/DARM, and recovered arm power via the equation

Parm = α1/α2 * π2 * mq * c

where mq is the mass of the ITMs/ETMs.  See the attached PDF for a more exposition to the above equation.  In the future we can just take the TF from Arm Power Transmission RIN to DARM.

Results:
α1 = 7.11 +- 0.07 × 10-13   (Attachment 2)
α2 = 6.6  +- 0.3  × 10-7    (Attachment 1)


From this measurement we can back out an approximate arm power gain.  The power in the arms is given by 
Parm = Pin * gP2 * garm2 / 2
where gP2 ≈ 43 is the power recycling gain and garm2 is the arm power gain.  

In a perfect, lossless world we would have garm2 = 282.  
In reality, we have garm2 ≈ 197

EDIT: Gabriele comments that for this low of an arm gain, our losses would have to be incredibly high, on the order of 5000 ppm.

EDIT: Stefan asked about the SRCL to Arm RIN fit differences between the arms.

There are two QPDs (A and B) at the ends of both arms.  Names are like H1:ASC-{X,Y}_TR_{A,B}_NSUM_OUT.  These QPD sums are used for monitoring the arm powers.
In Attachment 1, I refer to H1:ASC-X_TR_A_NSUM_OUT as X1, H1:ASC-X_TR_B_NSUM_OUT as X2, etc...
In the legend, we can see the individual fit values to each ArmTransRIN/SRCL TF to a α2/f2 model.  Each of these fits were made with a MCMC and had an uncertainty of around 0.8%

                                       QPD/SRCLctrl TFs
Model Value               X1           X2           Y1           Y2
----------------------------------------------------------------------------
α2               6.94 × 10-7   6.86 × 10-7  6.43 × 10-7  6.29 × 10-7

We can see that the X1 and X2 fits are close (within 0.6%), and Y1 and Y2 fits are close (within 1%), but the X and Y arm fits together are slightly different (within 4%).  
Looking at the PDF attached, we can say that the optical response of the Y arm to the SRCL dither γ is less than the optical response of the X arm.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


HEPI Offloading Measurement:


Results:

Change in HEPI offload for each arm during 2 to 20 watt powerup: (see Attachment 3)
ΔHEPI_X = -3287 +- 261 nm  (or +-8%)
ΔHEPI_Y = -3587 +- 246 nm  (or +-7%)

Change in X Arm Power from 2 to 20 W Input = 95 +- 8 kW
Change in Y Arm Power from 2 to 20 W Input = 104 +- 7 kW

Total X Arm Power at 20 W = 105 +- 8 kW
Total Y Arm Power at 20 W = 115 +- 8 kW

Estimated X Arm Power at 30 W = 158 +- 13 kW
Estimated Y Arm Power at 30 W = 173 +- 12 kW

Arm Power Ratio = Px/Py = 0.9 +- 0.1


As a double check of this arm power measurement I have been looking at HEPI offloading of tidal during INCREASE_POWER.
When the input power increases from 2 to 20 watts, HEPI moves due to the increase of radiation pressure by about 3000 nanometers in each arm.  
I fit a simple trend line to capture what the tidal servo was doing anyway, then subtract it at after around 400 seconds when the offload is complete, for 10 separate INCREASE_POWERs.  (Attachment 3)

We can use these results to estimate the arm powers, assuming all other things being equal in LSC->HEPI tidal offloading, and ΔHEPI_X = ΔLXarm:

HEPI Analysis

ΔPx = ΔHEPI_X * c/(4 k),
where k = DC compliance of the quad TST stage = 2.6 × 10-3 m/N.

If we trust the measured PRG of ~46 for each of these powerups, that would correspond to an arm gain of 
gXarm2 ≈ 230
gYarm2 ≈ 250


So this measurement reports much higher arm powers, but also a huge mismatch between arms (> 10%).



We can learn a lot more about the IFO from this measurement and others like it.  The end goal of this is to understand noise couplings to DARM, many of which are governed by these hard to measure numbers like arm power, power recycling gain, signal recycling gain, SRCL detuning and optical spring, DARM offset, coupled cavity poles, etc.  If we can make a more complete picture of the IFO, hopefully we will find some things that don't add up.

EDITTED for clarity of which measurement gave me which information, added the arm power calculations from HEPI offloading done with Sheila, and added comments from Stefan and Gabriele.  

A temporary FSS chassis (S1812014) was installed on ISCT6. I connected the +/- 18VDC to the chassis as well and left it powered that way. The accompanying +/- HV power supplies were placed on the floor inside the squeezer rack and connected to the chassis. The HV supplies were set to 180VDC as per their counterparts in the CER mezzanine and then turned off until they're needed. WP#8029

WP 8033
WP 8035

In order to do some cable clean up at EX, the AC power for the ALS laser was removed from the old laser kill circuit. The ALS laser is now controlled by a enable/disable interlock. Old power cable had about 20 ft of extra cable coiled up on top of ISCTEX. While removing power cable the ALS fiber was accidently moved, commissioners were notified.

While looking at grounding at field racks, Richard noticed the electronics bay racks and field racks did not have a true grounding cable tying both rack sets. New grounding cable has been pulled in at EX and EY.

F. Clara, R. McCarthy, E. Merilh

We reconfigured the h1guardian machine to not use swap and rebooted the system.   This was done with:

swapoff -a
remove the swap entry from /etc/fstab

It took a few minutes to come back.  TJ then started any nodes that did not come up initially.

On December 27th I moved the laser current up and away from multimode region (alog46181). Attached a plot to show that the pump power has been stabilized since. I will follow up with another intensity noise plot later. We may no longer need the intensity stabilization servo (doesn't hurt to have though). Also attached a 60-day plot to show that we've been having this issue for a long time. The green power drift coming out of the SHG was mistaken for misalignment issue at one point.

I checked on the HAM 7 annulus ion pump failure - tapped on the pump body and was able to make the red overload light disappear, but green LEDs are at 10 mA maximum, so the indication light on MEDM is still red. It's not worth the effort to replace this AIP since HAM 7 & HAM 8 annuli are combined and still being pumped with HAM 8 AIP, and HAM 7,8 will be removed after O3 for A+.

Craig Georgia Jenne Sheila

Edit: this is Georgia, accidentally posting as Keita!

• Moving spot on TRY (and TRX a bit)
  • As Sheila mentioned in alog-46727, we noticed that the spot positions on the ASC transmission QPDs had drifted over time, particularly on TR_B_PIT on the Y arm. This is a potential culprit for our scatter-shelf from ~20-60 Hz. We tried adjusting the spot position on the ETM by adjusting the A2L gain at the L2 stage, while the spot position was controlled by the dither loop.
  • We used a reference time from a 2W lock on December 8, before the scatter shelf was problematic, and adjusted the P2L and Y2L gains to move the spot position back towards the old position. We found the spot moved more on TR_B than TR_A.
  • Meanwhile Jenne realised that over time the TMS, particularly TMSY, had drifted according the the witness sensors. We adjusted the alignment sliders on TMSY to bring it closer to its position on December 8.
  • First attachment shows:
    • First column: Y and X QPDs during a lock on December 8
    • Second column: same QPDs during today's lock where we adjusted the A2L gains and the TMS
    • Third column: The P2L and Y2L gains on ETMY and ETMX
    • Fouth column: TMSY Pitch alignment, and PR gain
  • In the end we chose a spot position which gave slightly better PR gain rather than trying to return to exactly the same spot.
  • We got to nominal low noise (03:40:00UTC) and, at least qualitatively, the scatter shelf was still there. I tried to sit at NLN for a while so we could get a nice spectrogram, but we lost lock 10 mins later.
  • I brought the A2L gains back to where they were.
• Updated Green references
  • In the next lock, while we were locked at 2W with all the ASC loops on, Craig un-shuttered ALS and updated the QPD offsets using Kiwamu's script. We also updated the camera references though they were not very different.
  • We noticed a perplexing 0.5 Hz glitch on the Y arm transmission, when the alignment was bad (second attachment).
• After some soft loops weirdness, (lockloss after increasing CSOFT and DSOFT gain in ENGAGE_SOFT_LOOPS) we finally got to NLN again only to lose lock after 10 minutes again! Attachment 3 is a screenshot of some ASC signal at the lockloss.

• attachment 1: 
  • ISS Second Loop, 1" lens, 2-aperture baffle, and 2" steering mirror
• attachment 2:
  • HAM2 components, 1" lens, 2-aperture baffle, and 2" steering mirror, all circled

These were taken after the PZT mount swap, and show on the 2" mirror that the beam is high and to the right.  The ISS Second Loop QPD has been reading around 0.8-0.9 and with there being some light on the right side of the 1" lens, it seems possible that there's clipping in this path at the time of the pictures, and possibly up to today.  I did trend the sum of the Second Loop 1-4 DC sensors, and after the PZT swap, the sum of the firsgt 4 DC PDs dropped by 1/3, for a 20W lock, which also suggests clipping.  Work on the alignment of the beam to the ISS Second Loop may have happened today, or may be planned for tomorrow+.

Jenne, Craig, Georgia, Sheila, Kara

We had a couple of locklosses this afternoon when the dither/soft loops were engaging.  We have made some changes in the guardian that we think will help prevent this in the future.

• Housekeeping: we moved the turning on of all the dithers into PREP_ASC, we also deleted the gaurdian states that were for the slow ASC engaging, because these haven't been kept up to date and weren't doing things correctly.  We also made sure that the SOFT pitch blends are off. 
• We have split the SOFT and dither loops out of ENGAGE_ASC into a state called ENGAGE _SOFT loops.
• before we were turning on the CHARD loops at high bandwidth before engaging the dither loops.  This means that if the alignment isn't great, the A2L noise from the CHARD loops covers the dither lines in DARM.  We are now leaving CHARD in the lower bandwidth state (~40dB lower gain than used for power up) while the dither and soft loops are engaged, and then increasing their gain once the dither loops have converged.
• We also ran into the soft limiters on SRC1 P when the soft loops were engaging once.  We are now waiting for all the ASC except the soft loops to converge, turning off the soft limiters, and then engaging the soft loops so that the other loops can follow with their full gain.

We see that spot positions on the TR QPDs are rather different now than they were before the dither loops were commissioned (attachment), especially the Y B QPD.  We would like to try moving these spots while in low noise to see if we can reduce the fringe wrapping we are seeing, but in the past we have difficulty moving our spots after powering up because of the dP/d angle instability.  Georgia is trying right now to move the spot while we are at 2W to be more similar to the old spot.