After a quick coherent hunt between OPO frequency noise and PEM channels it seems that a lot of noise came from SQZT6 and ISCT6 table (2nd and 3rd row, 3rd column). Both ISCT6 and SQZT6 accelerometer shows highest coherence at ~1kHz compared to accelerometers and microphones everywhere else in the LVEA (along PSL fiber path). ISCT6 accelerometer also shows high coherence at ~300Hz (2nd row, 3rd column). Still unclear where the noise comes from, but it's probably safe to assume that whatever noise is giving us trouble they're all local to HAM6 area.

For future reference: attached a sweep sine of CHARD yaw including the coupling to all other yaw degrees of freedom. Apart from DHARD, all other error signals see the excitation quite well.

Time To Perform Task:  14min

TCSx:  Chiller level was at "Max Level" & no water added.

TCSy:  Chiller level was at 8.4cm & 1200 mL of water was added to [get] level to the "Max Level".

NOTE:  I only removed the GRAY chiller cap for TCSy Chiller.  I was not sure where/how to remove the mesh filter.  Thomas later informed me we are supposed to remove the Chiller Cap "Panel" to gain access to this Mesh Filter.  So there is a chance the level I filled to might be erroneous.

Laser Status:
Front End Power is 33.3W (should be around 30 W)
70W Output Power is 75.75W
Front End Watch is GREEN
70W Watch is GREEN

PMC:
It has been locked 2 days, 1 hr 6 minutes (should be days/weeks)
Reflected power = 17.14Watts
Transmitted power = 43.5Watts
PowerSum = 60.64Watts.

FSS:
It has been locked for 0 days 21 hr and 15 min (should be days/weeks)
TPD[V] = 2.075V (min 0.9V)

ISS:
The diffracted power is around 1.6%
Last saturation event was 2 days 1 hours and 6 minutes ago (should be days/weeks)

Discussed CDS crashes from the weekend (due to recent upgrad).  

Talking about Maintenance

• 4hrs (i.e. end at noon)
• Several model updates
• PSL:  RefCav alignment (Peter, starting before 8am)
• PSL:  GigE Camera (Cheryl)
• PSL:  VCO amp swap (Peter, Marc)
• PCAL:  tuneup for EX laser/aom (Travis)
• CDS:  upgrading operating systems for control room computers (Carlos)
• CARM chassis read back fix (Marc)
• PSL new FSS/PMC/Laser screens (Jeff B, Jason)
• New big coil & PEM troubleshooting in LVEA & EX/EY VEA (Robert)
• Beckhoff updates (Patrick)

General Coordination Tasks

• Opportunistic Mode Matching
• Regular Charge Meas
• Corner VE Leak checking
• Test Access Laser Kill Switch
• HAM2 Camera Adjustment

(Hang, Corey)

Ran through the Operator Check Sheet's CFC (Configuration File Change) Check.  h1asc was the one model which had a CFC Warning (red box on CDS Overview medm).  Ran the command "filter_diffs asc", and had a few diffs which came up (see attached output screenshot) related to Radiation Pressure Compensation (RPC) filter banks (as well as some "unused" filter banks, i.e. ASC_DC7_P).  Since this was probably commissioning-related, I grabbed Jenne to ask about the diffs, and she steered me to Hang.  Hang confirmed these were changes he did, and so we hit the Load Coefficients for this model (@16:39utc and this cleared the CFC Warning).

All looks nominal

TITLE: 10/08 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    Wind: 0mph Gusts, 0mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.20 μm/s
QUICK SUMMARY:

• Mostly recovered by the EY crash (see Dave's Alog).
• Jenne noticed both ALS lasers are off & Richard is checking on an Access Control "FB Error" for these guys.

Both ALS lasers are off.  It looks like they both think their interlocks were triggered / tripped?

I have remotely reset the h1susey computer. Procedure was: disable the Dolphin IXS600 port h1susey is connected to and then issue a 'chassis power reset' command via the remote management IPMI port.

The whole process only took about 5 minute to complete, so I was surprised to see the SWWD on SEI-EY had tripped (should take 10 mins to trip). Perhaps this was tripped before the reset?

I walked into the control room this morning to a symphony of names of optics being announced on the verbal saturation alarm.  The ISC_LOCK guardian had connection errors, and so was not able to recognize the fact that the lock was lost, so it had not run the DOWN state.  I stopped and restarted the ISC_LOCK guardian, which cleared the error long enough for it to start the DOWN state. 

While the DOWN state was resetting things, I checked the alog and see that Jamie predicted this behavior in alog 44383, since a similar thing happened at LLO recently. 

I think the take-away here is that if a front end computer has crashed, we should ensure that the IFO is in its DOWN state.  (To do this, in ISC_LOCK guardian, request 'manual' control, then select DOWN, then request 'auto' control.  You may also have to ask the node to STOP (pulldown menu that is usually on EXEC), then request EXEC again.)  I'm happy to receive a call to do this remotely if someone is busy with other recovery. 

Carlos and DaveB are already working on recovery of the susetmy computer.

This is a late alog of work done yesterday afternoon with Gabriele. 

Friday evening Daniel and I noticed that we have about 10 times more power on our REFL diodes than in O2 according to the readbacks, with 1mW on the in vac REFL LSC diode when we are locked at DC readout.  Yesterday we tried to check that the amount of DC power roughly makes sense, which it does.  Since this is one thing that is different than O2, we suspected it could be contributing to our fast lockloss problems.  

• The amount of power on the diodes:

  • There is a factor of 2 missing in the calibration for REFLAIR A, and an ND filter which attenuates the light by half.  While we were sitting at DC readout with 2W of input power and a recycling gain of 48, I measured 0.54mW of light heading towards the REFL air A diode, before the ND 0.3 filter which is mounted on the diode face.  That means that there should be 0.54*10^-0.3 = 0.27mW on the diode if the entire beam hits the diode and the ND filter is accurate.  Our readback says that there is 0.145mW on the diode, so there is a factor of 2 missing.  I also checked the centering of REFLAIR A, and noticed that we are overfilling the diode.  I didn't check the calibration for REFLAIR B but there may also be a missing factor there.

  • The two LSC REFL air diodes should each have half as much power incident on them as the in vacuum diode does, if there were no ND filter on REFLAIR A.  (see ISCT1 drawing and HAM1)  This means that the measurement of 0.54mW heading towards REFLAIR A is consistent with the 1mW readback of REFL_A with 2W locked interferometer. 

  • The amount of light we get with the interferometer unlocked roughly makes sense. Anamaria suggesting checking that the powers make sense with the IFO unlocked, which they almost do.  With PRM aligned, 2.1W of input power, and the interferometer misaligned our readbacks said that we had 2.3mW on REFLAIR A (which really means 8.6mW heading toward the diode based on the measurement above) 5.08mW on REFLAIR B (I didn't check this calibration), and 15.5mW on REFL.  According to the HAM1 drawing, we dump 90% of the refl light using M14 and 50% of the remaining light using M17 before splitting between AIR/in vac and LSC/ASC.  Including 80% efficiency through the IMC and input Faraday, we should expect to get 2.1W*0.8*0.1*0.5*0.25=21mW on LSC in vac, and about 10mW on both REFL AIR diodes.  I haven't included any losses in the REFL Faraday path since I don't know what they are.  

  • In O2 we had 1.8mW on LSC refl with 30W input and a PR gain of 29.5. Now with 2W of input power and a recycling gain of 48, we have 1 mW on the refl diode according to it's readback, which means we have ten times more power on the diode with the improved recycling gain for the same input power. Hang did a quick calculation for the increase in carrier power that we expect based on the improved recycling gain, 44370, and it seems plausible that we could have a factor of 10 more carrier light for the same input power with our improved recycling gain.  

  • We are now powering up with nearly twice as much 9MHz modulation depth as we used in O2.  (The 9MHz modulation depth was only slightly reduced by the EOM swap 41435 (for 45 see 41889 ))  In O2 we reduced the 9MHz modulation depth by 6dB before powering up, I believe that the morning crew tried to do this in the last half of this week but it unlocked the interferometer.

  • As we power up the power on the REFL diode is increasing roughly linearly, so we can expect that if we power up to 50W and maintain our recycling gain we will have about 25mW on the REFL diode.  

As a quick test yesterday afternoon Gabriele and I switched the CARM control from REFL to REFLAIR, since it has a fourth of the amount of light that in vac REFL has.

• I phased REFLAIR 9 using an excitation into the common mode board well above the ugf to move most of the signal into I.  The delay line in on "Int" mode which means that the front panel switches and not beckhoff control the delay. We are now using 3ns of delay on REFL air. (the 1 and 2 ns switches are flipped up)
• The cable to the common mode summing junction was connected to the Q output of the REFLAIR 9 demod, so I switched it to the I output.  I did not switch the cable which go to the whitening board, they are connected so that the cable with the yellow tape that says A is connected to I and the one that says B is connected to Q. 
• For in vac Refl, the switch on the front panel is set to Ext, which means that the delay is beckhoff controlled.  The cable that goes to the summing junction is connected to I, as it was when we phased REFL on monday night.  (44272 and comments)  The cables that go to the whitening chassis seem swapped however, the cable labeled A goes to Q while the cable labeled B goes to I.  
• Using a swept sine excitation I measured that the gain of refl air is 23dB lower than the gain of refl, and there is a phase of 0 degrees between I mon of the two demods.
• We switched the control of CARM over to REFLAIR after the ANALOG CARM step by reducing the gain in the REFL path by 6dB then ramping on the REFLAIR path.  The attached script can be used to do this.  
• We also edited the adjust_CARM gain function in ISC_library and added a parameter to lscparams to indicate that we are on reflair and ramp down the appropriate gain slider as we power up. 
• We tried switching to reflair and powering up a couple of times, and tried once to power up on in vac refl.  Each time we lost lock due to something very fast that happened within minutes of reaching 11W input power.  

Ideas for things to try next:

• Look at the spectrum of the RF monitor on REFL (and pop) and see if there are any frequencies that are large enough to potentially saturate the diode. 
• Try to debug the modulation depth reduction step so that we can try reducing 9MHz before powering up.  
• It seems that this problem with fast locklosses has been gradually getting worse over the last month, it is happening at lower powers and more frequently, yesterday was the worst that I have seen it.  

h1susey has crashed. Similar to h1lsc0 this morning it looks like the models are still running and the suspensions are still being damped. We have lost DAQ data from this machine and EPICS monitoring/controls.

If we need to restart this system, please call me on my cell phone and I can do this remotely.

In the attached plot we show the refl pd power as a function of power recycling gain (PRG). Here the x-axis is the PRG, the y-axis is the refl PD power, normalized by the power when the carrier is anti-resonant in CARM & PRC. In generating the plot we vary the arm losses as a parameter to compute the PRG and refl power simultaneously.

The figure shows spectra from the 4 angular IMC WFS DC channels with labeled peaks. Eight of the peaks between 80 and 1000 Hz are from optics or structures on the PSL table, two were from IOT2 and four haven’t been identified. The unidentified peaks were not excited by our PSL injections, even the global ones. It is possible that they are associated with structures that weren’t excited on IOT2 or in HAM2, or they may be associated with structures on the PSL table that we weren’t able to excite.

The identified sources are table resonances, periscope resonances and optic mount resonances. All optics that produced peaks in the spectrum were downstream of the PMC and were mirrors rather than through-optics like lenses, as would be expected.

The figure also shows the location of the sources in the PSL, as well as samples of the bowing and tapping data used to make the identifications.

Robert, Craig