Restarted web screen capture for h0 to pick up revised FMCS overview and fan sensor detail screens.  These now show all values since the new vacuum controls were installed.

One of the lockloss classes from tonight has been some kind of angular instability that we see very strongly on the AS camera right when we start to power up.  Not sure what it is, but it's the kind of thing that could be a BS angular instability, since the Michelson fringe separates, and the 2 bright spots orbit around each other for a second or so before we lose lock.  It looks primarily like pitch according to the oplevs, but there are certainly some yaw components in there.

Attached is a plot showing the lockloss, including the power at the AS port and the oplev (or wit) signals from each of the IFO optics.  If we call the frequency of the AS DC power fluctuations 2f, then it looks like perhaps BS and the ITMs are moving in pitch at 1f.

We tried once turning off the CHARD, MICH and SRC1 loops just before powering up.  We were able to make it to 3W and sit for a few moments, but then lost lock on the way to 5W, although this was a fast lockloss, probably just because we didn't have enough loops on.  We'd like to try powering up with all the loops on to 3W, to confirm that we can't even do that (we had always been going straight for 5W when we saw the instability). We'd also like to try with only 1 loop off at a time - MICH seems the most suspicious loop, but CHARD does seem to get noisy when the rotation stage is moving. 

As a side note, I've added tried to add a convergence checker to the Engage_SRC_ASC state, so that you no longer have to wait by hand before going to Part1.  You should be able to safely select Part1, and the guardian log will tell you what loop(s) aren't converged yet. The wait seems to not be working consistently yet, so maybe we can't depend on it yet, unfortunately.

Today we have seen at least three examples that might make us want to return to trying 90 MHz centering.  The first two screenshots attached show locklosses where we lost sideband buildups, which is usually a sign of misalignment of the BS or SRC.  (We are currently using a combination of AS36I A+B for SRM control 26785 and AS36BQ  for BS control).   You can see that the BS yaw control signal drifts along with the sideband powers, and that there is clearly also a signal in the 90MHz Yaw signals, which might indicate we could use these sensors to prevent this kind of run away and lockloss.  

The third one shows the lock that CHris and TJ left overnight last night, and you can also see sideband powers drifting a bit over the first  couple hours of the lock, which is also tracked by some signal in 90 MHz yaw.  

Jenne, Sheila, Chris, TJ, Evan

It seems that tonight we have been sabotaged by some code that we have been using for a long time (this haas only happened once that we caught it, although we have a lot of unexplained locklosses tonight).

In the attached screenshot you can see that the DRMI guardian was sitting at DRMI_3F_LOCKED (130) when it decided to go to LOCK_DRMI_1F (30).  There is a decorator in DRMI_3F_LOCKED that apparently returned LOCK_DRMI_1F, because it though DRMI was unlocked (it was fine as you can see from the power build ups in the top row). 

The code that checks for DRMI lock is: 

All rotating vacuum pumps in the LVEA are now off.

HAM11 and HAM12 annulus ion pump currents displayed on MEDM shown in red are, in fact, on-scale on local controllers @ 10mA

C. Cahillane, J. Driggers, S. Dwyer

There was an issue where during lock acquisition at the stage ENGAGE_SOFT_LOOPS, there was a filter (FM2) that was occasionally not being switched 'OFF' automatically by guardian for the filter modules ASC-DSOFT_P and ASC-CSOFT_P. 
In the Guardian code ISC_library.py, there is a function called FM_limit_checker_decorator(FM, sign, action, onstate, offstate) which takes in a filter module FM and changes the state to onstate under certain conditions and to offstate otherwise.  (action specifies the filter 'FM2', sign affects the conditions)

The issue was that FM_limit_checker_decorator always read in FM+'_OFFSET' value, even when the offset filter was flipped off.  I adjusted the FM_limit_checker_decorator code to check if the offset filter was flipped off before populating the FM+'_OFFSET' value.  This is why the code worked when the offset was 0, but could fail if the offset had value but was flipped off.

Jamie, TJ, Nutsinee

Two new TCS guardian nodes have been created. They handle CO2X and CO2Y output power (by adjusting the rotation stages). MIN_POWER outputs minimum power to the optics (0 W). PRE_HEATING outputs 0.1W for CO2Y and 0.35 W for CO2X. And NOMINAL_LOCKED_CO2_LEVEL is outputing 0W at the moment but can easily be changed in the future. NOMINAL_LOCKED_CO2_LEVEL (CO2 output = 0W) is to be used at 20W IFO power.

I have tested PRE_HEATING and NOMINAL_LOCKED_CO2_LEVEL when everything is in a good state (CO2 laser on, no interlock tripped) and they seem to work as I hoped. Once I have a chance to test these states when things go wrong (no laser, interlock tripped) and see that they behave as expected I will make ISC_LOCK talk to it. The state is left at PRE_HEATING at the moment since we mostly stay at 2 W and 10W nowadays.

Replaced the 1A fuses for CP7 and CP8 with a 2.5A.

1/2 open LLCV bypass valve, and the exhaust bypass valve fully open.
Flow was noted after 66 seconds, closed LLCV valve, and 3 minutes later the exhaust bypass valve was closed.

Made measurements of the oscillator power noise with a few photodiodes.

PowerNoise.png shows the free-running oscillator relative power noise
measured before the acousto-optic modulator.  This is more than 10 times
noisier than when the laser was installed in the H1 enclosure.  The
other trace in the plot is the out of loop of the relative power noise.
It is also about a factor of 10 higher than it should be.

    Whilst the power stabilisation was locked, I looked at the AC coupled
output of the photodiode and did not observe any oscillations.  The maximum
peak-to-peak variations were ~40 mVpp.

The X1 PT140 Pirani gauge is reading above the software interlock threshold to turn on the Cold Cathode gauge. Per Chandra's request I have bypassed the software interlock by forcing the variables in Beckhoff on h0velx (see attached screenshot).

Added 3194 channels. Removed 592 channels. (See attached channel list)

J. Kissel, S. Dwyer

As identified yesterday (LHO aLOG 26793), we've needed to *re*flip the ESD bias sign in order to resume bringing the accumulated charge on the test masses back down to zero. As such, we have reverted the requested bias voltage to what had been done in February (see LHO aLOG 25575) and performed all of the necessary changes in settings that are ancillary to the change.

Note, there are a few changes / additions in what needs changing from Kiwamu's February flip, so I'll re-list everything here:

[Bias Flip on ETMX]
- Changed H1:SUS-ETMX_L3_LOCK_INBIAS from -9.5 to +9.5 [V_DAC]
- (unlike before) The ISC_LOCK guardian (now) takes care of checking the sign of H1:SUS-ETMX_L3_LOCK_INBIAS, and adjusting the sign of 
   - H1:SUS-ETMX_L3_DRIVEALIGN_L2L_GAIN from +1.000 to -1.000  (fixes the ESD stage longitudinal loop gain to match the bias sign change)
   - H1:SUS-ETMX_L3_DRIVEALIGN_L2P_GAIN from +0.021 to -0.021  (fixes the F2P sign)
   - H1:SUS-ETMX_L3_DRIVEALIGN_L2Y_GAIN from +0.007 to -0.007(fixes the F2P sign)
   - H1:SUS-ETMX_L3_ESDOUTF_LIN_FORCE_COEFF from -124518.4 to +124518.4 (fixes the linearization force coefficient to match the new bias sign)
   accordingly. (Note that it's marked as a #FIXME, but these values are hard-coded into the guardian)
- (unlike before) Accepted the new values in *all* SDF snap files, 
   /opt/rtcds/userapps/release/sus/h1/burtfiles/
   - h1susetmx_down.snap
   - h1susetmx_safe.snap
  by loading in each table and accepting the values I've changed, followed by a commit to the userapps repo. 
   - This also involved *creating* a copy of the down.snap in the userapps repo, and changing the file in the 
  target/h1susetmx/h1susetmxepics/burt/ directory to a soft link, as was already the case for the OBSERVE.snap and safe.snap) 
   - All the changes in settings had been accepted on the OBSERVE snap on the Feb flip, so I didn't need to accept any new values)
- (like before) There's no need to make any further changes in the CAL-CS epics settings, because ETMX is not used in any capacity for calibration.

[Bias Flip on ETMY]
- Changed H1:SUS-ETMY_L3_LOCK_INBIAS from +5.0 to -9.5 [V_DAC] 
- (like before) The ISC_LOCK guardian takes care of checking the sign of H1:SUS-ETMY_L3_LOCK_INBIAS, and adjusting the sign of 
   - H1:SUS-ETMY_L3_DRIVEALIGN_L2L_GAIN from +30.0 to -30.0
   - Note that previously, this check had been in the ISC_LOCK state of just before "NOMINAL LOW NOISE" which seemed too late, so we moved this check into the DOWN state just under where it's checked for ETMX.
- (like before) there are no off-diagonal L3 DRIVEALIGN coefficients in use (and L2P / L2Y or P2L / Y2L), but if they were, we would have needed to flip those signs too.
- (unlike before), I've by-hand changed the linearization force coefficient sign, 
   - H1:SUS-ETMX_L3_ESDOUTF_LIN_FORCE_COEFF from +124518.4 to -124518.4
   Even though linearization is not used in nominal low-noise (not even during charging measurements), I think it's important to be consistent during these flips lest it bite us later if someone wishes to turn on the linearization.
- (unlike before) Accepted the new values in *all* SDF snap files, 
   /opt/rtcds/userapps/release/sus/h1/burtfiles/
   - h1susetmy_down.snap
   - h1susetmy_observe.snap
   - h1susetmy_safe.snap
  by loading in each table and accepting the values I've changed, followed by a commit to the userapps repo. 
 (This also involved *creating* a copy of the down.snap in the userapps repo, and changing the file in the 
  target/h1susetmy/h1susetmyepics/burt/ directory to a soft link, as was already the case for the OBSERVE.snap and safe.snap) 


[Making sure Calibration is unaffected]
- (like before) We must make sure that the CAL-CS replica of the ETMY actuation matches the SUS ETMY digital signal chain. However, because the CAL-CS model had not been restarted, and its SDF system had been kept up-to-date it did not lose the appropriate settings, so the settings for
   - H1:CAL-CS_DARM_FE_ETMY_L3_DRIVEALIGN_L2L_GAIN remained -30.0
   - H1:CAL-CS_DARM_ANALOG_ETMY_L3_GAIN remained +1.0
- (unlike before) Again, to be consistent, I've changed the sign of the linearization force coefficient,
   - H1:CAL-CS_DARM_FE_ETMY_L3_ESDOUTF_LIN_FORCE_COEFF from +124518.4 to -124518.4
- (unlike before) Accepted the new values in *all* SDF snap files, 
   /opt/rtcds/userapps/release/sus/h1/burtfiles/
   - h1susetmx_observe.snap
   - h1susetmx_safe.snap
  by loading in each table and accepting the values I've changed, followed by a commit to the userapps repo. 
 (Since there is no change between the "down" and "observation" state, a "down" .snap doesn't and need not exist.)  

Back in the day, Joe Betz had the idea of writing some guardian code that would keep the CAL-CS model up-to-date with the SUS model's digital settings. That would make this last step unnecessary. I'll ask what the status is on that. It still good to check all of these things anyways, though.

I attach a bunch of screenshots which should what I've accepted in each SDF file for future reference.

Sadly, because we've not locked the IFO on ETMY for quite some time, we cannot check the DARM open loop gain to confirm that all is well with the calibration. Further, the actuation strength will have likely changed and we're going to remeasure it anyways prior to ER9, so there's little point in double checking the actuation strength. So there's no further things to do this time around. However, if and when we do have a viable calibration, I'll remind you of the steps, which are at the bottom of LHO aLOG 22135.

Per https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=26806

CP5 LL has stabilized to its normal conditions. Still has a jittery output.

Lots of lock losses and FSS oscillations. Sheila and Jenne investigating.
14:15 UTC Chris S. working in LVEA moving barrels from bottom of stairs into the high bay
14:30 UTC Jeff B. to HAM6 and PSL rack
15:00 UTC Chris S. done in LVEA, working in high bay until 17:00
15:17 UTC Kyle to LVEA
15:17 UTC Peter to H1 PSL enclosure
15:28 UTC Jeff B. back
16:24 UTC Filiberto to LVEA to terminate cables for PT170 and PT180 BPG402 gauges
16:54 UTC Kyle back from climbing on HAM11, HAM12, BSC4
17:05 UTC Jamie restarting h1guardian0 for code upgrade
17:12 UTC Christina and Karen to mid Y
Peter done
18:20 UTC Turned on and off PSL noise eater in PSL rack
18:21 UTC Christina and Karen done at mid Y
18:33 UTC Jeff B. to LVEA to check dust monitor cabling
18:56 UTC Dave restarting OMCPI, SUSPI models, DAQ restart
18:58 UTC Jeff K. changing SUS ESD bias
19:19 UTC Filiberto out for lunch
19:21 UTC Jeff B. out of LVEA
19:26 UTC Sheila to LVEA to investigate FSS oscillations
19:39 UTC Filiberto to LVEA to continue cable work
19:53 UTC Gerardo to end Y to replace electric LLCV motor fuse
19:55 UTC Sheila back, made no changes
20:11 UTC Gerardo changed CP7 electric LLCV motor fuse, going to mid Y to overfill CP3, then end X
20:17 UTC Joe to LVEA to track down view port equipment
20:19 UTC Jeff B. to turn on dust monitor at PSL
20:48 UTC Gerardo changed CP8 electric LLCV motor fuse
20:48 UTC Filiberto done
21:00 UTC Jeff B. done
21:47 UTC Gerardo to BSC4 to look at PT140 cabling
22:20 UTC Gerardo back
23:03 UTC Gerardo to LVEA to measure PT140 voltage at vacuum rack

Guardian has been upgraded to r1519, and the h1guardian0 machine has been rebooted.

This upgrade includes a couple of minor bug fixes and a new guardian log server/client:

• Guardian nternal channel access server updates at same rate as the main process thread.  This should address the state not registered bug.
• Minor fixes, improvements, and tweaks to some of guardutil functions
  • fixed 'git-clone' function to retrieve the archive
  • improved the 'plot' function to plot node status info around a specified time
  • added 'state-hist' function to print guardian states over a specified time range
• new guardlog client/server
• changes to node logging on h1guardian0 to increase log size, and compress logs on rotation.  This requires the new log server for searching the logs

h1guardian0 reboot

After these changes were applied, all nodes were re-created to get the new changes to the logging, and the h1guardian0 machine was rebooted.  A couple of small issues were encountered during the reboot:

• Some of the manager nodes thought that they had requested states from their subordinates, but the subordinates where in the NONE state (no request).  My guess is that this might be due to some sort of race condition as the nodes are initialized (cas is initialized before the nodes starts processing requests?).   This happened with some of the SEI nodes and the ISC_LOCK.  It was trivially resolved by requesting INIT from the managers.
• BRS nodes have no initial request defined, so they can up with NONE requests.
• SEI_BS is managed by ISC_LOCK, but ISC_LOCK is not initially doing anything.  At some point during initialization SEI_BS stalled, so it just sat there stalled.  Not a big deal, but could maybe be handled cleaner.

New guardctrl log client/server

This version of guardian includes a new and improved log client/server.  The new server running on the h1guardian0 machine:

• has a new RESTful http interface
• is now able to process the now possibly compressed node logs
• handles queries with date range filters, filtering for log files and log lines within the files that correspond to the date/time search parameters
• output log lines with timestamps in UTC, local, or GPS time

See "guardlog -h" for more info.

Chris, Keita, Evan

Today we were able to lock the outer ISS loop with the modecleaner at 20 W (and no interferometer). We looked at several PSL/IOO PD signals (the FSS transmission PD, the ISS inner-loop PDs, the IM4 transmission PD, and the ISS outer-loop PDs) and tried to understand their behavior in different ISS configurations.

Naively one would expect all these signals (except the in-loop ISS PDs) to agree with each other, since they should all be out-of-loop sensors for the RIN leaving the PMC. Together, these signals monitor three of the four PMC ports: the FSS transmission sees the RIN of one port, the out-of-loop inner-loop ISS PD sees the RIN of another port, and IM4 trans and the out-of-loop outer-loop ISS PD sees the RIN of yet another port.

These are the behaviors we observed (see attached pdf):

• When neither ISS loop is closed, both the FSS and IM4 transmission PDs see the noise of the HPO (about 1×10⁻³/Hz^1/2 at 10 Hz). From 10 to 100 Hz, these PDs are coherent with each other and with the HPO PD. All four of the ISS PDs are swinging rail-to-rail in this configuration, so they are not useful. [Apricot traces.]
• When the inner ISS loop is closed, the FSS PD, IM4 PD, and outer-loop ISS PDs see a RIN that is a few times 10⁻⁵/Hz^1/2 at 10 Hz, with a 1/f slope. These three signals are coherent with each other between 10 and 100 Hz. However, the inner-loop PDs are in some parallel universe in which the inner-loop appears to be happily stabilizing the laser RIN to better than 10⁻⁶/Hz^1/2 at 10 Hz. The inner-loop PD signals have low coherence with the FSS, IM4, or outer-loop PDs between 10 and 100 Hz. [Red and blue traces.]
• When the inner and outer ISS loops are closed, the outer-loop PDs seem to behave sensibly. The out-of-loop RIN is slightly better than 10⁻⁷/Hz^1/2 at 10 Hz. The IM4 PD more or less agrees. However, the FSS RIN seems to show no change from the previous measurement (inner ISS loop only), and no longer has very much coherence with the IM4 and outer-loop PDs. The FSS has some coherence above 100 Hz with the outer-loop PDs. The inner-loop PDs now see the 1/f RIN of the FSS PD, and they are coherent with this PD. [Purple and green traces.]

We think that a possible explanation for these effects is that both ISS PDs are seeing some correlated noise that is not seen by either the FSS PD or the post-IMC PDs. In this scenario, the inner-loop ISS would suppress the HPO noise but impress this correlated noise on the light entering the PMC.

Briefly we entertained the idea that the light circulating in the PMC could be multimoded (either from the NPRO or the HPO), but judging from the RIN before and after the IMC, this seems to not be the case (png attachment).

One other idea is that some of the 808 nm light is getting through the PMC and onto the ISS.