BSC ISI guardians now running for BS, ITMX, ITMY, ETMX

After a lot struggle with the BSC ISIs this week, we finally have all commissioned BSCs under guardian control.

The BSC seismic stack is one of the more complicated systems we've deployed, as it currently consists of three nodes in a "managed" configuration: two separate nodes for each of the two ISI stages (e.g. ISI_ITMY_ST1, and ISI_ITMY_ST2), and a "chamber manager" (SEI_ITMY) that coordinates the activity of the two subordinate ISI stages.  USERAPPS location of the top level modules (and primary loaded library):

• USERAPPS/isi/common/guardian/SEI_ITMY.py     (USERAPPS/isi/common/guardian/isiguardianlib/BSC_MANAGER)
• USERAPPS/isi/common/guardian/ISI_ITMY_ST1.py (USERAPPS/isi/common/guardian/isiguardianlib/ISI_STAGE)
• USERAPPS/isi/common/guardian/ISI_ITMY_ST2.py (USERAPPS/isi/common/guardian/isiguardianlib/ISI_STAGE)

All single ISI stages share the exact same code, for each BSC stage as well the single HAM stage.  Here are the system graphs for the SEI manager and ISI stage 1:

The manager (left above) has three main requestable states: READY, DAMPED, ISOLATED.  The READY state puts both of the ISI stages in READY, the DAMPED state puts them both in DAMPED, and the ISOLATED state puts both into isolation levels that can be specified in the manager system description module (USERAPPS/isi/common/guardian/SEI_*.py).  At the moment all systems are using their default configuration, which is to operate the ISI stages in the HIGH_ISOLATED state, which corresponds to "lvl3" from the old "command" scripts.  This can't currently be changed on an individual BSC basis, due to a guardian core issue, but I'm working on it.

The degrees of freedom for which to restore cart bias offsets can also be specified on a per stage basis.  Note the ISI_ITMY_ST1 graph above has "RESTORE_ISO_CART_BIAS_*_RZ" states, and no corresponding states for the other degrees of freedom (X, Y, Z, RX, RY).  This indicates that we are restoring only RZ target cart bias offsets for this stage.

Notes about the BSC isolation procedure

There are a couple of important things to note about the current isolation procedure:

We had a lot of trouble dealing with the T240s in stage 1.  Their exterme sensitivity to pitch causes them to saturate if a large pitch cart bias offset is applied to the stage.  We there spend a lot of time trying to figure out how to gracefully bring them in and out of the loop, by reducing their gain and/or changing the blend filters during the isolation procedure.  All of theses things has problems, though:

• switching of the T240 analog gain does not synchronously switch the corresponding digital compensation filters.   This means that any attempt to change the T240 gains will cause the T240 watchdogs to trip.
• there were strange problems with changing the blend filter with the isolation loops closed.  The "SWITCH_ALL" buttons cause large glitches in the isolation loops.  Clicking each DoF manually doesn't cause glitches, but clicking them too fast causes various things to saturate.  Basically you have to manually switch each gain independently, with a significant delay between each DoF.

This makes things very difficult if we need to hold pitch offsets on any of the ISIs.  It looks like we need to fix the synchronous switching of the T240 compensation filters before we can handle pitch cart bias offsets.

However, after speaking with the integration team, it seems that it's ok to operate temporarily in a mode where we only restore RZ cart biases.  Any needed pitch offsets can be held in the suspensions for now.  Therefore we do not need to change the T240 gains or blend filters during the isolation procedure.  This is good news, since it means we can run with the current guardian behavior.  Obviously this will have to be fixed down the line, as we will likely want to hold pitch offsets on the ETMs, but we have a workable solution for the moment.

Guardian BSC isolation configuration and procedure

• All sensors are configured to be in HIGH gain mode.  This includes the T240s and L4Cs on stage 1, and the GS13s on stage 2
• Both stage 1 and stage 2 BLEND filters are configured to be in "Tcrappy" for all DOFs at all times.  "Tcrappy" are the low-frequency cross over blends that include the T240s on stage 1.  There is no changing of the blend filters at any point during the isolation procedure.
• All cartesian biases for all DoFs are restored to their "equilibrium" positions at the beginning of the isolation procedure.
• Only the RZ degree of freedom cart bias target is restored at the end of the isolation procedure.  The rest of the DoFs are left at their equilibrium position.
• Both ISI stages are configured to go to HIGH_ISOLATED (i.e. "level 3").
• Isolation procedure:
  • stage 1: DAMP
  • stage 2: DAMP
  • stage 1: equilibrium cart bias
  • stage 1: HIGH ISOLATE
  • stage 1: restore RZ cart bias
  • stage 2: equilibrium cart bias
  • stage 2: HIGH ISOLATE
  • stage 2: restore RZ cart bias

With the above configuration we can successfully isolate the BSCs without touching the T240s or blend filters.  It seems to be fairly robust.

Important operator notes:

• We don't yet have guardians for HPI.  This means that restoration from HPI watchdog trips require a bit of care.  If the ISI watchdogs are reset before the HPI, the SEI manager will restore isolation on the ISI before the HPI is restored.  This means that the ISIs will trip during the HPI isolation.  It is therefore important remember to  reset HPI watchdogs and restore HPI isolation before resetting the ISI watchdogs.
• It's possible that depending on the seismic environment, etc., the ISI could trip during the isolation procedure.  We have noticed, for instance, that some of the sensors might saturate during isolation if things are noisy.  Don't give up on guardian if this happens.  Just reset the watchdogs and let guardian try to isolate again.  It will likely work during the next attempt.  If it still doesn't, try waiting until things settle down a bit before resetting the watchdogs.
• If all else fails, the guardians can be either set to PAUSE (so they stop executing code) or can be shut off entirely (e.g. "guardctrl stop SEI_ITMY ISI_ITMY_ST1 ISI_ITMY_ST2"), at which point all the old command scripts can be used.  This should obviously be done only as a very last resort.

Notes about guardian SEI manager behavior

The SEI managers represent a new step for guardian, as their primary task is to control the state of other guardian nodes, which we haven't needed up to this point.  Getting their behavior robust has been quite tricky.  It necessitated a lot of work on the guardian built-in "manager" library, as well as a lot of experimentation with the SEI manager module itself.  The manager is designed to be fairly robust against commissioner noodling with it's subordinates, but that is also fairly tricky to get right in a robust and useful way.  Some things will need to be improved, but this is the behavior currently:

• The manager puts its subordinate ISI nodes into MANAGED mode during the INIT state.  This slightly modifies the graph traversal algorithm of the subordinate to defer to the manager after "jump" state transition.  The subordinate node MEDM control screens also turns purple in MANAGED mode as an indicator to the operator that the node is under the control of another guardian node.
• The INIT state now has a special feature that it can be jumped to directly from any state in the graph, after which the overall system REQUEST will be reset to whatever it was previously.  The SEI manager INIT state is programmed to set all subordinates to MANAGED, identify the current state of both ISI stages, reset things as needed, jump to the current state of the system, then proceed to the request.  Therefore, regardless of the state of the subordinates, requesting INIT from the manager should completely reset the state of the SEI BSC.
• The manager expects its subordinates to be alive.  If they're not, the manager will go into ERROR.  Once all subordinates are alive, clear the error (mode => LOAD) and re-exec the manager and it will pick up the subordinates.  You may need to move to request INIT in the manager to fully reset the the system.
• If a subordinate REQUEST is changed while in MANAGED mode, the manager will throw a notification and will immediately reissue it's old request.  The subordinate will have already started moving to the rouge requested state, but will immediately move to back to the manager requested state once it gets there.
• If a subordinate is removed from MANAGED mode, the manager will throw a notification, but will keep running as normal.  The manager will continue to monitor the state of the now unmanaged subordinate but will stop issuing new commands to the subordinate.  Since the manager will continue to process normally, it will therefore expect the subordinate to change it's state on its own.  The new human manager will therefore need to take over this task, as the manager will otherwise not see it's states completed.

While things seem to working well at the moment, I certainly don't claim that all the bugs have excised.  If the manager does get hung in some state, resetting to the INIT state should clear everything, reset the ISIs, and get things back on track.

TODO for SEI guardian

• turn on guardian nodes for ETMX once it's ready, e.g.: guardctrl start ISI_ETMX_ST{1,2} && guardctrl start SEI_ETMX (should then remove the "WIP" masks over the BSC ETMX SEI and ISI guardian boxes in the GUARD_OVERVIEW screen).
• fix synchronous gain switching for the T240s
• figure out how to handle T240s with pitch cart biases (either via T240 gain switching and/or blend filter switching)
• HPI guardians, for both HAMs and BSCs
• incorporate HPI nodes under SEI managers for HAMs and BSCs, for full SEI stack control
• allow specifying non-default isolation levels for ISI stages (probably something in guardian core needs to be fixed)
• further development of manager infrastructure

A. Staley, S. Dwyer, J. Kissel 

After speaking with Sheila and Alexa, trying to understand the spectrum in LHO aLOG 11026, I realized that -- because of so much unexpected excess noise -- the current ALS setup has diverged significantly from the baseline plan. As such, we've worked together to create a diagram that is to-the-best-of-(my)-knowledge accurate, and makes sure to show how every control signal gets around to its actuators, including passes through digital land. The only things I did not include were the demodulator boards between the REFL diodes (ARM REFL, MC REFL, and the two green XARM and YARM REFLs) and lame things like whitening, AA, and AI chassis that facilitate getting in-and-out of digital land.

In particular, I draw your attention to the unorthodox and new features that were not part of the original design plan:
(1) The PRM is misaligned. This way we can use IFO REFL red diode signal (after normalizing / linearizing it with the red arm cavity trans) to measure the arm and offload at low frequency to the CARM board in the second input. This is unorthodox because normally we would feed REFL straight to the CARM board in all analog and retain the large bandwidth, but because the noise is so large, it must be linearized in the front-end (the only place the Arm TRANS signal exists), and fed out one of the corner station LSC DACs. Note that this configuration is just a special technique for these particular series of measurements and is not designed to become a normal part of the lock acquisition scheme.

(2) The Voltage Controlled Oscillator (VCO) actuated, Phase-Locking Loop (PLL) that's wrapped around the corner-station COMM and DIFF Phase-Frequency Discriminators (PFD) are used to reduce the noise enough to keep the PFD in the linear, phase detecting mode (check out pgs 6-8 of the AD9901 datasheet -- it's a really nice explanation of how it works).

(3) Before the hand-off, there's another temporary step which is to use the Beckhoff system ("Slow ADC" in the diagram) to feed directly to the PSL VCO (again at low frequency) to reduce the noise enough for the COMM hand-off to work. 

This is most certainly an impressively complicated system!

Note that this diagram has been committed into the noisebudget repo here:
${NBSVN}/trunk/HIFO/Common/
and I imagine will become an "as built" DCC document in the glorious future when everything is working beautifully.

started at ~18:50 will run for 8 hours.

Yesterday I went to end X to investigate the beam shape for the ALS WFS.  (This is to follow up the measurements we made in alog 10774)

First I tried changing the alingment of the reaction chain, this didn't make a difference.  Then I tried changing the TMS QPD offsets.  This moves the lock point of the input alingment loops, changing the alingment of the green beam as it goes through TMS.  This certaintly changes the beam quality on the WFS, screenshots of several different offsets are attached.  In the end I left the offsets at ALS-X_IP_ANG_PIT at -50, ALS-X_IP_POS_PIT at 150. 

I am not sure why changing these offsets makes such a difference in the beam quality. 

This moved the QPD A by 0.07 counts and QPD B by 0.04 counts.  Pablo and I redid the pointing of TMS to the baffle PDs, but it hadn't changed more than the normal change between ISI trips.  However, switching these offsets does move the beamspot on ISCT1, I moved PR3 by 0.9urad in pitch to bring the spot back.  

After realinging the arm, we re checked the phasing of the WFS. We did this by putting an exictation on LSC-X_EXTRA_AO_1_EXC at 50Hz and looking at the signals of the individual segments, which seems fine.  This is saved in /home/sheila.dwyer/ALS?HIFOX/WFS/WFS_PhaseDTT_March282014.xml

Then we started the sensing matrix measurement in Alexa's alog 11073

This week, I tested the ETMX UIM P2P and Y2Y filters (described in alog 10912). The results are showing that filters are working as expected giving a single pendulum transfer function falling at 1/f^2 after the resonnance (f=0.45Hz for pitch, f=0.6Hz for yaw). Also the coupling between pitch and yaw is small (~5%) so unless this is too much, there's no need for diagonalization for UIM.

Attached pdf are showing P2P P2Y (first pdf) Y2Y Y2P (second pdf) transfer functions with/without the filter.

Left to do for next week :

ETMX : Long decoupling on UIM/PUM

• measure UIM L2PY for long decoupling 
• measure PUM L2PY for long decoupling

ETMY

• Install/comission oplev on ETMY
• Balance UIM/PUM coils
• Measure top to test mass L2PY P2PY Y2PY
• Measure UIM to test mass L2PY P2PY Y2PY
• Measure PUM to test mass L2PY P2PY Y2PY

ITMX/ITMY

• Measure/invert UIM P2P Y2Y

I got the medms populated so the cartesian location values are now valid.  These initial values today are the 'Aligned' numbers.  They will probably be a little different after these range of motion/linearity tests complete and the stress of the system are further released/shaken out.

Removed iLIGO RGA hardware from BSC10 and installed 10" blank in its place*installed ESD interlock gauge and 4.5"-2.75" zero-length reducer on BSC10*Installed 2.5" aLIGO RGA isolation valve and 10"-4.5" zero-length reducer on BSC6 (remaining aLIGO RGA components to be installed later) 

"Bumped" dial indicator cluster on BSC10's NE support tube -> readings from these dial indicators can be considered "bogus" as a result 

Running QDP80 overnight but couldn't start Turbo pump as its foreline pirani gauge isn't working now(?) -> I'll fix this gauge tomorrow while rough pumping the Y-end with the QDP80.

(Alexa, Daniel, Sheila)

We measured the WFS sensing matrix for both PIT and YAW. The matrices were computed by exciting ITMX/ETMX PIT/YAW at 1.5Hz with 0.3 amplitude, and looking at the WFS transfer function. The results can be found in the following DTTs saved in /ligo/home/sheila.dwyer/ALS/HIFOX/WFS/:

• WFS_sensing_matrix_ETMX_PIT_march28_2.xml
• WFS_sensing_matrix_ETMX_YAW_March28.xml
• WFS_sensing_matrix_ITMX_PIT_March28_2.xml
• WFS_sensing_matrix_ITMX_YAW_March282014.xml

Inverting the results gives...

YAW input sensing matrix:

-2.294    -3.155

0.529     -2.604

PIT input sensing matrix:

-0.494    -0.239

0.302    -0.489

These values are in microradians; however the factor of 1e-3 is taken care of in the gain of the filter module. We also adjusted the signs as follows:

• H1:ALS_X_WFS_DOF1_Y_GAIN = -0.001
• H1:ALS_X_WFS_DOF2_Y_GAIN = -0.001
• H1:ALS_X_WFS_DOF1_P_GAIN = -0.001
• H1:ALS_X_WFS_DOF2_P_GAIN = 0.001

Under this configuration, with all four DOFs engaged (DOF 1,2 P,Y) the WFS_A/B_I_YAW/PIT_OUTMON all converge to zero. The lock appears to be stable with a transmission betweeen 950 to 1050 counts. Futher investigation needs to be done...

Aidan. Dave H. Greg G. Thomas V.

We have, basically, finished installing all the electronics and cabling on the TCSX table. There are a few outstanding issues (summarized in the attached PDF). The ones that are preventing us from turning on the laser are listed below.

1. For some reason, the flow meter is not outputting a sense current (current proportional to the measured flow). This means that the CO2 controller, D1200745, is registering low flow and preventing the laser from being turned on. We're not sure if this is a faulty connection at the terminal block in the meter or if there is a deeper problem. We will continue investigating on Monday. The connections are shown in photo 1.
2. The TCS Beckhoff unit is, apparently, running but is not connecting to EPICS. We need to be able to access the software on/off switch to turn the laser on.
3. We don't currently have the viewport IR sensor that plugs into the CO2 laser controller, D1200745. We can bypass this input by changing a jumper on the board.

We hope to address all these early on Monday to get the laser running to check the output power.

Chiller settings:

We adjusted the bypass valve on the back of the TCSX chiller to achieve the following settings (to mimic what's at LLO)

Flow rate: 3.4GPM

Pressure: 60.5PSI

Laser housing leak.

The replacement laser installed on the TCSX table also has a slow leak from the laser housing (in the same place as the first one) - about 1 drip every 5 minutes. We will check the output power of this laser before we pull it. See photos 3 & 4

This trip happened around the time of a beckhoff restart.  A beckhoff restart causes a crash of the IMC guardian, which causes it to stop.  Its not clear to me why this should cause MC2 to get a large signal, but it seems to.  This causes a cascade of trips.  Even though I don't know why this happends, creating a safe state in the IMC guardian so that it handels missing channels better could help with the problem. 

Another solution is to have the SUS WDs not trip HEPI.  Can we get that fix soon?

HAM2 also tripped at the same time, Jeff and Hugh brought that back. 

Please mind the signs here at HAM4, there is an Optic suspended here.  Commissioning to commence shortly.

Day Shift Summary
LVEA Laser Safe

Apollo - Bolting dome on BSC10, working on view ports at End-Y
Mitch – Check ACB at End-Y
Hugh – Working on HAM5 HEPI
Richard – Y-Arm TCS cabling

08:35 Mitch – Going to End-Y
08:53 Dave – LVEA working on TCS cabling
09:00 Justin – Moving laser barriers around the HAM4 TCS table area
09:18 Andres – Working on dog clamp shelves next to HAM2
09:20 Hugh & Jim – Working on HAM5 HEPI 
09:29 Betsy & Margo – Going to End-Y for BSC10 closeout
11:26 Filiberto – Going to Mid-Y to pick up TCS panels
13:00 Justin – Transition LVEA to laser hazard
13:20 Dave & Thomas – Turning on TCS chillers
14:00 Filiberto – Going to End-Y to check ACB feed through
14:05 Andres – Working on dog clamps shelves next to HAM2
15:30 Jax – Briefly transitioning End-Y to laser hazard to make a PZT check
15:36 Michael – Going to End-Y to check chamber close status 

At Richard's request I disabled the End Y picomotors, so that they will not be used durring the pumpdown.

Margot and I performed all of BSC10 closeout tasks listed in my previous alog regarding this closeout.  We pulled the FC from the ETMy optic a little after 11am, and the door was going on at ~1:30.  It took "so long" (ha) because we ended up having to spend some extra time calling in the troops about one of the ISC viewports having what appeared to be a scratch on it's inside surface.  More details to come as needed.

This morning, SEI, TMS, and SUS gave a cursory "approval" of their systems in order to proceed with chamber close-out tasks.  So, after lunch, Margot and I applied First Contact to the ETMy-HR surface which will be pulled just before closing the door tomorrow.  The order of events this afternoon (and late morning in order to catch the alog up):

Jim locked the ISI and put final touches on chamber feedthru cable routing.

Apollo+Joe+Gerardo had a viewport party at the BSC10/6 area.

Margot and I locked the lowest masses of the ETMy QUAD.  Then we,

Secured the ACB locking bracket and used the wedge to swing it back and prop it into place.  See notes on this in attached alog.

Attached the FC QUAD cone to the structure and sprayed FC (standard 3 coats) on the HR surface.

Removed the cone from the suspension.  Finished FC layer reinforcement.

Removed the wedge from the ACB and restored it's nominal hanging position. Removed the ACB locking bracket screws.

Started stowing all left over bags of "junk" left outside of the chamber door on various table surfaces by SUS and every other team that visited this last month.  If you're missing something (and you noticed), I took ownership of it.

Jim arrived and unlocked the ISI.  The ETMy QUAD will remain locked during dome replacement.

Apollo started gearing up to remove walking plates from upstairs and crane the dome on the chamber.

Tomorrow's to-do list which Margot and I will do until the last step:

Inspect insides of viewports.

Adjust ACB to check magnet gap which may or may not have shifted slightly due to the swing back from today.

Clean the chamber, vacuum, etc.

Take final PCL swipes.

Blow various other optic surfaces.  Or not.

Swap in final 1" witness optic on QUAD structure.

Check/Remove all tools.

Remove ETMy-HR with N2 blow.

Set and lock all 64 QUAD EQ stops and nuts, accounting for buoyancy.

Set QUAD BOSEMs to account for buoyancy.

Quick QUAD health check with DTT TFs.

Lay final witness plates.

Exit chamber.

Finish stowing everything around door and move it out of the way.

Put door on.