With the upcoming TCS CO2 work, we needed to work out any bugs in software land before trying to turn on the laser. A few additions (rotation stage) and deletions caused the old code to not work very well and so we re-mapped the configuration and re-linked the software to hardware.  This solved the problem of getting the hardware channels all the way through to EPICS.

A few more details:
- In order to not trip all the watchdogs in HAM2/3, Sheila turned off the connection between ISC to SUS temporarily as well as putting the MC Guardian in a down->paused state.
- We created a new boot project for PLC3.
- Committed the new system manager to SVN.

(Sheila, Alexa)

Since the beam on the WFS seemed a little large, we moved WFSA closer to the lens by 5 inches and WFSB closer to the lens by 9 inches. Hopefully, this helps keep the sensing matrix more consistent when we recenter the beams. TBD...

Here is the list of commissioning task for the next 7-14 days:

Green team:

• Look for the source of the 1/f noise.
• Look for clipping in green transmission.
• Make a mode scan of the red higher order modes.
• Test the length feedback path to the ETM top suspension point to suppress microseism.

Red team:

• Center the red QPDs in EX.

Blue team (ALS WFS):

• Reduce beam size on sensor head.
• High bandwidth WFS.

Blue team (ISCTEY):

• PLL fiber locking.
• Add WFS.

SEI/SUS team:

• Commissioning optical lever.
• PUM range for ETMs.
• ETMY diagonalization.

Weekend report

model restarts logged for Sat 29/Mar/2014

no restarts reported for Sat

model restarts logged for Sun 30/Mar/2014

2014_03_30 00:42 h1fw0

Unexpected restart of h1fw0

Sheila, logged in as Alexa. 

I spent some time working on the ALS WFS today.  The guardian can now be used to turn on and off the WFS, although to get started we still need to align the cavity pretty well by hand and then center the WFS by hand.

 I was able to lock one DOF (DOF_2_Y) with a 1Hz bandwidth (transfer function attached, this was using a gain of -0.05).  However, when I tried this after recentering the WFS it was no longer stable, so I've set it back to -0.001

One difficulty I had today is that the sensing matrix changes enough as the WFS centering changes to make the loops become unstable.  When I arrived I centered the WFS, locked all 4 DOFs, then the centering changed and the pit DOFs moved away from 0 until they saturated the DAC. 

The second screen shot attached is the measurent of the sensing matrix for ITMX pitch that Alexa and Daniel did on friday, next to one I did today.  You can see they are totally different.  After recentering I measured the sensing matrix again, and got back a value similar to Friday's. 

Assembled aLIGO RGA on BSC6 and rough pumped RGA volume -> replaced gauge controller and foreline pirani gauge on MTP (gauge exhibiting failure mode of losing "enhanced" or stable readings in turbulent flow conditions, controller losing power to B channel pirani (foreline) intermittently) -> Demonstrated MTP "Safety Valve" closing as a function of foreline pressure setpoint when in "Turbo" mode -> Pumping BSC10 annulus 

2100 hrs. local -> Kyle leaving site now

BSC ISI guardians now running for BS, ITMX, ITMY, ETMX

After a lot struggle with the BSC ISIs this week, we finally have all commissioned BSCs under guardian control.

The BSC seismic stack is one of the more complicated systems we've deployed, as it currently consists of three nodes in a "managed" configuration: two separate nodes for each of the two ISI stages (e.g. ISI_ITMY_ST1, and ISI_ITMY_ST2), and a "chamber manager" (SEI_ITMY) that coordinates the activity of the two subordinate ISI stages.  USERAPPS location of the top level modules (and primary loaded library):

• USERAPPS/isi/common/guardian/SEI_ITMY.py     (USERAPPS/isi/common/guardian/isiguardianlib/BSC_MANAGER)
• USERAPPS/isi/common/guardian/ISI_ITMY_ST1.py (USERAPPS/isi/common/guardian/isiguardianlib/ISI_STAGE)
• USERAPPS/isi/common/guardian/ISI_ITMY_ST2.py (USERAPPS/isi/common/guardian/isiguardianlib/ISI_STAGE)

All single ISI stages share the exact same code, for each BSC stage as well the single HAM stage.  Here are the system graphs for the SEI manager and ISI stage 1:

The manager (left above) has three main requestable states: READY, DAMPED, ISOLATED.  The READY state puts both of the ISI stages in READY, the DAMPED state puts them both in DAMPED, and the ISOLATED state puts both into isolation levels that can be specified in the manager system description module (USERAPPS/isi/common/guardian/SEI_*.py).  At the moment all systems are using their default configuration, which is to operate the ISI stages in the HIGH_ISOLATED state, which corresponds to "lvl3" from the old "command" scripts.  This can't currently be changed on an individual BSC basis, due to a guardian core issue, but I'm working on it.

The degrees of freedom for which to restore cart bias offsets can also be specified on a per stage basis.  Note the ISI_ITMY_ST1 graph above has "RESTORE_ISO_CART_BIAS_*_RZ" states, and no corresponding states for the other degrees of freedom (X, Y, Z, RX, RY).  This indicates that we are restoring only RZ target cart bias offsets for this stage.

Notes about the BSC isolation procedure

There are a couple of important things to note about the current isolation procedure:

We had a lot of trouble dealing with the T240s in stage 1.  Their exterme sensitivity to pitch causes them to saturate if a large pitch cart bias offset is applied to the stage.  We there spend a lot of time trying to figure out how to gracefully bring them in and out of the loop, by reducing their gain and/or changing the blend filters during the isolation procedure.  All of theses things has problems, though:

• switching of the T240 analog gain does not synchronously switch the corresponding digital compensation filters.   This means that any attempt to change the T240 gains will cause the T240 watchdogs to trip.
• there were strange problems with changing the blend filter with the isolation loops closed.  The "SWITCH_ALL" buttons cause large glitches in the isolation loops.  Clicking each DoF manually doesn't cause glitches, but clicking them too fast causes various things to saturate.  Basically you have to manually switch each gain independently, with a significant delay between each DoF.

This makes things very difficult if we need to hold pitch offsets on any of the ISIs.  It looks like we need to fix the synchronous switching of the T240 compensation filters before we can handle pitch cart bias offsets.

However, after speaking with the integration team, it seems that it's ok to operate temporarily in a mode where we only restore RZ cart biases.  Any needed pitch offsets can be held in the suspensions for now.  Therefore we do not need to change the T240 gains or blend filters during the isolation procedure.  This is good news, since it means we can run with the current guardian behavior.  Obviously this will have to be fixed down the line, as we will likely want to hold pitch offsets on the ETMs, but we have a workable solution for the moment.

Guardian BSC isolation configuration and procedure

• All sensors are configured to be in HIGH gain mode.  This includes the T240s and L4Cs on stage 1, and the GS13s on stage 2
• Both stage 1 and stage 2 BLEND filters are configured to be in "Tcrappy" for all DOFs at all times.  "Tcrappy" are the low-frequency cross over blends that include the T240s on stage 1.  There is no changing of the blend filters at any point during the isolation procedure.
• All cartesian biases for all DoFs are restored to their "equilibrium" positions at the beginning of the isolation procedure.
• Only the RZ degree of freedom cart bias target is restored at the end of the isolation procedure.  The rest of the DoFs are left at their equilibrium position.
• Both ISI stages are configured to go to HIGH_ISOLATED (i.e. "level 3").
• Isolation procedure:
  • stage 1: DAMP
  • stage 2: DAMP
  • stage 1: equilibrium cart bias
  • stage 1: HIGH ISOLATE
  • stage 1: restore RZ cart bias
  • stage 2: equilibrium cart bias
  • stage 2: HIGH ISOLATE
  • stage 2: restore RZ cart bias

With the above configuration we can successfully isolate the BSCs without touching the T240s or blend filters.  It seems to be fairly robust.

Important operator notes:

• We don't yet have guardians for HPI.  This means that restoration from HPI watchdog trips require a bit of care.  If the ISI watchdogs are reset before the HPI, the SEI manager will restore isolation on the ISI before the HPI is restored.  This means that the ISIs will trip during the HPI isolation.  It is therefore important remember to  reset HPI watchdogs and restore HPI isolation before resetting the ISI watchdogs.
• It's possible that depending on the seismic environment, etc., the ISI could trip during the isolation procedure.  We have noticed, for instance, that some of the sensors might saturate during isolation if things are noisy.  Don't give up on guardian if this happens.  Just reset the watchdogs and let guardian try to isolate again.  It will likely work during the next attempt.  If it still doesn't, try waiting until things settle down a bit before resetting the watchdogs.
• If all else fails, the guardians can be either set to PAUSE (so they stop executing code) or can be shut off entirely (e.g. "guardctrl stop SEI_ITMY ISI_ITMY_ST1 ISI_ITMY_ST2"), at which point all the old command scripts can be used.  This should obviously be done only as a very last resort.

Notes about guardian SEI manager behavior

The SEI managers represent a new step for guardian, as their primary task is to control the state of other guardian nodes, which we haven't needed up to this point.  Getting their behavior robust has been quite tricky.  It necessitated a lot of work on the guardian built-in "manager" library, as well as a lot of experimentation with the SEI manager module itself.  The manager is designed to be fairly robust against commissioner noodling with it's subordinates, but that is also fairly tricky to get right in a robust and useful way.  Some things will need to be improved, but this is the behavior currently:

• The manager puts its subordinate ISI nodes into MANAGED mode during the INIT state.  This slightly modifies the graph traversal algorithm of the subordinate to defer to the manager after "jump" state transition.  The subordinate node MEDM control screens also turns purple in MANAGED mode as an indicator to the operator that the node is under the control of another guardian node.
• The INIT state now has a special feature that it can be jumped to directly from any state in the graph, after which the overall system REQUEST will be reset to whatever it was previously.  The SEI manager INIT state is programmed to set all subordinates to MANAGED, identify the current state of both ISI stages, reset things as needed, jump to the current state of the system, then proceed to the request.  Therefore, regardless of the state of the subordinates, requesting INIT from the manager should completely reset the state of the SEI BSC.
• The manager expects its subordinates to be alive.  If they're not, the manager will go into ERROR.  Once all subordinates are alive, clear the error (mode => LOAD) and re-exec the manager and it will pick up the subordinates.  You may need to move to request INIT in the manager to fully reset the the system.
• If a subordinate REQUEST is changed while in MANAGED mode, the manager will throw a notification and will immediately reissue it's old request.  The subordinate will have already started moving to the rouge requested state, but will immediately move to back to the manager requested state once it gets there.
• If a subordinate is removed from MANAGED mode, the manager will throw a notification, but will keep running as normal.  The manager will continue to monitor the state of the now unmanaged subordinate but will stop issuing new commands to the subordinate.  Since the manager will continue to process normally, it will therefore expect the subordinate to change it's state on its own.  The new human manager will therefore need to take over this task, as the manager will otherwise not see it's states completed.

While things seem to working well at the moment, I certainly don't claim that all the bugs have excised.  If the manager does get hung in some state, resetting to the INIT state should clear everything, reset the ISIs, and get things back on track.

TODO for SEI guardian

• turn on guardian nodes for ETMX once it's ready, e.g.: guardctrl start ISI_ETMX_ST{1,2} && guardctrl start SEI_ETMX (should then remove the "WIP" masks over the BSC ETMX SEI and ISI guardian boxes in the GUARD_OVERVIEW screen).
• fix synchronous gain switching for the T240s
• figure out how to handle T240s with pitch cart biases (either via T240 gain switching and/or blend filter switching)
• HPI guardians, for both HAMs and BSCs
• incorporate HPI nodes under SEI managers for HAMs and BSCs, for full SEI stack control
• allow specifying non-default isolation levels for ISI stages (probably something in guardian core needs to be fixed)
• further development of manager infrastructure

A. Staley, S. Dwyer, J. Kissel 

After speaking with Sheila and Alexa, trying to understand the spectrum in LHO aLOG 11026, I realized that -- because of so much unexpected excess noise -- the current ALS setup has diverged significantly from the baseline plan. As such, we've worked together to create a diagram that is to-the-best-of-(my)-knowledge accurate, and makes sure to show how every control signal gets around to its actuators, including passes through digital land. The only things I did not include were the demodulator boards between the REFL diodes (ARM REFL, MC REFL, and the two green XARM and YARM REFLs) and lame things like whitening, AA, and AI chassis that facilitate getting in-and-out of digital land.

In particular, I draw your attention to the unorthodox and new features that were not part of the original design plan:
(1) The PRM is misaligned. This way we can use IFO REFL red diode signal (after normalizing / linearizing it with the red arm cavity trans) to measure the arm and offload at low frequency to the CARM board in the second input. This is unorthodox because normally we would feed REFL straight to the CARM board in all analog and retain the large bandwidth, but because the noise is so large, it must be linearized in the front-end (the only place the Arm TRANS signal exists), and fed out one of the corner station LSC DACs. Note that this configuration is just a special technique for these particular series of measurements and is not designed to become a normal part of the lock acquisition scheme.

(2) The Voltage Controlled Oscillator (VCO) actuated, Phase-Locking Loop (PLL) that's wrapped around the corner-station COMM and DIFF Phase-Frequency Discriminators (PFD) are used to reduce the noise enough to keep the PFD in the linear, phase detecting mode (check out pgs 6-8 of the AD9901 datasheet -- it's a really nice explanation of how it works).

(3) Before the hand-off, there's another temporary step which is to use the Beckhoff system ("Slow ADC" in the diagram) to feed directly to the PSL VCO (again at low frequency) to reduce the noise enough for the COMM hand-off to work. 

This is most certainly an impressively complicated system!

Note that this diagram has been committed into the noisebudget repo here:
${NBSVN}/trunk/HIFO/Common/
and I imagine will become an "as built" DCC document in the glorious future when everything is working beautifully.

started at ~18:50 will run for 8 hours.

Yesterday I went to end X to investigate the beam shape for the ALS WFS.  (This is to follow up the measurements we made in alog 10774)

First I tried changing the alingment of the reaction chain, this didn't make a difference.  Then I tried changing the TMS QPD offsets.  This moves the lock point of the input alingment loops, changing the alingment of the green beam as it goes through TMS.  This certaintly changes the beam quality on the WFS, screenshots of several different offsets are attached.  In the end I left the offsets at ALS-X_IP_ANG_PIT at -50, ALS-X_IP_POS_PIT at 150. 

I am not sure why changing these offsets makes such a difference in the beam quality. 

This moved the QPD A by 0.07 counts and QPD B by 0.04 counts.  Pablo and I redid the pointing of TMS to the baffle PDs, but it hadn't changed more than the normal change between ISI trips.  However, switching these offsets does move the beamspot on ISCT1, I moved PR3 by 0.9urad in pitch to bring the spot back.  

After realinging the arm, we re checked the phasing of the WFS. We did this by putting an exictation on LSC-X_EXTRA_AO_1_EXC at 50Hz and looking at the signals of the individual segments, which seems fine.  This is saved in /home/sheila.dwyer/ALS?HIFOX/WFS/WFS_PhaseDTT_March282014.xml

Then we started the sensing matrix measurement in Alexa's alog 11073

This week, I tested the ETMX UIM P2P and Y2Y filters (described in alog 10912). The results are showing that filters are working as expected giving a single pendulum transfer function falling at 1/f^2 after the resonnance (f=0.45Hz for pitch, f=0.6Hz for yaw). Also the coupling between pitch and yaw is small (~5%) so unless this is too much, there's no need for diagonalization for UIM.

Attached pdf are showing P2P P2Y (first pdf) Y2Y Y2P (second pdf) transfer functions with/without the filter.

Left to do for next week :

ETMX : Long decoupling on UIM/PUM

• measure UIM L2PY for long decoupling 
• measure PUM L2PY for long decoupling

ETMY

• Install/comission oplev on ETMY
• Balance UIM/PUM coils
• Measure top to test mass L2PY P2PY Y2PY
• Measure UIM to test mass L2PY P2PY Y2PY
• Measure PUM to test mass L2PY P2PY Y2PY

ITMX/ITMY

• Measure/invert UIM P2P Y2Y

I got the medms populated so the cartesian location values are now valid.  These initial values today are the 'Aligned' numbers.  They will probably be a little different after these range of motion/linearity tests complete and the stress of the system are further released/shaken out.

This trip happened around the time of a beckhoff restart.  A beckhoff restart causes a crash of the IMC guardian, which causes it to stop.  Its not clear to me why this should cause MC2 to get a large signal, but it seems to.  This causes a cascade of trips.  Even though I don't know why this happends, creating a safe state in the IMC guardian so that it handels missing channels better could help with the problem. 

Another solution is to have the SUS WDs not trip HEPI.  Can we get that fix soon?

HAM2 also tripped at the same time, Jeff and Hugh brought that back. 

Margot and I performed all of BSC10 closeout tasks listed in my previous alog regarding this closeout.  We pulled the FC from the ETMy optic a little after 11am, and the door was going on at ~1:30.  It took "so long" (ha) because we ended up having to spend some extra time calling in the troops about one of the ISC viewports having what appeared to be a scratch on it's inside surface.  More details to come as needed.

This morning, SEI, TMS, and SUS gave a cursory "approval" of their systems in order to proceed with chamber close-out tasks.  So, after lunch, Margot and I applied First Contact to the ETMy-HR surface which will be pulled just before closing the door tomorrow.  The order of events this afternoon (and late morning in order to catch the alog up):

Jim locked the ISI and put final touches on chamber feedthru cable routing.

Apollo+Joe+Gerardo had a viewport party at the BSC10/6 area.

Margot and I locked the lowest masses of the ETMy QUAD.  Then we,

Secured the ACB locking bracket and used the wedge to swing it back and prop it into place.  See notes on this in attached alog.

Attached the FC QUAD cone to the structure and sprayed FC (standard 3 coats) on the HR surface.

Removed the cone from the suspension.  Finished FC layer reinforcement.

Removed the wedge from the ACB and restored it's nominal hanging position. Removed the ACB locking bracket screws.

Started stowing all left over bags of "junk" left outside of the chamber door on various table surfaces by SUS and every other team that visited this last month.  If you're missing something (and you noticed), I took ownership of it.

Jim arrived and unlocked the ISI.  The ETMy QUAD will remain locked during dome replacement.

Apollo started gearing up to remove walking plates from upstairs and crane the dome on the chamber.

Tomorrow's to-do list which Margot and I will do until the last step:

Inspect insides of viewports.

Adjust ACB to check magnet gap which may or may not have shifted slightly due to the swing back from today.

Clean the chamber, vacuum, etc.

Take final PCL swipes.

Blow various other optic surfaces.  Or not.

Swap in final 1" witness optic on QUAD structure.

Check/Remove all tools.

Remove ETMy-HR with N2 blow.

Set and lock all 64 QUAD EQ stops and nuts, accounting for buoyancy.

Set QUAD BOSEMs to account for buoyancy.

Quick QUAD health check with DTT TFs.

Lay final witness plates.

Exit chamber.

Finish stowing everything around door and move it out of the way.

Put door on.