Successful upgrade of SEI guardian code today to achieve full SEI deployment

Yesterday, we:

• installed new guardian core (v918) and cdsutils (v237)
• HPI guardian nodes deployed for all active HPI systems: HAM 2,3 and BSC 1,2,3,9,10
• upgraded ISI single-stage code which includes monitoring of T240 saturation state: all ISI stage nodes were restarted
• upgraded BSC chamber manager nodes to include handling of BSC HPI: all BSC managers restarted
• deployed HAM chamber managers for HAM 2,3 that coordinates HAM ISI and HPI

In other words, we now have full SEI guardian for both HAMs and BSCs.  This is a fairly big milestone, as now essentially all seismic systems are fully automated (modulo the output HAMs which are still undergoing installation).

SEI guardian operation

The most important new UI aspect of these improvements is that all interface to the seismic systems for each chamber is now done through the chamber managers, e.g. SEI_HAM3, SEI_ITMY, etc.  They are the manager nodes that coordinate the activity of the HPI and ISI in each chamber.  All HPI and ISI nodes should now be operating in managed mode, which means that there activity is controlled by their respective chamber manager.  Managed nodes are purple.  Here is a shot from the guardian overview screen, showing the BS manager SEI_BS in blue, and the one HPI and two ISI nodes in purple:

All interaction with the seismic systems should now go through the managers.  Here's the graph of the SEI_BS manager, which is identical to the graphs of the rest of the BSC chamber managers:

The dark blue states above are the requestable states, e.g. DAMPED or FULLY_ISOLATED.

In general, hte managed subordinate nodes (i.e. the ISI and HPI nodes) should not be touched while they're in managed mode (i.e. purple) unless you know what you're doing.  Otherwise you run the risk of interfering with and potentially confusing the manager, and therefore distrupting the automation.  That said, the managers are designed to be fairly robust against noodling with their subordinates (see alog 11077 for more information about manager behavior).  Probably the most important thing to keep in mind, that might help you out of a jam or to reset things:

If any of the subordinates are out of wack, for whatever reason, e.g. they are not in the correct MODE or STATE or REQUEST etc., command the chamber manager to go to the INIT state

The manager INIT state will:

• reset all subordinates into MANAGED mode (i.e. reacquire them to being managed)
• check the STATEs of the subordinates to determine the overall state of the system
• REQUEST the appropriate states of the subordinates to recover the system back to a known configuration

SEI guardian code structure

All SEI guardian code is installed in:

$USERAPPS_DIR/isi/common/guardian

At the top level of this directory are the main guardian nodes for HPI, each ISI stage, and the chamber managers.  These modules all load their code from the SEI guardian library:

$USERAPPS_DIR/isi/common/guardian/isiguardianlib

This library includes the primary packages for HPI, ISI_STAGE, HAM_MANAGER, and BSC_MANAGER.  These load support modules/packages also part of isiguardianlib.  For example, the ISI_ITMY_ST1 guardian is defined by the ISI_ITMY_ST1.py module:

from isiguardianlib.ISI_STAGE
import * prefix = 'ISI-ITMY_ST1

The first line loads all code from the isiguardianlib.ISI_STAGE package, and the second line sets the channel access prefix.  All the SEI modules are structured in this way.

Jamie, Dave, Fabrice,

We have implemented the modifications in the isi2stage master model that were necessary to:

1)  Allow  good functioning of the guardian

2)  Investigate the subtraction scheme we have been talking about to improve teh RZ (Yaw) isolation.

We are testing these changes so we can decide which ones to propagate (ECR, svn commit, install on other units, other sites...)

Details of the changes:

- Added a block in the ST1 block, to filter the T240RZsignal induced by the Z drive. This block is called T240 subtract

- Changed the way the WD "ignore" the T240 when they are not in loop. This is now done in the T240 block instead of being done in the c code. It permits to keep the c code generic for all other types of sensors. The c code is back to 9 inputs. I removed the 10th input in Stage 1 and Stage 2 "Muxes" entering in the Simulink. We must check that everything i s compatible with the HAM-ISI and BSC-ISI models

- Changed the logic to calculate the state of Stage 1 ISO. It is now done with boolean operators. The output returns a "0" if all gains are 0, and a "1"  if any of the gains in non-zero. Added an Epics output to monitor this variable.

- Added some logic to reset the T240 counter when the isolation is tuned on.

The model has been compiled, installed and started on ITMY.   The T240 counter does reset when the isolation is turned on. However, on the last test I did with the guradian before to hand the unit back to commissioning, it tripped on the T240s. To be investigated tomorrow morning.

Other problems to be fixed:

- the GS13s trip when the boosts of the HEPI horizontal loops are engaged. We must check if it's a matter of adding phase margin or adding some ramping time.

- all the safe.snaps must be updated to record the T240 saturation counter parameters

- for some reasons, the cont2b filters in ITMY Stage 1 X,Y, Z had disappeared. I assumed they were just constant values of "1" and added such filters with foton. To be checked.

- some HEPI ETMX boost filters were not loaded. We loaded constant values of "1" so we could keep working on the guardian. To be fixed.

Hosken, Grabeel, Vo

CO2X:
- Laser left on running overnight again.
- Prepare gig-E camera infrastructure for wiring and software.

CO2Y:
- Rotation Stage working, turns out that the motor power cable  at the feed through was not plugged in.
- Starting to connect up temp. sensors on table, looking for the custom clamps that mount the sensors.
 
HAM4 HWS:
- Alignment laser is the wrong wavelength for the dichroic mirror which is HR at 650-900nm and HT at 500-550nm.
- Lens mounts readied for install in-vac

We twice observed a filter module turn on almost immediately, even though it was supposed to ramp over 15seconds.

It is not reproducable in the sence that it seems to be a random occurance, and usually ramps with the propper time.

8:00-9:42 Getting ready to move clean room in LVEA (Craning) – Apollo
8:30 Water supplier on site – Paradise
8:35 Going into the LVEA to search for parts – Corey
8:40-14:00 Continue assembling/suspending/balancing of the ACB suspension- Betsy/Travis/Margot
8:48-13:02 Heading into the LVEA for SRM alignment – Jason
9:00-11:50 Joining Jason on SRM alignment – Jeff B
9:05-9:44 Retrieving items from LVEA - Gerardo 
10:04-12:02 Heading in to the LVEA for TCSx table work – David
10:09-15:43 ICS rack inventory in LVEA – Filiberto
13:14 Restarting ISI model for ITMY – Dave/Fabrice
13:16 Restarting ASC model and the DAQ – Dave/Fabrice
13:32-14:20 Back to HAM5 to connect BOSEMs – Jeff B
14:05 Transitioning the LVEA to Laser HAZARD
14:13 Back to West bay (LVEA) – Betsy/Travis/Margot
14:37 Back to the LVEA for TCSY table work – David/Greg

 LVEA is Laser HAZARD!

NOTE: Today while going over the CDs Check sheet, I noticed that the Right BOSEM on PRM was reading almost 0. I talked to Alexa and concluded that it was because the suspension has been yawed.

Heated volume is adjacent to but isolated from unpumped volume seen by PT246A,B gauges -> Ignore readings from PT246A,B until further notice (2 weeks?)

Last week I took some in chamber, in vacuum acceptance measurements for TMSx and TMSy. ISI tables were both isolated during the measurement.

Attached are the 6 Dofs transfer functions measurements as well as the osems spectra for both suspended tables :

1. TMSX TF compared with the model
2. TMSX spectra for each osem, as well as in the euler basis
3. TMSY TF compared with the model
4. TMSY spectra for each osem, as well as in the euler basis
5. TF comparison of both H1 TMSs and L1 TMSs

To note :

The difference seen in the TF of H1 TMSx is explained in previous alogs
TMSx spectra shows excess 60Hz noise in F1 F2 F3 and LF osems (see 2nd attachment)
Damping for the yaw degree of freedom, for both TMSs is increasing the motion at the resonnance (see last page of 2nd and fourth attachment). I will turn the yaw damping off for both TMSs for now.

Took a first look at the SRM surrogate (SRM-s) today and lo-and-behold, no adjustment was necessary.    The cookie cutters did their job and placed the suspension within spec for X, Y, and Z axis positions.  The position errors are below:

• X axis (lateral): -1.5 mm (south)
  • Tolerance: ±4.6 mm
• Y axis (longitudinal): +1.3 mm (west)
  • Tolerance: ±3.0 mm
• Z axis (vertical): +0.9 mm (high)
  • Tolerance: ±3.0 mm

I have begun and mostly completed the setup for the rough pitch/yaw alignment of the SRM-s.  This will be completed tomorrow morning and the rough pitch/yaw alignment will follow.

Fabrice and Dave:

We successfully made a new h1isiitmy model using the latest isi2stagemaster model changes. We restarted h1isiitmy with the new code

Chris, Sheila, Dave:

I found that I had mistakenly restarted an incorrect h1asc model on Tuesday. After the first DAQ restart that day I noticed that the DAQ picked up a newer INI fle for h1asc and I assumed (which we should never do) that the last "make install" was correct. It appears that later changes were not installed. 

I did a clean "make" and "make install" on h1asc and restarted the model.

Dave:

Finally I did a DAQ restart to support the above changes (remembering to wait into the next 10min period to flush out the second trends)

Here is the restart sequence in detail:

2014_05_15 13:16 h1isiitmy
2014_05_15 13:19 h1asc
2014_05_15 13:22 h1broadcast0
2014_05_15 13:22 h1dc0
2014_05_15 13:22 h1fw0
2014_05_15 13:22 h1fw1
2014_05_15 13:22 h1nds0

8:15-8:30

Fred, Richard, Cyrus, Filiberto, Aaron, Bubba, Gerardo, Jeff B, Bubba, Travis, Hugh, Mitchell, Justin, Mike, Thomas, Jeff K, Fabrice, Alexa, Steban, Robert, Jamie, etc.


•	SRM alignment on HAM5 - (Jeff B/Jason)
•	TCS tables work continues. – (David, Thomas)
•	Getting things ready to transport the OFI - (Gerardo)
• 	Craning activity in LVEA / Moving clearoom from west bay -(Apollo)
• 	ICS rack inventory in LVEA – (Richard/Filiberto)
•	Commissioning work continues (Commissioners)
•	Testing and assembly operations continue in Staging building –(Jim/Hugh/Scott)
•	Guardian work in progress -(Jamie)


        LVEA is Laser SAFE

Richard warned me that the Fiber polarization had probably drifted at EY due to some work done around the table. Indeed it had drifted to up to 33% for the wrong polarization. However, Stefan had increased the threshold to 36. I returned the threshold down to 30 and adjusted the controller. I also had to adjust the EX polarization when I turned the controller on, which forced both polarizations to drift. All is good now.

Craning activity in the LVEA at the moment. A cleanroom is being moved - Apollo

model restarts logged for Wed 14/May/2014
2014_05_14 12:37 h1fw1

unexpected restart of h1fw1

Richard showed me that there is a liner coupling from any of the electrodes to the OPLEV even when there's no bias anywhere, and the bias-less coupling is on the same order as the coupling with bias.

I started flipping the bias of the common ("DC") electrode while keeping all four quadrants at zero count output, and indeed there's a linear coupling, and the effect is not subtle.

In the attached, I changed the DC bias from 125000 to 0 to -125000, then to 0 and finally back to +125000. The ETM points up by 0.3urad in the first step, then further up 0.5urad in the second step (instead of going back to the original position). It's 0.8 urad peak to peak!

OTOH, when I set the four quadrants such that top two outputs -50900 counts and the bottom +50900 to tilt the optic legitimately while the DC bias is kept at 125000 counts, I can only move it by -0.5 urad peak to peak (second attachment).

This could still be electronics offset, so I looked at the ESD readback. The readback is not calibrated, but four quadrants always read from -75 to -200 counts when I'm outputing zero, the 300volt-ish thing reads 30000-ish counts, so the voltage offset for the quadrants should be 2V-ish at most. That's not large enough, I think.

I think that the ISI trip happened before the suspension trip, this was while we were locking diff.  I also thought that the T240s were green before I untripped the watchdog, but maybe they weren't because the ISI tripped again as it was isolating. 

This afternoon, after Jeff balanced the coils, I took measurements on the main chain of ETMY for filter design.
Results attached are showing Length/Pitch/Yaw drive to Pitch/Yaw Response for the top mass, UIM and PUM. Y axis is in urad/cts

Templates live in the usual sus path H1/ETMY/SAGL1/Data/2014-04-16_H1SUSETMY_P2PY_WhiteNoise.xml