model restarts logged for Wed 24/May/2017
2017_05_24 13:04 h1ecatx1

Restart of EX Beckhoff slow controls PLCs in attempt to clear an error (did not clear the error)

model restarts logged for Tue 23/May/2017
2017_05_23 09:57 h1iopseih23
2017_05_23 09:59 h1hpiham2
2017_05_23 09:59 h1hpiham3
2017_05_23 09:59 h1isiham2
2017_05_23 09:59 h1isiham3

Restart of all models on h1seih23 after timing glitch in CER.

After the initial work on installing the air bleed in the cooling circuit, the flow rate in head 3 was lower
than it was before.  There is no reason why the air bleed should affect things.  All the other heads are the
same as previously.  I lifted the lid off the oscillator and looked at the turbine sensor.  Nothing unusual
was seen.  I tapped the flow sensor to see if it might dislodge anything stuck inside but the flow rate did
not change.

    The reduced output power of the laser is real.  I checked the alignment of the monitoring photodiode
that is located near the diagnostic breadboard and it appeared more or less okay.

    I also checked the iris underneath the IO periscope.  The beam appeared to be off there too.  Unfortunately
there are no other irises earlier in the beam bath, so it not easy to tell where exactly the beam may be off
centre.

TITLE: 05/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Environment
OUTGOING OPERATOR: N/A
CURRENT ENVIRONMENT:
    Wind: Calm
    Primary useism:~ .01um/s
    Secondary useism: ~.2um/s
QUICK SUMMARY: ops lazy script isn't doing transition (-t).

Based on my HAM2 install pictures of the IMC Trans beam in vacuum steering mirror (ROM RH2), it’s position is closer to the blue oval (that represents the front of the optic) in the attached diagram, than the location of the mirror in the DCC drawing.

The beam shift in yaw of -7mm on IM1 has also translated the beam on ROM RH2, and it’s my belief that the IMC Trans beam is now clipping on the edge of that optic.

One of the things we ran into today was that the ALS laser had an error.  One of the diode currents has drifted outside of the tolerance, which caused an error.  (Trends coming tomorrow).  

• One of the things that made this hard was that the error was intermittent and the error messages flash very briefly making it hard to read them.  
• A second thing was that although we could trend and see that the laser head error flag was toggling between 0 and 1 for the last few days, the error code has been consistently 0.  This meant that we weren't able to see which error it was
• I looked at the code and started checking for error conditions in the trends, and we saw the offending diode current.  Increasing the tolerance made the errors go away.  Jenne Jeff and company proceeded with locking. 
• Daniel and I looked at the error handling code and the ALS Laser library, and realized that in the ALS Library the error message isn't set, and the error list is made but not added to the Error Handler when it is initialized. In the Error Handler library, there is an if, else if statement which assumes that there is either an error code or an error list, and sets the error code within the conditional statements.  We edited the Laserhead library so that there will be an error list once it is updated. It might also be possible to take the error code setting outside the conditional but that would require more thought.  
• It would also be possible to edit the ALS autolockers so that they are less stringent in forcing us to fix errors.  

So, we should do an update of the beckhoff code sometime soon.  We had not encountered this problem before because we haven't had any errors from the laser head before.  

V. Adya, J. Driggers, J. Kissel

We're heading out with all DRMI ASC loops closed and DRMI relocking reliably, with the IFO breaking lock just before RESONANCE (we only tried reducing the CARM offset a few times). We'll pick up here tomorrow. We noticed Bounce and Roll Modes were quite high, so they'll likely need attention tomorrow morning.

We leave the IFO locked in DRMI with ISC_LOCK guardian requested DOWN in case of earthquake. Anyone who's willing to pick up where we left off tomorrow morning is welcome, but do pay attention to Bounce and Roll modes -- they may just need some tender love and care before we go further. 

Great work today team!

V. Adya, J. Driggers, S. Dwyer, K. Izumi, J. Kissel, J. Warner

A mid-recovery status report: 
After slowly working our way through initial alignment, we've been able to successfully and stably lock DRMI despite strong winds, but we're not yet at a good enough alignment for DRMI WFS to come on. 

No major problems from initial alignment up to this point, just gotchas really:
- One needs the 2nd light-pipe open to find a green beatnote to align PR3
- One needs to turn ON the high voltage for the fast shutter to open so you can see light on WFS and run corner station initial alignment WFS
- If you're starting from a post-vent fresh alignment, don't trust the SR3 cage servo reference, so turn it off.
The lock acquisition sailed through ALS, and what trouble we're having with PRMI & DRMI is all just finding the right alignment for the IFO. Namely we're slowly closing each DRMI ASC loop by moving sliders to reduce error signals to where we can turn on the loops.

We've gotten so far as to close all loops except for PRC1 (i.e. MICH, INP1, PRC2, SRC1, SRC2), and it's just the slow process of moving PRM by hand while the rest of the ASC system brings the optics with your moves.

Slowly but surely!

I attach my detailed, blow-by-blow notes for future reference, up to this point.  

Aidan (remotely), Nutsinee, TJ (in spirit), Kiwamu,

The return beam of the Hartman system for ITMX (HWSX, alog 36332) was successfully found on the HWS table today after people performed the full initial alignment process once.

We then centered the HWSX return beam to the Hartman camera. We are ready to repeat the measurement of the hot spot (34853).

(some details)

We followed the procedure written in T1700101-v2.

Nutsinee and I went to the HWS table after the initial alignment had been done. First of all, looking at the Hartman beam with respect to the two irises on the table, we confirmed that the Hartman beam had stayed in a good shape. We then started checking the periscope mirrors to look for the green light down from the X arm. We immediately found the green light on the top periscope mirror. Carefully inspecting the spot position on the mirror, we determined that the beam was well centered with a precision of about 5mm or so.

We steered the top and bottom periscope mirrors to make the green light centered on both irises. Finally, we steered the steering mirror in front of the Hartman camera while watching the stream live image of the camera. We found the Hartman return beam on the stream image right away. We then touched the top periscope mirror and the steering mirror to fine-tune the beam/aperture locations.

The attached are the images after we finished aligning the return beam to the camera with and without the Hartman plate.

WP 6652
ERC 170062

The BRS ion pump read back signals are now tied into the vacuum rack. Field cabling from each controller was pulled to connector header H (Pins 37-40) inside the vacuum rack. The Vacuum Controls Chassis was modified to add the new read back signals to terminal 15. Channel 1 is the voltage read back and channel 2 is the pressure read back.

Controllers for the BRS ion pumps are not the same.

End Y - Gamma SPCe controller
End X - MiniVAC Controller

F. Clara, G. Moreno, R. McCarthy, P. Thomas

Following the installation of the OS and HWS_CODE on the H1HWSMSR1 computer for HWSY we found we couldn't get the HWS code to run properly. It turned out that the new computer had an old BASHRC file (where the environment variables for the HWS code are defined). I fixed this by duplicating the BASHRC file from H1HWSMSR. I've attached it below. 

The two computers are identical in all respects but one. Due to the way the HWS code currently runs, each computer creates EPICS channels for ITMX and ITMY. In order to avoid duplicate channels messing up EPICS, I've made the following two changes:

On H1HWSMSR (the HWSX computer) in .bashrc:

• export OPTIC0=ITMY0_HWS 

  • a fake ITMY0 channel is created

On H1HWSMSR1 (the HWSY computer) in .bashrc:

• export OPTIC1=ITMX0_HWS 

  • a fake ITMX0 channel is created

I valved-in the running pump cart to the HAM11/HAM12 annulus volume for a few hours today.  The HAM11 controller had went off scale high some time ago for no good reason as both the pump and the controller have low hours.  It stayed off scale even while being assisted by the pump cart so I "assisted" the pump body with a few impacts (tough love) at which point it recovered and stayed on scale.  

1615 hrs. local -> I valved-out the pump cart.  We'll see if it has "learned its lesson".

  Peter K., & Jeff B. 

   This morning we replaced the leaking ball valve assembly reported in aLOG #36343. A last check of the new assembly, just before opening the GVs, showed no leaks. Will take a look at it during the next window to verify all is well.   

J. Driggers, K. Izumi, S. Dwyer, J. Kissel, V. Adya, T. Shaffer

We've been able to find green alignment quite easily. Yes!!

However, once we were able to get ALS X well aligned and locked, with Green WFS and ITM camera alignment systems ON -- the ALS X Beckhoff state machine turned into a blinking MEDM light show. The arm would remain locked, with good alignment, the ALSX guardian would go into fault.

The obvious symptoms were several momentary errors on the (from the ALS Overview Screen) Beckhoff screens for PDH, Fiber PLL, and VCO that each complained of each other.
We started with slow calculated attempts of trying to disable various parts of the state machine, e.g. 
    - using H1:ALS-X_FIBR_LOCK_LOGIC_FORCE to force the fiber PLL lock, or 
    - by hitting reset (H1:ALS-X_VCO_CONTROLS_CLEARINT) on the H1:ALS-X_VCO_TUNEOFS to reset the frequency finding servo.
we then degraded to a bit of button mashing, after which the state machine would just restore everything to what it was before we started.

Finally, Sheila showed us how to dig down an alternate path for finding errors via the sitemap > SYS > EtherCAT overview and follow the error messages from there. However the screens only show explanatory text when there are errors present, which makes tracing a momentary error frustrating at best. Our path down this rabbit hole was
    sitemap >
        SYS >
            EtherCAT overview > ECAT_CUST_SYSTEM.adl
                X-End PLC2 (because it showed a text "ALS-X; ISC-EX") > H1_X1PLC2.adl
                    Als (which had no text) > H1ALS_X1_PLC2.adl
                        X (had no text) > H1ALS_X1_PLC2_X.adl
                            Potential BUG: On this screen the "Lock" and "Refl" were showing constant errors but "Laser" ended up being the problem
                            Laser (maybe showed only momentary text) > H1ALS_X1PLC2_X_LASER.adl
                                Head > H1ALS_X1_PLC2_X_LASER_HEAD.adl
                                    After careful scrutiny of this screen we found that the ALS-X laser diode 2's powr monitor
                                        H1:ALS-X_LASER_HEAD_LASERDIODE2POWERMONITOR
                                    was bouncing between 2.038 and 2.039, with is just hovering along the edge of the user defined tolerance of
                                        H1:ALS-X_LASER_HEAD_LASERDIODEPOWERTOLERANCE == 0.2
                                    from the user-defined nominal
                                        H1:ALS-X_LASER_HEAD_LASERDIODEPOWERNOMINAL == 1.842
                                    After increasing the threshold on deviations from the nominal from 0.2 to 0.5, the entire state machine became happy and normal.

This is a problem we'd never seen before, but upon further inspection while writing this log (because we found it hard to believe that laser diodes would produce *more* power than before), I took a look at the 15 day trend of this laser power vs. the X VEA temperature (as measured by the PCAL Receiver's Temperature Sensor), and indeed, the laser power follows it nicely. We should be prepared for the HVAC upgrade to be impacting a lot more than just suspension alignments (LHO aLOG 36331).

Lesson Learned The state machine for the ALS system is really hard to debug when there are momentary errors. 
    - We should change the beckhoff error reporting to be latching
    - We should change these automatically generated screens to *always* display text, so that one can navigate around them with comfort
    - There may actually be a bug in the reporting system

(This concludes WP #6644) 

Kyle

RGA scans were taken this morning followed by the spinning down of the Vertex, YBM and XBM MTPs (main turbo pumps), isolating the scroll fore line and stopping the scroll pumps, "burping" GV5 and GV7's gate annulus volumes into the adjacent ion-pumped annulus volumes and then opening GV5 and GV7.  

Note:  

GV5's gate separated without a fuss but approximately 3/4 of the way into its "open" stroke it came to a halt with a "clunk" confirmed by a stoppage of audible displaced air out of the electro-pneumatic selector manifold .  I assumed that it was at the mechanical stop but CDS was indicating a "yellow" field.  After a minute or so of contemplation, and on its own, it resumed stroking to the mechanical stop.  GV5 has always been worrisome but seems to be getting worse with use.  GV7 behaved weird as well in that the gate seemed to separate as usual and CDS transitioned to "yellow" but the adjacent vacuum pressures indicated no gate separation.  With additional increase in pneumatic air pressure the vacuum pressures finally responded but the displaced air out of the electro-pneumatic selector manifold was barely audible during the stroking.  Even with 55 psi applied, it took noticeably longer than normal for it to open.  This is consistent with the "dribble' of displaced air.  I did note that the red light showing which side of the piston the instrument-air was being applied to was not illuminated.  I don't know if this is a new problem or if the bulb has been burned out for a while????  

Vern S. asked that I post these scans so as to make them available for review to all who are interested prior to opening GV5 and GV7.  So, here they are. 

h0rgacs_SEM_analog_05242017a is a scan of the combined Vertex+YBM+XBM volumes while they are being pumped via IPs1-6.  Note that, for this scan, I am running 1mA filament current and 1300V SEM voltage.  For scan h0rgacs_SEM_05242017b I reverted the settings back the Quadera default of 2mA filament current and 1500V SEM voltage.  I think that the scans that Chandra R. had taken prior to this vent exercise had used the as found Quadera default settings.  As such, if you are comparing "before" to "after" scans, you will want to use h0rgacs_SEM_05242017b.