J. Kissel

I've completed the HAM-HEPI and HAM-ISI model upgrades as described by the step-by-step instructions yesterday (and today) in 18078. It should be noted, that with the amount of top-level model modifications and rearranging, one should definitely reserve two days for this update -- one for making the changes for the models which can be done "offline," and one for the model restarts and debugging.

We still have some minor MEDM screen debugging to do, but we should be able to get this done by the end of the day, and commit when ready. 

However, for LHO, this closes out the following ECRs and integration issues:
- Change the count for the saturation counters so that all of the HAMs have the values used now in HAM6. E1500186 :: II 1018
- Put a 10 sec delay on the connection monitor for the SUS Payload WD alerts. E1500196 :: II 1028
- Install parallel feed-forward paths from the Stg 0 L-4Cs and the HEPI L-4Cs; Disconnect the L4Cs from the HAM Sensor correction. E1500179 :: II 1026
- Clean up STS-2 wiring/ update the STS -> HAM inputs so that we don't overwrite last two ADC channels (which are reserved on ADC0). E1400386 :: II 942
- Put the QPD part into the HAM model for the HAM Optical Levers. E1400452 :: II 971
- Add monitor channels to the frames for the STS and displacement sensor signals in the Sensor Correction Path (done for HAM-ISIs and HAM-HEPIs only with this update). E1500216 :: II 1046
- Fix L4C routing bug in HPI template (done for HAM-HEPIs only with this update) (Bug fix -- no ECR) :: II 1045


There were a few hitches including 
- discovering how the IOP bit logic works, and that it needed inversion before going into the library part
- releasing that I needed updates to the isi/common/src directory as well as the model and medm directories
- remembering that the BSC HEPIs would be affected by the hepitemplate update
- a nasty hiccup in that somehow the version of the h1isiham6.mdl I was editing was in my home directory (where I'd never [intentionally] saved it) instead of in the userapps repo, so I was compiling something I hadn't yet edited
- a fix to the HAM3 and HAM5 HEPI models because of the differing cardNum vs block name identification of the ADCs
but I persevered in the end. 

All affected top-level models have been committed to the userapps repo for demonstrative purposes when LLO performs their upgrade.

Jim W had finished his TFs, so we went ahead with a DAQ restart. The restart failed due to duplicated slow channels in the corner station Beckhoff EDCU files. Both H1EDCU_ECATC1PLC1.ini and H1EDCU_ECATC1PLC12.ini had the following two channels defined:

• H1:PSL-ERROR_CODE
• H1:PSL-ERROR_FLAG

for now I have removed them from both INI files.

h1dc0 actually starting running, with an EDCU channel count of zero. This is unusual, it would normally not start. However the other DAQ systems did not restart.

I restarted h1dc0 via port 8087 as usual, it ran correctly but the other DAQ components did not restart. I used monit to restart the frame writers, NDS and broadcaster. All systems then came online.

The EDCU is now green (I had to manually removed TCS_ITMY from the guardian INI file)

J. Kissel

In prep for the HAM-ISI and HAM-HEPI Model upgrade (see SEI aLOG 708, WP #5171, E1500207, and a slew of ECRs and Integration Issues), I've put together a step-by-step instructions of all of the everything that I need to do today and tomorrow. 
Most of these upgrades were tested at Stanford and MIT, so I don't expect too many problems with the infrastructure, but they can only test/describe so much:
- they only have single chambers, not many chambers each with different top-leve needs
- those chambers don't have SUS on them that need to stay aligned to anything, 
- they don't have IOP watchdog systems, 
- they don't have settings that the really care about in any of their SDF systems, 
- they don't have multiple STSs to store with distibution chassis, and
- they don't have optical levers on their HAM-ISIs 
So this will be the first full-scale test and I wanted to get my ducks in a row before I touched anything.

See attached.

Biweekly crash recovery

Logged in. Closed system manager. Said no to save current changes. Restarted computer. Errors in EPICS IOC. Closed EPICS IOC.
Restarted computer again to get screenshot of errors in EPICS IOC (see attached).

Closed EPICS IOC.

Opened LIGO TwinCAT Target Configuration GUI
Selected H1ECATX1 -> SYS, PLC1, PLC2, PLC3.
Clicked Stop. Clicked Update from source. Clicked Compile. Failed (see attached)
Ran svn update on C:SlowControls. No changes at end X. (see attached)
Ran svn st on C:SlowControls. No local modifications.

Opened LIGO TwinCAT Target Configuration GUI
Selected H1ECATX1 -> SYS, PLC1, PLC2, PLC3.
Clicked Stop. Clicked Update from source. Clicked compile. This time it completed successfully. Clicked Activate and run.
Got message: No boot project created since last download. Exit anyway? Clicked OK. Said it completed successfully.
Clicked Restart EPICS database.
PLC1 and PLC3 came back. PLC2 did not (see attached) Why is the location Corner?

Opened PLC2 from target. Logged in and selected run. PLC2 came back.

Went to /ligo/cds/lho/h1/burt/2015/04/28/00:10
Ran:
burtwb -f h1ecatx1plc1epics.snap
burtwb -f h1ecatx1plc2epics.snap
burtwb -f h1ecatx1plc3epics.snap

Work Permit 5175
Per ECR-E1400386 V5

In conjunction with the Model changes made by Jeff K.  we have removed the STS-2 signal cables from the STS-2 Fan out chassis for each BSC  to the Ham 3 and 5 AA chassis.  Since Hams share the front ends we did not need to duplicate the signal path.  Six cables were removed.  

J. Kissel

In order to get the latest and greatest hardware injection infrastructure from LLO (for the ODC part, see LLO aLOG 17911), I've 
- svn updated the hardware injections library part,
/opt/rtcds/userapps/release/cal/common/models/CAL_INJ_MASTER.mdl
- added an SHMEM IPC sender called H1:CAL-INJ_ODC_IPC to the top level of
/opt/rtcds/userapps/release/cal/h1/models/h1calcs.mdl
- svn updated the library part
/opt/rtcds/userapps/release/sys/common/models/ODC_MASTER_PARTS_V2.mdl
- added an SHMEM IPC receiver called H1:CAL-INJ_ODC_IPC to the top level of
/opt/rtcds/userapps/release/sys/h1/models/h1odcmaster.mdl
and connected it to the last input
- compiled, installed, and restarted both models
- committed the top-level models to their respective location in the svn.

This closes WP 5172.

Did not in the end move it except vertically a few mm as I cut out holes in the vinyl floor under the feet.  It is back under the igloo and stabilizing.

Jeff, Jim, Dave [WP5174]

h1alsey and h1alsex were rebuilt against RCG branch-2.9 (all other models are built against RCG tag-2.9.1). The new rfm_delay flag is turned ON for these two models. This should eliminate the RFM IPC receive errors seen by the h1asc model. I verified that all RFM IPC channels sent by the ALS models are only received by the ASC model.

I have disabled the cron job which was performing a DIAG_RESET on the h1asc model every minute. If we get any IPC errors on the ASC they will latch ON.

I have updated the model_info.txt file (lists all running models and their DCUID number). I have regenerated the H1CDS_FE_RCG_VERSIONS_CUST.adl MEDM screen (attached). All models are running 3994 except ALS which are running 3999. Later builds show a modified RCG (3994M and 3999M) reflecting a change in the standard environment scripts to add ALS and CAL to the RCG model search path.

J. Oberling, E. Merilh

We replaced the laser for the ETMy optical lever laser that had overheated last week (see alog 17986).  Realignment was not necessary as the optical lever had maintained alignment.  New laser SN is 193.

Work Permit 5173

Today we swapped both EX and EY ISC AA AI chassis with V6 AA/AI chassis.  There were 2 AA and 1 AI at each end station.  We also verified the PEM and PCal were already at V6.
Tomorrow we will continue with SUS and SEI.

WP #5164, 5170

Installed version branch/gds-2.16.17.2 GDS tools for Ubuntu, which includes a bug fix for printing problems in diaggui and foton.  Adds an option to foton for use with non-aLIGO filter development.  Adds enhancements to diaggui to search for awgtpman servers on multiple networks, and to restrict the search for awgtpmans to specific models (not for general use).

Installed minor updates to dataviewer and command line nds tools to recognize NDS protocol version 12.2, which was installed with the RCG 2.9.1 update last week.  Should be no functional change to those tools.

H. Radkins, J. Kissel,

Sadly we had to break the very-long-standing RF lock. I've brought all IFO / ISC guardians to DOWN, and I've reloaded the IMC guardian in hope to receive T.J.'s new "OFFLINE" state for the IMC, but there's some bug there. I've left it in INIT and misaligned MC2. Hugh is beginning to bring the platforms to OFFLINE.

Evan, Dan, Sheila. Kiwamu

This afternoon we moved the picomotors on both transmon IR QPD sleds, although we moved Y much further than X.  This was to reduce the clipping and saturation Keita pointed out in 18077.  After that we excited TMSY in both pitch and yaw, and found error signals for the ITMs that are insensitive to TMS again. 

We have closed all the loops in full lock and things seem stable,  we were able to increase the power at 12:00 UTC on the 28th and saw that the recycling gain stayed stable.  We stopped at 6 Watts because the Y QPD is near saturation.  We should probably do more picoing (most of the light is on one quadrant), and reduce the whitening gain. 

The attached screenshot shows the settings we have now, the DSOFT and CSOFT loops could have gain of 1, but we were cautious. 

We had some difficulty turning on the ASC loops earlier tonight.  One thing that we noticed was that we were able to close INP1 each time, even when we didn't close PRC2 or CHARD (or SRC).  We also had trouble closing the CHARD PIT loop, although YAW was fine. Once we manualy adjusted the alignment to improve the recycling gain and closed the ITM loops, we closed CHARD PIT with no problems.  

We have left the ASC engage commented out in the guardian, because it is not working right now.  We also attempted earlier in the evening to take Keita's advice and manually aling the green PZT durring a full lock with good recycling gain.  We had difficulty getting the build ups high while minimizing the WFS signals, we adjusted the QPD offsets to keep the build ups high but this resulted in a bad recycling gain.  Probably a more thorough attempt is waranted. 

ETMX ESD tripping

Since saturday, the ETMX ESD has been tripping almost every lockloss.  We also need to toggle BO 4 to reset it, which is not required to reset ETMY.  

A follow up from the alog17973

A HWS operates at 1Hz at the corner station and 57Hz at the End stations. The first plot attached is a quiet DARM and PEM magnetometer reference during a full interferometer lock when none of the HWS cameras were on. The second plot shows spectra from the same channels when only the ITM HWS were on. It is difficult to determine the 1-Hz line due to high DARM noise floor while there is no clear evidence in the corner station magnetometer channel. The third plot shows the DARM spectrum and the PEM floor magnetometer when both ITM abd ETM HWS were on. A tiny peak just above the noisefloor appears in the EX magnetometer together with a peak in DARM. The choice of the chosen magnetometer axis was arbitary. The last three plots are the trend of HWS camera switches.

(This will not be reloaded into the working untill the morning)

As per request, I added a new 'OFFLINE' state to the IMC_LOCK Guardian.

This state will disengage the Common Mode Servo Board inputs (H1:IMC-REFL_SERVO_IN1EN and H1:IMC-REFL_SERVO_IN2EN) and misalign MC2.

Upon exiting this state, it will reengage the inputs and realign MC2 before heading through the DOWN state to aquire lock again, if desired.

J. Oberling, R. Savage, E. Merilh

Summary

After looking at the diode box chiller interlock signal H1:PSL-IL_DCHILFLOW and the actual flow sensor H1:PSL-OSC_DCHILFLOW we are starting to wonder if there might be a problem with the flow sensor.  It seems the interlock trips don't correlate to a drop in flow, but the other way around: an interlock trip triggers the drop in flow, which makes sense if the interlock is programmed to shut off the chiller in the case of an interlock trip, except this is not always the case and there doesn't seem to be a change in the flow rate before the interlock trips.  Details follow, investigation continues.

Details

In an effort to determine why the PSL tripped so many times over the weekend (see LHO alog 18063 and associated comments) we looked at the diode box chiller interlock signal H1:PSL-IL_DCHILFLOW and tried to correlate that to actual drops in the water flow of this chiller using the signal H1:PSL-OSC_DCHILFLOW.  Some examples from the weekend trips are attached below (I unfortunately forgot to save an overall picture of all 5 PSL trip events (6 diode box chiller flow interlock trips); will go back to the control room and attach as a comment).

PSL diode interlock trip event #1 - The interlock trips and after a few seconds either resets itself or someone was really on the ball.  The flow rate does not drop until several seconds after the initial trip (and after the interlock reset), and there is no change in the flow rate before the interlock trips.  Flow rate does not drop to zero but restores itself slowly.

PSL diode interlock trip event #2 -  2 interlock trips in quick succession, lasting just a few seconds each, once again with no change in the flow rate before the first interlock trip.  The flow rate of the chiller did not change until several seconds after the first initial trip; the second trip could be a result of the lower flow rate at the time of the trip.  Flow rate does not go to zero but restores itself (chiller shut off then back on?).

PSL diode interlock trip event #3 - One long trip for >8 minutes.  Hard to see on this plot but the interlock tripped a full 2 seconds before the flow actually stopped.  No change in flow rate before the interlock tripped.  Flow rate does go to zero this time and is restored shortly before the interlock resets.

PSL diode interlock trip event #6 - 3 interlock trips in short succession, each lasting just a few seconds.  Unlike the others, there is no associated drop in the chiller flow before or after the trip; the flow is being reported as within an acceptable range for operation with no obvious reason for the chiller flow interlock to trip.  Problem with the flow sensor?

Investigation ongoing, more to come.

Summary:

Y ARM IR QPDB on TMS, which is used for CHARD and CSOFT and also for estimating recycling gain, is severly clipped. This makes both the ASC itself as well as the power recycling gain estimate fishy (LSC-TR_? comes from QPDB).

Fix it (or try fixing it) before using such a fishy signal for ASC.

Details:

Attached is the trend of one lock where Evan increased the power from 3W to 16W. The alignment of the cavity was changing during this due to power up operation.

If you see the IR QPD SUM on TMS (middle panel),  X_A, X_B and Y_A are all sort of proportional to each other, but Y_B is not, so it's apparent that it's clipping.

If you make B/A ratio for X arm it's dead flat, but that's definitely not the case for Y arm (top panel). The ratio for Y should be about the same as that of X, but Y ratio is always smaller and it gets worse as the power goes up. 

Also, at around t=370 sec, there's a kink in YB that is not related to the power kink. Though the clipping was definitely going on at t=300 and probably earlier, it seems like the clipping got worse at t=370.