Below are RPN and Modescan scans for both LO and HI power.

David.M, Jenne.D, Richard.M

We tested some of the electronics for the L4C geophones which are soon to be installed as an array. These were three L4C Concentrator Chassis (D1600144, labelled '1', '2', and '3' on front) and two L4C Interface Chassis (D1600148, labelled '1' and D1600152, labelled '2').

These are all working correctly. The only alteration made was to swap the 9 pin plugs for 'L4C 6' and 'L4C 7' on the board of Interface Chassis D1600148 (currently labelled as Interface Chassis '1' on front), since they were the wrong way around.

I've attached a pdf with the pin layout for each chassis for reference:

Here are the trends from the past 12 days. The past couple of weeks have been agitated by a number of chiller trips and ongoing work.

Jenne, Sheila, Evan, Kiwamu

Today we aimed to have a long stable lock at 30 Watts. We did have one lock of more than half an hour at 30 Watts, which we lost due to commisioner error.

• We measured the coupling of test mass pitch to arm power fluctuations.  More information will be coming later. 
• We noticed that we can loose lock because of the CSOFT instability driven by power fluctuations when we power up too quickly for our soft loops to keep up.  We rewrote the INCREASE POWER state to pause and wait for soft loops at 10 Watts intervals.
• In 3 out of our last 4 locking attempts, we have lost lock in DARM offset.  The one time we made it Jenne did the steps in the guardian by hand.  There don't seem to be any changes in the code for this state....
• We had a few locklosses engaging the ISS 2nd loop.  The thresholds needed to be adjusted, now it is working.
• We did some A2L tuning today.
• We had a couple of sudden locklosses, where our recycling gain was slowly decaying, and it looks like the PR2 (REFL WFS) loop and SRC2 (AS_C to SR2 and SRM) loops ran away in the last fractions of a second.  This seems like the lockloss we had because of the sign flip in the REFL WFS, because we increased the bandwidth of the PR2 loop it is now a fast lockloss.  We measured the open loop gain of PRC2 both at 30 Watts (where the recycling gain is decaying) and at 2 Watts, the optical gain is lower at the 30 Watts measurement which was taken right before one of these locklosses. 
• We decided based on this to try moving soft loop offsets, but we want to picomotor the TMSY QPDs before moving the offsets...  we are leaving this for tomorow...  We turned off the CHARD blends so that we can move picomotors, and are leaving them off for now, but should turn them back on when we are done with picomotors.

Also, the ISS 1st loop is oscillating pretty much every time we loose lock and needs to be fixed by hand. we are leaving IFO undisturbed at 23 Watts 7:50 UTC

 
Activity Log: All Times in UTC (PT)

23:00 (16:00) Start of shift
00:08 (17:08) Dave – Rebooting h1fw0, and nds0 due to h1fw0 instability, (See aLOG #27232)


End of Shift Summary:

Title: 05/16/2016, Evening Shift 23:00 – 07:00 (16:00 – 00:00) All times in UTC (PT)
Support: Jenne, Sheila, Evan
Incoming Operator: N/A 

Shift Detail Summary: IFO has been locked and unlocked at various stages during the evening shift while the commissioning team is working on improvements and updates. 

Sheila, Jenne, Evan

We have suspected for some time that the POPAIR B diode (which we use to extract POPAIR18 and POPAIR90 signals) is saturated in full lock, because these signals do not scale appropriately when increasing the PSL power. Now that we are seriously trying to use POPAIR90 to diagnose DRMI angular behavior during power up, we would instead like to use a signal that we can trust.

We tried a few different things today:

• Take the rf monitor for POP45 (in-vac) and feed it into the POPAIR90 demodulator. This seems to give a signal that scales appropriately with the PSL power (good), but the SNR is worse (bad).
• Take the rf monitor for POPAIR45 and feed it into the POPAIR90 demodulator. Same performance as above.
• Insert a 12 dB coupler directly into the POPAIR45 line. Feed the coupled output into the POPAIR90 demodulator. This gives a signal that scales appropriately with the PSL power and has comparable SNR to POPAIR90.

The last of these is the configuration that we have now. Whitening and dark offsets were set to give appropriate signal levels.

Careful examination will actually show that there appears to be a bit of saturation with this POPAIR45 signal as the power is increased. According to the POPAIR_A_LF channel, there is 83 mW (!) of dc power on this PD at 25 W. Probably we should be closing the POP beam diverter at this power, or else we need to swap some optics on the table.

Ideas for how to improve:

• Put the coupler on POP45 instead, since the POP PD has four times less light than POPAIR A. Requires rechecking the phasing of POP45 for MICH and SRCL in full lock. Has the advantage of giving us POP90 information even when the POP beam diverter is closed.
• Investigate whether POPAIR B can be made to give sensible signals. Perhaps we should modify it, just as we did in order to make REFLAIR B give sensible signals.

After this afernoon's DAQ restart h1fw0 became very unstable. In the past rebooting the Solaris QFS writer has helped. We power cycled h1ldasgw0 at 17:14. We quickly got h1fw0 and h1nds0 processes restarted after the NFS server was available. Too early at the moment to see if the problem is fixed.

Transition Summary:

Title:  05/16/2016, Evening Shift 23:00 – 07:00 (16:00 – 00:00) All times in UTC (PT)
State of H1: IFO locked at ENGAGE_ISS_2ND_LOOP. Commissioners are working IFO. 
Commissioning: Working on improving IFO locking and stability.
Outgoing Operator: Ed
 
Activity Log: All Times in UTC (PT)

23:00 (16:00) Start of shift

T.J, Ed, Jim, Patrick, Dave:

The ALS_XARM node was reporting no connection to the epics channels H1:ALS-X_LOCK_ERROR_FLAG and H1:SYS-MOTION_Y_SHUTTER_A_OPEN. This was preventing any further IFO locking. Logging into h1guardian0 as user controls and issuing caget commands, we were able to get H1:ALS-X_LOCK_ERROR_FLAG's value, but not H1:SYS-MOTION_Y_SHUTTER_A_OPEN (reported data "not complete"). Both caget's complained that two IOCs were serving the data, the h1slow-h1fe epics gateway (spans between the slow controls 10.105 network and the front end 10.101 network) and the IOC itself, h1ecatx1. On the other hand, caget on a control room workstation could get both channels.

Since restarting the end-x Beckhoff slow controls system seemed the most impact, we followed the sequence:

reload ALS_XARM node

restart ALS-XARM node

restart h1slow-h1fe gateway

reboot h1guardian (then we remembered that we had decided that power cycles were preferred)

power-cycle h1guardian0

restart h1ecatx1 epics IOC*

* Patrick reminded us that the Beckhoff IOC could be restarted without the PLCs needing a restart, so this was far less impactful than anticipated. This ultimately fixed the problem.

h1guardian0 was scheduled for a reboot tomorrow to load patches, so this reboot was brought forward one day.

We still don't know why h1ecatx1 IOC was accepting all CAGET commands from the control room network caget's, only partial caget's from h1guardian0, and none from the guardian nodes.

postscript: when we power-cycled h1guardian0 we kept the epics gateway off to ensure CA clients would connected directly to h1ecatx1 and not use the epics-gateway. The nodes did not start up the same way as the previous reboot, with more nodes not connecting the channels. The h1ecatx1 channels were in the list, but so were channels on h1hwsmsr (a 10.105 machine) and h1seib3 (on the same front end network). After about 15 minutes TJ reported these channels reconnected, and the only thing we had done was to restart the epics gateway, or perhaps wait long enough.

J. Kissel, B. Weaver
Work Permit: #5880
FRS Ticket: #5489
Integration Issue: #FRS 5506
ECR: E1500045

As a band-aid solution to the ganged-coil-driver-state-switching-lock-losses-during-acquisition, recently identified to be PRM (see LHO aLOG 27158), we're going to modify the M3 stage of the HSTS's library part to have individual control over coil switching similar to what was done with the M2 stage of the Beam Splitter some time ago (see E1500045).

Because of the breakdown of suspension-type library parts, we must changes to
/opt/rtcds/userapps/release/sus/common/models/
HSTS_MASTER.mdl
MC_MASTER.mdl
and instead of copying the un-library-parted BSFM_MASTER M2 COILOUTF bank, we created a new library part to be copied into the HSTS_ and MC_MASTER_M3 stages, called
/opt/rtcds/userapps/release/sus/common/models/FOUROSEM_COILOUTF_MASTER_INDIVIDUAL_CTRL.mdl
We've confirmed that these updated models compile, and they've been committed to the userapps repo.

We'll work on MEDM screens tomorrow morning, once we've compiled the updated models and we have new channels. There will be some settings (namely the coil driver state settings) that will be lost from the channel name switch, so we'll make sure to replace those in the SDF system accordingly. We'll also make our best attempt EVER at preserving the alignment of these SUS after the restart.

Tega, Ross, Jim, Dave:

We installed new code for the models h1susitmpi, h1susetmxpi and h1susetmypi. The new code required a DAQ restart, which TEAM-PI obtained permission from TEAM-COMMISSIONING for this afternoon.

The purpose for the change was to make the code more efficient and claw back some CPU time on the h1susitmpi model which was running long (15-16uS for a 64k model). This was successful, it is now running in the 9-10uS range. ETMX is unchanged at 7uS, there is hint that ETMY is running one micro-second longer from 3uS to 4uS.

Now restored. Restarting the h1hwsmsr computer did the trick.

Here's more detail on what happened:

I came in this morning noticed a connection error on DIAG_MAIN. Opened up the HWS ITMs code and saw that every channel went white. There was no restart in the morning that could have affected TCS. I powercycled the msr computer and was able to rerun the code.

Dave also mentioned that various computer crashes between 2-4 AM local was normal during ER6. He didn't see this problem during O1. We also looked at the memory and CPU usage. Nothing is overloaded

we had some system problems over the weekend, here is a summary of the time-line. (These would appear to be unrelated events)

• Sat 14th 18:15 FMCS H2O-RO alarm
• Sat 14th 22:49 PSL turned off
• Sun 15th 03:05 h1hwsmsr IOC failed
• Sun 15th 04:52 h1ecatx1 IOC reported buffer error

The last issue caused Guardian problems with the ALS_XARM node. We did the following to try to fix it:

• stop-start the node
• reboot h1guardian0
• restart the h1slow-h1fe epics gateway
• power cycle h1guardian0 with epics gateway off

The power up of h1guardian sans epics-gateway gave even more CA connection errors, with HWS IOC and h1seib3 FEC. These were very confusing, and seemed to go away when the h1slow-h1fe epics gateway was restarted which added to the confusion. We need to reproduce this error.

After Patrick restarted the h1ecatx1 IOC the guardian errors went away.

Quick conclusion:

Bad news: The further away the rotation stage has to travel, the less accurate the measured requested power becomes.

Good news: The uncertainty is within 1-3%.

Slightly more details:

• The first set of measurement I ran the random walk script to let the RS rotate between -25.4 and -19 degree (roughly 0 W and 3 W) 100 times. The result yields ~1% deviation at 3W.
• The second set of measurement I ran the same code to let the RS rotate between -25.4, -19, -7, and 20 degree (which correspond roughly to 0 W, 3 W, 20 W, and 55 W) 66 times (people needs the IFO back and this RS measurement is taking forever). The result yields 2-3% deviation of the mean. 
• Both measurement overlapped at 0W.
• ISS and FSS was locked during the time of measurement, IMC was left unlocked so it doesn't kick FSS. ISS integrator was off according to Jason. ISC_LOCK and LASER_PWR guardian nodes were paused.

It looks like we need to do something about the triple coil drivers that we switch in lock, especially PRM M3.  We have lost lock a good fraction of the times that we have tried to switch in the last month or so.  Screenshot is attached, I also filed an FRS ticket hoping that someone might make an attempt to tune the delays while people are in the PSL tomorrow morning.  FRS ticket #5489