I've spent some time doing tilt-decoupling measurements on the BSCs in the last couple days, trying to figure out how to properly do the gain matching for the ST1 Z drive from RZ T240 subtraction.For these measurements: I put both stages in high blends, with both stages fully isolated, turned off all sensor corrections, used a 4th order butterworth bandpassed white noise excitation in awggui to drive at the error point of the isolation loops. In this configuration, I would expect the St2 motion to be the same as the St1 motion up to about 1hz

While doing this I looked at the Stage 2 motion,and there is some funny business there. First attached plot shows the transfer function from St1 Z T240 to the St1 X/Y T240 & St2 X/Y GS13s. At low frequency, a Z to RX/RY coupling would show up in X/Y as a g/w^2 slope. This might be what is happening on ITMY in the first image, below 100mhz. Above 100 mhz, something else is going on, but at least the St2 motion generally agrees with the St1 motion.

In the second image, ITMX below 100mhz looks more like 1/f slope, and the above 100mhz stuff is even worse. For some reason St2 seems to be moving a lot more that St1. 

For both plots, Z-X tfs are red, Z-Y tfs are blue, solid lines are St1 drive to St2 sensors, dashed lines are St1 drive to St1 sensors. 

It's possible that the Z/RX-RY tilt decoupling is bad, so I'll try to remeasure tomorrow. I think this might also be me rediscovering a problem RichM found a while ago in the SEI log and that I didn't understand when we were actively talking about it. Probably also related to an issue that Camillo and Arnaud were looking at for Camillo's BSC model.

I'm assuming the peak at .55 hz in the St2 dofs is some interaction with the quad. 

HAM & BSC spectra look as expected.  The one which has high high-freq noise is ETMx (this suspension is currently being worked on).

Added 50mL to the Crystal Chiller
Diode Chiller was fine (no water added).
Filters were clean & free of debris.

After changing the fluid filter 2+ weeks ago, the fluid has been recirculating just at the Pump Station with valve caused back pressure at 80psi. This is about the nominal pressure during normal running and is done to check for leaks at the filters just changed.

With approval from vent captain, the valve configuration has been changed to return flow to the VEA.  The flow is still bypassing the HEPI Actuators and their flex lines from the 4-way valves.  The Pump Stations are running under local servo control at 20psi.  Happy to run in this way for a week or two and then will be ready to switch to nominal configuration when allowed.

09-05-18 15:17 Christine in the LVEA
09-05-18 15:17 Tyler and Mark in the LVEA and heading to MY
09-05-18 15:18 Ken in the LVEA
09-05-18 15:19 Hugh running measurements on ITM
09-05-18 15:35 Carlos oplog Chris to the LVEA
09-05-18 15:35 Kyle in the LVEA
09-05-18 15:42 contractor to Maintenance shop
09-05-18 16:57 Mark and Tyler craning IPs near IMC beam tube
09-05-18 17:02 MarkP to the Vault
09-05-18 17:02 Richard to the LVEA
09-05-18 17:08 JeffB done in LVEA heading to both mids and ends to cap off the old house vacuum lines
09-05-18 17:21 Jim using both ITMX and ITMY SEI
09-05-18 17:22 TJ using SRM and PRM for guardian testing
09-05-18 17:28 MarkP and Fil to the Vault
09-05-18 17:34 Betsy and Travis to EX
09-05-18 18:01 Richard to CER to test a fiber
09-05-18 18:53 JeffB back from out buildings
09-05-18 18:53 beamtube IP craning is done
09-05-18 18:54 Mark and Tyler and Chris to LVEA to crane the Genie Lift
09-05-18 18:54 Travis and Betsy back from EX
09-05-18 19:14 Mark Tyler and Chris done craning in the LVEA
09-05-18 19:14 Mark and Tyler moving the Snorkel lift from MX to MY
09-05-18 19:38 MarkP and Fil back from the Vault

Conclusion--Driving the HEPIs +- 500micro units suggests no interferences

Details--Jim unlocked the HEPIs for the ITMs yesterday and as we want to be sure of no interferences, the platforms were strokes in X Y RX & RY (X Y & Z should suffice) by 500um/urad.

The ISIs were put into DAMPED as tilting of the platform will trip on the T240s.  With the HEPI ISOLATED, offsets were ramped into the isolation filters.  With all the cartesian loops closed with large DC gain, interferences impeding motion will be evident in the local sensors as the loops are satisfied.  Interferences will be evident with clipping and coordinated slope changes--see 40171.

The four attached plots show the ITMX first with horizontal drives and then vertical drives; ITMY are the third and 4th plots.  The lower graph has the 4 local sensors and the upper shows the cartesian position.  All traces indicate things are fine with no obvious clipping or slope changes.  The horizontal drives on the ITMY show the peaks shifting; I'm not sure what this means if anything...will keep thinking.  The positions end up at zero again.  Maybe this does indicate an interference as the X drive nears the end of the drive offset but not obvious enough to clearly clip.  Maybe this is worth a closer look driving further and trying open loop.  Will think and look some more and will report as required.

Yesterday I received a call from American Rock Products regarding a blast/explosion they were going to conduct (I'm assuming we have a connection with the company and they previously let us know when they will be making some noise.).  They said the blast was to occur between noon & 2pm PDT (7-9 UTC).  They said this was at the Kiona Pit.  I called them to get a little more information.  They said it had been a few years, but in the past they would give us a call whenever they were planning a blast.  They said this was at a pit mine on Kennedy Road.  Googling around, I'm fairly certain this is the American Rock Products quarry on Kennedy Rd between Candy & Red Mountains.  This blast is ~13-miles from the LVEA & ~12-miles from EY.

(Not sure why I am their contact number, but here is their contact info:  Kim Terlson of American Rock Products (509)547-2380)

• VAC: PT146 is high
• Tyler and Mark are in the LVEA, going to MY
• Ken is in the LVEA

Cheryl and Ed aligned the ALS beam path using IO_MB_M1 and ALS_M1 (see the first attachment for the mirror names from D0902114). After that, the main beam was rotated such that the beam will not even clear IO_MB_WG1. Second attachment shows this, the beam was hitting the edge of an iris placed in front of IO_MB_WG1 (iris is not on the PSL layout diagram). This is a bit more than an inch of a shift mostly in YAW, and the distance between the iris and IO_MB_M2 is about 32 inches, so this is like ~1/32 rad ~2 degrees (huge).

This seems to qualitatively agree with that the beam position moved towards +Y direction on IO_MB_M1 at some point after new PMC installation (3rd attachment) given that the steering mirror positions cannot be changed due to the PSL mount design (though the detailed numbers depend on how much the shift on IO_MB_M1 was, what was done to EOM path after the beam shift and how good the ALS path alignment is). With the beam position change on IO_MB_M1, we cannot satisfy both the main path and the ALS path at the same time using only IO_MB_M1 and ALS_M1.

There was some suspicion that the EOM motion/rotation could have caused the beam rotation downstream. There could only be a miniscule change due to that compared with O(1deg) we're talking about, and thus I don't see this as an urgent issue. See attached comment.

As such we need to move on to align the main beam path downstream of IO_MB_2 using two irises that Cheryl placed in that path as a fiducial. We first turn IO_MB_1. If that is not enough we iterate using IO_MB_2 and IO_MB_1. We might need to displace EOM using 5-axis mount so the beam cleanly goes through EOM (we need to measure in/out power like Koji did). Then we go back to ALS path and move ALS-M1 and ALS_M2.

[TJ, Jamie]

minor guardian upgrade

A minor version upgrade of guardian (1.2.1) was installed on the primary guardian host (h1guardian1) today.  Changes in this version:

• disallow INIT as initial request state
• fix date search in guardctrl log
• reduce log spew during connection errors
• clarify cas server processing in main daemon loop

The guardian host, h1guardian1, was rebooted (see below) so all nodes are now running this new version.

The new version is available for the workstations as well, but has not been deployed yet.

code archiving re-enabled

Code archiving has been re-enabled for all guardian nodes.  This is the feature that commits all system user code into a git repo (in /ligo/cds/lho/h1/guardian/archive) upon node start or LOAD.  The ownership for the archives were changed to be owned by the 'guardian' user (uid 1010).  This will protect the archive from accidental overwrite by users in the controls group.

The way to access the archives is via 'guardutil', i.e.:

profiling guardian machine reboots

TJ and I performed a couple of full reboots of h1guardian1 to see how things come up after reboot.

Before re-enabling the code archives, the reboot process was relatively smooth and nodes came up fairly quickly and with very few issues.  The oddest issue, was that SUS_ETMY showed the following odd exception, that did NOT go away with a LOAD, but did go away after a node restart:

2018-05-08_20:44:44.859908Z SUS_ETMY [INIT.main] next state: ALIGNED
2018-05-08_20:44:45.027281Z Traceback (most recent call last):
2018-05-08_20:44:45.027281Z   File "_ctypes/callbacks.c", line 315, in 'calling callback function'
2018-05-08_20:44:45.031773Z   File "/usr/lib/python2.7/dist-packages/epics/ca.py", line 583, in _onConnectionEvent
2018-05-08_20:44:45.050185Z     if int(ichid) == int(args.chid):
2018-05-08_20:44:45.050185Z TypeError: int() argument must be a string or a number, not 'NoneType'
2018-05-08_20:44:47.094551Z SUS_ETMY W: Traceback (most recent call last):
2018-05-08_20:44:47.094551Z   File "/usr/lib/python2.7/dist-packages/guardian/worker.py", line 464, in run
2018-05-08_20:44:47.094551Z     retval = statefunc()
2018-05-08_20:44:47.094551Z   File "/usr/lib/python2.7/dist-packages/guardian/state.py", line 246, in __call__
2018-05-08_20:44:47.094551Z     main_return = self.func.__call__(state_obj, *args, **kwargs)
2018-05-08_20:44:47.094551Z   File "/opt/rtcds/userapps/release/sus/h1/guardian/SUS.py", line 355, in run
2018-05-08_20:44:47.094551Z     if is_aligning():
2018-05-08_20:44:47.094551Z   File "/opt/rtcds/userapps/release/sus/h1/guardian/SUS.py", line 132, in is_aligning
2018-05-08_20:44:47.094551Z     if ezca.is_offset_ramping(chan):
2018-05-08_20:44:47.094551Z   File "/usr/lib/python2.7/dist-packages/ezca/ezca.py", line 519, in is_offset_ramping
2018-05-08_20:44:47.094551Z     return LIGOFilter(sfm_name, self).is_offset_ramping()
2018-05-08_20:44:47.094551Z   File "/usr/lib/python2.7/dist-packages/ezca/ligofilter.py", line 556, in is_offset_ramping
2018-05-08_20:44:47.094551Z     return self._offset_ramp.is_ramping()
2018-05-08_20:44:47.094551Z   File "/usr/lib/python2.7/dist-packages/ezca/ramp.py", line 62, in is_ramping
2018-05-08_20:44:47.094551Z     return self.__current_ramp_status()
2018-05-08_20:44:47.094551Z   File "/usr/lib/python2.7/dist-packages/ezca/ramp.py", line 31, in __current_ramp_status
2018-05-08_20:44:47.094551Z     return self.__is_ramping(self.rampmon_pv.get())
2018-05-08_20:44:47.094551Z   File "/usr/lib/python2.7/dist-packages/ezca/ligofilter.py", line 217, in is_ramping
2018-05-08_20:44:47.094551Z     return bool(int(value) & const.READONLY_MASKS[_RAMP_TYPE].bit_mask)
2018-05-08_20:44:47.094551Z TypeError: int() argument must be a string or a number, not 'NoneType'
2018-05-08_20:44:47.094551Z SUS_ETMY [INIT.run] USERMSG 0: USER CODE ERROR (see log)

After re-enabling the code archives, we ran in to issues with startup timeouts.  With the code archives enabled, each node hits the code archive (which is mounted via NFS at /ligo) at launch.  When h1guardian1 is rebooted, that's 120 nodes all hitting the NFS mount simultaneously.  This slowed down access considerably, and caused the startup process for many nodes to hit the systemd startup timeouts that are now in place.  More than half the nodes were terminated during startup because they failed to check in in a timely manner while accessing their archive.

The solution we put in place was to just extend the startup timeout to 3 minutes, from 3 seconds.  On successive reboot all nodes eventually came up and none were terminated because of startup timeout.  Here's the systemd drop-in that configured the timeout:

There were only two nodes that had problems on the last reboot (after the timeout increase was put in place), and they both failed because of what appeared to be an EPICS port bind timeout:

2018-05-08_22:59:48.654796Z Starting Advanced LIGO Guardian service: SQZ_CLF...
2018-05-08_23:00:28.914421Z SQZ_CLF Guardian v1.2.1
2018-05-08_23:00:28.914421Z CAS: listen() error Address already in use
2018-05-08_23:00:28.914421Z terminate called after throwing an instance of 'int'
2018-05-08_23:00:30.378973Z guardian@SQZ_CLF.service: Main process exited, code=dumped, status=6/ABRT
2018-05-08_23:00:30.379381Z Failed to start Advanced LIGO Guardian service: SQZ_CLF.
2018-05-08_23:00:30.379414Z guardian@SQZ_CLF.service: Unit entered failed state.
2018-05-08_23:00:30.379421Z guardian@SQZ_CLF.service: Failed with result 'core-dump'.

This is not a failure we've noticed before, so unclear why it came about now.  The problem was cleared after a node restart.

It might be nice to add a randomized node startup delay at boot, so that all the nodes aren't all fighting for resource all at the exact same time.  It's easy enough to add a randomized delay at startup, by using e.g. StartExecPre= to execute a script that just sleeps for a random amount of time.  But It's not clear how to do that at boot only, so that it doesn't affect normal node restarts.

I haven't logged all of the variac reductions since the start of the rampdown but a good summary would be to say that about -2% in the morning and -2% in the evening for the weekdays.  Also, -2% each of the weekend days.  I expect to connect the electronics tomorrow afternoon and energize the filament at that time.  I hope to capture some data even while everything is above ambient as insurance in case anything goes "south" before the official scans early next week. 

Will resume in the morning

After getting the new ETMX QUAD suspension back into chamber and attached to it's upper structure on the ISI yesterday, I recabled the reaction chain.  Today, Fil HIPOT tested the ESD so that we were not finding any issues late in the closeout.  Sure enough, 2 of the 5 connections failed the HIPOT testing (although continuity passed).

We attempted to reseat the cable, pins, and sockets multiple times, but nothing helped.  We also inspected for some rogue metal bits which may be shorting the sockets/pins but didn't find anything.  Attached are pix of the male end which passed HIPOT when tested alone (not connected to the AERM traces), and a picture of the AERM sockets which seem to not show anything glaring.

I could move the PIN 3 (BIAS) socket in and out with tweezers about ~a mm, which probably isn't good.

Fil is checking on parts stock to see if we have enough to re-terminate the whole connector.  Getting in touch with SYS.

What to do, what to do...

(Related: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=41418)

Relevant DCC: D1600298.

According to the circuit diagram (first attachment from V3 of the above DCC entry), the control voltage ("Switch") and the readback voltage ("Switch Readback") are always opposite, i.e. when control=high readback=low and vice versa. This was not the case for BOOST stages as measured at the rack. High/low is inverted in readback and "high" is only 3V. When I disconnected the readback cable from the chassis, 3V changed to 6V.

I opened a spare chassis and found that the wiring for boost logic is indeed different. Second attachment shows some LEGO-type mid-air jumper job (yellow circles) as well as pins 2 and 3 short-circuited (red) for the Max4659 connected to the readback of BOOST binary inputs. Third attachment shows the same thing in a human-friendly form, but I don't know if pin 8 was completely cutt off from GND (or pin 2 from +15V).

Is this intentional? At least this explains the readback logic.

I still don't know exactly why High voltage reads 3V (readback cable connected) or 6V (disconnected) at the chassis. Voltage pulling sounds a bit too much to me.