Large ~11.3 Hz Displacement in HAM2/HAM3 SUS

J. Kissel, C. Vorvick

While Cheryl and I were aligning the IMC this afternoon, we noticed that we were getting much faster fringing than we saw yesterday. However, comparing spectra of all suspensions in HAM2 / HAM3, in L, T, and V from yesterday evening after work (2017-11-07 02:20 UTC) vs this evening after work (2017-11-08 00:20 UTC) show that there is a large, high Q feature at ~11.3 Hz. Comparing the spectra from over the weekend, in which HAM2 was unlocked, it becomes obvious that the problematic feature dominating the RMS is the HEPI cross-beam resonance, transmitted directly through the locked platform.

Our best supposition is that yesterday, the lower frequency motion (0.5 Hz < f < 5 Hz) was much *larger* than it currently is, dominating the RMS velocity of the IMC, resulting in slower fringing.

So, we'll think more about how we can slow the suspensions down enough to get alignment done tomorrow.

Change to the SUS guardians (proposed)

S Dwyer, S Cooper, J Kissel, TJ Shaffer

We've been thinking about how to better protect our suspensions and optics during earthquakes, and one thing that would help to keep them from banging around would be to keep top mass damping on whenever we can.  Brian Lantz is working on changes to the ISI watchdogs (see 39241), and Jeff K has proposed some changes to the sus watchdogs for this and other reasons (38948). What I am suggesting here is a relatively simple change to the suspension guardian to allow the suspensions to stay damped if a lower stage trips but not the top masses.

Currently, the suspension guardian reacts to any watchdog trip on the suspension by turning off all damping, so if any of the lower stages trips the top mass damping will be turned off.  Also, if M0 trips the reaction chain will become undamped, and the other way around. I am proposing a simple change to the guardian, shown by the graph image that is attached. 

Currently, if any watchdog trips the suspension guardian is redirected to the TRIPPED state, where a function called reset_safe turns off all the outputs to the suspension, including alignment offsets and damping, then a different function turned off the master switch.  This was probably done out of convenience, so that when the watchdog is reset by an operator, the guardian can follow the same set of steps to bring it back to its nominal state no matter which watchdog had tripped.  It is unnecessary in most cases to turn everything off like this: we could have the guardian respond to each watchdog trip differently, and turn on the controls again differently in each case. 

What I would like to do instead is have the tripped state do nothing (thanks TJ for catching the masterswitch), and rely on the watchdogs to do their job by turning off any stages which need to be turned off.  In the old graph, the tripped state was a protected state (redirect=False, red outline) which means it will not move on to another state until it returns true (which it will not do unless the watchdogs are reset).  I want the operator to be able to request that the suspension turn everything off (or the manager), which they would be able to do in the new graph by requesting reset or safe.  To make sure that the guardian will not move through its entire graph with a watchdog tripped, I've made RESET  a protected state which will not exit until the watchdog is untripped. 

With the graph, if an earthquake hits and trips some suspension watchdogs, the top mass damping will be left on unless the top mass trips.  When an operator resets the watchdogs, the suspension will turn off all outputs, and then re-damp, realign, ect.  This means that we will be better off if an EQ hits when no one is here.  The operators can do nothing until the EQ passes if only lower stages trip, if the top masses trip the operators should try to un-trip them as soon as possible.  (Jeff's changes to the SUS watchdogs will hopefully make it easier to untrip the suspensions.)

Ops Day Shift Summary

WP 7200 updating the CDS EPICS CA Gateways

Carlos and Jonathan.

We did some updates to the EPICS CA Gateway cdsegw0 under WP 7200.  The goal was to replace it with a newer machine running a current operating system.  The new system would be running Debian 9, EPICS 3.15.3, the EPICS CA Gateway, and move the config into puppet control.

The current system ran 6 EPICS gateway processes, with 4 of the processes sharing the same server IP/port.

The new system had problems running in the same configuration.  We could read the vacuum network from all of the interfaces, but could not read the FE/slow controls/and aux gateways. Further investigations are required to state exactly why.  At this time we are not sure if it was a configuration setting that did not get transferred over, or just a difference in EPICS base versions as we moved from 3.14.x to 3.15.3.

After discussion with Dave we questioned the need for so many gateway processes.  In previous revisions of the system, the gateways were needed to rate limit access to VxWorks boxes.  These boxes have all been replaced, and the gateways are really are only needed to provide a read-only view of the vacuum channels.  The plan was adapted to run the gateway server with only the vacuum network gateway processes, turning off the server -> FE, server -> slow controls, server -> aux gateways.  This was done this morning, using the old system with a smaller number of EPICS CA gateway processes running.

If this works out we will likely open a new work permit and replace the old gateway box with the new box, but without attempting to multiplex 4 gateways onto one interface (which is the problem we ran into).

cell phone alarm texter reconfigured after epics gateways were removed

Jonathan, Dave:

Following this morning's removal of three epics gateways on the internal gateway machine (the three gateways with SIP=10.20.0.5 and clients on h1fe, h1aux and h1slow) my cell phone texter crashed and would not reconnect to any subnet other than h0ve. We found that the CA env vars were not set and the code was dependent upon a gateway on the cds vlan (10.20.0/24). Defining the env vars in the startup script fixed this issue.

IMC ISC Control of Suspensions Completely Disabled

D. Barker, S. Dwyer, J. Kissel, E. Merilh, C. Vorvick

While resuming alignment of the IMC this morning, we were confused and surprised to see the IMC mirrors and PZTs being driven around without our doing. The problem was that the MC WFS servos had triggered at around 19:10 UTC after we opened the light pipe and began shooting light down to MC2 TRANS, the SUM of which serves as the trigger PD**. Because the P and Y path from these servos' output -- all the way to each MC1, MC2, and MC3 mirrors -- is, by default, always on, this non-sense drive was driving around the suspensions and PZTs. 

Because we don't want anything but humans driving the suspensions for the foreseeable future (until chamber doors start to go back on) I've taken the following disabling measures:
- Turned of the input to the MC1, MC2, and MC3 top mass (M1 LOCK) P and Y filter banks, 
    - Accepted this change in each suspensions safe.snap SDF file.
- Turned up the trigger on/off thresholds for the MC WFS (H1:IMC-IMC_TRIGGER_THRESH_ON / H1:IMC-IMC_TRIGGER_THRESH_OFF) from 40 / 20 to 5000 / 500.
    - Accepted into the safe.snap file of h1ascimc 
- Un-managed MC2 from the IMC_LOCK guardian.
- Moved the IMC_LOCK guardian to PAUSED so that it's no longer running (just a precaution, really, the MC WFS are all triggered via front-end logic).
- Cleared the history on all MC WFS servos.
- Restored all alignment offsets (PSL PZTs, MC1, MC2, and MC3) to there values reported at the close of business yesterday (see LHO aLOG 39299)

An as-of-yet unsolved mystery is why we're getting so much more sum counts on the MC2 trans (during yesterday's flashing we hit 2 or 3 [ct] max, now we've hit 25 - 30 [ct]).

**The trigger is not just MC TRANS SUM, but H1:IMC-MC2_TRANS_SUM_OUT / H1:IMC-PWR_IN_NORM_MON. The demoninator, H1:IMC-PWR_IN_NORM_MON, is set to 0.1. So with a sum of 25-30 [ct], the triggered value becomes 250-300, and surpasses the (former) threshold of 40 [ct].

Sheila has opened FRS Ticket 9387 so as to better reflect the trigger logic on the IMC / IMC WFS MEDM screens.

For the record: the above unsolved mystery of strangely high MC2 TRANS SUM was a result of an LED Work Lamp being left on pointed on the +Y (global IFO coordinates) of HAM3, which Jim had turned on to rotate and align the new PR2 scrapper baffle (aLOG pending). Once we arrived at HAM3 for our afternoon session, we turned off the lamp, and the SUM returned to there expected value of 2-3 counts (supposedly [uW]) with the parachute cover OFF (while looking in chamber at MC2's iris) and 0.5-0.9 [uW] when the parachute cover is on.

h1nds1 configured to serve archived raw minute trend data from aligo epoch

Dan, Dave:

Prior to this morning's DAQ restart I reconfigured h1nds1 (default NDS) to serve the archived raw minute trend data which Dan had recovered over the weekend onto the new SATABOY-RAID (a ZFS files system with compression enabled). I tested this to verify minute trend data is now available from the aLIGO DAQ epoch which is 12:25 PST Wed 16 Jan 2013. Attached plot shows a pem seismic channel for the whole year of 2013.

The raw minute trends are stored on a compressed ZFS file system, built on a re-purposed SATABOY RAID. Compression is done in software on the Oaracle/Sun Microsystems 4270 computer. Getting 500 days of minute trend data took h1nds1 only 5 seconds to retrieve and plot.

ISH HAM models restarted, DAQ restart

WP7206 Brian, Jim, Dave:

ISI-HAM[2-6] models were modified, compiled, installed and restarted. The DAQ was then restarted at 10:05 PST.

PSL Weekly FAMIS Tasks (FAMIS 3675 & 8447)

This morning I performed the weekly PSL FAMIS tasks.

HPO Pump Diode Current Adjust (FAMIS 8447)

With the ISS OFF, I adjusted the HPO pump diode operating currents; this week required a larger increase than usual from all 4 HPO DBs, this is summarized in the below table.  As usual, I attach a screenshot of the PSL Beckhoff main screen for future reference.

	Operating Current (A)
	Old	New
DB1	51.3	51.6
DB2	53.7	54.0
DB3	53.7	54.0
DB4	53.7	54.0

I did not adjust the operating temperatures of the DBs.  The HPO is now outputting 155.0 W and the ISS is back ON.  This completes FAMIS 8447.

PSL Power Watchdog Reset (FAMIS 3675)

I reset both PSL power watchdogs at 17:55 UTC (9:55 PST).  This completes FAMIS 3675.

PSL Weekly Report - 10 Day Trends FAMIS #6173

Apparent LASER trip (10/31) and corresponding aLog. Environmental data seems to have quieted. An incursion by Peter, yesterday, is also apparent. All other data appears to be normal.

WHAM4 ISI Balanced and left Unlocked with soft covers held away from table

I'll detail the rebalancing after I review the previous payload to document changes--Exec Summ: removed many kgs.

SUS RMS MAX thresholds set to be the same for all quads

S Cooper, S Dwyer, J Kissel

Sam has been looking into what trips in what order durring large earthquakes, and one thing he has run into is that we have different RMS thresholds for tripping different suspensions, meaning that for some optics R0 will trip while M0 might not for a particular EQ.  I have gone through the QUADS and set the RMS MAX to be the larger setting where settings differed between optics. SDF screenshot showing changes is attached.

IO, as of Monday, Nov. 6th

Today:

• IM4 pitch adjusted
  • required removing the IM4 SiC baffle
  • required loosening both lower EQ stop brackets to allow the inner SiC baffle to clear the EQ stop brackets
  • the pitch adjustment screw/sleeve was turned less than 1/2 of a turn
  • current IM4 pitch, with zero alignment bias, is -124urad
  • loosened blade attachment blocks to correct a ~-1100urad yaw with zero alignment slider
  • current IM4 yaw, with zero alignment bias, is -144urad
• IM2 yaw adjusted
  • loosened blade attachment blocks to correct the 2000 slider drive in yaw
  • IM2 yaw, with zero alignment bias, was ~-1500urad
  • current IM2 yaw, with zero alignment bias, is -36urad
• IMC irises placed in HAM2 and HAM3
• beam from PSL aligned to the IMC with the PZT
• IMC is flashing

Tomorrow:

• brief start to the day with laser safe - still some irises to place downstream of the IMC
• once that's done, will need the beam to continue the IO alignment

- Cheryl JeffK, JeffB, PeterK

On The IM4 Cage Baffle Assembly D1002745
This aLOG is merely to provide data on the troublesome IM4 baffle assembly such that it might be improved in some vent in the future.

As one may recall, in order to adjust the pitch alignment of the IMs / HAUX / HAM Auxiliary suspensions, one must remove their cage protection baffle. Recall that each of the IMs have their individualized cage protection baffle,
    Optic   Former    Baffle
            Name      Drawing / Assembly
    IM1     SM1       D0902378
    IM2     PMMT1     D0902380
    IM3     PMMT2     D0902382
    IM4     SM2       D1002745
where IM1's baffle has a large opening, IM2 and 3's are (from what I can tell) identical, and IM4 has much larger coverage, and has a special insert D1002722 in order to provide further protection for the suspension wires against stray IFO REFL beams during lock loss.

It is this special insert that gives us the most trouble when taking the IM4 baffle assembly on and off, because it does not clear the suspension cage's front (HR-side) earthquake stop brackets (D1000539, D1000540, D1002360, D1002361) when they're secure. One needs to loosen bolts and back off these EQ stop brackets to remove this baffle.

Attached are several pictures:
(1) & (2) show the baffle assembly after removal from the IM4 cage
(3) show the drawing numbers for each baffle part
(4) shows the metal erosion at the baffle's contact points with the EQ stop brackets
(5) shows IM4 after the baffle has been removed, highlighting the EQ stop brackets
(6) & (7) show the particulate that erodes off of the baffle every time we remove & re-install it.

The pains experienced during removal and re-install of this baffle assembly are not new. Kiwamu and I had trouble in the last vent when we were mechanically relieving the alignment sliders (see the bottom of LHO aLOG 12811) and during original install by Cheryl and Deepak (see 5633)

LLO aLOG 3814 seem to imply that something was done chamber-side to L1 IM4 to make the baffles fit better.