J. Kissel

ETMX UIM Driver's rocker switch tripped last night at 8:44:30 UTC (Feb 06 2015 00:44:30 PST), which cause all OSEM output on the L1 / UIM stage (and the sensor signals as well) to fail. 

I've turned on the rocker switch, and the stage appears functional.

Other points:
-------------
There is no smoking gun for this chassis power outage. There are not enough diagnostic tools in the frames to look at all possible suspects (power surge to the chassis [no power monitors], chassis overheating [no temperature sensors], an inconsequential component on say -- a monitor board -- smoking [only one of the four monitor signals for each channel are stored], too large a current request [the NOISEMON is stored, not the FASTIMON]).

Output request of DAC (from any ISC) does not appear to coincide with trip, but difficult to tell with dataviewer. There is an Longitudinal request to drive really hard, but it's unclear whether it's a the cause of or a result of the driver's power being shut off.

Output request is roughly 42 [mA] (according to FAST_I_MON) on all four coils leading up to trip, switch trips on 3 [A]. Even the sudden, very large output request does not cause any current larger than the 42 [mA] -- as reported by dataviewer / EPICs. Reqrettably, only the severely-high-passed-at-5-[Hz] noise monitor signals are stored in the frames, but I attach the time series of one of the coils anyways to indicate the exact time. It shows a nice smooth ramp down at the time of the chassis power down. 

Serial number of this box is S0900303. I attach pictures of the box after I'd turned it back on. When I first approached the box, the front "power" LED lights were OFF, and the rocker switch was in the opposite (powered off) position. 

I'm not sure if there is good way to make this power shut-down control-room visible. I did notice that all the coil driver monitor signals were frozen at -16 [ct], and that the OSEM sensor speed dials were zero for this stage and this stage alone, which is what immediately clued me in that it was a flaw with the entire coil-driver chassis or satellite amplifier. 

This last happened (according to collective analog CDS crew's memory) on H1 SUS ITMX UIM driver, some time ago, 19 November 2013. See LHO aLOG 8635.

Alexa, Kiwamu

In response to the commissioning alog from the last night (alog 16501), we spent an hour or two to study the DRMI lock acquisition with ETMs miasliged in this morning. The key item we checked this time was the BS limiter. We took a few lock acquisition samples with and without the BS limiter which was suggested by Ryan.

It seems that the limiter improved the locking time. We will tentatively keep the limiter on.

The alignment was adjusted initially so that we observed consistently high build up in each lock. We tried repeating the same test with the arm cavities aligned, but we ran into some aligment issues which prevented us from completing the test with the arm cavities aligned. Note that this limiter method is something we used to use in the past, but  apparently we decided not to use it for some reason at some point.

(The test without the limiter)

(The test with the limiter)

(The limiter)

The limiter resides in BS_M3_ISCINF_LIMIT and the value was set to 5x10⁵ cnts.

To aid in evaluating whether there is any potential wire heating happening with SR3, please see the attached SW rendering of the HAM5 SR3 view as seen from straight in front of SR3.  While there is not a "wire shielding" baffle in front of the SR3 (which there is on PR3), there is a stray light control baffle.  The aperature of the baffle is slightly larger than the distance between the wires on either side of the optic.  From the point of view in the rendering, the +X wire of the SR3 is exposed to incident beam.  The point of view seems to be pretty representative of the incident beam coming from the BS.  Of course, we could have made an error in placing this baffle which may expose more or less of the wire than shown in the rendering. 

(Note, when zoomed in, the wire shown in the rendering is slightly off since the tiny secondary prism in red was added after the fact and the wire trajectory was not updated.  The wire of course goes over this red prism, not through it as shown.)

The rendering attached can also be found on the dcc at:

https://dcc.ligo.org/DocDB/0096/D1201315/004/SR3%20CAD.JPG

Thanks, Eddie S!

no restarts reported. Conlog reporting offline.

Aidan, Alastair.

The ITMX CO2 flipper mirror screen was showing the annular mask turning on and off quite frequently. This was an issue with the alignment of the mask proximity sensor. We went out to the table and rotated the sensor so that it wasn't close to hitting anything.

The flipper mirrors are currently disabled and out of the beam path. The ITMX CO2 central mask is permanently fixed into place.

SEI - Jim wants to turn on more loops in both arms today (yesterday just in Y).

         Hugh/Jeff K. working on HEPI pump servo

SUS - ETMX some sensors not working (by the time I typed this out it has been fixed).

ISC - Ongoing

PSL - Ongoing

CDS - MY 3IFO inventory

Facilities - Bubba will be in the LVEA moving SEI stuff

        Beam tube cleaning on going

A core switch in the OSB hung up over night. A reboot of the offending switch returned service to wired network, wireless, and phones at approximately 08:35AM PT.

Sheila, Elli, Evan, Ryan, Lisa

Apparently today was a particularly bad day for the whole locking business. 
To get the DRMI locking with the arms off-resonant, we had to retune the locking gains
as previous settings were not working, meaning we could not get a single lock of DRMI + off resonant arms in ~ 20 min.

After carefully aligning the DRMI (with no arms), we changed the locking settings as follows:

DRMI ---> DRMI with off-resonant arms

MICH  3 ---> 3 
PRCL 15 ---> 8 (it was 11)
SRCL -50 ---> -40 (it was -45)


 It would be extremely helpful to test these new settings again several times, and do some systematic lock acquisition studies from DRMI to DRMI with off-resonant arms in the morning. It still takes too long to acquire lock. 

We then went through the locking sequence, and stop at some point to give Evan and Elli a chance to measure the opto-mecahnical TF of the DHARD alignment DOF, so we could design a good filter to increase the bandwidth of DHARD, if necessary.  
We broke the lock in the process, and Sheila struggled to get ALS DIFF working right after that. 

While investigating the problem, Ryan found out that we could not get any actuation at all from the ETX L1 stage.  Please investigate/fix this in the morning, a manual reboot at the end station might be required.   

On a positive side:

* Evan and Elli got a good measurement of the DHARD plant and designed a compensation filter for that, so we should be ready to test it;

* ETMY L1 has now a working L2P filter, so we could eventually try DARM actuation to both end masses, if necessary.

J. Kissel, H. Radkins,

Digging deep on the HEPI Pump Servo noise, we measured the voltage amplitude spectral densities of both pressure sensor signals right as they enter the Pump Servo box with a breakout board, some clip leads, and an SR785. The results confirm our suspicion -- the pressure sensor voltage is WELL below what is reported by EPICs. So, it's either the Athena II's ADC noise (1e-2 [V/rtHz] ?? -- seems too high even for a crappy ADC), or some nastiness in turning these channels into EPICs channels. See attachment.

In order to get the pressure sensor signal voltage above the readout noise floor without saturating the ADC, one would need either need to AC couple the signal with a ~ 10 [mHz] high-pass and gain up the signal by an additional factor of 1000, or add a whitening filter; something like two zeros at 1 [mHz] and two poles at 50 [mHz]. I don't know anything about electronics, but I think I know enough that such filters would be difficult to construct at best. 

These pressure sensors clearly were not meant to be anything more than DC sensors, so if we intend to keep using them as is, we should lower the unity gain frequency of the pump servo to WELL below 10 [mHz] (along the lines of what Hugh started to do in LHO aLOG 16466). Up to this point, we've been simply injecting terrible ADC noise into the pumps. This explains why the *very* low frequency response and DC values add up, but anything to do with the spectral densities are just injecting this bad ADC noise into the pumps. We still have to make sure the PID parameters are being employed in the units we expected them to be by the CALC record, but tomorrow's another day.

Note -- I'm not yet sure how Rich took his data, but from what I see, both the supply and return sensors are above my measurement noise floor, but the return pressure is significantly (as much as a factor of 5 @ 10 [mHz]) below Rich's eye-ball fit to the noise he measured in SEI aLOG 688.

--------
Details:
Measurement --
With the pump servo in manual mode (i.e. feedback PID loop OPEN), we inserted a DB9 breakout board in-line with the pressure sensor inputs to the HEPI pump servo (channels 1 [supply] and 2 [return], successively). As directed by the pinout in D0901559, independent clip-doodles were connected to pins 4 & 6, the positive and negative legs of the differential signal (with the shield / black clips clipped to pins 7 & 8, which are the pressure preamp's GND returns). These two legs were then fed into the A and B ports of Channel 1 of the SR785, which was set to A-B mode. Other relevant parameters:
Freq: SPAN 3.125 [Hz], LINES 800, LINEWIDTH 3.9 [mHz]
Input: INPUT CONFIG A-B, DC Coupled, INPUT RANGE 20 dBVpk
Disp Step: UNITS -- db units OFF, Pk units RMS, PSD units ON

This measurement was done with the supply line sensor, the return line sensor, with one of the 100 [ohm] resistor plugs from , and finally the SR785 noise floor was determined by sticking 50 [ohm] plugs on the A and B inputs (using the same input range for all measurements -- though I did confirm that the return ASD did not change, and was still above the SR785 noise floor when the input range of 2 dBVpk).

Calibration --
The signal chain goes as follows:
     Pressure Sensor            Pressure Preamp          Diff 2 S.E.              ADC            EPICs Calc Record
100e-3 / 300 [V_diff/PSI]    151 [V_diff / V_diff]    2 [V_se / V_Diff]    2^16 / 20 [ct/ V_se]  0.0032 ["PSI"/ct]
The SR785 voltage was measured just before the Diff 2 S.E., instrumentation-amplifier, factor-of-2 in the HEPI pump servo. 
So 
- to calibrate the 100 [ohm], input terminated signal ASD as reported by EPICs into single-ended ADC input voltage, one needs
EPICS ["PSI"] ( * (1/0.0032) [ct/"PSI"] * 20 / 2^16 [V_se / ct] =  0.0477 [V_se / "PSI"] ).
- to calibrate the differential SR785 voltage into single-ended ADC input voltage, one needs
SR785 Voltage [V_diff] ( * 2 [V_se / V_diff] = 2 [V_se / V_diff] ).
- to calibrate the differential SR785 back to PSI at the sensor,
SR785 Voltage [V_diff] ( * (1/151) [V_diff / V_diff] * 300 / 100e-3 [PSI / V_diff] = 19.8675 [PSI / V_diff] )

What did else did I learn today? 
- The EPICs calc record defines the sampling frequency to be 10 [Hz]. EPICs data is normaliy stored at 16 [Hz]. That means the features seen in the EPICs spectra (the ~1.6 [Hz] and harmonics notches) could just be Fourier reflections NOT features in the plant. I don't know what the native rate of the Athena II ADC is, I have an email out to Ben. But there is certainly some sort of nasty aliasing / imaging nonsense going on.

- The .ini files that determines which channels are stored in the frames for these pump servos lives here:
/opt/rtcds/lho/h1/chans/daq/
H1EDCU_HEPIPUMPL0.ini
H1EDCU_HEPIPUMPEX.ini
H1EDCU_HEPIPUMPEY.ini

- The CALC record that calibrates the EPICs channels from [ct] to [PSI], and also defines the sampling rate which the records are picked off of the ADC lives in
/opt/rtcds/lho/h1/target/h1hpipumpl0/h1hpipumpl0epics/db/
a.12.db

- An SR785's high-pass for its AC coupling mode is at 160 [mHz], so for these low frequency measurements, I sure-as-heck need to be DC coupled, and that limits my choice of input range, since *actual* DC signal is clearly what dominates these pressure sensors. 

Earlier today, with Ryan's encouragement, I turned on more of the I/ETMY isolation loops. Currently these 2 ISI's are running a configuration like LLO's, with all DOFs engaged on St1 and X,Y,Z,RZ running on St2. The St1 RZ loop is running a high (750mhz) blend with no T240. St2 is running the same 250mhz blend on Z and RZ as it was previously running on X and Y. The Stage Guardians for these two chambers have also been modified to turn on these loops, in case we have some traumatic event.

FF is now running on all test mass chambers as well with X&Y running on ETMY, Y on ITMY and X on E/ITMX.

Alastair, Elli, Jamie

We're working on Guardian today, and wanted to install a beam dump at the output of the Y-arm table so that we can play with that laser without injecting power to the CP.  While doing this we discovered the laser was running only at 1/2 power.  The history to this is that a few months ago the same laser was running at half power, and after swapping in a different driver, went back to full power.  The "not working" driver was sent to Caltech where it was found to be working fine.  So this new episode was actually quite useful in finding that a problem still exists.

We went out to the LVEA and found that the power connector at the patch panel is badly connected.  We couldn't effect a permanent solution in the 1hr window we had, but were able to get the laser back to full power again.  These connectors will likely need replaced long term, and we'll start working on a solution to that.

The ouput of the table was found to be dumped to a beam dump already (it has been like this since installation of the table was completed).

Minor changes to the drift monitor threshold updater script:

The log files are now recorded to: "/opt/rtcds/userapps/release/sus/common/medm/sus_driftmon_logs/", and have the following file name structure: SUSDRIFT_{OPTIC NAME}_{START_TIME}-{DURATION}.log (e.g. SUSDRIFT_ALL_1107199664-15.log, SUSDRIFT_MC1_1107199664-15.log)