The MC2 suspension occasionally ran into a situation where some part of the suspension was kicked intermittenly for some unknown reason. I noticed this behavior this morning. I could not identify what was kicking.

The worst part is that the kicks (or glitches) seem to be gone since sometime around 14:14 local. Very bad.

The symptoms are:

• Once it runs into a glitchy period, the suspension glitches mostly in yaw. This is visible in the witness sensors and even in the IMC REFL camera.
  • Usually not big enough to unlock the IMC. However, very bad for lock acquisition which relies on the stability of the laser power.
• According to the witness sensors at the bottom stage, glitches are as large as 20 urad in yaw in pk-pk and 5 urad in pitch.
• This happened even when all the active loops were off (LSC, ASC, and damping loops).
  • Note that I left the static alignment biases on -- so there were some finite constant DAC counts at the top stage.
• When it came back to the nominal situation where no glitches, the actual alignment changed roughly by 5 urad in yaw.
• Evan also noticed a similar behavior yesterday evening for the first time.
  • Maybe this issue is new and started happening very recently ?

(Checking various channels)

Apparently this behavior is different from the two DAC issues we have seen in this past week (alog 18453 outputting a high constant voltage, alog 18569 outputting a nonlinear signal). As mentioned above, I disabled ail the active loops including the LSC, ASC and the top stage damping loops. Even in this condition, the suspension still kept being kicked interminttently. I looked at VOLMONs on all the stages, but did not find any suspicious activities. Also, I checked motions of the HAM3 ISI in L, P and Y using the suspension-coordinate-witness-signals (i.e. SUS-MC2_M1_ISIWIT_L(P, Y)MON), but did not  find such a fast signal. In addition, I checked the bottom stage witness sensors of PR2 as well to see if this is some kind of table motion, but the PR2 was very quiet and no glitches were found at the time when MC2 was glitching.

(A test without LSC or ASC signals)

I attach a second trend of some relevant channels. This is a two-hours trend, with the top stage damping loops fully on and with no LSC or ASC signals. You can see that the wintess sensors of the bottom stage showed glitches in the first 1/3 of the trend and suddenly stopped. Since the damping loops were engaged during this periode, the VOLMONs showed some reaction of the loops and I think they are just reactions and not the cause of the glitches.

Actually, now the top stage RT OSEM makes me think this is it. Its VOLTMON showed a noticable discrete jump (by 14 counts) in the attached trend right around the time when glitches stopped.

Thank you, Dave, Richard, John, Gerardo and those who helped us today even though it was a holiday. These pictures are for you guys.

Richard, Kiwamu, Dave:

Following Richard's suggestion, Kiwamu powered down the IO Chassis and removed the lid. After about 10 minutes the IO Chassis and then the CPU were powered up. I had removed h1seih45 from the rtsystab file to prevent any models auto starting. After several minutes I verified the Gen Std cards were still visible on the bus. I then started the IOP model. It came up with a minor IRIG-B excursion, which came back down in a few minutes. The Gen Std cards were still with us. I then started the user models, one at a time, with no problems.

Kiwamu and Evan are now untripping the watchdogs and starting isolation. Hopefully this will get us through to tomorrow.

I've put h1seih45 back into rtsystab. Kiwamu noted that the ADC and DAC Gen Std cards have LEDs which are on when powered up, and the DAC cards LEDs go out when the IOP is started. We should see what these LEDs mean.

SudarshanK, DarkhanT, RickS

We performed a routine calibration of Photon calibrator photodioides (both transmitter module PD-TxPD and receiver module PD-RxPD) at Xend on 20th May and Yend on 22nd May. We will post the results of the calibration soon.

Richard, Kiwamu, Dave:

the IOP model on h1seih45 has failed, it cannot see any General Standards cards in the IO Chassis when doing a software bus scan (lspci), but it can see the Contec Binary IO cards:

root@h1seih45 ~ 1# lspci -v |grep 3101

root@h1seih45 ~ 1# 

root@h1seih45 ~ 1# lspci -v |grep 3120

root@h1seih45 ~ 1# 

root@h1seih45 ~ 1# lspci -v |grep -i contec

13:00.0 Multimedia controller: Contec Co., Ltd Device 8682 (rev ff) (prog-if ff)

21:00.0 Multimedia controller: Contec Co., Ltd Device 8682 (rev ff) (prog-if ff)

2c:00.0 Multimedia controller: Contec Co., Ltd Device 8632 (rev ff) (prog-if ff)

Kiwamu went into the CER and confirmed that the h1seih45 chassis is powered up, the ADC interface cards have their LEDs lit, fans are on. We powered down the CPU and the Chassis (keeping the latter down for a minumum of 30 seconds) and then powered them back up.

This is where it gets strange. As soon as I could log back into h1seih45 I could see the General Standards cards on the PCI bus (six ADCs, two 16bit DACs). But, when the IOP model started, they disappeared from the bus and once again all I could see are the Contec cards.

controls@h1seih45 ~ 0$ lspci -v|grep 3101

Subsystem: PLX Technology, Inc. Device 3101

Subsystem: PLX Technology, Inc. Device 3101

Subsystem: PLX Technology, Inc. Device 3101

Subsystem: PLX Technology, Inc. Device 3101

Subsystem: PLX Technology, Inc. Device 3101

Subsystem: PLX Technology, Inc. Device 3101

controls@h1seih45 ~ 0$ lspci -v|grep 3120

Subsystem: PLX Technology, Inc. Device 3120

Subsystem: PLX Technology, Inc. Device 3120

controls@h1seih45 ~ 0$ lspci -v|grep -i contec 

13:00.0 Multimedia controller: Contec Co., Ltd Device 8682

Subsystem: Contec Co., Ltd Device 8682

21:00.0 Multimedia controller: Contec Co., Ltd Device 8682

Subsystem: Contec Co., Ltd Device 8682

2c:00.0 Multimedia controller: Contec Co., Ltd Device 8632

Subsystem: Contec Co., Ltd Device 8632

< about here the models try to startup >

controls@h1seih45 ~ 0$ lspci -v|grep 3101

controls@h1seih45 ~ 1$ 

Richard suggested we power the sytem down for an extended time (10 minutes) to cool everything down and run with the lid off. Kiwamu is doing that at the moment.

I'll disable the autostartup of the models, so we can manually step through the startup process.

Kiwamu and Dan reported that there is no potable water in the OSB. It looks like the RO system went into fault on May 22 around 5:30pm and this morning the potable water tank went dry.

Kiwamu visited the water room for me and reset the RO system so it now appears to be operating normally and making water. However, the building is not yet pressurized.

I suspect the Naimco water skid will need to be reset.

Dan, Kiwamu, Evan

List of things done today:

• The ETMX UIM coil driver had tripped off sometime yesterday, and we drove down to the end station to turn it back on.
• In ETMY L3 drivealign, there is now compensation for the LVLN driver summing network. The compensation is a zero at 152 Hz and a pole at 3.2 kHz. This brings the high-frequency DARM phase response back to its previous shape. (See attachment.)
• In the DARM_OFFSET state, the DARM gain is now increased by a factor of 2 and the low-frequency boost (FM7) is engaged. This makes DARM more stable during the transition to dc readout. However, this throws off the procedure for determining the scale factor between OMC DC and AS45Q, so sometimes the transition causes the DARM gain to decrease.
• A high fifth-harmonic violin mode was causing the ETMY ESD to saturate, so there is now a stopband filter in the ETMY L3 LOCK filter bank (since DARM is full).
• LOWNOISE_ESD_ETMY has been rewritten for the new transition procedure. It has been tested (once) and works.
• We took a crossover measurement of ETMY L2. We tried to get ETMY L1 as well, but it seems currently that it's quite low (below 1 Hz). The gain in the L1 LOCK filter can be pushed up from 0.16 to 0.22 without issue (and maybe even further).

The CW hardware injection code psinject now runs without crashing at LHO.  (Thanks, Peter Shawhan, for debugging the launch script.)  The injection pathway is turned off, but the injection code has been sending 13 standard CW signals into the CAL model since approximately 21:30 PDT.  Psinject is still running many hours later, so the code seem stable.  Additionally, all injection code has now been migrated to svn here:

https://svn.ligo.caltech.edu/svn/dac/hwinj/Details

I've tested the svn version of tinj (for transient injections) at LHO and it seems to work properly, though, additional tests are planned.  The latest version includes additional interaction with EPICS channels.  During the course of today's debugging, I determined that GRB alerts do not appear to be recorded in the channel H1:CAL-INJ_EXTTRIG_ALERT_TIME.  I have contacted Andrew Williamson to sort out why.

Kiwamu, Peter, Dan, Evan

We made it back to dc readout tonight. 3 W PSL power, DARM feedback to ETMX.

We are in the process of trying to engineer the new transition to ETMY. We were hampered by the intrusion of a large amount of 1/f³ noise in DARM, with ASD 1×10⁻¹⁴ m/Hz^1/2 at 10 Hz. The initial portion of the lock was fine, and then the noise came. We trended various channels and looked at coherences, but could not nail down the cause. The noisy time starts at 2015-05-23 9:41:00 UTC. After about an hour, the noise went away again.

The ASC was a bit touchy. It could be engaged by hand, but with the guardian it seemed that engaging the PRM pointing loop caused several of the other loops to go unstable. Also, we gave up on the ITM pointing loops. They increased the recycling gain, but made the arm and sideband buildups less stable.

We found that there was no tidal signal being sent to ETMY. At first blush it appears to be an IPC issue.

Following Xavier's timing measurements, the following FECs have their DAC Duotone* still turned on:

• h1lsc0
• h1iscex
• h1iscey

During parts of this week this was also turned on for h1susex and h1susey. These have been turned off as these DAC channels are used to drive the R0_COIL_RT signal. For the systems listed above, neither the DAC channel nor its ADC readback counterpart are being used by the models (PCAL reads the ADC channel but only archives it to DAQ).

* - reminder: if the DAC Duotone drive is enabled (via the GDS_TP MEDM screen for the IOP model), relays in the IO Chassis interface cards for the first DAC and the first ADC route the output of the last channel of the first DAC to the second-to-last channel of the first ADC. The IOP model takes the Duotone (digitized from the last channel of the first ADC) and writes this to the last channel of the first DAC.

Jim, Dave:

We are seeing a DAC card in h1susey which passes autocal on power up, but subsequently fails autocals if the IOP model is restarted. This is the same card (fifth card of five) which stopped driving its output to the ESD (see Richard's alog earlier this afternoon).

To check if autocal failure and lack of output are related, we used the card which always fails autocal (SN 101208-05) and looked at its output. In summary, its output drive is good and not related to its autocal success.

Details: DAC is the only card in x1susex. h1iopsuseywdt is the IOP model, it has a SWWD which required reset of its DACKILL part. A new user model called h1cds18bitdactest.mdl was created (DCUID=100) which drives all eight DAC channels using filtermodules. The DAC was connected to the DTS 18bit DAC AI chassis (lower eight channels) and a breakout board and DVM used to determine the output DC voltage. Voltages were driven by putting an offset on the filtermodules.

07:15 Richard and Peter to end Y
08:13 Sudarshan to end X to pick up PCAL equipment
08:56 TJ misaligning SR3 for brief period
08:57 Rich, Calum, Ben to end Y, ESD
08:58 Richard to end Y
09:17 Sudarshan, Darhan to end Y, PCAL calibration
09:18 Hugh moving BS HEPI
10:08 Richard to end Y
10:12 Nutsinee to HAM1 HWS table
12:23 Richard to end Y
12:47 Richard back

Relocking got as far as CHECK_IR. Further progress has been interrupted by an earthquake.

Peter, Calum, Ben, Rich

Took transfer functions (1Hz to 10kHz, 255 points in to the DAC Drive Input on the front panel of the ESD driver, out of the respective SHV connector going to the ETM) of all the quadrant paths of the newly installed LN Driver.  The results are stored in:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Measurements/ElectronicsMeasurements/
under the following file suffixes

UR -> TFSR785_22-05-2015_102054.txt
LR -> TFSR785_22-05-2015_102804.txt
UL -> TFSR785_22-05-2015_103546.txt
LL -> TFSR785_22-05-2015_105053.txt

Everything looks good as compared to the Spice model.

Also, we measured the output noise of each channel at 20Hz with and without DC bias.  Results are all consistent with the design (<40nV/rtHz at 20Hz and above) and Spice model.

0V bias/-7.4V bias
UR -> 21nV/rtHz/32nV/rtHz
LR -> 22nV/rtHz/30nV/rtHz
UL -> 23nV/rtHz/37nV/rtHz
LL -> 28nV/rtHz/37nV/rtHz

Analyzer Noise Floor 10nV/rtHz @ 20Hz

Typical output noise at higher frequencies is:
20nV/rtHz @ 50Hz, 22nV/rtHz @ 100Hz, 20nV/rtHz @ 150Hz

Noticed that there is still a discrepancy in the DAC monitors for the user model vs. the IOP.  Richard has the details and Jeff will likely follow up next week.

Once Richard sorts out the somewhat touchy vacuum interlock (he's working on that now and knows the problem), the entire system is now functional and ready for use.