During the period of bad IFO state which started at approx 12:12 PDT, the ITMX SWWD triped the SEI-B3 system. In the plot below, Ch1 shows the ITMX SUS top stage F1 OSEM signal, which rapidly exceeded its 95mV trip limit. This started the SUS SWWD counter. Five minutes later, Ch2 shows the signal going to the SEI-B3 drop to zero (the BAD state). This started the SEI SWWD counter, shown in Ch3 as going from one (GOOD) to three (1ST COUNTER). Four minutes later the SEI SWWD transitioned to four (2ND COUNTER) and one minute later tripped the SWWD, which zeroed all SEI-B3 DAC outputs. This shows up on the RMS plot as a slightly elevated signal, but SUS continues to be rung up. At 12:27 the operator intervened and manually panic'ed the SUS SWWD, at which point the RFM started decreasing.
Here is a timeline of the CDS issues we had today (all times local):
10:37 h1oaf0 models stop running
11:11 h1oaf0 models restarted
11:15 h1calex and h1caley models restarted with IRIGB channels added
11:55 DAQ restarted due to cal model changes
12:12 SUS ITMX rings up
12:22 SEI-B3 SWWD trip
12:27 SUSB123 manual trip of DAC
Kyle, Gerardo In and out of X-end VEA ~1030 - 1230 hrs. local Added 1.5" O-ring valve in-series with existing 1.5" metal angle valve -> Wrapped RGA with heater tapes and foil -> Elevated pump cart off of floor (resting on foam) -> NW40 inlet 50 L/s turbo (no vent valve) backed by aux. cart (no vent valve) -> Begin 100C bake In and out of X-end VEA between 1405 - 1425 hrs. local, 1450 - 1455 hrs. local and 1600 - 1605 hrs. local. NOTE: Will need to enter X-end VEA to make adjustments Saturday morning
~1710 -1800 hrs. local I realized that I had a CFF inlet 50L/s turbo on the shelf as well as UHV 1.5" valve -> Swapped out 1.5" O-ring valve and NW40 inlet turbo for their dryer cousins -> resumed bakeout
LVEA: Laser Hazard IFO: Locked Observation Bit: Commissioning 07:45 Cleared IPC errors on H1SUSTMSY & H1IOPASCO 08:00 IFO locked at Low Noise, 24.2w, 68Mpc 08:35 Karen – Cleaning in the LSB cleaning area 09:16 LockLoss – Unknown 09:17 Kiwamu & Sudarshan – Going into LVEA to make ISS Outer Loop Servo measurements 09:19 Robert – Going into LVEA to setup for PEM injection 09:22 Jason – Going into LVEA to check PR3 OpLev 09:25 Kyle & Gerardo – Going to End-Y to get pump cart 09:30 ISI-ITMY WD trip – Robert doing PEM setup work in Beer Garden 09:53 Kyle & Gerardo – Finished at End-Y – back to CS 10:00 Kyle & Gerardo – Going to End-X to connect pump cart 10:17 Kyle – In LVEA to take valve off Pump Cart 10:20 Kyle – Out of LVEA 10:27 TJ – Going to End-X to reset BRS 11:07 Dave – Restarting models after OAF crash 11:12 Stefan – Going into MSR to work on atomic clock 11:27 Stefan – Out of MSR 11:30 Sudarshan – In LVEA plugging monitoring equipment 11:50 IFO Locked at Low Noise, 24.1w, 65Mpc 11:55 Dave – DAQ restart – NOTE: DAQ restart did not take IFO out of lock 12:12 IFO LockLoss – DAQ restart/Guardian/etc 12:36 Kyle & Gerardo – Back from End-X 12:39 Vendor on site to restock machines 12:47 Bubba – Test fire pumps 14:28 Dave & Stefan – At End-Y to install attenuator 14:50 Dave & Stefan – Going to Mid-Y to power cycle PEM 15:05 Dave & Stefan – Back from Mid-Y 15:15 Dave & Stefan – Going to End-X to install attenuator 15:30 Dave & Stefan – Back from End-X
Today I performed the monthly fire pump test. All numbers and flows were acceptable.
Robert, Vinny, Stefan C, Dave:
The recently installed IRIGB signals were clipping the ADC due to the 10X gain in the PEM AA chassis. We installed a 10X attenuator in the line at the IRIG-B chassis in both end stations, the IRIGB signal range is now 0 - 5000 counts.
We also power cycled h1pemmy to see if this would remove the 64Hz noise being seen in the SEIS channels there. The procedure was: stop all models, power down h1pemmy, power down IO Chassis, power up IO Chassis, power up h1pemmy. We got lucky on the auto-code start and the timing came back OK with no further IOP restarts required.
Unfortunately this power cycle does not seem to have fixed the noise.
Scott L. Ed P. Rodney H. 7/30/15 The crew relocated lights, fans and cords to begin cleaning the next section starting at HSW-2-043. 51 meters of tube cleaned ending l2 meters east of HSW-2-041. I repaired several of the extension cords and a cord on the vacuum used for cleaning the support tubes. 7/31/15 Ed took today off. The support vehicles and all related equipment were relocated after lunch and a total of 61 meters of tube cleaned today ending 13.7 meters east of HSW-2-038. The past ~200 meters of tube have been especially dirty with feces and urine. We have been giving these areas extra attention. 1239.5 meters of Y-Arm cleaned to date.
Stefan, Jim, Andres, Filiberto, Dave
WP5392
New h1calex and h1caley models were installed today, these read out ADC3-CHAN30 as the IRIGB signal. These signals were added to the commissioning frame and the frame broadcaster. The DAQ was restarted at 11:55 PDT to install this change.
Something happened at lock loss and the IFO did not reach the defined DOWN state.
Symptoms:
Dave, Jaime, Sheila, operators, and others are investigating.
Let me try to give a slightly more detailed narrative as we were able to reconstruct it:
So what's the take away:
bug number 1: guardian should have caught the NDS connection error during the NDS restart and gone into a "connection error" (CERROR) state. In that case, it would have continually checked the NDS connection until it was re-established, at which point it would have continued normal operation. This is in contrast to the ERROR state where it waits for operator intervention. I will work on fixing this for the next release.
bug number 2: The operators didn't know or didn't repond to the fact that the IMC_LOCK guardian had gone into ERROR. This is not good, since we need to respond quickly to these things to keep the IFO operating robustly. I propose we set up an alarm in case any guardian node goes into ERROR. I'll work with Dave et. al to get this setup.
As an aside, I'm going to be working over the next week to clean up the guardian and SDF/SPM situation to eliminate all the spurious warnings. We've got too many yellow lights on the guardian screen, which means that we're now in the habit of just ignoring them. They're supposed to be there to inform us of problems that require human intervention. If we just leave them yellow all the time they end up having zero affect and we're left with a noisy alarm situation that everyone just ignores.
A series of events lead to the ISC_LOCK Gaurdian to not understand that there was a lockloss.
To prevent this from happening in the future, Jamie will have Guardian continue to wait for the NDS server to reconnect, rather than stopping and waiting for user intervention before becoming active again. I also added a verbal alarm for Guardian nodes in Error to alert Operators/Users that action is required.
(If i missed something here please let me know)
I ran BruCo on the quiet data period reported here. The report can be found at the following link:
https://ldas-jobs.ligo.caltech.edu/~gabriele.vajente/bruco_1122369690/
I'll look into the table and provide a summary later today.
Between 11:10 and 11:25 I adjusted the frequency offset and the phase offset in the timing diagnostic system's cesium clock in MSR so that the 1PPS time difference between it and our GPS backed timing distribution system equals 0±100ns and the long-term drift rate is near zero. Last time I adjusted the timing offset (3 weeks ago), I subtracted 68e-15 from the factory-set frequency offset of 1078e-15; this was apparently the wrong direction, as evidenced by a very visible doubling in the cesium clock's drift rate in the weeks since that adjustment. Since the last measurement of drift rate was calculated from a month's worth of data and was ostensibly accurate, I went ahead and changed the frequency offset to (1078+68)e-15. The manufacturers instructions are vague on the correct adjustment direction and the tech support reps aren't very knowledgable, so I'm going to write up a short technical paper on Cs-III calibration with specific instructions for how to go about this. I zeroed the phase using a 1PPS signal from the MSR master. The GPS timing jitter is O(10ns), which is well within the 100ns resolution of the CsIII phase syncing utility. I used a short (3') BNC cable to minimize delay. I also went ahead and installed the most recent Monitor3 software on the Lenovo X61 Thinkpad in MSR and put the Cs-III instruction manual on that computer's desktop, so it should be easy for someone else to make finer adjustments to the frequency offset once we have another month or two worth of 1PPS drift data.
at 10:37 PDT the IOP model on h1oaf0 received a timing error, all models stopped running at this time(h1pemcs, h1oaf, h1tcscs, h1odcmaster and h1calcs). The calculated IFO range went to a high value and the DARM FOM display went noisy. Only one external model receives IPC from the h1oaf0 computer, h1omc has a dolphin channel from the calcs, these channels went into 100% error.
There was PEM work in the CER around this time, this could have been the cause of the glitch. All the models were restarted and all is running correctly now. The intent bit was not set during this time.
BRS software crashed on the 25th, so today I went to EX to restrart the code. I kept the damper commented out, so the damper is currently OFF. I will turn it back on when it seems like it will be calm down at EX for a bit (vac team is down there right now).
Within a min afer a lockloss, the ISI ETMY ST1/2 WDs tripped. The medm said that it was ST1 T240 and the plots showed a slow drift to the WD trip level (plot attached). HEPI moved 170 um after lockloss before tidal began to bleed down. A further investigation is ongoing by the SEI team.
Looks like this is a trip we will suffer whenever the drive to the HEPI is large enough at the time of lockloss. That doesn't necessarily mean the length of lock but it is certainly related. If the tide turns around and the offset given to HEPI heads back to zero, we could be fine after a long lock stretch.
The attached 30 min second trend plots are around the three most recent locklosses from some low noise state. In each case you can see the T240 tilting towards trip level but only the most recent trip was the tidal offset large enough to 'tilt' the T240 long enough to hit the trigger. The HEPI output from ISC is in nm.
Is the problem the bleed off too fast? It seems this is 2um/sec. Is there horizontal to tilt coupling that needs to be addressed? Is it a problem anyway? The T240 will likely take a couple minutes at worse to settle and another minute for the ISI to reisolate.
CDS: EE shop work to prep for Ring Heater installation PSL: Wants to tweak PR3 OpLev at first available opportunity FAC: Continued beam tube cleaning Y-Arm VAC: Moving pump cart to End-X for weekend long RGA bake out
Evan, Matt, Lisa We took ~25 min of undisturbed data starting at Jul 31 2015 09:21:13 UTC , after the MICH FF improvement (and all the changes described earlier implemented), with the interferometer locked in low noise ~ 65 Mpc (according to SensMon). We saw a big glitch during this period (~9:28), a huge one right before (~9:18, right after the mich tuning was done), and another one at the end of the undisturbed period. We went back to commissioning mode as Evan is starting some noise injections for his noise budget.
I did injections into MICH (from 10:41:40 to 10:47:00) and SRCL (from 11:15:00 to 11:22:40) with all FF on.
About 30 s after turning off the SRCL injection, a number of photodiode signals associated with the AS port dropped suddenly by about 5% (in particular: AS DC, AS90, ASA sum, ASB sum, ASC sum, and AS45Q; notably, the OMC DCPDs, POP DC, MC2 sum, and IMC input power did not see this). After that, the interferometer glitched for about 30 minutes. This event happened at 11:23:31.05 and lasted for only a few milliseconds.
I did an injection into PRCL (from 12:21:50 to 12:29:00).
I did an injection into the ISS around 12:52:00. I think this measurement needs to be redone by injecting into the error point of the outer loop rather than the inner loop. With the outer loop closed it is now difficult to get good coherence between intensity and DARM below 300 Hz.
PSL Peri PZT servo ON/OFF test:
IFO was still locked when I came in in the morning, so I briefly switched the IMC WFSB YAW to PSL PZT feedback path at around 7:50 AM local time.
In the attached, references are with the servo on, current traces are without the servo.
Middle finger (300Hz bump) is greatly reduced in OMC DCPD, but the coherence for right and index finger slightly increased due to gain peaking (expected), and there is another gain peaking at 620Hz (again expected). The last one could be reduced further by adding a deeper notch but that might increase the 350Hz bump gain peaking a bit.
The servo was turned on again at about 8AM+-5min.
Sudarshan, Matt
The new ISS filter was seen to oscillate at ~1MHz with no input signal. This was fixed by adding 50pF cap into the feedback of the output opamp. This cap, in combination with the 10k resistor, puts a pole at 300kHz, which is fine for this loop. The second loop is currently closed (with the IFO at high power) and appears to be working properly.
Also, independently of the repair that Matt desribed above, we found that an op-amp in the very first stage kept railing at -10 V. It is AD620 stuffed in U10 of the 2nd loop servo box (D1300439-v2). We have no idea when this op-amp had died. Swapping the op-amp with a spare in the EE shop fixed the issue. We will update the e-traveler accordingly.
Matt, Evan
Why do the TMSX RT and SD OSEMs have such huge spikes at 1821 Hz and harmonics? These spikes are about 4000 ct pp in the time series. In comparison, the other OSEMs on TMSX are 100 ct pp or less (F1 and LF shown for comparison).
Also attached are the spectra and time series of the corresponding IOP channels.
On a possibly related note: in full lock, the TMSX QPDs see more than 100 times more noise at 10 Hz than the TMSY QPDs do.
From Gabriele's bruco report, the X QPDs have some coherence with DARM around 78 Hz and 80 Hz. A coherence plot is attached.
It seems similar to the problem from log 12465. Recycling AA chassis power fixed the issue at the time.
Quenched the oscillation for now (Vern, Keita)
We were able to clearly hear some kHz-ish sound from the satellite amplifier of TMSX that is connected to SD and RT. Power cycling (i.e. removing the cable powering the BOSEM and connecting it again) didn't fix it despite many trials.
We moved to the driver, power cycled the driver chassis, and it didn't help either.
The tone of the audible oscillation changed when we wiggled the cable on the satellite amp, but that didn't fix it.
Vern gave the DB25 connector on the satellite amp a hard whack in a direction to push the connector further into the box, and that fixed the problem for now.