nutsinee.kijbunchoo@LIGO.ORG - posted 07:35, Tuesday 24 November 2015 (23691)
Lockloss 15:28 UTC
Haven't investigated the cause but the ALS beams don't look too good.
Haven't investigated the cause but the ALS beams don't look too good.
Quiet night so far. Useism is on the rise. A HPI-ETMX_IPS location mon was increasing the beginning of the shift but now seems to flatten out.
Added ~ 5 gallons of fuel to generator -> found blown fuse for 1 of 2 variacs supplying heat to 1 1/2" tubes + gate of 10" gate valve -> replaced 5A fuse -> OK now -> T/C for ion pump indicates 188C -> adjusted output (decreased) for both variacs supplying heat to ion pump (150C is target)
Title: 11/23 Eve Shift 24:00-8:00 UTC (8:00-16:00 PST). All times in UTC.
State of H1: Observing
Shift Summary: EX ISI is causing troubles again, but some changes to ASC seem to be keeping things under control for now
Onsite Support: Evan, Jeff, Jenne
Activity log:
IFO was unlocked when I arrived. Evan and Jeff were trying to diagnose EX.
1:00 I start initial alignment, this is hampered by AlignIFO and IMC nodes not cooperating. IMC node seemed to hang and refuse to lock occasionally, I had to re-request Lock. Lock X-arm IR was also uncooperative.
2:40 Finally to NLN. Except for EX slowly ringing up and then down, all's quiet.
A few details on the Xarm locking, in case it's helpful for someone else in the future:
We had the XARM gain turned up to 0.15 (which many people have been doing lately, when the Xarm is being fussy (nominal is 0.05)), however when it would catch lock, it would oscillate significantly. I think this was because the optical gain was too low (highest flashes and average transmission were about 0.8), and so we couldn't afford the 2 integrators that come on immediately after the lock is acquired. I had Jim disable the filter module triggering (by setting FM1 and FM2 of the XARM to "Man" rather than "Trig"), and we were easily able to acquire a smooth lock. After a few seconds the ASC brought the arm power to just above 1.0, and we switched the triggering settings back to normal. Alternatively, we probably could have turned the XARM gain up even higher, to make sure we weren't so close to the low edge of the phase bubble, but there didn't seem to be a reason to explore this more thoroughly.
The IFO just got to NLN, so I switched to Observe. Shortly after, the ring EX ring up reappeared, so we have dropped out of Observe again, and are tuning some ASC to try to settly the IFO down.
Getting ready to go back into Observe, Jenne had me reduce the gains on PRC1 Y with a -20dB filter, and reduce the gains on DHARD P (from 10 to 7) and Y(from 15 to 10). The low frequency ground motion is coming up, so we may be getting an earthquake.
Not a cause for alarm. Just noting for commissioners interest that after doing initial alignment tonight, I went straight to Lock PRMI, which acquired quickly, but the transition to DRMI failed. This is just the first round of trying to relock for tonight. DRMI just acquired for a second time, then promptly dropped the whole IFO out. Looking to be a long night.
J. Kissel While trying to characterize problems with HEPI / TIDAL / ISI / ASC (LHO aLOGs 23676 and 23676), I tried taking a swept-sine transfer function with three data points between 0.001 and 0.005 to charaterize the HEPI / UIM cross-over frequency (recall that the TIDAL crossover frequencies were never calibrated beyond "roughly," LHO aLOG 16255). However, upon starting the excitation, DTT gave the IFO a good kick on lots of ASC and LSC DOFs. It did *not* break the lock, but it certaintly came close, and any higher an excitation amplitude would have certaintly lost it. From a look at the time serious, it looks like DTT *tried* to ramp the signal, but chose a much faster ramp than is necessary for such a waveform. Attached are the start of the waveform, a zoom in on the ramp, and then a screenshot of the DTT params.
After the maintenance of GraceDB this afternoon (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23665) I restarted the ext_alert process on h1fescript0 using Monit.
Generator will run continuously until Wednesday afternoon to supply power for the bake-out of the new ion pump -> Will be in @ ~0100 hrs. local tonight to refuel
Title: 11/23 Day Shift 16:00-24:00 UTC (8:00-16:00 PST). All times in UTC.
State of H1: Locked at NLN but in Commissioning
Shift Summary: After a couple of EQ related locklosses, we ran into an issue with the ISIs using 45mHz blends causing them to ring up, resulting in lockloss. Evan, Hugh, JeffK, and presumably Jim (when he arrives for shift) working on resolving this. We could revert back to 90 mHz blends, but with microseism increasing, that might not last long either.
Incoming operator: Jim
Activity log:
16:37 Joe and Chris to X arm for beamtube sealing
18:52 Lockloss
19:25 Observing
20:11 Joe and Chris back
23:00 Lockloss
A few short locks that I did not log as we were looking in to the osc. issue.
Evan, Hugh
As seen by JimW and others, the ETMX was moving ALS excessively especially wrt ETMY. Attached are the in line outputs from the Isolation loop for EX and EY. This two hour trend plot shows an EQ that took the IFO out of lock around 1/3 of the wasy through. The ring up of the ISO X at ETMX is very clear in the second have of the plot. The second attachment shows uncalibrated spectra of the X Drive two hours ago and during the ring up where it appears to be elevated in the 40mHz region. Oh what the heck, I'll put them on the same snap.
We deisolated the DOF and then reisolated. This action seems to have addressed the problem for now. Well not for long. Rung up again with IFO locked. Took it out of Observing and tried turning of boosts and adjusting gains.
E. Hall, J. Kissel, T. Sadecki, H. Radkins, J. Warner
The IFO had lost lock again from the same problem about ~20 minutes later. We noticed LLO was down, so we tried pushing buttons hoping to rectify the problem.
We tried several things, all on hunches we'd had from all signals we saw oscillating at ~30-40 [mHz], which included HEPI / ST1 ISI / PRC1 Yaw and TIDAL -- so this is NOT just an "ISI ETMX Loop goes unstable" problem, its a nasty, slow, cross-coupled interaction of which the ISI is only one of the players.
Things we "know" that informed our hunches and button mashing:
- This only shows up when the ISIs are on 45 [mHz] blends.
- Symptoms point us toward ETMX, because it stays rung-up after lock-loss
- The above mentioned channels are visible wobbling at these 30-40 [mHz] frequencies
- There seems to be a 6-8 minute period envelope to the oscillations
Our hunches include
(1) Bad IMC_F-to-UIM or UIM-to-HEPI tidal cross-over when ISIs are in 45 [mHz] blends, i.e. the tidal offload is at a higher frequency than the ISI's inertial blend frequency, so the ISI's isolation loops are fighting actuation from tidal
(2) PRC1 Yaw ASC loop (which keeps PRM aligned to the PRC's cavity axis) unstable, which modulates the power into the arms. This modulates the classical radiation pressure on the ETMs in common, which excites CARM / Tidal. This then puts excess motion into the Tidal feedback and offload.
(3) Excess motion on HEPI from tidal is causing excess tilt, because HEPI tilts when you request it to drive in longitudinal. With the ISI ST1 in 45 [mHz] blends, and the RX&RY blend still in the 250 [mHz] blend configuration, the excess tilt from HEPI is coupling into the ST1 CPS, tilting the ISI, and fooling the X&Y DOFs into thinking there's excess X&Y motion, and drives excess X&Y. This pushes the test mass in X&Y, exciting some DARM / CARM action, which then spills back to HEPI.
Note that these are all sort of half-coupling mechanisms that, if I waved my hands hard enough, you might believe. But, we have no measurements (i.e. Sys ID, Loop Transfer Function measurements) to prove any of these statements.
Based on these hunches, we twiddled the following knobs:
- Reducing UIM to HEPI offload UGF (H1:LSC-${X,Y}_TIDAL_CTRL_UGF) from 0.002 to 0.001 [Hz] --> no noticable effect, but we were watching a 10 [minute] strip tool, so out patience may not have been high enough. The Oscillation was present, but not ringing up.
- Reducing UIM to HEPI offload UGF by half again to 0.0005 [Hz] --> same result
- Returning UIM to HEPI offload UGF to 0.001 [Hz], and reducing the IMC-F to UIM offload UGF (H1:LSC-${X,Y}_COMM_CTRL_UGF) --> We might have started to see a reduction in amplitude.
- Reducing the PRC1 loop gain (via the POP A DC to PRC1 ASC input matrix H1:ASC-INMATRIX_Y_3_21) from 1.0 to 0.5 --> This seems to reduce the oscillations in all signals.
While Evan and Jim coninued to twiddle knobs, I and LLO was down, I suggested we try to characterize some of these low frequency loops. However, a third of the way through my initial characterization of the UIM to HEPI tidal offload (which had already had a bad start, see LHO aLOG 23680), the oscillations had come back to a point of no return. Unclear whether they were because we just didn't reduce the original oscillation fully, or whether the bad start to the excition kicked everything enough to restart a bigger oscillation, or if it was my measurement itself slowly driving things around that re-triggered the oscillation enough to eventually saturate Tidal / the UIM. However, it was apparent that it was the 6-8 minute envelope that brought things over the edge, not the 30-40 [mHz].
After that lock-loss, Jim had found IR in the arms to be pretty bad, so he has since gone into initial alignment.
Stay tuned...
I attach some visual aides that I'd used to discuss the hunches with Hugh / Evan / Travis in case it helps anyone else. (THC = Tilt Horizontal Coupling).
Went to Commisioning Mode at 22:48 UTC because ETMx and ETMy ISIs began oscillating. Evan and Hugh tried a few things, but to no avail and lock was lost.
Laser Status:
SysStat is good
Front End power is 31.47W (should be around 30 W)
Frontend Watch is GREEN
HPO Watch is RED
PMC:
It has been locked 5.0 days, 21.0 hr 50.0 minutes (should be days/weeks)
Reflected power is 1.726Watts and PowerSum = 25.12Watts.
FSS:
It has been locked for 0.0 days 0.0 h and 37.0 min (should be days/weeks)
TPD[V] = 0.9939V (min 0.9V)
ISS:
The diffracted power is around 8.366% (should be 5-9%)
Last saturation event was 0.0 days 0.0 hours and 37.0 minutes ago (should be days/weeks)
Performed a run spanning the time period from 1131828057 (Nov 17 2015 20:40:40 UTC) to 1132088085 (Nov 20 2015 20:54:28 UTC)
The following parameters were used for this run:
No scheduled injections were found for this period.
There were only two single-IFO injections found; both were CAL-INJ resets in H1:
Both injections were found to occur only in ODC HOFT and GDS HOFT.
No notable anomalies were found.
(Note to operators: Don't touch modulation depth, it doesn't do anything good while making it difficult for everybody to analyze the glitches.)
Summary
1. Output level of RF AM stabilization went back and forth between two levels since our last incursion to the PSL room.
2. Looking at RF AM glitch in LSC-ASAIR_B_RF90, LSC-ASAIR_A_RF45 and ASC-AS_RF45, it seems like the RF level going to EOM is constant but the modulation phase changes.
The first attachment shows a jumbo RF45 glitch reported in alog 23618 at around 11-21 2015 11:44 UTC. Nobody was touching the modulation depth slider.
Top left shows that the RF AM control signal jumped up by about 0.2% of its DC value and then came back after about 4 seconds. During this 4 second window, the signal was full of large glitches. This was felt by the control and error signal of the loop (top and middle, left ) but not by out-of-loop AC and DC (top and middle, middle row).
This was also felt by the spare unit in the CER (middle right) and the demodulator at the ISC rack that Evan installed (alog 23567, top right). For the CER unit, the jump was about 2E-5 of the DC level. For Evan't demodulator, it's AC-coupled, but using the gain of 1000 for SR560 and using 175mV DC level before amplification, the jump seems to be 5mV/175mV/1000~3E-5, which is consistent with the CER unit. Since these are much smaller than what the RF AM stabilization see (0.2%), it seems to me that the glitch in CER and ISC rack is a result of something bad in the EOM modulation path propagating back to the CER.
Interestingly, ASAIR_A_RF45_I_ERR jumps but Q_ERR doesn't (bottom, middle). The latter should be dominated by the DARM length offset. If the RF level coming out of the EOM driver jumps, I would expect both Q and I jump. This seems to me that the RF AM stabilization is working fine, but there's some phase jump associated with the AM glitch and the effect is more evident in the phase that has smaller signal.
Similarly, ASAIR_B_RF90_Q_ERR jumps but I_ERR doesn't (bottom, left), ASC-AS_A_RF45_I_SUM does but Q_SUM doesn't.
The second attachment shows the second jumbo glitch, but this was not reported by the operators because it happened when IFO is not locked. You can see that the LSC-MOD_RF45_AM_CTRL_OUT jumped by similar amount as the first event, but it never came back. No IFO signals were available of course, but you can see that it's not like there's a huge electronics coupling for LSC-ASAIR_B and ASC-AS_A_RF45.
It seems like there are two states, whatever they are, and the RF AM stabilization is responding to the transition from one to the other (or maybe the stabilization is going crazy). It could be an impedance change somewhere but I'm not sure.
Whatever the cause is, it cannot be after the EOM driver. If something is going on after the driver, the reflection should propagate back to the rms detector of in-loop and out-of-loop differently, and I cannot imagine the out-of-loop channels look flat as they do here.
So, something is going on, it could be anywhere between the distribution amplifier for this specific path in CER to the EOM driver itself, including these two end points. We could swap the distribution amplifier spigot of the spare and PSL in CER, I doubt that it does anything but we can further narrow down the trouble spot.
The third attachment shows that this going back and forth behavior was present before. This is from a week ago noted in Ed's alog. The amplitude was much smaller, but we can see that the output level jumped at around 77 seconds, then back at around 123 sec.
LLO had called and said they would be down for awhile. I'm going to finish doing a swept-sine through the INJ_HARDWARE filterbank from 5-500Hz. Previously I had only finished 500-2000Hz. The IFO intent mode has been turned off. This is WP#5595.
Previous swept-sine measurement was: aLog 23307 The following was done: * psinject turned off at 5:20 UTC * swept sine measurement started at 5:21 UTC * swept sine measurement ended at 5:48 UTC * psinject turned on at 5:49 UTC I've attached a plot of the transfer function and coherence between INJ_CW and DELTAL_EXTERNAL_DQ. WP#5955 is now complete.
Attached are results of computing a transfer function from H1:CAL-INJ_CW_OUT to H1:GDS-CALIB_STRAIN, using the same method used for the higher-frequency swept sine on Nov 11. The transfer function is generally flat and well behaved below 500 Hz and above ~35 Hz. The feature at 167 Hz appears to be an artifact of the swept sine measurement at that point being artificially split into several points. Figure 1 - Spectogram of H1:CAL-INJ_CW_OUT during swept sine Figure 2 - Spectogram of H1:GDS-CALIB_STRAIN during swept sine Figure 3 - Transfer function and coherence More detailed results can be found here.
The DTT template for doing the 5 to 500Hz swept-sine measurement is in the SVN here: template_cw_inj_sweptsine_5to500_20151121.xml The DTT template for doing the 500 to 2000Hz swept-sine measurement is in the SVN here: template_cw_inj_sweptsine_500to2000_20151111.xml
