Dan, Sheila, ITMX
While testing the vioin mode damping tonight, we had an abrupt lockloss, and for a short period my bandpass filter at 501.094Hz was sending noise x120dB into the ITMX L2 coils. This tripped the coil current watchdog, just like a previous event on ETMY a few weeks ago. (There are now limiters on all the violin mode damping filters...)
Unlike ETMY we can't reset this watchdog by toggling the RMS Reset epics channel. Anyone know how to clear this error?
We suspect something in hardware is preventing a software reset. In the CDS highbay, the SUS Independent Watchdog electronics box has red lights that won't reset when we use the 'Fault Reset' button.
I asked the question about how to clear it when I was designing an alert on the Ops Overview screen for it. As I recall, all you have to do is change the value in the field from a 1 to a 0 and then back again? Also, I think Stuart Aston told me that watchdog would eventually go away.
So, normally changing that value back and forth WOULD be the way to clear it according to Stuart. This did not work. He then suggested I simply power cycle the PUM so with Sigg's approval I did and that seems to have worked. The L2 RMS watchdog trip on ITMX has been cleared. On a side note, this 'SUS Ind WD' chassis is ONLY cabled to the BIO I/O chassis for ITMY. It IS NOT documented on any of the current drawings and it isn't commissioned. It's current state can't be reset with the button on the front so I assume that it would have to be power cycled to be cleared. That being said, it isn't really connected to anything.
Currently experiencing the same behavior with ETMX L2 UL. Toggling the RMS WD reset does nothing.
Also, the ETMX indicator on the ops screen does not indicate this fault (the border is still green).
Dan, Kwiamu, Sheila
We noticed last night that we had almost no gain at low frequencies in the DHARD loop. This is a little strange, and doesn't really make sense with the suspension model and our control filters. The beofre measurement, which doesn't have great coherence at low frequencies is shown in the first attachment. We have a ugf around 3 Hz with about 27 degrees of phase, however we also probably have some lower ugfs and not much DC gain. This doesn't make much sense given our control filter (alog 17006).
We added a boost, which is two poles at 0.001 Hz, and a pair of complex zeros at 1.5 Hz with a Q of 3. The situation for this boost is a little tricky, we can't afford to loose gain below 1 Hz and we can't afford to loose the phase that we would loose if we used real zeros. We ended up with the loop shown in the second attached screenshot. We have 3dB of gain margin on the low side, 6 dB of gain margin on the high frequency side, and 17 degrees of phase margin. We could probably make this more stable by moving up the roll off of our control filter, which currently has a pair of conplex poles at 12 Hz.
We are leaving the loop as it was for lock acquisition, the unconditionally stable loop is usefull durring CARM offset reduction, but engaging this boost on resonance.
Sheila, Kiwamu, Dan
We have had several occasions where we are trying to make a driven measurement, we can clearly see our drive orders of magnitude above the DARM noise, but we get no coherence according to DTT. The first screenshot attached shows the coherence from my drive into the DHARD pitch loop to several channels that in principle all contain the same information:
H1:CAL-DELTAL_EXTERNAL_DQ, H1:LSC-DARM_IN1, and H1:LSC-DARM_OUT_DQ. You can see all of these channels have different coherences, but I don't understand why. Since I'm not driving the DARM filter bank, DARM IN and DARM OUT should be exactly the same information with a linear filter applied, and the CAL channel is the sum of these two channels with some calibration applied.
Hmm. Since OUT is the worst, followed by IN and then CAL, I am wondering if we get the single treatment?
I have added a couple of medm screens which will enable us to have a quick overview of all the oplevs. One screen for all SUS oplevs and another for all HAM_ISI oplevs. Screenshots attached.
The current limit settings beyond which indicators turn red are as follows:
1) for SUM signals the range of good values is [10 to 40000]
2) for PIT and YAW signals the range of good values is [-30 to +30]
3) for individual segments of the QPDs the range is [-10000, 0]
I plan to include BLRMS for PIT and YAW for all oplevs asap.
Suggestions for improving these screens are welcome!!
Today we added an output to OM3 from the ASC model, and recalculated the DC centering control in HAM6 so that three degrees of freedom are controlled by the three tip-tilt actuators. The DOFs we chose to control are two from the OMC (POS and ANG) and the spot position on AS_A. Initially we went with AS_B, but this scheme led to a worrisome amount of beam jitter on AS_A and we decided to switch.
The AS_A spot position is controlled through the DC3 filter bank in the ASC model. The 3x3 control matrix for {DC3, POS, ANG} --> {OM1, OM2, OM3} was calculated using a measurement of the sensing matrix from last September. (This was done for both pitch and yaw, of course.) The loop gains are set very roughly to generate only a modest impulse when the input switches are turned on. We'll measure these loops using a single bounce later this week.
The Guardian has been updated to use this new topology. The OMC QPDs loops are now enabled as part of the AS DC centering shortly after DRMI lock acquisition. Note that all three loops need to be closed to maintain the correct spot position on AS_B!
With this new centering we've turned on all the ASC loops while on resonance and things seem ok. Since we're not directly controlling the spot position on the WFS that is used for MICH and SRC1 we should check that unsuppressed beam jitter doesn't couple into those error signals.
Attached is a screenshot that records the control matrices. Note that we're only using DC3, not DC4, in the ASC output matrices.
Daniel, Kiwamu,
One of the to-do items today was to study how the rotational stage behaves. There have been some suspicion with the rotational stage that:
Also,
We did a simple test today to study the above issues. It looks that the variation in the adjusted laser power can reach roughly 500 mW at most when the requested power is around 10 W. We also experienced a slipping-like behavior.
(Random walk in requested angle)
To check if it behaves good or bad, I wrote a script which randomly request the angle and automatically collects the resultant power for each random walk. This time I made 200 random steps within +- 90 deg. The result is shown below:
The data points are divided into half i.e. 100 samples each in chronological order. The red dots are the ones from the 1st half and the blue dots from the 2nd half. The difference is that we hit the "search for home button" between the two measurements. Since Daniel incidentally updated and rebooted the rotational stage code during the measurement, we had to hit the "search home" button as the reboot gave us a warning sign. As shown in the above plot, clearly the two data sets have different angle offsets which are visible as sine curves in the bottom residual plot. Because of that, the resultant residuals became bigger and reached 500 mW at most. Probably if I fit only one of the halves, the residuals would be smaller. At the moment, I have no good explanation of why the angle offset seemed to be shifted. I am guessing that it is not from the actual optical hardware as each data set was consistent.
Some notes:
(Calibration)
See the attached screen shot shown below for some details. Based on all the measured data from the random walk test, I fit the necessary parameters and updated the calibration of the rotational stage. Also the safe.snap in target/h1ecatc1/h1ecatc1plc1epics/burt
is updated accordingly.
Jason and I looked at the rotation stage was looked at yesterday morning. The retaining ring holding the optic in place was firm. So it would seem unlikely that the waveplate is slipping within the rotation stage. A closer examination when we can block the beam going into the IMC would be required. The part of the rotation stage that actually does the rotating, did not feel loose within its frame. Perhaps the shaft encoder is intermittent?
To prepare for controlling DARM using ETMY, I copied the changes the Keita has made to damping and L2P decoupling for ETMX (alog 16968) to ETMY . The attached screen shot shows the transfer function from an length excitation on L1 to the pitch optical lever, red is before copying from ETMX and green is after. The lower plot shows the op lev spectrum durring the excitation.
I also checked the L2P and L2Y on L2, this was done simply by putting an offset onto the length drive which used about half the DAC range, and looking at the impulse response using the oplev. The arrangement of filters that was already engaged seems to do well for L2P, it reduces the impulse response by more than a factor of 10 compared to having no L2P decoupling on. for Yaw there was no difference really between the filter I found engaged and having nothing on, for the time being I'm leaving it on although it could be turned off.
Dan, SHeila, Jeff K remotely
The coil driver for ETMY L1 shut itself off this afternoon, this might have been while I was trying to measure the L2P coupling there. We switched the rocker switch and things are fine. The symptom was that we didn't seem to be drivig even though everything on the medm screen looked fine, the OSEM readback all had low values. This is the same situation we had at End X a while ago alog 16511
This is one of the "errors" that we probably need better monitorinng of. As a short term solution maybe we could add this to TJ's list, to check if all OSEM readbacks are low for a certain stage.
Just in the case that you weren't aware, that rocker switch is in fact a 3A circuit breaker. So whatever was going on with that stage was taxing that driver pretty hard.
I have already put it in the SYS_DIAG guardian. I had it on Pause during the time that coil driver turned off so I wasn't aware that it had.
Richard, Jim, Dave:
at 12:05 the h1seib2 (BS) front end system went offline. We saw no green leds on the IO Chassis One Stop card, and saw red leds on the card itself. We power cycled h1seib2 and its IO Chassis (we prevented the models from running). The computer could not see any cards in the chassis.
We powered down both h1seib2 (BS) and h1seib3 (ITMX) systems, and switched the One Stop fibers at the IO Chassis (h1seib2 connected to h1seib3's chassis and vice versa). Confusingly now both FE computers can see all the cards in both chassis. We then powered everything back down and connected the fibers back correctly. At this point the problem has gone away, h1seib2 sees its cards again. We started the user models, the only problem was with h1iopseib3 which had a run-away IRIG-B time with went up to 1500 and took an hour to come back down.
Richard says that the One Stop card in the h1seib2 IO Chassis showed a red led when the computer was not powered up, but the h1seib3 chassis did not do this. So we are still suspicious of the h1seib2's IO Chassis One Stop card, and if this problem re-appears we will consider replacing it.
An interesting aside, I looked at the IO Chassis One Stop cards in the CER and found that while most have 4 green leds on, some have 1 and one has 2.
NTP and Atomic Clock WP#5105
Dave,Jim, Vern:
The NTP server was moved in the timing rack to a lower slot. The lid to the Beckhoff chassis was reattached. The new Symmetricom Caesium Atomic Clock was racked in the location vacated by the NTP. The atomic clock was connected to UPS power and achieved sync up after 30 minutes of running.
Installation of MSR Timing Comparator
Dave, Jim:
We installed the MSR timing comparator chassis in the timing rack. We connected its timing slave fiber optics cable to the timing master (port 3) and then powered it up to +12V DC. It connected to the master with no problems
PEM CS model change WP5042
Dave, Robert:
I added the missing two narrow band radio channels to the h1pemcs model. To keep all the radio channels together on the 3rd ADC, I moved the temperature, lowfreq mic and mainsmon channels down by 2 channels in the model, and did the same with the BNC cables at the AA chassis. I checked that the low freq mic and mainsmon signals look the same in minute trends before and after the change.
OAF IPC receivers
Dave:
I removed the two IPC receivers in the h1oaf model which used to come from the LSC model but were removed. These are H1:LSC-CAL_IMC_[CTRL,F]. I grounded these channels in the h1oaf model.
DAQ restarts
Dave, Dan, Daniel, Sheila:
Several DAQ restarts were needed today: New PEM-CS model, new OAF model, new LSC model, new ASC, SUSHTTS models, new Beckhoff C1-PLC2 code, new guardian nodes.
For the ASC and HTTS models: we added IPC sender/receiver pairs to provide an ASC drive to OM3.
Also, we added the following ASC channels to the DAQ (but not to the science frames):
As outlined in alog 17077 we should shift the IMC frequency to the reference cavity AOM up by ~450 KHz after we are locked, the common tidal feedback path is engaged and after the green light is no longer needed. To this end an offset has been added after the IMC_F filter module. This way we can use kHz rather than counts. It also employs a ramping feature, so the offset can be turned on smoothly.
New channels are H1:IMC-F_OUT_OFFSET for the frequency offset in kHz, H1:IMC-F_OUT_OFFSET_SW for engaging the offset, H1:IMC-F_OUT_TRAMP for the ramp time in sec, H1:IMC-F_OUT_RAMPING for the status of the ramping process, and H1:IMC-F_OUT_MONITOR as the new error signal for the common tidal servo. H1:IMC-F_OFFSET_OUT is available as a new fast testpoint which includes the offset.
(Fil, Mani, Daniel)
Adding the 10 MHz frequency reference to the RF synthesizer in the mechanical room turned out to be more involved than we planned. First, we noticed that some of the RF distribution cables were badly terminated and needed to be redone. We then found that no baluns were added—generating one big ground loop through the RF cabling.
This is the current setup:
We noticed that thin coax cables were used to route the RF signals from R1 to R2. We should replace those with the proper thick coax which is used everywhere else in the distribution system.
We still need to route a DB25 cable from the RF distribution amplifier to the RF amp concentrator in the CER. This way we have both frequency and RF power readbacks available. The EtherCAT system was udated with the new monitor channels for the TCS RF distribution amp. This will make it straight forward to diagnose a problem with this RF synthesizer.
A new medm screen was made for the TCS RF distribution (see attached).
Kyle, Gerardo Legacy pumps long past their nominal life expectancy -> Damaged knife edge of 2 1/2" O-ring isolation valve at HAM2 ion pump attempting to align flanges which were kept sprung misaligned by rigid non-perpendicular piping -> Had to remove damaged valve and replaced it with an elbow (will repair valve in house and keep as spare) -> Leak tested new joints -> Will now need to pump with pump carts for a few more days before energizing new ion pumps. Note: HAM1/HAM2 annulus volume was fully vented for a period of time -> no pressure change in Corner Station
Added 326 channels. Removed 57 channels. The following 10 channels are unmonitored: H1:GRD-TCS_ITMY_LOGLEVEL H1:GRD-TCS_ITMY_MODE H1:GRD-TCS_ITMY_NOMINAL_S H1:GRD-TCS_ITMY_REQUEST H1:GRD-TCS_ITMY_REQUEST_S H1:GRD-TCS_ITMY_STATE_S H1:GRD-TCS_ITMY_STATUS H1:GRD-TCS_ITMY_TARGET_S H1:SUS-SR2_LKIN_P_OSC_SW2R H1:SUS-SR2_LKIN_Y_OSC_SW2R
Hugh and Krishna had found the BRS software had crashed pretty much on schedule, so Sheila and I went to the end-station to restart it. Pretty straightforward. Alogs 13817 and 15005 will get you everything you need to do it. The BRS is rung up right now because we went out to check the mass damper positions, but it should settle down in 30 minutes or so.
K. Venkateswara
I took a look at the BRS_RY_IN channel and it looks like BRS started to get damped but then the damping failed (see attached pic). It is currently oscillating with 500 count ampitude. I'm not sure why this happened but it currently looks like the mean position of the damping turn-table has changed by 45 degrees causing it to neither damp nor drive BRS. A repositioning of the turn table might help. If not, I'd suggest turning off the damper temporarily.
Looks like Jim and Sheila did everything right, but the damper is a bit finicky.
Looks like it did damp down eventually. I have no idea what happened, but it is possible that the damper turn-table got stuck, moved to a new mean location and then somehow got stuck again and returned to a reasonable location. In any case, looks like BRS is damped now and working well.
Dan, Kiwamu,
At one point last night, we noticed that a roll mode rang up at ~14 Hz. The peak height in the DARM spectrum (ffted with a 0.1 Hz BW) exhibited a anomalously high value of more than 10-12 m/sqrt Hz.
So we worked on damping the peak, this time with the AS_WFS_A_45Q signal (alog 16864) which is a state-of-the-art Livingston technique. We were successfully able to damp the mode to an ordinary height of about 10-14 m/sqrt Hz level in the DARM spectrum.
(Identification process)
First of all, see how terrible the roll mode was:
The red curve is from Mar-17-2015 8:51:00 UTC and we were intentionally stayed on the rf DARM rather than the DC readout to combat the roll mode. Even though there was some confusion in identifying which suspension had been rang up, eventually we confirmed that the roll mode was from ITMX by taking coherence between AS_WFS_A_45Q and the OPLEV_PIT signal of each test mass. In fact, the ITMX oplev was able to see the roll mode in its spectrum with a peak ~ 10 times higher than the noise floor while the rest of the oplevs did not see a visible peak.
In addition, after the successful damping on ITMX, the AS_WFS_A still showed a bit high peak but with a slightly lower frequency (~ 200 mHz ) than that of ITMX. We then identified that this was in turn from ETMY by looking at the coherence again. This lead us to another damping work on ETMY which we were also able to damp.
(Damping setup)
The AS WFS_A signal was routed from the ASC model with a gain of +1 (in ASC_OUTMATRIX_TESTMASS_DAMP). I found that both were damped well with a positive gain in SUS-E(I)TMX(Y)_M0_DAMP_R.
As for ITMX, I ended up with FM3 (-100 dB), FM4 (bp13.9) and a gain of +40. Although I had to start from a gain of +1 in order not to saturate the DAC due ot the high amplitude in the roll mode. Then I gradually increased the gain as the DAC counts reduced. Once I went to a gain of more than 10, the mode was damped relatively quickly probably on a time scale of about a couple of minutes.
As for ETMY, I ended up with FM4 (bp13.9) and a gain of +0.0008. At one point I went to a gain of +0.005, but this actually started increasing the peak height. So I had to set the gain back to + 0.0008. This loop could damp out the mode on a time scale of a couple minutes as well.
(Why they rang up ?)
We don't know. One observation is that before the modes rang up, Sheila was trying to increase the recycling gain by manually touching the ITMs.
What's the exact frequency (or at least to ~0.01 [Hz] precision)? Remember, I was able to distinguish two mode frequencies in LHO aLOG 16868: a 13.18 [Hz] mode or the 13.81 [Hz] mode, which I'd narrowed down to an ITM and an ETM mode respectively. It would be good to catalog is the ITMX mode you say rung up. @DetChar -- you're more than welcome to answer this, if Dan or Kiwmau can't get back to it quickly. The time you should use is listed in the bottom corner of the DTT screen capture.
since the color of the Time text is the same as the reference, I guess that's not the right time. Better to use the time of the RED live trace.
Looks like about 13.98 Hz, accurate to 0.01 Hz. I used 5 mins of data, 120 sec FFT, overlap of 0.9. I used the time mentioned in the log entry (8:45 UTC Mar 17) since the data didn't look right around the time in the DTT screen. We don't have a nice tool for quickly marking the peaks in spectra. I'll look into making one.