edmond.merilh@LIGO.ORG - posted 14:15, Tuesday 27 January 2015 (16307)
PSL Diagnostic Breadboard Reports w/ISS RPN
Non-image files attached to this report
J. Kissel, H. Radkins, R. McCarthy We tried a few things in order to debug STS B (GND_ITMY), which had been revealed to have a factor-of-2 too low gain after an event at Tuesday, 8:30a Dec 23 2014 (see LHO aLOG 16237). Nothing was successful at fixing the problem. However, we have ruled out the STS Distribution Chassis, the STS Interface Chassis, and the ADC that reads in the STS by swapping a few cables around at the rack: - The factor of two followed the field cabling, i.e. when STSB / ITMY was plugged into STSA / HAM2's readout, the factor-of-two moved to STSA / HAM2 readout. - Richard also made sure that cables were secure both going in and out of the satellite amplifier. - Finally, he injected (differential) +10 [V_{DC}] into the appropriate STS Interface Chassis channel, and found the readout displaying +16384 [ct], as expected (see T1100538). This implies that the fault is either in the STS itself, the orange cable from the STS to its satellite amplifier, the satellite amplifier itself, or in the long cable run from the satellite box to the racks. Richard suspects that the custom D50 connection at the STS Interface Chassis, but will continue to investigate tomorrow. Attached is evidence of the mornings activities, but more importantly that the gain factor is *exactly* what is was before after restoring everything, so we'll continue to use the matched gains determined in LHO aLOG 16208.
This morning we ran a new field power cable from the CER DC power rack to the ISC racks by HAM6. This is to prep for the installation/testing of the Fast Shutter later this week. We also installed a HV Kepco power supply in the CER. In order to terminate the new field cable at the floor rack, we temporarily powered down the OMC HV Piezo power supply from around ~11am-12pm. The power for the OMC Piezo is now restored.
We turned off the sensor correction to do some trouble shooting on the STS2_B. Ed got these turned back on ~11pst. All HAMs have ground XYZ to ISI-ST1, BSCs have ground Z to HEPI and ground XY to ISI-ST1.
ISS erratic. I unlocked, adjusted REFSIGNAL and re-locked. Holding steady for now. https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=13781
The original log entry was made before 12:00P.
With permission from Jeff K I have remotely updated the EPICS settings related to the ODC for the PSL. This change had no impact on any non-ODC EPICS settings, or other settings. I have committed updated safe.snap files for h1psliss, h1pslfss, and h1pslpmc, as logged here. This should bring the H1 PSL ODC in line with the equivalent at LLO.
When we center the REFL DC with the servos, we often find that the RM1 has tripped. This was true yesterday evening, and today Jan 27th, 2015 19:43 UTC. No investigation has been done by the commissioning team.
This noise source (pump) will likely be running for the next 6 - 8 weeks
Doug went to the floor and used a rope to pull the instrument air line etc. more from the oplev pyron and the 10-min oscillation is gone.
alog from yesterday: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=16276
This is yet another temporary fix that is good enough for a while, just like the original "fix" which stayed until today. This sounds like a good target for the detector group to keep track of.
Last week I found all of the BSC ISIs with their St2 GS-13s in low gain. Looking in conlog, I found that the gains had been switched at 07:05 PST on the 22nd, then I found the error and corrected it at 08:59 PST on the 23rd, meaning there was an entire night where the all of the ISI's were running in a different from nominal state. This would be a good window to look at the performance
Additionally, last week, Arnaud tried running the TM ISI's with all of their RZ loops turned on. The largest window I found for this was from about 10:20 to 20:00 PST.
All BSC ISI's have otherwise been in their nominal configurations since then (ie RZ loops off, rdr blends, sensor correction on, all instruments in high gain (except the BS)).
J. Kissel, A. Pele (WP #5025) We are to make the IFO's fast shutter functional within the next week, in order to prepare and protect the OMC DCPDs from the energy leaked from a FULL IFO lock lock. In its current design / implementation, this shutter is fast and loud as it closes and reopens, which sends large impulses to the optical table on which it sits (in this case H1 ISI HAM6). In order to prepare for this "toaster" being activated, I've increased the number of consecutive saturations (computer cycles above the user-predefined threshold) that are acceptable for the H1 ISI HAM6 GS13s and Actuators before its watchdog trips to 4096 [samples] / 4096 [Hz]= 1 [sec] and 8192 [samples] / 4096 [Hz] = 2 [sec], respectively (where it had previously been 10 [samples] / 4096 [Hz] = 0.0024 [sec] for these channels). See attached screenshot of the relevant WD block innards. This has been done and in place at LLO for a few weeks with no obviously adverse affects, (see 16430 and 15998), we were merely replicating their change. As such, they've seen a drastic reduction in HAM6 ISI trips after lock-losses, and we hope to see the same benefits. Note, the change involves *unhooking* (disabling and breaking) the /opt/rtcds/userapps/release/isi/h1/models/h1isiham6.mdl model link to the generic HAM ISI model library part /opt/rtcds/userapps/release/isi/common/models/isihammaster.mdl and replacing the WD_SAFE_COUNT tag input to each GS13 and ACT blocks (inside the HAM6 WD block) with hard-coded constants of 4096 and 8192. I've committed the unhooked model to the repository. For the restart, I - Brought the OMC suspension to SAFE - Turned off all output to OMs 1-3 - Asked the SEI_HAM6 manager to take the chamber to DAMPED - Paused the SEI_HAM6 manager (to make sure ISI_HPI is left alone and happily running) - Changed the ISI_HAM6 subordinate to exec - Requested ISI_HAM6 subordinate to READY - Turned OFF ISIHAM6's master switch - Recompiled, Reinstalled, Restarted h1isiham6 front end process on h1seih16 computer - Requested ISI_HAM6 subordinate to DAMPED, confirmed IOP output - Changed the ISI_HAM6 subordinate back to managed - Changed the SEI_HAM6 manager back to exec (who came back, happily fooled that it never left the DAMPED state) - Requested SEI_HAM6 manager to return to ISOLATED. - Restored OMC suspension to ALIGNED - Restored alignment and damping to OMs 1-3 (and a random Y offset of 2000 [ct] on the TEST filterbank). Note, also, since I merely changed constants in the model, the change did NOT require a DAQ restart. I also attach trends of the P and Y of the HAM6 suspensions (as measured by their local OSEM sensors), confirming that they've been restored to the same alignment.
J. Kissel Proving that watchdog still over-protects, the watchdog tripped twice after I had made this model upgrade. It has tripped on both on CPS saturations (still set to 10 [samples]) *and* the modified GS13 (now set to 1 [second]). Attached are screen-shots of the trips. No clue what the cause of the trips were, but it was maintenance day and several people were in and out of the LVEA (canned guess without any evidence to support this was it). Trip Times: GPS 1106417935.822998 (GS13s) GPS 1106416251.409424 (CPS)
Loaded the updated tidal code into iscex/y.
I will be doing some work on the connection between CDS and GC this morning which will result in some brief interruption to access to CDS from the outside world (and vice versa). This includes any services hosted at lhocds.ligo-wa.caltech.edu. A follow up entry will be posted when work is complete.
Work is complete, things should be operating as normal again. Also, I omitted the work permit number in my original entry, that is WP5022.
Switched to manual control at 1510utc. Made one setpoint tweak at 1514. It looks like it will probably run without further adjustment (maintaining the ~70psi differential pressure) for a hour or more.
J. Kissel, H. Radkins, Corner Station HEPI Pump Servo Control was restored at 9:21 PST (17:16 UTC). For the record -- wasn't collection an Open Loop Transfer Function as indicated by the title, we merely wanted a few hours of data with the pump servo off to assess out-of-loop noise and impact on platform motion.
Alexa, Elli, Sheila, Rana, Evan
Today we worked some more to make the sqrt(TRX+TRY) handoff more robust.
Now we can pretty reliably complete the transition by hand. We are working on implementing the transition in the guardian.
We have tried to reduce the CARM offset while locked on sqrt(TRX+TRY), but we cannot seem to get beyond 2 or 3 times the single-arm power without blowing the lock. The next step is to go through the CARM reduction sequence more carefully and characterize the OLTF of the CARM loop in this state.
As discussed in LHO#16252 et seq., we suspect that the common-mode board has gain-dependent offsets. If this is true, it explains why the interferometer can be knocked out of lock while ramping down or turning off ALS COMM.
To boost the CARM loop's immunity to these offsets, we redistributed gains as follows:
LSC-REFL_SERVO_IN2GAIN). Correspondingly, the gain on the AO into the IMC board was reduced from +16 dB to −4 dB (IMC-REFL_SERVO_IN2GAIN).LSC-REFLBIAS_GAIN was increased from 0.8 to 8).
Even with these changes, success is not guaranteed. The gain steps can be heard very clearly when listening to LSC-CARM_IN1, and turning off ALS COMM on the summing board will blow the lock about half the time. We can get better results by using the gain slider on the common-mode board, but we can still hear the gain steps.
LSC-REFLBIAS, we engage FM3 (1 Hz pole, 40 Hz zero) and FM9 (1.6 Hz pole, 40 Hz zero). The gain is 8 ct/ct.LSC-TR_CARM, we set the offset to −0.5 ct.LSC-REFL_DC_BIAS with a gain of 30 ct/ct.LSC-REFL_DC_BIAS to 50 ct/ct.A lockloss plot is attached for an unexplained lock loss during the CARM offset reduction after handing off to sqrt(TRX+TRY). Other lock loss times, all 2015-01-27 UTC: 03:56:53, 04:05:25, 05:36:16, 06:40:59, 07:56:02, 08:37:20. The last four are during CARM offset reduction after the handoff is complete.
For a sqrt(TRX+TRY) offset of −1.5 ct, the gain we need for REFL_DC_BIAS is 50 ct/ct. Our OLTF looks good here. However, when we reach an offset of −2, we seem to lose lock, even though there is no obvious nastiness in the CARM spectrum.
Today, locking DRMI without arms was pretty painless. In contrast, DRMI+arms lock acquisition was very, very slow for most of the morning and afternoon. After about 5pm local, it became painless as well. This may have been correlated with our changing the trigger settings to be twice as high with arms (compared to no arms), since the POP18 buildup is twice as high. But we haven't investigated this systematically.
Last night Sheila requested DRMI_LOCKED on the ISC_LOCK guardian when she left for the night at 1am to che k if this configuration is stable.
DRMI with green in the arms and IR held off resonance stayed locked overnight, and the power slowly degraded untill 4.30am when the lock dropped. Cuardian relocked DRMI untill 6am when DRMI lock dropped and did not recover. This shows that this configuration is fairly stable.
After changing our ALS gain rampdown to be in the CM board rather than ahead of it, we never broke lock due to the rampdown. However, we only tried it in this new way twice.
The lock loss plot attached above was typical of our current lock loss mystery. After moving to a new CARM offset, the CARM error signal seems stable (no peaking in the spectrum) and the loop shape looks good. The lock breaks without any characteristic sound - just a sudden lock loss. Investigation of the 5 LSC error and control signals shows no instability in the 100 ms before lock loss. The lock loss happens several seconds after the CARM offset is done ramping. Since the signal from the Thorlabs Transmon PDs is only 2000-3000 counts, we think that they are not saturating.
No guess yet about what is happening. Any speculations on lock loss causes is welcome.
We struggled with the ETMX bounce mode all of today. It seems to have started ringing up around 3 AM last night (according to the ETMX OL 3-10 Hz BLRMS trend). We spent a couple of hours damping it around noon today, but then it slowly started growing again today. We've now installed a 60 dB stopband filter centered at 9.77 Hz in the OL loops to see if this will stop the ringups. We have also installed a resonant gain filter for 9.77 Hz in the CARM loop to reduce the arm power fluctuations during the offset reduction.
To better monitor the bounce mode, we set up the ETMX OL lockin screens to demod the OLPIT signal at 9.67 Hz. So now one can trend the 0.1 Hz beat note in the lowpassed output of this to see what the Bounce mode peak height is at all times. Probably should make a dedicated BLRMS for each suspension with an OL to monitor its bounce mode height.
I am taking a look at the lock losses Evan listed in this entry. All the correction signals seem good except the BS one. This is, for example, lock loss 4_05 UTC, with different zoom of the correction signals during CARM offset reduction. Probably a campaign of loop measurements for the vertex DOFs can help.
More lock loss science. As far as I can tell, 5:36 and 6:40 are similar to the previous lock loss 4:05, while in 7:55 looks like CARM is indeed the culprit..
(BS oplev seems OK)
Given the prior lock loss science by Lisa, I speculated that the BS oplev loops were doing something bad such as glitches. (Note that the people did not use the ASC loops last night so that the oplev damping loops on BS had been engaged all the time). Looking into the last five lock losses that Evan posted, I am concluding that the oplev damping loops were not glitching or disturbing the MICH loop. The attached are 60 seconds full time series of various BS oplev-related channels for the five lock losses. The 4:05 event is the only one which clearly showed DAC saturation and the rest of them did not saturate the BS DAC before the lock loss. The oplev sum sometimes shows a fast transient, but it happens right after each lock was lost -- indicating that the transient was caused by the motion on the oplev QPD as the BS was kicked and it is NOT initiating the lock loss.
(TRX seems always lower than TRY)
I don't know if this is related to the cause of the lock losses, but I found that TRX have been consistently lower than TRY by roughly 10 % regardless of how big the CARM offset was. It is unclear if this discrepancy is from an unintentional offset in the ALS diff operating point or some kind of calibration error in TRs. In any case, we should fix it in order to reduce DARM coupling in the TR_CARM signal path.
What is the CARM offset and offset reduction in physical units? (pm or Hz of the arm cavitites)
Peter, we can use Kiwamu's plot to convert this offset into physical units (alog 15389). An offset of -0.5cts gives about 800 pm. We were able to bring the carm offset to about 300 pm stabily.