When we center the REFL DC with the servos, we often find that the RM1 has tripped. This was true yesterday evening, and today Jan 27th, 2015 19:43 UTC. No investigation has been done by the commissioning team.
This noise source (pump) will likely be running for the next 6 - 8 weeks
Doug went to the floor and used a rope to pull the instrument air line etc. more from the oplev pyron and the 10-min oscillation is gone.
alog from yesterday: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=16276
This is yet another temporary fix that is good enough for a while, just like the original "fix" which stayed until today. This sounds like a good target for the detector group to keep track of.
Last week I found all of the BSC ISIs with their St2 GS-13s in low gain. Looking in conlog, I found that the gains had been switched at 07:05 PST on the 22nd, then I found the error and corrected it at 08:59 PST on the 23rd, meaning there was an entire night where the all of the ISI's were running in a different from nominal state. This would be a good window to look at the performance
Additionally, last week, Arnaud tried running the TM ISI's with all of their RZ loops turned on. The largest window I found for this was from about 10:20 to 20:00 PST.
All BSC ISI's have otherwise been in their nominal configurations since then (ie RZ loops off, rdr blends, sensor correction on, all instruments in high gain (except the BS)).
J. Kissel, A. Pele (WP #5025) We are to make the IFO's fast shutter functional within the next week, in order to prepare and protect the OMC DCPDs from the energy leaked from a FULL IFO lock lock. In its current design / implementation, this shutter is fast and loud as it closes and reopens, which sends large impulses to the optical table on which it sits (in this case H1 ISI HAM6). In order to prepare for this "toaster" being activated, I've increased the number of consecutive saturations (computer cycles above the user-predefined threshold) that are acceptable for the H1 ISI HAM6 GS13s and Actuators before its watchdog trips to 4096 [samples] / 4096 [Hz]= 1 [sec] and 8192 [samples] / 4096 [Hz] = 2 [sec], respectively (where it had previously been 10 [samples] / 4096 [Hz] = 0.0024 [sec] for these channels). See attached screenshot of the relevant WD block innards. This has been done and in place at LLO for a few weeks with no obviously adverse affects, (see 16430 and 15998), we were merely replicating their change. As such, they've seen a drastic reduction in HAM6 ISI trips after lock-losses, and we hope to see the same benefits. Note, the change involves *unhooking* (disabling and breaking) the /opt/rtcds/userapps/release/isi/h1/models/h1isiham6.mdl model link to the generic HAM ISI model library part /opt/rtcds/userapps/release/isi/common/models/isihammaster.mdl and replacing the WD_SAFE_COUNT tag input to each GS13 and ACT blocks (inside the HAM6 WD block) with hard-coded constants of 4096 and 8192. I've committed the unhooked model to the repository. For the restart, I - Brought the OMC suspension to SAFE - Turned off all output to OMs 1-3 - Asked the SEI_HAM6 manager to take the chamber to DAMPED - Paused the SEI_HAM6 manager (to make sure ISI_HPI is left alone and happily running) - Changed the ISI_HAM6 subordinate to exec - Requested ISI_HAM6 subordinate to READY - Turned OFF ISIHAM6's master switch - Recompiled, Reinstalled, Restarted h1isiham6 front end process on h1seih16 computer - Requested ISI_HAM6 subordinate to DAMPED, confirmed IOP output - Changed the ISI_HAM6 subordinate back to managed - Changed the SEI_HAM6 manager back to exec (who came back, happily fooled that it never left the DAMPED state) - Requested SEI_HAM6 manager to return to ISOLATED. - Restored OMC suspension to ALIGNED - Restored alignment and damping to OMs 1-3 (and a random Y offset of 2000 [ct] on the TEST filterbank). Note, also, since I merely changed constants in the model, the change did NOT require a DAQ restart. I also attach trends of the P and Y of the HAM6 suspensions (as measured by their local OSEM sensors), confirming that they've been restored to the same alignment.
J. Kissel Proving that watchdog still over-protects, the watchdog tripped twice after I had made this model upgrade. It has tripped on both on CPS saturations (still set to 10 [samples]) *and* the modified GS13 (now set to 1 [second]). Attached are screen-shots of the trips. No clue what the cause of the trips were, but it was maintenance day and several people were in and out of the LVEA (canned guess without any evidence to support this was it). Trip Times: GPS 1106417935.822998 (GS13s) GPS 1106416251.409424 (CPS)
Loaded the updated tidal code into iscex/y.
I will be doing some work on the connection between CDS and GC this morning which will result in some brief interruption to access to CDS from the outside world (and vice versa). This includes any services hosted at lhocds.ligo-wa.caltech.edu. A follow up entry will be posted when work is complete.
Work is complete, things should be operating as normal again. Also, I omitted the work permit number in my original entry, that is WP5022.
Zero'd the Match gains starting at 1549utc. Jim reports the X leg of the sensor dropped by ~1/2 on 23 Dec when it appears the zero button was pressed. We will press it again to see if that might bring it back. I can't find a log of a button press but we had been struggling to get that STS2 signal down to reasonable levels. We're not sure this will work but we'll see.
model restarts logged for Mon 26/Jan/2015
2015_01_26 00:02 h1fw0
2015_01_26 04:29 h1fw1
2015_01_26 05:28 h1fw1
all unexpected restarts. Conlog frequently changing channels report attached.
Switched to manual control at 1510utc. Made one setpoint tweak at 1514. It looks like it will probably run without further adjustment (maintaining the ~70psi differential pressure) for a hour or more.
J. Kissel, H. Radkins, Corner Station HEPI Pump Servo Control was restored at 9:21 PST (17:16 UTC). For the record -- wasn't collection an Open Loop Transfer Function as indicated by the title, we merely wanted a few hours of data with the pump servo off to assess out-of-loop noise and impact on platform motion.
Alexa, Elli, Sheila, Rana, Evan
Today we worked some more to make the sqrt(TRX+TRY) handoff more robust.
Now we can pretty reliably complete the transition by hand. We are working on implementing the transition in the guardian.
We have tried to reduce the CARM offset while locked on sqrt(TRX+TRY), but we cannot seem to get beyond 2 or 3 times the single-arm power without blowing the lock. The next step is to go through the CARM reduction sequence more carefully and characterize the OLTF of the CARM loop in this state.
As discussed in LHO#16252 et seq., we suspect that the common-mode board has gain-dependent offsets. If this is true, it explains why the interferometer can be knocked out of lock while ramping down or turning off ALS COMM.
To boost the CARM loop's immunity to these offsets, we redistributed gains as follows:
LSC-REFL_SERVO_IN2GAIN). Correspondingly, the gain on the AO into the IMC board was reduced from +16 dB to −4 dB (IMC-REFL_SERVO_IN2GAIN).LSC-REFLBIAS_GAIN was increased from 0.8 to 8).
Even with these changes, success is not guaranteed. The gain steps can be heard very clearly when listening to LSC-CARM_IN1, and turning off ALS COMM on the summing board will blow the lock about half the time. We can get better results by using the gain slider on the common-mode board, but we can still hear the gain steps.
LSC-REFLBIAS, we engage FM3 (1 Hz pole, 40 Hz zero) and FM9 (1.6 Hz pole, 40 Hz zero). The gain is 8 ct/ct.LSC-TR_CARM, we set the offset to −0.5 ct.LSC-REFL_DC_BIAS with a gain of 30 ct/ct.LSC-REFL_DC_BIAS to 50 ct/ct.A lockloss plot is attached for an unexplained lock loss during the CARM offset reduction after handing off to sqrt(TRX+TRY). Other lock loss times, all 2015-01-27 UTC: 03:56:53, 04:05:25, 05:36:16, 06:40:59, 07:56:02, 08:37:20. The last four are during CARM offset reduction after the handoff is complete.
For a sqrt(TRX+TRY) offset of −1.5 ct, the gain we need for REFL_DC_BIAS is 50 ct/ct. Our OLTF looks good here. However, when we reach an offset of −2, we seem to lose lock, even though there is no obvious nastiness in the CARM spectrum.
Today, locking DRMI without arms was pretty painless. In contrast, DRMI+arms lock acquisition was very, very slow for most of the morning and afternoon. After about 5pm local, it became painless as well. This may have been correlated with our changing the trigger settings to be twice as high with arms (compared to no arms), since the POP18 buildup is twice as high. But we haven't investigated this systematically.
Last night Sheila requested DRMI_LOCKED on the ISC_LOCK guardian when she left for the night at 1am to che k if this configuration is stable.
DRMI with green in the arms and IR held off resonance stayed locked overnight, and the power slowly degraded untill 4.30am when the lock dropped. Cuardian relocked DRMI untill 6am when DRMI lock dropped and did not recover. This shows that this configuration is fairly stable.
After changing our ALS gain rampdown to be in the CM board rather than ahead of it, we never broke lock due to the rampdown. However, we only tried it in this new way twice.
The lock loss plot attached above was typical of our current lock loss mystery. After moving to a new CARM offset, the CARM error signal seems stable (no peaking in the spectrum) and the loop shape looks good. The lock breaks without any characteristic sound - just a sudden lock loss. Investigation of the 5 LSC error and control signals shows no instability in the 100 ms before lock loss. The lock loss happens several seconds after the CARM offset is done ramping. Since the signal from the Thorlabs Transmon PDs is only 2000-3000 counts, we think that they are not saturating.
No guess yet about what is happening. Any speculations on lock loss causes is welcome.
We struggled with the ETMX bounce mode all of today. It seems to have started ringing up around 3 AM last night (according to the ETMX OL 3-10 Hz BLRMS trend). We spent a couple of hours damping it around noon today, but then it slowly started growing again today. We've now installed a 60 dB stopband filter centered at 9.77 Hz in the OL loops to see if this will stop the ringups. We have also installed a resonant gain filter for 9.77 Hz in the CARM loop to reduce the arm power fluctuations during the offset reduction.
To better monitor the bounce mode, we set up the ETMX OL lockin screens to demod the OLPIT signal at 9.67 Hz. So now one can trend the 0.1 Hz beat note in the lowpassed output of this to see what the Bounce mode peak height is at all times. Probably should make a dedicated BLRMS for each suspension with an OL to monitor its bounce mode height.
I am taking a look at the lock losses Evan listed in this entry. All the correction signals seem good except the BS one. This is, for example, lock loss 4_05 UTC, with different zoom of the correction signals during CARM offset reduction. Probably a campaign of loop measurements for the vertex DOFs can help.
More lock loss science. As far as I can tell, 5:36 and 6:40 are similar to the previous lock loss 4:05, while in 7:55 looks like CARM is indeed the culprit..
(BS oplev seems OK)
Given the prior lock loss science by Lisa, I speculated that the BS oplev loops were doing something bad such as glitches. (Note that the people did not use the ASC loops last night so that the oplev damping loops on BS had been engaged all the time). Looking into the last five lock losses that Evan posted, I am concluding that the oplev damping loops were not glitching or disturbing the MICH loop. The attached are 60 seconds full time series of various BS oplev-related channels for the five lock losses. The 4:05 event is the only one which clearly showed DAC saturation and the rest of them did not saturate the BS DAC before the lock loss. The oplev sum sometimes shows a fast transient, but it happens right after each lock was lost -- indicating that the transient was caused by the motion on the oplev QPD as the BS was kicked and it is NOT initiating the lock loss.
(TRX seems always lower than TRY)
I don't know if this is related to the cause of the lock losses, but I found that TRX have been consistently lower than TRY by roughly 10 % regardless of how big the CARM offset was. It is unclear if this discrepancy is from an unintentional offset in the ALS diff operating point or some kind of calibration error in TRs. In any case, we should fix it in order to reduce DARM coupling in the TR_CARM signal path.
What is the CARM offset and offset reduction in physical units? (pm or Hz of the arm cavitites)
Peter, we can use Kiwamu's plot to convert this offset into physical units (alog 15389). An offset of -0.5cts gives about 800 pm. We were able to bring the carm offset to about 300 pm stabily.
Similar to the HEPI's done last week, I took the text file generated by Barker, which is extracted from the Guardian logs, and removed the truncated channels from that list. The MASTERSWITCH was also added to the list. This I used for the do not monitor channel list. I saved this file with the safe.snap files in /opt/rtcds/userapps/release/isi/h1/burtfiles.
Next the safe.snap is modified to include the monitor bit on all channels not RO and not fields. This is then loaded into the FE with the LOAD TABLE ONLY button.
After confirming a successful load and noting the DIFF list, the safe.snap is then modified with the donotinclude list. When theTABLE is LOADED again, most of the diffs go away and the remaining differences represent the differences we care about wrt most recent safe.snap file. Either the channel is set wrong or the snap needs updating.
All HAM-ISI are good except HAM3, there remains some blend diffs as we test configs for the 0.6hz peak issue.
I'll work on the BSC-ISI soon.
This exercise exposed a problem: if you have more than 40 diffs, it only shows 40 and it only lists 40. It makes you think there is only 40 when there are more. I've filed bug 796.
I have confirmed that the guardian reports (displays a notification) when the filter module is not in the correct state. This may not be true for other subsystem Guardians. Betsy and I saw this was the case on some ETM filter module that Dave's list reports that guardian touched. The monitoring that Guardian does for the SEI channels may need to be coded explicitly.
This has nothing to do with the new SDF functionality, nor has guardian behavior changed. The SEI guardian systems were programmed from the beginning to notify on changes to any of the controller filter banks. This is specific to SEI and is not being done by any of the other subsystems.