Significant exhaust temperature response (<-30C) 47 seconds after changing the manual mode %open value of the LLCV (liquid level control valve) from its as-found valve of 17 to 50 -> restored this value back to 17 following this exercise.
The DMT calibration code has been updated to gstlal-calibration 1.0.9-1.el7. This produces the low-latency GDS hoft.
Numerous entries of the last few days mention the BRS X being down.
First notice is LHO aLog 31792 when PEM went to EndX. A couple hours later Jim mentioned that it was very rung up and he disabled the damping. On Thursday Jim worked on the BRSX but was unsuccessful at getting it to damp reliably. On Saturday and Sunday, TJ logged that the BRS was ringing down--it was but just passively.
This morning after reenabling the Damping, it was going nowhere. The Damping neither helped nor rung it up further so I'm unsure what it was doing.
After IFO dropped lock, remote logged into the BRS computer and looking at the encoder file showed it contained gibberish. At the End Station, opened up the box and saw the Damper Masses sitting at the 0 position. I thought I had cleared out the gibberish in the Encoder file but upon restart, the damper just started 360s, this might be what was going on for the past few days.
With Krishna on the phone and some guidance that needs updating, I managed to clear the encoder file and reset the damper encoder angle. Once this was successfully done, the damper quickly did its job and got things under control.
For the operator--see T1600103 for some troubleshooting. After today's lesson, this needs some updating, and it contains nothing about the Encoder file problem. This is on my to-do list.
When I went to the end station on Wednesday, the damper table wasn't moving at all because it had been disabled. When I came in on Thursday, I got it to turn on once, and it seemed to servo normally for a while, but then it stopped and I couldn't recover it. It may have gone crazy on Wednesday afternoon while Anna-Maria and Robert were down there, but it wasn't moving at all after Wednesday night.
Jim's observation is not inconsistent with our conclusion that the problem was with the corrupted encoder file, I think. What I am more curious about is what caused this corruption in the first place. After the damper was upgraded, the last time we had a similar problem was when we had a series of restarts of the Beckhoff computer (see 29871). It would be interesting to look at the history of BRS status bits over the last ~2 months to see if we can get a clue.
Attached trends are all of the "bit" channels I could find associated with BRSX. First trend is the last 10 days, second is the last 90. Where the AMPBIT shows excursions away from 1 lines up with the two times when the encoder file has been corrupted. If this is diagnostic, maybe we could monitor this in the DIAG guardian.
Paul Altin
Full DQ shift report here.
Nutsinee's Saturday report of an alog no longer being viewable and needing to be reposted reminds me that anyone can subscribe to "L-mail" and get an email everytime an entry is created and edited (for both main entries and comments). If an entry goes missing, you can easily reconstruct the entry from raw HTML via your email application.
Thanks, Dave! I was wondering how to do this.
As a further note, there are few instances where the entry is actually gone unless you go out of your way to delete it. There's an issue related to changing the Section/Task on a comment to mis-match it from the parent report (causes it to not display inline in the normal page -- short story, don't change the section/task, only modify with tags on your comment). If you're missing a post check your drafts for the missing report or comment and flip its status to posted if that's your intent. This appears to be the case with #31847.
I have been investigating the spike in the computed value of kappa_tst shown here in the summary pages: https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20161122/cal/time_varying_factors/ A closer look reveals a seeming correlation between the DARM line coherence and the values of kappa_tst (see the attached plot). Currently, kappa_tst computed values are gated with PCALY_LINE1 and SUS_LINE1, the only lines used to compute kappa_tst. Things to note in the plot: 1) kappa_tst diverges from the expected range about a minute after the DARM line coherence uncertainty skyrockets, just the amount of time it takes to corrupt the 128 running median. 2) kappa_tst as computed by CALCS also diverges when the DARM coherence goes bad. 3) Ungated kappa_pu behaves similarly to kappa_tst, but since it is gated with the DARM coherence, the GDS output is not corrupted.
Here is a plot of offline-calibrated data that includes the same time, this time adding DARM cohreence gating of kappa_tst. Note that the the GDS timeseries now looks good, and kappa_tst is well behaved.
After John Worden noticed multiple gauges were flatlined, we discovered that pulling the fuse to depower PT180 gauge for bake-out inadvertently flatlined LX beckhoff vacuum system this past weekend, including CP2 PID loop. CP2 LLCV locked at 60% open (1.7x its nonimal value) due to integrator high limit. Strange that the % full dipped from ~92% to 91.7% (Kyle noted a step change in PT120 (cold cathode gauge) after pulling PT180 fuse). Bubba noticed significant ice build up on exhaust line this morning. Now we know why. There was significant LN2 consumption as a result of overfilling for four days. Dewar level fell from 49% to 32%. The exhaust pressure was also not reading actual values. Thanksfully we opened all CP exhaust bypass valves as new nominal state last week to prevent overpressurization!
There was no indication of faults from the vacuum overview MEDM screen. PT180 appeared to be powered with a reasonable pressure value. The LVEA subscreen, however, did show a fault for PT180. I filed FRS 6772 to have faults moved to front overview page. We will work with Patrick to create an alarm handler for future events where perhaps a fuse blows on its own.
PT180 was manually disabled in beckhoff this morning which caused the other components to read as-is, showing PT180 reading red and zero and CP2 and LX gauges to reflect their faults. Richard will write an instruction manual so we can do this in the future for planned events.
My suggestion for the first thing to do would be to add the corresponding error channel to the phone/text alarm handler for each vacuum channel already in the phone/text alarm handler.
Once again I must profess my gratitude for this amazing team!
Kyle pulled the fuse for the LX PT180 Inficon BPG402 gauge on Nov. 22 (alog 31814). For reasons not entirely clear to me, this put all of the input channels for LX into an invalid state with their values flatlined. In Beckhoff this error was was only indicated by the WcState for each channel. To fix this, in the TwinCAT software on h0vaclx, I right clicked on 'Pressure Gauge PT180 (BPG 402)', clicked on 'Disable', then went to the 'TwinCAT' menu and ran 'Activate Configuration'. When it asked whether to restart in run mode I said yes. This ended up crashing the EPICS IOC, which I restarted. I burtrestored the IOC to 6:00 AM on Nov. 22 (local time). For each readback channel there is a channel with the same name that ends in _ERROR. These channels reported this condition, however they are not on any of the overview MEDM screens or in any alarm handler. This should be changed. So currently, the PT180 pressure gauge is powered off and disabled in Beckhoff. The other LX channels should be valid again.
My suggestion for the first thing to do would be to add the corresponding error channel to the phone/text alarm handler for each vacuum channel already in the phone/text alarm handler.
Following up on the DetChar report of ETMx oplev laser glitching causing problems, I confirmed (using the DetChar summary pages) that the laser was indeed showing signs of glitching. I also noticed the ETMy laser was also showing signs of glitching. I increased the power for both lasers to hopefully get them into a thermal state they are happy with. I used the Current Monitor (outputs a voltage to monitor the current delivered to the laser diode) port on the back of each laser as a guide. The changes were:
Seeing as I had to open the cooler for both lasers they will both need 4-6 hours to return to thermal equilibrium. At that point a determination can be made as to whether or not this helped at all. The attached 3 hour trend of the oplev SUM for both optical levers gives hope; after the power increase the SUM signal is quieter for both oplevs. I will leave WP 6348 open for now as more adjustment may be necessary to fix the glitching.
Further power adjustment for both lasers this morning as both were still showing signs of glitching (better than yesterday morning, but not gone). Changes are:
As before these lasers will need ~4 hours to return to thermal equillibrium before I can assess whether or not this latest tweak helped.
I have turned of the "silent lines" (suspension actuator lines canceled by Pcal lines) at ~18:45 Nov 28 2016 UTC. All values for the line frequencies, amplitudes, and phases have been zeroed. These changes have been accepted in the observe and safe SDF files for both SUSETMY and CALEY.
Had doubted what I had been told last week by an INFICON service tech. that our gauge could only tolerate 80C bake when the external electronics were removed. I was able to confirm this in INFICON's published Operating Manual BAYARD-ALPERT PIRANI GAUGE tina46e1-a (2010-03) which also specifies this 80C limit even when the external electronics are removed (why?)-> I don't see anything in the materials exposed to vacuum list that would explain this low temperature limitation so I sent them an email requesting that they confirm.
Todd S. from INFICON responded to my email request and confirmed the 80C bake temp. limit even in the absence of the external electronics - > OK, fair enough.
A follow up email from company tells us that there is a build-in calibration board at the sensor.
I've scheduled 5 more burst injections. They're spaced 70 minutes apart. The first one begins at 8:10 UTC (12:10 PT). Here are the updates to the schedule file:
At LHO, the first two injections failed due to an unlocked IFO. The last three were successful. However, the last two injections (ADI-e_12_0.4.txt and ADI-e_16_0.9.txt) were the only ones coincident with L1.
The relevant sections of the INJ_TRANS log are attached.
For some reason a +50 degree filter was turned on in the ETMY top mass damping on October 12th, it was not good and we got no real top mass damping. Nutsinee and I just turned it off, so the settings are now the same as for ETMX. This fixed the problem
J. Kissel, B. Weaver We've investigated what we could of this, but there is no aLOG evidence that these phase adjustment filters were intended to be used in the design (what's now left is the same semi-low noise design I made a few years back: see G1300537). They also don't make any sense upon first glance individually, but after staring at the design strings and names it has slowly become clear. We can also narrow down who would attempt such sophisticated trickery without aLOGging, maintaining SDF, committing filter changes to the svn repo, or otherwise cleaning up after they're done... There are four filters that I've now cleared/completely removed from the H1SUS ETMY M0_DAMP_V filter bank: FM6 BS1_4 ellip("BandStop",5,1,40,1,4) FM7 -90deg zpk([-9.8],[9.8],1,"n") FM8 +50deg zpk([5],[50],1,"n") FM9 bouncetest ellip("BandPass",4,1,55,9.5,11)gain(45,"dB") Guessing what the intent was by the frequencies in question it looks like someone was exploring damping the highest vertical mode (a.k.a. "*the* bounce mode") from the top mass, using top mass OSEMs as the error signal. I'm not surprised it didn't work! Potentially related aLOGs: Corey's midnight issue on Thanksgiving: LHO aLOG 31828 My cleanup of ODC state vector Oct 25, where I blindly accepted the new state as OK: LHO aLOG 30849 The last attachment shows a trend of the filter state. The bottom trace is merely a report of the current filter state, and the top trace shows the user-entered EPICs record that is compared against when establishing whether ODC thinks there's OK status. The bottom trace confirms Sheila's suggestion that the filter bank status was changed on Oct 12th 2016 at ~16:50 UTC (which is Oct 12 2016 09:50:00 PDT).
Here is the temp trend (in seconds). Not sure why the dip prior to fill. First I've seen that.
I trended CP3 exhaust TC temp gauges over 30 days to see if the dips noted above have always been there. Looks like it started on Nov. 15th. The only change to CP3 noted was a Dewar fill while LLCV was set high to 21%. LN2 was coming out the exhaust so I lowered to 16% while the LN2 truck was still filling Dewar. The previous day on the 14th I had filled CP3 by setting LLCV to 100% and noted this was too much flow. Is this an instability in pump reservoir where it burps every 20-40 minutes? Note that from Nov. 19-21 signal was smoother. Could be a function of LLCV setting and how full exhaust line is. Fills lately have been really fast.
CP3 exhaust is also noisy since Nov. 21st.