Robert, Kiwamu,
This is a summary of the recovery of the IMC alignment after the PSL PZT mirror mount swap today (for which I think Robert will make a separate report).
Synopsis-
IMC seems to have shifted its waist location horizontally by 0.25 mm based on the change seen by the suspension witness sensors. This apparently was large enough to reduce the amount of light at IMC-TRANS by a factor of two presumably due to a worse in-vac clipping in HAM2. Nevertheless, as of now, we seem to be able to lock the interferometer on DC readout without new issues.
Recovery process-
We temporarily placed an iris after the top periscope mirror and before the beam tubing. After the swap, we checked two existing irises that had been on the table and the new one at the top of the periscope to coarsely recover the alignment. Then checking the spot on the PSL wall coming out from the light pipe between the PSL and HAM1, we did a fine alignment. This was good enough to put us back to a position where we see the main light on IMC REFL camera. A final alignment was then done by engaging IMC ASC which automatically servoed the alignment to an optimum. The temporary iris was removed before we left the PSL.
After the successful lock of IMC and successful engagement of IMC ASC, we noticed that the IMC TRANS is smaller by a factor of two. This seems to be due to a slight change in the horizontal direction in the uncontrolled degree of freedom. Here is a table listing several changes in some relevant sensors.
before | after | diff | |
MC1 PIT witness | -3 | -2 | +1 urad |
MC2 PIT witness | 510 | 507 | 0 |
MC3 PIT witness | -841 | -842 | -1 urad |
IM4 PIT | -0.33 | -0.34 | almost 0 |
MC1 YAW witness | -1035 | -1043 | -8 urad |
MC2 YAW witness | -672 | -672 | 0 |
MC3 YAW witness | -996 | -987 | +9 urad |
IM4 YAW | 0.25 | -0.08 | - 0.33 cnts |
As shown above, the only appreciable change is that in DOF2 of IMC YAW (highlighted by red texts). Using Kawazoe's formula (P1000135), one can find that this amounts to a lateral shift of the spot position by 0.25 mm toward HAM3 or away from PSL. [EDIT] Keita pointed out that the direction of the move that I initially reported was wrong. So the correct statement is that the beam shifted by 0.25 mm toward the PSL or away from HAM3.
Things we didn't optimize-
~1555 - 1605 hrs. local -> Kyle in and out of LVEA More data - I'm letting the gauge volume pressure accumulate overnight - I noticed that the pirani gauge mounted between the two in-series turbo pumps has changed again and is now showing 1.8 x 10-3 torr. This a few hours after the most recent decrease in heating power. Being positioned between these two turbos, i.e. at the inlet of the downstream turbo, this gauge should be "off-scale low" and unchanging regardless of what temperature the, now baked parts, are. If this pressure change is real (which it isn't) it would have to be due to a newly developed leak (which it isn't). The real reason to isolate the pumps from the gauge volume at this stage is two-fold; firstly, the off gassing/gas removal bang for the buck is over now that we would be pumping on it at, or near, room temperature and secondly, by doing so, we now have (2) closed valves between the site vacuum and any failure modes of the temporary pumping setup that could result in a venting of the pump line.
Frontend Watch is GREEN
For the record, these up-times are low because we've just gone through a PSL incursion to install new, stiffer, better damped, mounts for PZTs on the PSL periscope mirrors (aLOG pending, for now see ops report in LHO aLOG 31917). No problem here.
Attached are the long trends of the ETM charge measurements with this morning's data appended. Note in the first 2 plots that the ETMy charge looks slighly ramped up as it heads away from zero in all 4 quadrants - if indeed ramping, it could be due to the nice IFO duty cycle we have which drives through the ETMY L3. For this reason, Jenne wrote into the ISC_LOCK.py Guardian to set the ETMY L3 bias gain in DOWN to be the opposite of what it is in-lock in an attempt to de-charge it when not in low noise use. She also made sure the there was an appropriate re-setup step for this bias gain where needed in the guardian script.
This is a similar change that Sheila made in alog 31172 to "de-charge" ETMX via flipping the gain sign during unused times - in that case during low noise locking.
Change to the guardian script has been committed to SVN.
J. Kissel, S. Dwyer We've further modified the LOWNOISE_ETMY_ESD state to wait for this LOCK_BIAS gain change to finish ramping before moving to the transition. Also, now the transition of control to ETMY (after the bias has successfully flipped sign) happens in the run portion of the state as opposed to the main, with a couple of more counters that wait for all the appropriate 30 second ramps to finish. This means that it's going to take 30 seconds longer to go through this state. Your patience is appreciated! We've successfully transitioned over to ETMY using this further updated code.
TITLE: 11/28 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Jim
SHIFT SUMMARY: After the lockloss, we have been in DOWN for PSL PZT work, with other tasks happening opportunistically.
LOG:
16:12 Joe D beamtube patching Yarm
17:37 Lockloss, possibly due to coincident PSL Noise Eater OOR
17:58 Kyle to LVEA PT180
18:06 Jason to EX for OpLev work
18:10 Robert and Kiwamu to PSL PZT work
18:15 Joe D done
18:20 Kyle out
18:22 Jeff B forklifing chiller
18:25 Hugh to EX BRS work
18:30 Jason done at EX
18:33 Jason and Betsy to TCSy table to check flow sensor
18:54 Jason to EY OpLev, SEI_CONF to SC_OFF_NOBRSXY
19:08 Jeff B done
19:21 Jason done
20:02 Hugh done, SEI_CONF set to WINDY_NO_BRSX (Hugh said not to use BRSX yet)
22:08 Switched SEI_CONF back to WINDY since Hugh reports that BRSX is back up
22:34 Dave restarting models
23:55 Kyle to LVEA to close valve
A mini circuits amplifier was placed on the following RF signals going into the PEM QUAD I&Q Demodulator Chassis:
Cable PEM_CS_Radio_Ebay_45MHZ (AMP1)
Cable PEM_SPQ_PWR_HAM2-3_2 (AMP2)
Cable PEM CS_Radio_Ebay_9MHz (AMP3)
Cable PEM SPQ_PWR_HAM2-3_1 (AMP4)
The cables were re-routed from the demodulator chassis to the network rack (next to the PEM/OAF Rack). Four mini circuits amplifiers ZHL-1A+ were monted on a rack tray. A 5 dB attenuator was installed at the output of each amplifier.
Amplifiers are being powered by a DC power supply.
Special Monday Maintenance:
WP 6350 ALS and LSC model change
Keita, Dave:
Removed 3*2048Hz channels from each of h1alsex, h1alsey and 1*16384Hz lsc channel from DAQ Frame.
WP 6354 SUSPROCPI model change
Terra, Dave:
removed 28*2048Hz channels from DAQ frame.
WP 6353 ECR E1600351 Add GDS channels to DAQ broadcaster
Jess, Dave:
Added 41 channels to the DAQ frame broadcaster.
Additional data size is 2*16kHz and 39*256Hz. channels.
DAQ restart
Dave:
Did one DAQ restart to install the above changes.
Attached files show the DAQ channels removed from frame due to model changes, and the added channels to the broadcaster.
The online h(t) pipeline has been restarted at Hanford at approximately GPS second 116440990. This does not include any filter change, but does include the following changes:
The full command line is below.
gstlal_compute_strain
--data-source=lvshm
--shared-memory-partition=$LIGOSMPART
--filters-file=/home1/calibration/svncommon/CalSVN/ER10/GDSFilters/H1GDS_1163857500.npz
--ifo=H1
--frame-duration=4
--frames-per-file=1
--write-to-shm-partition=$HOFTSMPART
--compression-scheme=6
--compression-level=3
--control-sample-rate=4096
--expected-fcc=341.0
--partial-calibration
--coherence-uncertainty-threshold=0.004
--kappas-default-to-median
--apply-kappatst
--apply-kappapu
--apply-kappac
Significant exhaust temperature response (<-30C) 47 seconds after changing the manual mode %open value of the LLCV (liquid level control valve) from its as-found valve of 17 to 50 -> restored this value back to 17 following this exercise.
Here is the temp trend (in seconds). Not sure why the dip prior to fill. First I've seen that.
I trended CP3 exhaust TC temp gauges over 30 days to see if the dips noted above have always been there. Looks like it started on Nov. 15th. The only change to CP3 noted was a Dewar fill while LLCV was set high to 21%. LN2 was coming out the exhaust so I lowered to 16% while the LN2 truck was still filling Dewar. The previous day on the 14th I had filled CP3 by setting LLCV to 100% and noted this was too much flow. Is this an instability in pump reservoir where it burps every 20-40 minutes? Note that from Nov. 19-21 signal was smoother. Could be a function of LLCV setting and how full exhaust line is. Fills lately have been really fast.
CP3 exhaust is also noisy since Nov. 21st.
The DMT calibration code has been updated to gstlal-calibration 1.0.9-1.el7. This produces the low-latency GDS hoft.
Numerous entries of the last few days mention the BRS X being down.
First notice is LHO aLog 31792 when PEM went to EndX. A couple hours later Jim mentioned that it was very rung up and he disabled the damping. On Thursday Jim worked on the BRSX but was unsuccessful at getting it to damp reliably. On Saturday and Sunday, TJ logged that the BRS was ringing down--it was but just passively.
This morning after reenabling the Damping, it was going nowhere. The Damping neither helped nor rung it up further so I'm unsure what it was doing.
After IFO dropped lock, remote logged into the BRS computer and looking at the encoder file showed it contained gibberish. At the End Station, opened up the box and saw the Damper Masses sitting at the 0 position. I thought I had cleared out the gibberish in the Encoder file but upon restart, the damper just started 360s, this might be what was going on for the past few days.
With Krishna on the phone and some guidance that needs updating, I managed to clear the encoder file and reset the damper encoder angle. Once this was successfully done, the damper quickly did its job and got things under control.
For the operator--see T1600103 for some troubleshooting. After today's lesson, this needs some updating, and it contains nothing about the Encoder file problem. This is on my to-do list.
When I went to the end station on Wednesday, the damper table wasn't moving at all because it had been disabled. When I came in on Thursday, I got it to turn on once, and it seemed to servo normally for a while, but then it stopped and I couldn't recover it. It may have gone crazy on Wednesday afternoon while Anna-Maria and Robert were down there, but it wasn't moving at all after Wednesday night.
Jim's observation is not inconsistent with our conclusion that the problem was with the corrupted encoder file, I think. What I am more curious about is what caused this corruption in the first place. After the damper was upgraded, the last time we had a similar problem was when we had a series of restarts of the Beckhoff computer (see 29871). It would be interesting to look at the history of BRS status bits over the last ~2 months to see if we can get a clue.
Attached trends are all of the "bit" channels I could find associated with BRSX. First trend is the last 10 days, second is the last 90. Where the AMPBIT shows excursions away from 1 lines up with the two times when the encoder file has been corrupted. If this is diagnostic, maybe we could monitor this in the DIAG guardian.
Paul Altin
Full DQ shift report here.
Nutsinee's Saturday report of an alog no longer being viewable and needing to be reposted reminds me that anyone can subscribe to "L-mail" and get an email everytime an entry is created and edited (for both main entries and comments). If an entry goes missing, you can easily reconstruct the entry from raw HTML via your email application.
Thanks, Dave! I was wondering how to do this.
As a further note, there are few instances where the entry is actually gone unless you go out of your way to delete it. There's an issue related to changing the Section/Task on a comment to mis-match it from the parent report (causes it to not display inline in the normal page -- short story, don't change the section/task, only modify with tags on your comment). If you're missing a post check your drafts for the missing report or comment and flip its status to posted if that's your intent. This appears to be the case with #31847.
After John Worden noticed multiple gauges were flatlined, we discovered that pulling the fuse to depower PT180 gauge for bake-out inadvertently flatlined LX beckhoff vacuum system this past weekend, including CP2 PID loop. CP2 LLCV locked at 60% open (1.7x its nonimal value) due to integrator high limit. Strange that the % full dipped from ~92% to 91.7% (Kyle noted a step change in PT120 (cold cathode gauge) after pulling PT180 fuse). Bubba noticed significant ice build up on exhaust line this morning. Now we know why. There was significant LN2 consumption as a result of overfilling for four days. Dewar level fell from 49% to 32%. The exhaust pressure was also not reading actual values. Thanksfully we opened all CP exhaust bypass valves as new nominal state last week to prevent overpressurization!
There was no indication of faults from the vacuum overview MEDM screen. PT180 appeared to be powered with a reasonable pressure value. The LVEA subscreen, however, did show a fault for PT180. I filed FRS 6772 to have faults moved to front overview page. We will work with Patrick to create an alarm handler for future events where perhaps a fuse blows on its own.
PT180 was manually disabled in beckhoff this morning which caused the other components to read as-is, showing PT180 reading red and zero and CP2 and LX gauges to reflect their faults. Richard will write an instruction manual so we can do this in the future for planned events.
My suggestion for the first thing to do would be to add the corresponding error channel to the phone/text alarm handler for each vacuum channel already in the phone/text alarm handler.
Once again I must profess my gratitude for this amazing team!
Two addenda.
Firstly, to cope with the fact that IMC TRANS decreased by a factor of two, I have edited the IMC_LOCK guardian and lowered all the FM_TRIG_THRESH_ON and _OFF values by a factor of two. In addition, I manually changed the IMC-MCL_TRIG_THRESH_ON and _OFF values by a factor of two as well. They don't seem to under control of any guardians. IMC locks fine with these new settings. The guardian is then checked into SVN.
Secondly, the spot positions on the PSL wall seem to have shifted by 1-2 mm towards the West. No obvious change was found in the vertical direction. See the attached picture. The new positions are recorded with black 'X' marks as shown in the picture.
Jenne and I did a repeat of what we tried a few weeks ago after different PSL work: (30918) we restored the optics to their old values using the witness sensors, then moved the piezo to maximize build up without turning on the IMC WFS. This brought the spot back to its before position on IM4 trans, although the MC2 trans sum was low, so we think that as expected only the input beam has moved, not the mode cleaner optics or IMs. However, we can't fix the input beam change simply by moving the PZT.
We let the MCWFS run to increase the mode cleaner transmission, and watched the spots on both the ISS QPD and IM4 trans. We walked IM3 and IM2 in yaw to bring both QPDs back to the spot positions before this morning's PSL work, and now Cheryl is doing inital alingment.
To move this degree of freedom, we moved IM2 1.284 urad in yaw for every urad that we moved IM3. Since this is a degree of freedom that our inital alingment and ASC don't control, it may be a good idea to try moving this degree of freedom in full lock to see what impact it has on our noise and recycling gain. For the record, today we moved IM3 -2390 urad, and IM2 -3030 urad.