{Kyle, Gerardo, Chandra}
We spent Tuesday maintenance morning hunting for a big air leak causing the pressure in vacuum system to bottom out at mid e-8 Torr range, and finally found what we were looking for. The leak measures 7.3e-5 Torr-L/s of He at an original 12" CFF blank flange next to NEG #1. This is a new leak since the post O2 vent and we are still unsure how to started. The N2 gas load from this range of leak even with all six ion pumps ON (15,000 l/s total) may be too big to ignore beam tube gas noise (according to M. Zucker), so a vertex vent may be required before O3. If all goes well with the fix, we could vent and be pumping back down in one day and open beam tube gate valves ~ one week later.
The procedure for leak testing included soft closing GV 1,2, valving in main turbo, backed by leak checker (background measuring 1e-8 Torr-L/s of He), valving out four IPs and three NEGs, and then spraying flanges for many seconds each with helium. We started spraying the (new) NEG adaptors (BT side of their isolation GVs) since we saw an increase in signal last time we leak checked near the RGA, which is very near this 12" blank. Gerardo noticed an increase in signal when Kyle was spraying around NEG 1 and asked him to spray that 12" blank where the He signal rose significantly. In the end we bagged the conflat joint with ameristat and sprayed until the He signal maxed out at 7.3e-5 Torr-L/s. We also sprayed the weld at the joint where the port tube mates with beam tube, and also the short tube seam, and did not see a rise in signal. The leak is likely coming from the copper gasket knife edge.
We valved the four IPs back into main volume but will leave NEGs valved out due to high N2 load. We scanned the RGA the entire morning to capture He rise. Text file attached - not - file size too big.
See the attached map. I added another sensor 10' downwind from the previous 'most' downwind sensor. The _4 was shifted downwind and the now on-line _3 is 10' downwind from the fence--just look at the map.
Thanks Hugh! Would you like these new sensors to be added to the DetChar summary pages? Unfortunately the LHO cluster is down at the moment, so I can't find an example page for you, but if you would like the sensors to be added please let me know.
After the install of the new long range RFM, Dave and I have worked on adding channels and code to calculate common mode ground motion, to eventually use for improving earthquake lock robustness. Right now both endstation ISI's are sharing their ground beamline supersensor (gnd sts - brs), ground Z, ETM suspoint length and St1 beamline CPS. The first two channels are needed for the common mode calculations, the suspoint is used for the OAF suspoint cavity length calculations. I don't know what the CPS is used for, but I kept it in the list of channels being shared. Previously all 4 of these channels were shared via muxing through the endstation PEM models, but only the suspoint channel was being used for anything. I've now added some code to the OAF to take the corner and endstation ground seismometers and calculate common mode ground velocities (i.e (ex_z+ey_z +cs_z)/3= site-wide common mode z ground motion). The plan is to eventually take these cm signals and subtract them from the ISI sensor correction and only isolate the platforms from the local differential motion, but that will require more invasive ISI & HEPI model changes. Attached screen shot shows the section I added to h1oaf. I also added 3 256hz dq channels for the cm x/y/z outputs. After Dave restarts the models I'll throw together an MEDM for the filter banks. The plan for now is to just figure out how to get the common mode calculations right and get data to try some offline subtractions.
Rebooted the machine at 18:54 UTC (11:54 PST). All nodes naturally recovered except SUS_IM4 and SUS_ETMX. These nodes did not time out like we have seen in previous reboots, due to the timeout duration being extended (alog44092). Instead, the logs show that the nodes started and then stopped after "terminate called after throwing an instance of 'int'". I'm not sure where this error is coming from yet. A restart of the nodes recovered both.
Screen shots of the status logs and grd logs of the two failed nodes are attached.
Also, zotws20 cannot run many of the guardctrl commands. I ssh'd into zotws6 and "list -s" ran just fine. They are both running Debian 9.5. but the kernal release and version are different:
zotws20 = 4.9.0-7-amd64 | #1 SMP Debian 4.9.110-3+deb9u1 (2018-08-03)
zotws6 = 4.9.0-8-amd64 | #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21)
Attached a screen shot of the error.
This has nothing to do with the kernel. I'm not sure if it's related, but I note that the site environment variables are not being set appropriately on this machine. In particular the GUARDCTRL_HOST variables is not being set properly to point to "h1guardian". This might be a red herring, but after setting the variable correctly I can see the listing fine:
jameson.rollins@zotws20:~ 0$ GUARDCTRL_HOST=h1guardian guardctrl list
ALIGN_IFO
ALS_COMM
ALS_DIFF
ALS_XARM
ALS_YARM
BRSX_STAT
BRSY_STAT
DIAG_CRIT
DIAG_EXC
DIAG_MAIN
DIAG_SDF
FAST_SHUTTER
HIGH_FREQ_LINES
HPI_BS
HPI_ETMX
....
Danny, Hang, Stefan, Daniel Daniel hooked up two IFR function generators to simulate the ALS DIFF and COMM beat notes. 1) Just injecting a lot of noise (up to 100kHz broadband) into ALS_COMM did not produce enough cross-talk - only about 1/200 of the injected signal in COMM shows up in DIFF (plot 1) 2) However, when we started to simulate COMM noise by lowering the beat note strength enough that the PLL started missing beats, the noise also showed up in DIFF at about 1/4th the strength (about the ratio we see when COMM locks) (plot 2). 3) In situation 2, changing the COMM PLL feed-back gain actually reduced the noise coupling - I don't know whether that was already tried with the arm signal. In all plots, the references are from arm locks yesterday. The COMM is always noisy (brown). DIFF has a noisy (dark blue, COMM locked) and a quiet (black, COMM unlocked) reference.
At 10:08 UTC (03:08 PDT) it looks like h1lsc0 kernel panic'ed. Strangely we are getting bad DAQ data status from a strange collections of machines (all end stations which are on dolphin, corner ISC and susauxb123!) see attached image.
h1lsc0 console is showing a kernel panic error (I'll post it later).
I'll recover all systems execpt LSC before we start today's upgrade.
Opened corresponding to FRS Ticket 11528. Likely we can quickly close the ticket, since the system has been recovered, but I leave it open for Dave to comment as he sees fit.
Here is the list of things I did until I was able to reset the DAQ status of these front end machines:
h1susauxb123: start streamers
h1lsc0: power computer down (after disabling its ix-dolphin switch port)
DAQ: restart
h1susauxb123: computer reboot. This system came back with no errors
h1asc0: computer reboot: This system came back with no errors
At this point, I could fix the DAQ mx stream on all the other models just by issuing a /etc/start_streamers.sh command.
Entered the enclosure to find the status screen indicating that the NPRO had fainted however
the NPRO power supply indicated that the NPRO was on and the laser was up and running. Not
sure what the discrepancy is due to.
I left things the way they were.
The LVEA has transitioned to LASER SAFE.
This is under work permit #7836.
A note in passing ISCT6 and SQZT6 doors I locked closed.
Daniel, Nutsinee
Today we installed a phase frequency discriminator chassis (U7 on SQZ rack near ISCT6). The pump/PSL beat note RF signal and 158MHz LO went in and the output went to OPO common mode board. We unplugged the PZT output and only used the fast path error signal to feed into the 158MHz VCO. The 158MHz VCO now follows the beat note frequency. Here's a quick look at the control signal. More noise hunting+modeling is required to be able to pinpoint where all of them came from. We suspect temperature to be responsible for low frequency noise.

Sadly locking at 0 Hz beat note doesn't mean high 3MHz peak when using PSL LO. A feed forward from IMC land is required. A model has been modified to sum the OPO fast control signal with this feed forward signal and the output will go to the OPO PZT somehow (don't remember what we decided on). Once this is complete then we can work on the 3MHz hand-off.
We started having trouble handing off in DRMI_TO_POP. - The symptom is a ~10000ct offset in H1:LSC-POP_A_RF9_I_NORM_MON just before hand-off. - We checked all the phasing of POP9 and REFL 9,27,135 - they all look reasonable. - However the whole thing seems to be related to a CARM fringe offset. The offset only appears when switching CARM_TO_ANALOG. - Moreover, while sitting at the end of CARM_TO_ANALOG, we jiggled the common mode board gain, and suddenly the offset dropped out. I am suspecting some sort of saturation in the common mode board might be the culprit for this one.
The SOFT loop engagement has in the past always been a source of trouble, mostly because the initial offsets can be really big. Together with the double-integrator design (in control filters and suspension top stage filters) this usually is a recipe for disaster. This should now be a thing of the past: - re-ordered the filter modules to keep the switchable integrator and lead filters ahead of the always-on low-pass (see alog 44124). - Set error point limiters before the control modules, but after the input matrix (in the blend filters) to 0.02. - During engagement only a 0.1Hz pole integrator in FM5 (along with the DC gain, inverted plant and 10Hz LP in FM1, FM9 and FM10) are on. Plot 1 (red traces) show the total control filter frequency and step response. After this filter, we still have a gain of 2.5 and the suspension top stage 1/f integrator. - Thus, during engagement with limiters on, the max signal to the suspensions can only be 0.02 (limiter) x 10000 (control filter) x 2.5 (gain) = 500 cts This is small enough that the suspensions don't run away. Instead they perform a constant velocity move to shrink the SOFT error signals. - The Guardian then looks for a) convergence of the SOFT control signals below 100cts, and b) convergence of the SOFT error signals below 75% of the limiter (or 0.015) When these conditions are met, Guardian turns the limiters off and turns on the true integrator and lead filter (changing the transfer function and step response to the blue trace in plot 1). - Finally, after waiting for the 10 sec filter ramp time of the lead filter, Guardian turns up the gain to the nominal values of 4 for DSOFT and 6 for CSOFT. Plot 2 shows the turn-on sequence for all SOFT control and error signals
(Betsy, for Betsy and Kissel)
This afternoon, Kissel and I embarked on the second round of SDF house cleaning. With the IFO set to down for a short bit, and after enlisting help from commissioners, we resolved most changes in the SAFE files. Things that were "interesting" yet accepted:
- New Test Mass L3 LOCK P/Y FM7 Stop27 filter (commissioners needed to add this along with last week's Stop28 filter but this still needs to be sorted out)
- Accepted new IMC PZT offset, but these will likely need to change during the subsequent lock due to hysteresis (if this model goes down, currently not expected to)
-Quite a few ALS settings accepted or reverted (Sheila was surprised just now to find that the ALS COMM PLL OFFSET slider was unresponsive...)
- Sheila and Kissel found that the PRM M2 Coil Driver state and the M2 Lock Output SW SDF settings were saved in the wrong state so were accepted (correct state: coil driver in State 3 and output OFF)
- OMC DCPD Gains found in the unbalanced state so fixed (both A and B gains set to 1000)
But the weirdest was:
- Some strangeness in the SDF setpoints for LSC REFL RF9 Phase and ITM Baffle PD Gains/Offsets (on different PLCs!) which stated that the settings were changed within a few minutes of Sept 19 13:49, yet the SDF as-found EPICS values were correct. So we accepted the correct values. No one in the room could confirm these older setpoints, but we did find an alog (44064) pointing to SDF model work around that time...
We also accepted all of the optic DRIVEALIGN offsets (alignment sliders) so that restore should go smoothly.
It's as reconciled as it's gonna get for tomorrow's CDS boots of LSC, ASC, SUS.
Prep. for leak testing tomorrow
The process of adding channels requested by WP7832 to the GDS broadcast list requires removing old channels some of these new ones replace. I took the opportunity to scan the complete channel list in H1BROADCAST.ini and found that 20 channels no longer exist in the running DAQ (most don't exist, some are commented out in the INI files). These were removed from the file.
WP7832 adds 16 new channels to GDS. These will be installed during tomorrow's DAQ restart.
Below is a table where we compare the readings on the HWS with different CO2 and RH lensing which ideally should be the same if the amount of CO2 actually corrects the applied amount of ring heaters. There are a few caveats to the measurement though, the ITMX HWS are particularly sensitive to the beamsplitter's pointing (long term and short term drift). But what this tells us is that the step down in common lens brought the spherical power closer to the "cold" state before any TCS was applied which maybe makes sense on why the PRC gain got better after Stefan made an adjustment.
|
|
|
|
Before TCS Work |
-13.0 |
-15.0 |
50 Watt Preloading (RH + CO2) |
-20.0 |
|
25 Watt Preloading (RH + CO2) |
+2.0 |
+20.0 |
25 Watt RH but reduced CO2 common Lens |
-12.5 |
+10.0 |
Daniel, Nutsinee
The OPO is now locked to pump laser/PSL beat note with beat note error signal (from TTFSS preamp) goes to input of the OPO common mode board and 158MHz to common mode board LO. We bypassed the PZT driver box completely to reduce the gain to the OPO PZT (PZT driver box has a gain of x22, and a pole at 400Hz). 10Hz low pass filter was added between OPO common mode board output and in-vac PZT driver to give it a low frequency roll-off, which (according to the sketchy writing on the box) attenuated the signal by a gain of 10. Haven't looked at the spectrum or transfer function (will add that to the comment later). A stable beat note frequency is needed for 3MHz locking (with PSL being an LO).
Problem: It will lock +-50kHz sideband instead of 0Hz beat note frequency. Next step is to try using a PFD instead of an IQ demod.
Correction: The error signal and LO went into an IQ demod board and the output of that went to input of the common mode board.
Attached is a picture taken back in mid December of 2017 of the flange which was found to be leaking today. In it, you can see that most of its flange fasteners had been removed at that time. Our best guess is that this was done in preparation to install a NEG pump at this location but, for whatever reason, was later aborted. We are unsure as to the chain of events but there was likely miscommunication involved as we would have never, knowingly, re-flanged this joint using the original non-vacuum, zinc-plated hardware (that had been provided by the OMC mode cleaner tube manufacture) that is still present today. Considering that the right hand and left hand were working independently, it is possible that the original, compressed, gasket may still be present and that is why we have such a big leak!
FRS 11544 ticket filed.