In order to check the ESDs we perform some charge measurements. ETMY ESD was Ok, ETMX ESD was restored after visit to the end station and program reset after that. Now ESDs work Ok. Charge measurements data are in attachment.
I suspect the accumulator check last week left the schrader valve on one or more of the seven accumulators with a slow leak. Not a big operational deal but there will likely be more couping into the platform motion from pump pressure variations. I don't know what the pressures in the accumulators are now and it will be informative to check them. I suspect they are significantly down but next Tuesday may be the soonest I can check.
The attached plot really does not show anything about this but I like to show the daily glitch in the pressure signals as I reported in 19416. Also, the diurnal likely temperature response of the motor speed is obvious.
Bottom Line--Theory: The pump was cavitating at some level and corrected by slowly spinning down and back up.
Details--Checked the accumulators last Tuesday and found they all looked good. But you have to bring all the pumps down to check/set the accumulator gas pressure. I noticed the bubbling/gurggling/rock-rolling-around sound in pump #8 right away after bringing the pumps back up and reported in alog 19699 showing the drop in output from this pump. The pressure out from this pump was down ~10psi even though the pump was spinning ~20% faster: see first attachment of 10 days showing the pressure out at the pumps for the four CS PS skids. Also the upper right channel is the PID servo drive to the motor controller increasing the speed in response to the loss of pressure at PS1_PRESS1. It is also evident that this pressure sensors is noisier by a lot due to this cavitation.
While the UGF of the PID is quite low, ~10mhz, starting the pumps back up with the PID will ramp it up quite quickly. I suspect this start up had been just a little too fast for PS1, this time, and some noisy air bubbles would not clear the pump.
We increased the VFD Accel and Decel settings from 1 to 300sec to stop the pump spinning over 5 minutes. After a quick look at the spider coupler, the pump was spun back up over five minutes and the noise was gone and the output efficiency and noise level was recovered. Sweet!
Take away: Anytime the pumps are restarted, check the trends to confirm pump is operating as usual, or better. Audible changes at the pumps mean something--investigate. A slow spin up of the pumps is better.
J. Kissel, B. Weaver, D. Barker, J. Batch After the completion of the upgrade of the BIOS of the fast front ends for the ETM SUS, we had tried compiling the front-end models against RCG branch 2.9, such that we could inherit both fixes to the True RMS part. It hasn't been documented anywhere other than the initial ID of the bug (see LHO aLOG 19658), but RCG 2.9.5 (released by Rolf Bork on Friday afternoon) fixes the "initialization problem" with the true RMS part, and a further fix to the part's "accumulation issues" had been checked into the branch 2.9 last night by Matt Evans. The hope was that we could get both fixes by compiling against the branch. Beginning with the H1 SUS ETMY model (which had previously been compiled against RCG tag 2.9.4), it successfully restarted with the existing code. However, when we compiled against the branch, during the restart the BURT restore of the safe.snap SDF file failed. We were able to reproduce this twice: for many channels (275 the first time, and 277 the second time, according to the SDF system; often with filter bank settings, but not matrix settings), the correct safe.snap was loaded, and then ~3 seconds later come other bad value was restored in its place. We were finally able to successfully restart and automatically restore with the code compiled against the RCG tag 2.9.5. The attached conlog screenshot documents this problem with a single one of these bad channels. The restart at 9:20 PDT is the restart with existing code (had been compiled at 2.9.4). The restarts at 9:41 and 9:58 are with the code compiled with branch 2.9. The (successful) restart at 10:04 is compiled against the tag 2.9.5. Further, ~30 minutes later, we noticed that ETMY's IOP DAC channels were zero, while the user model was requesting output. To resolve this, we had to kill all user models on the front end, and restart the IOP model. This theoretically should have been done already, given that, after the BIOS upgrade, the all front end models start from scratch. But, we shrug, and say "must not have stuck!" Indeed this same problem (code compiled against RCG 2.9.5, successfully restored, and subsequent zero IOP DAC outputs) occurred on ETMX as well, so we also did a second full restart of all models in that front end after the BIOS upgrade restart.
RickS, Sudarshan, Darkhan
We've adjusted PCALX, PCALY and ESD calibration line drive levels to give a signal to noise ratio of about 100 (for 8 s FFT) over the mean science mode spectra from ER7.
The following lines have been adjusted:
| Line | Frequency (Hz) | Old drive level (ct) | New drive level (ct) |
| PCALX | 33.1 | 38 | 700 |
| PCALX | 534.7 | 6000 | 19300 |
| PCALY | 36.7 | 40 | 320 |
| PCALY | 540.7 | 5000 | 9900 |
| ESD | 34.7 | 0.50 | 0.22 |
| ESD | 538.1 | 0.20 | 2.40 |
I added pcal lines at 325.1 Hz, 3000 ct for X and 331.9 Hz, 1500 ct for Y. These are maybe not optimal because they are in the region of the PSL piezo mirror mount, but I wanted to stay close to the DARM pole and out of the bottom of the bucket.
Based on noise levels around calibration lines during ER7, to get SNR of 100 (with 8 s FFT) , drive levels adjustments needed for LLO calibration lines:
I inserted a nototch filter for each of the two lines that Evan added last night. Because of the limitation in the number of the IIR coefficients, I had to put one in suscomp (in FM7) and the other in ResG (in FM3). The previous notch that we put for the cavity pole tracker (alog 18401) was removed from FM7 since we have not been using it.
Recovery from its biweekly crash. Note that it took down the end X ESD high voltage because the ethercat Pirani gauge runs on it. I burtrestored to 5:10 this morning. Filiberto is turning the high voltage back on.
Will change channel 2 at next opportunity
I rebooted h1conlog1-master and h1conlog1-replica to check that they could be without issues. The only thing I saw was an 'unable to contact caRepeater' for the Conlog process, which should be fine. Both are back up and running.
Elli, Nutsinee
CO2X laser mysteriously tripped this morning (*probably* due to PEM cable pulling). It untripped itself but the output power was never returned and the temperature kept dropping. We had to go in and restart the laser at the rack. Just wanted to point out that the laser READY status on Ops Overview screen doesn't always mean it's ready. Always keep an eye out for the laser power.
As Daniel noted we plugged in the 1PPS signal from the Trimble timing unit that was installed last week into the Master timing system in the MSR. This is on Antenna cable 1. I then moved the Master timing antenna to the weather station mast. I did not change the cabling as this was already on 1/4" Heliax coax cable and the unit has 3/4" connection which would be hard to make up the the 1/2" heliax and N connector. We then Moved the NTP server antenna to the weather station mast using antenna cable 2. We needed to adapt the N connector the TNC at the antenna head. This completed the intrusive GPS moves. Next will be to connect the Symmetricom GPS unit in the rack to another antenna. This will done during another maintenance period.
the workstation in the front right corner of the control room doesn't work well, sometimes it stops working completely, sometimes it is unresponsive for a few minutes then comes back.
Aidan. Nutsinee
Summary:
The CO2 ITMX ISS AC anti-whitening filters where previously (naively) set up with the complete inverse of the ISS whitening filters in the electronics. This included a pole at zero, or an integrator. This caused problems at LLO when turned on as residual DC in the input to that channel was steadily integrated over time. We fixed this problem at LLO by moving the pole from 0 Hz to 1 Hz. I added the same fix here. Also, these filters were missing from the ITMY channels, so I added them at LHO as well.
Details:
In Foton, I loaded H1TCSCS.TXT and, in the following four channels,
I set the first three filter modules to:
The key change being the removal of a pole at zero and the addition of a pole at 1Hz.
After saving the file, I hit "Clear History" and "Load Coefficients" on all four channels.
Additional information:
The ISS AC channels tell us the intensity noise on the CO2 lasers. We're particularly interested in the region 10-200 Hz where we may couple noise into DARM.
The master GPS clock has been switched from the internal GPS unit of the master/fanout chassis to the external Trimble unit. This simply required to connect the 1PPS ouptut of the new GPS unit to the 1PPS input of the master/fanout chassis. Plot 1 shows that this proceeded with a small 100ns transient in the OCXO error signal but otherwise went smooth without interruption of the timing distribution system.
Plot 2 shows 1 day of second plots trending the difference between the atomic clock and the internal GPS (DIFF_1), the difference between the NTP server and the internal GPS (DIFF_2) and the difference betrween the Trimble and the internal GPS (DIFF_3). The atomic clock was shifted by +1µs and the NTP server by +0.65µs to shift the histograms towards the origin. This measurement was done before the switch over still using the internal GPS as the master. The Trimble unit seems to have about twice the variance compared to the internal GPS and the NTP server. The difference between the Trimble unit and the internal GPS is about 25ns.
J. Kissel, B. Weaver Betsy and I have updated the end-station SUSAUX top-level models to include Stuart's new library block for the HV and LV ESD analog readback monitors, as per WP 5373, ECR E1400232, II 859. with the changes as described in LHO aLOG 18819, and obeying wiring diagram D1400177 , specifically p11. They been compiled against RCG 2.9.5, we'll wait for Dave to confirm that we want 2.9.5 or 2.9.6, and will install later in the morning, as per today's schedule (LHO aLOG 19770).
This models have been installed, compiled against RCG 2.9.5 (given the issues with RCG branch 2.9; see LHO aLOG 19793). All channels appear functional, but now we see that the digital monitor channels that were in the QUAD AUX model (which are white on the AUX overview) are now in the QUAD MASTER model have turned into bits (which are now green / grey and alive); see attached. @Stuart -- should we remove these white channels from the AUX overview screen? Note -- the cable for the digital monitoring doesn't yet exist, so the new digital monitor bits are at the moment meaningless.
h1susaux ex and ey model updates as per the above have been committed to svn.
As a precaution to working on the timing system ALL of the SUS and Seismic systems have been taken down using guardian. The addition of the 1pps into the timing system should not cause a problem but if it did this should keep equipment from potential issues as a result of the front ends freezing.
There we now have several burt references for alignment, a good 'ol snapshot never hurt anybody!
Evan, Stefan - Loops worked - simply lowered DHARD gain by 6dB. - Had some low frequency transients right after - not clear whether they are related to the matrix - seemed to diminish later on.
This is the spectrum at the end of today's work. We'll spend tomorrow looking at the calibration, but tentatively this spectrum was >65 Mpc after engaging the outer ISS loop (it was about 60 Mpc) beforehand.
There is also some fake 80 Mpc stretch where we were examining the settings in the online calibration. Our uncertainty about the calibration has increased as a result of this investigation.
Sudarshan, Jeff K.
The ESD calibration line at 538.1 Hz is moved up to 540.5 Hz, 0.2 Hz away from the Pcal Line at Yend (540.7 Hz) to estimate the ESD actuation strength. This is a temporary arrangment and the ESD line will be moved back to its original position next week after we have some locked data to analyze. The SDF table is updated accordingly.
The ESD line is switched back to its original position at 538.1 Hz. SDF table is updated accordingly.
Kyle, Gerardo 0850 - 0900 hrs. local -> In and out of Y-end Kyle, Gerardo Connected LD to exhaust of Y-mid turbo and sprayed helium on turbo fittings -> Found that this turbo has a "Leak valve" a.k.a. a solenoid vent valve which is either not connected or absent altogether on the site's other Main Turbo Pumps (MTPs) -> The turbo had been left levitated but not spinning since John W. and Gerardo's earlier investigation -> As such, this vent valve would be open -> Found that the clamp for the blank on this valve was loose -> Tightened -> With the turbo levitated but stopped, the QDP80 running but valved-out, safety valve and turbo bypass valve opened we sprayed the turbo's CFF and NW fittings with no obvious "smoking gun" -> It is likely that tightening the blank on the vent valve may have been the leak on the turbo side of the 10" gate valve For our next session we will vent the turbo volume and remove and replace the vent valve with a blank followed be a leak testing of the CFF joints on the Y-mid volume Also, set IP9 to Fixed 7000V -> This leaves only the X-mid still in Step Mode (currently 5000V)
Upon noticing the vent valve mounted on the Y-mid turbo, Gerardo and I examined the X-mid turbo and found that it's vent valve was not installed. Knowing that we would never use it and not ever noticing it on any of the other MTPs, I assumed that the Y-mid having one installed must be the exception. Since then, I have surveyed the MTPs at the Corner Station and found that they too have them installed, albeit blanked off like the MTP at the Y-mid. Thus, the X-mid MTP is the exception not the Y-mid MTP
