The high voltage cables for the cosmic ray detector were pulled from the CER (PEM-C1) to BSC3. Cables need to be terminated with MHV connectors at both ends. Four BNC cables were pulled last week. Electronics are currently being modified for installation in the CER PEM rack.
Started installation of Low Voltage Noise ESD Driver chassis at EX. Field cables were pulled from electronics bay (SUS-C1) to field rack (SUS-R1). Items for next Tuesday: 1. Modify H1:SUS_ESD_06 cable and pull from SUS-C1 to SUS-R1. 2. Install driver chassis in same slot (SUS-R1) as unit at EY. 3. Build ±24V power cable for driver chassis.
Ever since the 3ifo long-term storage containers have been utilizing CP2's dewar boil-off for purge gas we have been getting alarms due to poor PID control following a Praxair LN2 delivery. Today I valved-out the 3ifo line (0830 - 1600 hrs local) for the Praxair LN2 delivery and the control of CP2's level was significantly improved.
Times PST
7:40 Jeff B to LVEA for N2 dewar and CC work
8:11 Jim B to EY for timing fanout and BIOS work
8:13 Jeff B done
8:16 Hugh and Jim to HEPI mezzanine for pump station work
8:19 Andre to LVEA for cosmic ray detector cabling
8:20 Praxair on site
8:31 Vinny to EY for PEM work
8:33 Karen and Cristina to both ends
8:36 Kyle to both mid stations
9:00 Richard done w/ GPS work
9:15 Vinny to LVEA for tiltmeter work
9:23 Kyle back from mids, going to LVEA
9:37 Jim B done at EY, going to EX for BIOS work
9:40 Dave doing EY software updates
9:48 Kyle done in LVEA
9:58 restarting EY SUS
10:00 Karen and Cristina done
10:15 Nutsinee out to CER to reset TCS-X laser
10:20 Fil to EX for ESD LVLN install
10:38 Jim B done
10:45 Praxair on site
10:45 Kyle touring LN2 dewars
10:47 Hugh done
11:04 Nutsinee and Elli to LVEA TCS sensor check
11:38 Kyle done
12:08 DAQ restart
14:45 Dave to MX for PEM power supply replacement
14:52 Richard to CER
15:14 Dave back
J. Kissel The renewed number of IPC errors caused by timing errors from the new fast SUS front ends are causing a problematic number of ISI trips when there are no suspension payload trips. I remembered questioning this issue back in March, and rediscovered that the BSC-ISIs are mishandling the error signals from the Dolphin IPC parts -- see the original identification of the problem LHO aLOG 17044. The state of the payload watchdog flag is still exactly as quoted in that aLOG. We need to solve this problem, because it's unclear how/when we'll fix the timing errors on the SUS. I'm half-inclined to just make this change *now,* but after reading the aLOG, I don't remember which is the right way to fix the problem. I've tried calling Stanford, but I didn't get an answer, will send an email.
Thank you Evan for helping clear out the accumulated ASC/LSC SDF Diffs last night. After the reboots from today, I used SDF to check for settings which were not restored (for some reason) today. The only offenders I found were that the L2 PIT OLDAMP loop outputs were turned off on ITMX, ITMY, and SR3. After trending that they have been on for the last week or 2 (commissioners confirm they should be on), I switched them back on.
In order to check the ESDs we perform some charge measurements. ETMY ESD was Ok, ETMX ESD was restored after visit to the end station and program reset after that. Now ESDs work Ok. Charge measurements data are in attachment.
I suspect the accumulator check last week left the schrader valve on one or more of the seven accumulators with a slow leak. Not a big operational deal but there will likely be more couping into the platform motion from pump pressure variations. I don't know what the pressures in the accumulators are now and it will be informative to check them. I suspect they are significantly down but next Tuesday may be the soonest I can check.
The attached plot really does not show anything about this but I like to show the daily glitch in the pressure signals as I reported in 19416. Also, the diurnal likely temperature response of the motor speed is obvious.
Bottom Line--Theory: The pump was cavitating at some level and corrected by slowly spinning down and back up.
Details--Checked the accumulators last Tuesday and found they all looked good. But you have to bring all the pumps down to check/set the accumulator gas pressure. I noticed the bubbling/gurggling/rock-rolling-around sound in pump #8 right away after bringing the pumps back up and reported in alog 19699 showing the drop in output from this pump. The pressure out from this pump was down ~10psi even though the pump was spinning ~20% faster: see first attachment of 10 days showing the pressure out at the pumps for the four CS PS skids. Also the upper right channel is the PID servo drive to the motor controller increasing the speed in response to the loss of pressure at PS1_PRESS1. It is also evident that this pressure sensors is noisier by a lot due to this cavitation.
While the UGF of the PID is quite low, ~10mhz, starting the pumps back up with the PID will ramp it up quite quickly. I suspect this start up had been just a little too fast for PS1, this time, and some noisy air bubbles would not clear the pump.
We increased the VFD Accel and Decel settings from 1 to 300sec to stop the pump spinning over 5 minutes. After a quick look at the spider coupler, the pump was spun back up over five minutes and the noise was gone and the output efficiency and noise level was recovered. Sweet!
Take away: Anytime the pumps are restarted, check the trends to confirm pump is operating as usual, or better. Audible changes at the pumps mean something--investigate. A slow spin up of the pumps is better.
J. Kissel, B. Weaver, D. Barker, J. Batch After the completion of the upgrade of the BIOS of the fast front ends for the ETM SUS, we had tried compiling the front-end models against RCG branch 2.9, such that we could inherit both fixes to the True RMS part. It hasn't been documented anywhere other than the initial ID of the bug (see LHO aLOG 19658), but RCG 2.9.5 (released by Rolf Bork on Friday afternoon) fixes the "initialization problem" with the true RMS part, and a further fix to the part's "accumulation issues" had been checked into the branch 2.9 last night by Matt Evans. The hope was that we could get both fixes by compiling against the branch. Beginning with the H1 SUS ETMY model (which had previously been compiled against RCG tag 2.9.4), it successfully restarted with the existing code. However, when we compiled against the branch, during the restart the BURT restore of the safe.snap SDF file failed. We were able to reproduce this twice: for many channels (275 the first time, and 277 the second time, according to the SDF system; often with filter bank settings, but not matrix settings), the correct safe.snap was loaded, and then ~3 seconds later come other bad value was restored in its place. We were finally able to successfully restart and automatically restore with the code compiled against the RCG tag 2.9.5. The attached conlog screenshot documents this problem with a single one of these bad channels. The restart at 9:20 PDT is the restart with existing code (had been compiled at 2.9.4). The restarts at 9:41 and 9:58 are with the code compiled with branch 2.9. The (successful) restart at 10:04 is compiled against the tag 2.9.5. Further, ~30 minutes later, we noticed that ETMY's IOP DAC channels were zero, while the user model was requesting output. To resolve this, we had to kill all user models on the front end, and restart the IOP model. This theoretically should have been done already, given that, after the BIOS upgrade, the all front end models start from scratch. But, we shrug, and say "must not have stuck!" Indeed this same problem (code compiled against RCG 2.9.5, successfully restored, and subsequent zero IOP DAC outputs) occurred on ETMX as well, so we also did a second full restart of all models in that front end after the BIOS upgrade restart.
RickS, Sudarshan, Darkhan
We've adjusted PCALX, PCALY and ESD calibration line drive levels to give a signal to noise ratio of about 100 (for 8 s FFT) over the mean science mode spectra from ER7.
The following lines have been adjusted:
| Line | Frequency (Hz) | Old drive level (ct) | New drive level (ct) |
| PCALX | 33.1 | 38 | 700 |
| PCALX | 534.7 | 6000 | 19300 |
| PCALY | 36.7 | 40 | 320 |
| PCALY | 540.7 | 5000 | 9900 |
| ESD | 34.7 | 0.50 | 0.22 |
| ESD | 538.1 | 0.20 | 2.40 |
I added pcal lines at 325.1 Hz, 3000 ct for X and 331.9 Hz, 1500 ct for Y. These are maybe not optimal because they are in the region of the PSL piezo mirror mount, but I wanted to stay close to the DARM pole and out of the bottom of the bucket.
Based on noise levels around calibration lines during ER7, to get SNR of 100 (with 8 s FFT) , drive levels adjustments needed for LLO calibration lines:
I inserted a nototch filter for each of the two lines that Evan added last night. Because of the limitation in the number of the IIR coefficients, I had to put one in suscomp (in FM7) and the other in ResG (in FM3). The previous notch that we put for the cavity pole tracker (alog 18401) was removed from FM7 since we have not been using it.
Recovery from its biweekly crash. Note that it took down the end X ESD high voltage because the ethercat Pirani gauge runs on it. I burtrestored to 5:10 this morning. Filiberto is turning the high voltage back on.
Will change channel 2 at next opportunity
I rebooted h1conlog1-master and h1conlog1-replica to check that they could be without issues. The only thing I saw was an 'unable to contact caRepeater' for the Conlog process, which should be fine. Both are back up and running.
Elli, Nutsinee
CO2X laser mysteriously tripped this morning (*probably* due to PEM cable pulling). It untripped itself but the output power was never returned and the temperature kept dropping. We had to go in and restart the laser at the rack. Just wanted to point out that the laser READY status on Ops Overview screen doesn't always mean it's ready. Always keep an eye out for the laser power.
J. Kissel, B. Weaver Betsy and I have updated the end-station SUSAUX top-level models to include Stuart's new library block for the HV and LV ESD analog readback monitors, as per WP 5373, ECR E1400232, II 859. with the changes as described in LHO aLOG 18819, and obeying wiring diagram D1400177 , specifically p11. They been compiled against RCG 2.9.5, we'll wait for Dave to confirm that we want 2.9.5 or 2.9.6, and will install later in the morning, as per today's schedule (LHO aLOG 19770).
This models have been installed, compiled against RCG 2.9.5 (given the issues with RCG branch 2.9; see LHO aLOG 19793). All channels appear functional, but now we see that the digital monitor channels that were in the QUAD AUX model (which are white on the AUX overview) are now in the QUAD MASTER model have turned into bits (which are now green / grey and alive); see attached. @Stuart -- should we remove these white channels from the AUX overview screen? Note -- the cable for the digital monitoring doesn't yet exist, so the new digital monitor bits are at the moment meaningless.
h1susaux ex and ey model updates as per the above have been committed to svn.
We continue the charge measurements on ETMs. Results for ETMX are consistent with negative trend, now the charge is from 10 to 20 [V] Effective Bias Voltage for all the quadrants. Results for ETMY do not not show a significant trend (probably, the data are beginning to be consistent with positive trend). Charge is below the 10 [V] Effective Bias Voltage for all the quadrants. Note: We had positive bias on ETMX and negative bias on ETMY after discharging procedure. So it seems possible that charging is caused by the bias voltage.
Has the bias on ETMX and ETMY remained positive and negative respectively for the duration of this observation?
Bias was the same for this and next charge measurements. It was changed on July, 22: alog 19821 Today we have the first measurements after changing the bias sign: alog 19848
Hang, Matt, Evan, Stefan, Rana
WE mostly were chasing weird locklosses and instabilities, but also managed to implement a few noise improvements:
Of the locklosses, some were just due to marginal points in the transitions and loop margins. The main issue over the past few days turned out to be that the TIDAL servo was turned OFF somehow on Friday evening. After switching that back on for ETMY, we have returned to having long locks. The highpassing of SRCLFF has removed the high power pitch instabilities that we were seeing.
We were able to get > 40 Mpc with 10W input. The butterfly mode of ETMY @ 6053.81 Hz is preventing us from turning on the OMC DC whitening right now, so we don't know how good our range really is, but our low frequency
We also got a change to reimplement a small amount of the ASC changes from before last maintenance day:
Next:
Note, Rana reports that the tidal was OFF at the far right switch on the IMC_F Filter Module (picture attached shows this switch now on as it should be).
It should not be on during data taking.
Also added warnings for when the cameras and the frame grabbers are on.
I've commented out the HWS and frame grabber on warnings because we want to use them during commissioning. We should uncomment this for the science run though.
Kyle, Gerardo 0850 - 0900 hrs. local -> In and out of Y-end Kyle, Gerardo Connected LD to exhaust of Y-mid turbo and sprayed helium on turbo fittings -> Found that this turbo has a "Leak valve" a.k.a. a solenoid vent valve which is either not connected or absent altogether on the site's other Main Turbo Pumps (MTPs) -> The turbo had been left levitated but not spinning since John W. and Gerardo's earlier investigation -> As such, this vent valve would be open -> Found that the clamp for the blank on this valve was loose -> Tightened -> With the turbo levitated but stopped, the QDP80 running but valved-out, safety valve and turbo bypass valve opened we sprayed the turbo's CFF and NW fittings with no obvious "smoking gun" -> It is likely that tightening the blank on the vent valve may have been the leak on the turbo side of the 10" gate valve For our next session we will vent the turbo volume and remove and replace the vent valve with a blank followed be a leak testing of the CFF joints on the Y-mid volume Also, set IP9 to Fixed 7000V -> This leaves only the X-mid still in Step Mode (currently 5000V)
Upon noticing the vent valve mounted on the Y-mid turbo, Gerardo and I examined the X-mid turbo and found that it's vent valve was not installed. Knowing that we would never use it and not ever noticing it on any of the other MTPs, I assumed that the Y-mid having one installed must be the exception. Since then, I have surveyed the MTPs at the Corner Station and found that they too have them installed, albeit blanked off like the MTP at the Y-mid. Thus, the X-mid MTP is the exception not the Y-mid MTP
