Greg, Dan, Dave, Jim:

The MSR single mode fiber optics cable which connects the Q-Logic switch to the patch panel for the MSR-LDAS link which was showing errors was replaced this afternoon. Dan is reporting that so far it looks like the errors have stopped, suggesting the old cable was faulty. The MSR Q-Logic switch was reconfigured to enable the single mode port and disable the multi mode port which was connected the two MSR switches.

Timing Upgrades [WP5370]

Richard, Filiberto, Andres, Jim, Dave:

The timing source for the MSR timing master was transitioned from internal GPS receiver to external 1PPS. The transition went smoothly. The broken timing fanout at EY was replaced with a spare. The timing MEDM screen is now GREEN with no errors.

BIOS changes of end station SUS front end computers [WP5374]

Jim:

The BIOS settings for h1susex and h1susey were modified to be identical to the LLO computers. This was an attempt to clear the IOP glitching seen since the faster computers were installed last Tuesday, it did not clear the glitching.

RCG upgrade of end station SUS and OAF [WP5372]

Jeff, Betsy, Jim, Dave:

the h1susetmy and h1susetmypi models were built against RCG branch-2.9 to install the TrueRMS fix. We found an EPICS initialization problem and reverted the models back to RCG-2.9.5. Rolf has found the error so we can try this upgrade again.

New h1susetmxpi model [WP5365]

Dave, Jim:

the first roll-out of the h1susetmxpi model was made. To remove the "DAQ too small" error and the writing of channels to the commissioning frame which were not needed, I removed all DAQ Channels definitions in PI_MASTER.mdl. At this point the RCG chose two fast channels and set them to "acquire=0" in the ini file. This solves the problem without writing data at 64kHz to the frame. I applied this change to both h1susetmypi and h1susetmxpi

Add Beam Tube Accelerometers to PEM mid stations [WP5375]

Robert, Vinny, Dave:

the models h1pemmx and h1pemmy were modified to change the 11th ADC channel from a generic input to the BEAMTUBE_CRYO acceleromter channel. They were added to the science frame at 2kHz.

MSR Front end Rack Network Switch Reboot [WP5371]

Jim:

As part of the FE channel access problem investigation, the Netgear switch which supplies the FE-VLAN to the front end computers, boot, build and guardian was rebooted. The downtime was about a minute and all systems reconnected seamlessly afterwards. It does not appear to have fixed the problem.

MX PEM power supply failure

Jim, Dave, Richard, Filiberto:

While restarting h1pemmx models I noticed the ADC data was zero, starting around 9pm Friday 17th July PDT. We found the -18V power supply to be off. We replaced it with a spare and the AA chassis is now operating correctly.

Server reboots

Patrick, Dave, Jim

The following servers were rebooted as part of preventative maintenance: all conlog machines, h1guardian0, cdsegwe0, h1hwinj0, h1hwinj1.

after the first reboot of h1guardian0 the guardian nodes did not auto start. After a second reboot guardian started correctly.

Travis, Nutsinee, Jeff K., Betsy, Ed, Cheryl, Hugh, Jim, Stefan, Sheila, Elli, pretty much everyone in the control room!

This log is meant for fellow operators, but commissioners might find it useful as well.

- We started relocking in the afternoon. Locking green went well. There was an argument about whether to request LOCKED NO SLOW or LOCKED WITH SLOW FEEDBACK (as stated in the Initial Alignment Wiki). I went for LOCKED NO SLOW because I knew it worked two Saturdays ago but I didn't know when was the last time wiki got updated (and I also don't know what LOCKED WITH SLOW FEEDBACK was). It is a good idea for the person who's maintaining the Ops wiki to put in the date and the content that was being updated (this was already brought up by TJ, but I just wanted to emphasize how important it is if we are going to use the wiki as the reference so we know we can trust it). 

- Everything went well up until MICH DARK LOCKED. The counts on the BS output were awfully high and the AS port wouldn't go dark. The solution to this in a normal situation would be to request DOWN and wait until the beam splitter becomes settle. However, Stefan found that there was a change in the LSC MICH filter and the Guardian tried to turn on the filter that no longer existed (Guradian turning on a blank filter wasn't a problem, the problem was a filter was supposed to be there). He reverted the filter to the way it was the night before. Then everything worked again. I wasn't sure if this difference showed up in the SDF or not.

- Mode Hopping! Soon as you approach DRMI LOCKED, get the IFO_ALIGN window ready. You will have to touch the beam splitter if the RF18 (pink) is low. You have to act quick! Or the interferometer will give up on you. Use AS port camera as a reference for pitch/yaw and watch "pink" and "purple" to see if you go the right way.

- We lost lock at LOW NOISE ESD ETMY. After that we had difficulty relocking ETMY green. Power dropped after WFS was engaged. Shortly after Travis found that the TMSY slide bar was off from Kissel's screenshot in the morning (or last night?). After we put TMSY yaw back to where it was and the arm was able to lock in green. Not only the screenshot never hurt, but it has also proven to be helpful! 

- We also switched ISI Windy Blend filters on for ALL Quads because the wind was reaching 35 mph (and we thought might have lost lock because of that). However, TURNING ON WINDY BLENDS TRIPPED BOTH ETMX AND ETMY ISI. Jeff demands all ISI filters are set the same while Sheila thinks it's okay to leave the Windy blend at the corner station on. Personally I would turn on the corner station windy blends iff I'm desperate. Since the wind wasn't too bad we switched the filters back to "Quite" (I'm sure it should spelled "Quiet"). This configuration worked fine the night before.

- Then we lost lock at LOW NOISE ESD agian. At least we got all the way there right after maintenance period!

- Ed was updating the Wiki during the relocking. It should be as up-to-date as it possibly can right now.

From the time the end station sus machines were upgraded to the faster model, we have been seeing gliches on the SUS IOP model, which translates to IPC error over to SEI and ISC front ends. Sometimes these trip the ISI watchdogs. This has happened 11 times in the past week. Attached is a two week minute trend which shows the following:

top right plot, TIMING or ADC error rate on h1iopsusey whch starts when the new computer was started 9am Tues 14th July PDT

bottom left plot, receive errors on the ISI ETMY model, these latch on until DIAG_RESET

top left plot: ISI ETMY trip flag, this initiates the watchdog trip

bottom right plot: ISI ETMY watchdog trips, latched until user untrips. 4 = full trip, happens each time trip flag is set to one.

The high voltage cables for the cosmic ray detector were pulled from the CER (PEM-C1) to BSC3. Cables need to be terminated with MHV connectors at both ends. Four BNC cables were pulled last week. Electronics are currently being modified for installation in the CER PEM rack.

Started installation of Low Voltage Noise ESD Driver chassis at EX. Field cables were pulled from electronics bay (SUS-C1) to field rack (SUS-R1). Items for next Tuesday:

1. Modify H1:SUS_ESD_06 cable and pull from SUS-C1 to SUS-R1.
2. Install driver chassis in same slot (SUS-R1) as unit at EY.
3. Build ±24V power cable for driver chassis. 

Ever since the 3ifo long-term storage containers have been utilizing CP2's dewar boil-off for purge gas we have been getting alarms due to poor PID control following a Praxair LN2 delivery.  Today I valved-out the 3ifo line (0830 - 1600 hrs local) for the Praxair LN2 delivery and the control of CP2's level was significantly improved.

Times PST

7:40 Jeff B to LVEA for N2 dewar and CC work

8:11 Jim B to EY for timing fanout and BIOS work

8:13 Jeff B done

8:16 Hugh and Jim to HEPI mezzanine for pump station work

8:19 Andre to LVEA for cosmic ray detector cabling

8:20 Praxair on site

8:31 Vinny to EY for PEM work

8:33 Karen and Cristina to both ends

8:36 Kyle to both mid stations

9:00 Richard done w/ GPS work

9:15 Vinny to LVEA for tiltmeter work

9:23 Kyle back from mids, going to LVEA

9:37 Jim B done at EY, going to EX for BIOS work

9:40 Dave doing EY software updates

9:48 Kyle done in LVEA

9:58 restarting EY SUS

10:00 Karen and Cristina  done

10:15 Nutsinee out to CER to reset TCS-X laser

10:20 Fil to EX for ESD LVLN install

10:38 Jim B done

10:45 Praxair on site

10:45 Kyle touring LN2 dewars

10:47 Hugh done

11:04 Nutsinee and Elli to LVEA TCS sensor check

11:38 Kyle done

12:08 DAQ restart

14:45 Dave to MX for PEM power supply replacement

14:52 Richard to CER

15:14 Dave back

J. Kissel

The renewed number of IPC errors caused by timing errors from the new fast SUS front ends are causing a problematic number of ISI trips when there are no suspension payload trips. I remembered questioning this issue back in March, and rediscovered that the BSC-ISIs are mishandling the error signals from the Dolphin IPC parts -- see the original identification of the problem LHO aLOG 17044. The state of the payload watchdog flag is still exactly as quoted in that aLOG.

We need to solve this problem, because it's unclear how/when we'll fix the timing errors on the SUS. I'm half-inclined to just make this change *now,* but after reading the aLOG, I don't remember which is the right way to fix the problem. I've tried calling Stanford, but I didn't get an answer, will send an email.

Thank you Evan for helping clear out the accumulated ASC/LSC SDF Diffs last night.  After the reboots from today, I used SDF to check for settings which were not restored (for some reason) today.  The only offenders I found were that the L2 PIT OLDAMP loop outputs were turned off on ITMX, ITMY, and SR3.  After trending that they have been on for the last week or 2 (commissioners confirm they should be on), I switched them back on.

In  order to check the ESDs we perform some charge measurements. 
ETMY ESD was Ok, ETMX ESD was restored after visit to the end station and program reset after that. Now ESDs work Ok.
Charge measurements data are in attachment.

I suspect the accumulator check last week left the schrader valve on one or more of the seven accumulators with a slow leak.  Not a big operational deal but there will likely be more couping into the platform motion from pump pressure variations.  I don't know what the pressures in the accumulators are now and it will be informative to check them.  I suspect they are significantly down but next Tuesday may be the soonest I can check.

The attached plot really does not show anything about this but I like to show the daily glitch in the pressure signals as I reported in 19416.  Also, the diurnal likely temperature response of the motor speed is obvious.

Bottom Line--Theory: The pump was cavitating at some level and corrected by slowly spinning down and back up.

Details--Checked the accumulators last Tuesday and found they all looked good.  But you have to bring all the pumps down to check/set the accumulator gas pressure.  I noticed the bubbling/gurggling/rock-rolling-around sound in pump #8 right away after bringing the pumps back up and reported in alog 19699 showing the drop in output from this pump.  The pressure out from this pump was down ~10psi even though the pump was spinning ~20% faster: see first attachment of 10 days showing the pressure out at the pumps for the four CS PS skids.  Also the upper right channel is the PID servo drive to the motor controller increasing the speed in response to the loss of pressure at PS1_PRESS1.  It is also evident that this pressure sensors is noisier by a lot due to this cavitation.

While the UGF of the PID is quite low, ~10mhz, starting the pumps back up with the PID will ramp it up quite quickly.  I suspect this start up had been just a little too fast for PS1, this time, and some noisy air bubbles would not clear the pump.

We increased the VFD Accel and Decel settings from 1 to 300sec to stop the pump spinning over 5 minutes.  After a quick look at the spider coupler, the pump was spun back up over five minutes and the noise was gone and the output efficiency and noise level was recovered.  Sweet!

Take away: Anytime the pumps are restarted, check the trends to confirm pump is operating as usual, or better.  Audible changes at the pumps mean something--investigate.  A slow spin up of the pumps is better.

J. Kissel, B. Weaver, D. Barker, J. Batch

After the completion of the upgrade of the BIOS of the fast front ends for the ETM SUS, we had tried compiling the front-end models against RCG branch 2.9, such that we could inherit both fixes to the True RMS part. It hasn't been documented anywhere other than the initial ID of the bug (see LHO aLOG 19658), but RCG 2.9.5 (released by Rolf Bork on Friday afternoon) fixes the "initialization problem" with the true RMS part, and a further fix to the part's "accumulation issues" had been checked into the branch 2.9 last night by Matt Evans. The hope was that we could get both fixes by compiling against the branch.

Beginning with the H1 SUS ETMY model (which had previously been compiled against RCG tag 2.9.4), it successfully restarted with the existing code. However, when we compiled against the branch, during the restart the BURT restore of the safe.snap SDF file failed. We were able to reproduce this twice: for many channels (275 the first time, and 277 the second time, according to the SDF system; often with filter bank settings, but not matrix settings), the correct safe.snap was loaded, and then ~3 seconds later come other bad value was restored in its place. We were finally able to successfully restart and automatically restore with the code compiled against the RCG tag 2.9.5.

The attached conlog screenshot documents this problem with a single one of these bad channels. The restart at 9:20 PDT is the restart with existing code (had been compiled at 2.9.4). The restarts at 9:41 and 9:58 are with the code compiled with branch 2.9. The (successful) restart at 10:04 is compiled against the tag 2.9.5. 

Further, ~30 minutes later, we noticed that ETMY's IOP DAC channels were zero, while the user model was requesting output. To resolve this, we had to kill all user models on the front end, and restart the IOP model. This theoretically should have been done already, given that, after the BIOS upgrade, the all front end models start from scratch. But, we shrug, and say "must not have stuck!" Indeed this same problem (code compiled against RCG 2.9.5, successfully restored, and subsequent zero IOP DAC outputs) occurred on ETMX as well, so we also did a second full restart of all models in that front end after the BIOS upgrade restart. 

RickS, Sudarshan, Darkhan

We've adjusted PCALX, PCALY and ESD calibration line drive levels to give a signal to noise ratio of about 100 (for 8 s FFT) over the mean science mode spectra from ER7.

The following lines have been adjusted:

Recovery from its biweekly crash.

Note that it took down the end X ESD high voltage because the ethercat Pirani gauge runs on it.

I burtrestored to 5:10 this morning.

Filiberto is turning the high voltage back on.

J. Kissel, B. Weaver

Betsy and I have updated the end-station SUSAUX top-level models to include Stuart's new library block for the HV and LV ESD analog readback monitors, as per WP 5373, ECR E1400232, II 859.
with the changes as described in LHO aLOG 18819, and obeying wiring diagram D1400177 , specifically p11.

They been compiled against RCG 2.9.5, we'll wait for Dave to confirm that we want 2.9.5 or 2.9.6, and will install later in the morning, as per today's schedule (LHO aLOG 19770).

We continue the charge measurements on ETMs.
Results for ETMX are consistent with negative trend, now the charge is from 10 to 20 [V] Effective Bias Voltage for all the quadrants.
Results for ETMY do not not show a significant trend (probably, the data are beginning to be consistent with positive trend). Charge is below the 10 [V] Effective Bias Voltage for all the quadrants.

Note:  We had positive bias on ETMX and negative bias on ETMY after discharging procedure. So it seems possible that charging is caused by the bias voltage.

Hang, Matt, Evan, Stefan, Rana

WE mostly were chasing weird locklosses and instabilities, but also managed to implement a few noise improvements:

1. SRCL FF better shaping via fitting and better measurements.
2. A2L decoupling gave us 10-30 Hz improvement.
3. MICH FF tuned up a bit. New measurments taken; fitting and filter generation yet to be done.
4. ASC tune up: hard/soft modes for CARM.
5. ETMY Butterfly mode identified and notched.

Of the locklosses, some were just due to marginal points in the transitions and loop margins. The main issue over the past few days turned out to be that the TIDAL servo was turned OFF somehow on Friday evening. After switching that back on for ETMY, we have returned to having long locks. The highpassing of SRCLFF has removed the high power pitch instabilities that we were seeing.

We were able to get > 40 Mpc with 10W input. The butterfly mode of ETMY @ 6053.81 Hz is preventing us from turning on the OMC DC whitening right now, so we don't know how good our range really is, but our low frequency

It should not be on during data taking.