In support of the CP4 bake-out, I've created a temporary Rate-Of-Change (ROC) medm and linked it as the last item on the VE pull-down on the SITEMAP. It is calculating the ROC for a CP3 thermocouple, which is actually located at CP4. When the 1-HOUR measurement exceeds 1.0degC/hour in either direction, the gray rectangle turns RED, indicating a cell phone alarm has been issued.

Note, ROC also calculates the LN2 consumption rate of EY-CP7-Dewar as a test channel (consumption is about 1% dewar capacity per day).

I have a script check_guardian_nodes_against_medm_screen which reports: nodes which exist but are not on MEDM screen, nodes which are on MEDM screen which do not exist. It produced one report: CS_BRS on the overview but does not exist. I have removed this node from the MEDM file.

Note: INJ_TRANS is white on the overview, Jamie is working on restarting this node.

WP7399

The process of offloading raw minute trend files from h1tw0 to the DAQ SATABOYS has been started. The first part of the procedure was to switch writing the current data to a new directory on h1tw1 and adding the directory which contains the last 6 months of data to the NDS archive_raw_minute_dirs list. This was done and both NDS were restarted at noon.

The next step is to use h1fw1 to copy of files from h1tw1's SSD RAID over to the SATABOY RAID (a fully compressed ZFS specifically built to hold archived raw minute). I started this process at 12:52 PST. Note that this transfer uses ionice to reduce this transfer's impact on the frame writing.

Posted below are plots for the PSL Chiller water usage for the past 2.5 years. 

   Water usage in the Crystal chiller appears to be constant between 300ml and 100ml per week. It should be noted water is not added every 7 days, but as it is needed and per the operator's FAMIS task. The average is one week between top offs. The water usage in the Diode chiller is not tracked as it's reservoir is much larger and only needs occasional attention. 

Service timeline of the two chiller:    

   The chiller known as Gromit (S/N Crystal 64150, Diode 63793)was in service from May 2015 until May 2016. It was put into backup for annual maintenance rotation.  

   The chiller known as Wallace (S/N Crystal 44801, Diode 44807) was in service from May 2016 until September 2016. It was put into backup for service due to Crystal chiller trips. The faulty Crystal chiller controller was replace. 

   The Chiller known as Gromit was in service from September 2016 until February 2018. It was put into backup for annual maintenance rotation. 

   The Chiller known as Wallace is in service as of February 2018.  
    

The CP4 enclosure temperature ramp is slowing down and the heater has been running at 100% for the last day. I've been increasing the temp of the GN2 flowing into CP4 so it's not a heat sink. I'm also trying to increase flow by increasing head pressure of Dewar (valve is wide open).

Here is another snapshot of Y-arm pressures over five days. We don't see a response in Y2 from changes in CP4 temperature, but Y1 side trends closely with CP4. GV11 (nearer to mid-station) has an outer o-ring leak in gate, but we evacuated this annulus and valved out. GV12 (Y2 boundary valve) has both inner and outer o-ring leaks on gate - also evacuated and valved out.

When BSC6 was replaced with spool at mid-Y several years ago, GV11 was exposed to air and the spool was not baked after installation (was it baked at factory?), so we expect outgassing from these components during the CP4 bake, but the sharp cuffs in Y1 pressure are not what we would predict with temperature changes in steel.

CP4 air temp trend over five days in enclosure is also attached.

Kyle and I replaced a bad solenoid valve on the compressed air drying tower skid outside after getting alarms 'after hours'. Loss of compressed air caused the safety valves on the three turbo stations to close and thus spin down turbos. It also caused GV 6,8 to sag. After we regained compressed air we spun turbos back up, but YBM turbo is being difficult and keeps shutting itself down due to vibration (a known issue). We tried all the tricks with bypass valve and air admittance to load rotors. We'll let it spin 100% down tonight and try again tomorrow. I closed all three turbo isolation gate valves in a timely manner so the corner should not have been affected from back streaming. Pressure rose for a period while turbos were valved out. I didn't have time to check on in-vacuum high voltage equipment - don't think anything was turned on.

FYI: we did not replace the solenoid portion of the valve at the drying tower.

The guardian upgrade has been completely aborted, due to the unexplained segfaults described in elog thread 40765 .  The entire guardian system is back to the exact same configuration as it was two weeks ago (guardian 1.0.3, h1guardian0 ubuntu12).  All nodes are up and running nominally.

h1guardian1 has been left in place to run demon excision tests: 20 fake guardian nodes are running under valgrind in the hope of catching a crash that would point to the problem.  These tests should not affect interferometer operation in any way.

In our hunt for issues, we swapped a BOSEM on ETMY Side R0 (reported below).  The BOSEM wasn't the problem tho.

OLV has been updated in medm:

• 08-03-18 15:48 JeffB in optics lab
• 08-03-18 16:01 Bubbas contractor for pump room work arrived - red truck
• 08-03-18 16:09 Richard to CER
• 08-03-18 16:49 Terry to HAM6, ISCT6
• 08-03-18 16:49 Richards contractor for the card access system arrived
• 08-03-18 17:14 JeffB done in optics lab and cleaning area
• 08-03-18 17:17 Karen done with cleaning at EY
• 08-03-18 17:18 Christina done with cleaning EX
• 08-03-18 17:18 Ken at EY for card access work
• 08-03-18 17:50 Richards card access contractor at EX and MX today
• 08-03-18 17:56 Mark to CER for cosmic ray equipment
• 08-03-18 17:56 Niko to HAM6
• 08-03-18 17:57 Dave looking at python error on verbal alarm machine
• 08-03-18 18:01 Sheila to optics lab and HAM6
• 08-03-18 18:10 as of 18:10 we are having an earthquake and also moving guardian nodes
• 08-03-18 18:11 many guardian nodes are in fault but Jim beleives all ISIs are locked or in damped
• 08-03-18 18:16 Robert to EX
• 08-03-18 18:17 Robert to EX to take pictures
• 08-03-18 18:18 Jim and Mark to EY to lock the ISI
• 08-03-18 18:23 Chandra to MY
• 08-03-18 18:43 Travis and Betsy to EY
• 08-03-18 19:00 Karen to MY
• 08-03-18 19:16 Jim to EX to start Pcal baffle install
• 08-03-18 19:38 Karen done at MY
• 08-03-18 19:43 Jim back from EY
• 08-03-18 19:49 Richard to EY to talk to Ken
• 08-03-18 20:21 Betsy and Travis back from EY
• 08-03-18 20:24 Robert to EY to take pictures
• 08-03-18 20:32 TJ, Sheila, Terry, Gerardo out of LVEA and HAM6 for lunch
• 08-03-18 20:45 Jim, Mark, and Niko, to EX to work on Pcal baffles
• 08-03-18 20:52 Dave and Fil hard cycling EY SUS, optic and TMS in damped
• 08-03-18 21:38 Richard and Fil swapping binary card at EY, after power cycle didnt fix
• 08-03-18 21:38 Kyle to EX to check on purge air, turn up
• 08-03-18 21:41 Chandra to MY
• 08-03-18 21:49 Dave testing a CS to EY connection, ETMY pitch
• 08-03-18 21:57 Terry heading to HAM6
• 08-03-18 22:01 Fil and Mark to MX to pull cable for new access system
• 08-03-18 22:05 Kyle back from EX
• 08-03-18 22:10 Matt to LVEA to take a picture
• 08-03-18 22:12 Nutsinee to HAM6
• 08-03-18 22:14 Dave has restored EY
• 08-03-18 22:17 Mark testing a table on HAM4
• 08-03-18 22:26 Matt to MY to look for IO 3IFO parts
• 08-03-18 22:32 Sheila to HAM6
• 08-03-18 22:45 TJ to HAM6
• 08-03-18 22:46 TJ to HAM6
• 08-03-18 22:53 Matt back from MY
• 08-03-18 22:56 TJ back from HAM6
• 08-03-18 23:08 Richard to LVEA with contractors
• 08-03-18 23:17 Roberts back from EY
• 08-03-18 23:54 crew still at HAM6
• 08-03-18 23:55 Jim and crew still at EX
• 08-03-18 23:58 Richard's contractor are probably still on site at MX

With the connecting and disconnecting the cables in  hunting and fixing ground loops at EY the side coil on the RO failed to drive the suspension.  Investigating it was obvious we had no coil connected. (Could not see 40 Ohms on the pins)  Check the feed through and everything appeared good and tight but it must have been seated oddly.  Reseated and tightened the connection and the coil was fine.  40 Ohms at he Sat amp 45 at the rack.

Fil and I also took this opportunity to replace the Binary IO card that had failed for the read back.  All signals seem to work now.

Adjusted CP4 Dewar pressure relief valve (aka economizer valve) by turning screw 2 turns clockwise to increase head pressure. It was at 12 psig with tank 62.1% full. This may help increase GN2 flow. However, the PSI document specifies a head pressure of 5 psig for regen exercise. 

It will take a day or so for the head pressure to stabilize to new setting.

Dust6 appears to be off, JeffB checking

After ~3 days of ground loop hunting, fixing, breaking, fixing as well as a relocking of the ISI, we finally are getting good results on the 18 DOF transfer functions for all 3 suspended chains in BSC10 (ETMY main, ETMY reaction, and TMS).

Attached are results of the TMS.  Peaks from today overlap peaks from previous closeout sessions.

Kissel has cast an eye on these and calls them good.

Will post the ETM set shortly.

Next up:  First contact and electrometer optic readings, chamber clean-out, then closeup!

After a lot of experimentation, I have found a way to improve the attenuation of frequencies below 9 Hz in the calibration by 1-2 orders of magnitude, without significantly increasing the computational cost or latency of the pipeline. Here is a list of what I've changed and what I've kept the same:

• In the actuation path, before the actuation filter, I have added a separate high-pass filter using Matlab's firls() function (documentation). firls() designs a linear-phase FIR filter that minimizes the weighted integrated squared error between the desired frequency response in specified intervals and the approximated response of the filter. The filter I've found most effective is designed to have a frequency response of 1 from 9 Hz to the Nyquist rate, and 0 from DC to 8 Hz.  I found improvement when I weighted the stop band more heavily by a factor of 100.  This filter was 2 seconds in length for the tests that I ran.
• I shortened the actuation filter itself from 6 seconds to 4 seconds. (The main reason we needed such a long filter before was to prevent spectral leakage, which is now taken care of by the previous high-pass filter.) It still contains half of a Hann window to the fourth power to roll off low frequencies, in order to further attenuate those frequencies. I found that this roll-off actually worked better than any version of least squares filter I tried in its place.
• In the inverse sensing filter, I replaced the half Hann window to the fourth power with Matlab's least squares filter using firls(). For some reason, this worked better in the inverse sensing filter than the Hann window stuff, even though it did not work better for the actuation. I designed this filter to have a frequency response of 0 from DC to 5 Hz (weighted by a factor of 100) and a frequency response of 1 from 9 Hz to the Nyquist rate. This is what seemed to produce the best results. I did not add a separate high-pass filter in the inverse sensing path, as it seemed unnecessary, and filtering at 16384 Hz gets expensive.

Of all the things I tried, this is what worked the best. Reasons I did not make this even better include:

• I haven't figured out a better way yet
• It is possible to do better with longer or more filters, but since we are doing this in the time domain with FIR filters, adding more filtering gets computationally expensive and adds latency. It is necessary to balance computational cost with quality, as we do not want the pipeline falling behind in low-latency or taking forever to produce offline data on the clusters.

Several plots are attached to show the new features. The first 5 plots are the frequency responses and comparisons to the ideal models for each of the filters used. The last 3 plots are comparisons of C01 data with data produced using the new filters. The attenuation is better by about 1-2 orders of magnitude, and there is just a very small amount of ripple added below 20 Hz.