Bubba and Vern on site for NOVA Film Crew Shoot with Jana Levin.

Working in peace and quiet. Vertex pressure is looking good @ 2.7e-7 Torr.

FRS 8166

Filament #1 on the Pfeiffer RGA seems to be burned out. It turns off a second after turning it on. I switched to filament #2. It would not stay on either. The default emission current was set to 2 mA. I lowered it to 0.1 mA. The 
filament (#2) stayed on at that value. Then I raised to 0.3 mA (you have to 
click in a different window to activate the new setting btw). Then I 
raised to 1 mA where the filament automatically turned off. Went back to 
0.3 mA and that's where it's sitting for now. Pfeiffer suggested we let it 
warm up and slowly ramp up to 2 mA. These steps are sometimes needed when the filament hasn't been used for a long time (not the case here). Something else is going on.

The error message we were getting from the filament turning off was 
"emission error" or E002. Pfeiffer said a burned out filament will usually 
yield a different type of error, but sometimes these fail such that the 
filament still registers a current - when the broken filament cools it 
reconnects. Here is a manual describing emission error:

ftp://ftp.aerodyne.com/ACSM/ACSM_Pack/Quadara2/Manuals/English/QUADERA/quadera.pdf

We verified the RGA turbo is ON and pumping out the RGA volume. One theory is that during the bake of the RGA (during oven load bake) we sprung an air leak and now the turbo can't keep up with the leak rate which is causing a high head pressure - too high for a filament to tolerate. Note that one of the two turbos on VBOC is at 33C, which is a little high (two fans are blowing on it). The other is at 27C. Also, the foreline pressure is reading 0.018 Torr. Normally at 0.016 or 0.017 Torr. We should add HV pressure gauges to the system for times like this.

Also, one of the electronics units has a recessed pin socket. The other 
two spare units yielded the same error (filament turning off almost 
immediately). Maybe instead of replacing the SRS with Pfeiffer we should 
replace the Pfeiffer with SRS because of these pin issues due the frequency of removing and reinstalling the electronics in between bakes.

J. Kissel, NOVA Film Crew

I've modified the main displays on the front wall at the request of the NOVA film crew. The lower screen is displaying the GW150914 waveform and the background of the DTT session showing the Live Sensitivity on the Upper screen has been changed to black.

There is zero intention of this being permanent, but I figure there will be no harm in leaving it as such over the weekend since the content normally displayed on these screens only has meaning when we have light in the arms, no one is intending to commission the corner station over the weekend, and there'll be no operations specialists on shift over the weekend. So, it should just be the film crew, Dr. Levin, and their site liaison, and they all want it this way!

I'll restore the displays to normal Monday.

Gerardo, Jonathan, Patrick

We used dataviewer to plot the raw, second and minute trends for the following channels from May 18 2017 16:38:59 UTC - May 18 2017 20:38:59 (attached):

H0:VAC-EY_Y3_PT410A_PRESS_TORR
H0:VAC-EY_Y3_410_PIRANI_INTLK
H0:VAC-EY_Y3_410_PWR_REQ

H0:VAC-EY_Y3_410_PIRANI_INTLK should only be either 0 or 1. In the second trends it takes a value of ~ 2.2.
In the seconds trends H0:VAC-EY_Y3_PT410A_PRESS_TORR jumps from 0 to 1, which is not realistic for a pressure reading.

• gate valves are closed
• LVEA is laser hazard

Activities: all times in UTC

• 16:00 - Jim, taking EX ISI platform ofline and then back to reset the target position, needed due to temperature changes
• 16:06 - Karen to the optics lab and clean storage area
• 16:16 - Jim running a measurement on EX ISI, needs quiet in the building, runs about 2 hours
• 16:20 - Jason to LVEA to pull HAM oplevs
• 16:35 - Jenne to optics lab
• 16:42 - Betsy to LVEA, parts
• 16:51 - Karen out of LVEA, heading to VPW
• 16:55 - Jason and Ed, done with removing HAM oplevs
• 17:17 - Amber and film crew into the LVEA
• 17:27 - Richard and more film crew into the LVEA
• 17:39 - Jenne to LVEA
• 18:28 - Robert to EY
• 19:28 - Betsy to My and EY
• 20:00 - Jenne and film crew in LVEA
• 20:28 - Betsy back
• 20:45 - Corey to LVEA
• 21:09 - Corey's tour arrives, out of LVEA
• 21:42 - Bubba and film crew on the roof
• 22:25 - Bubba and film crew back back from the roof
• 22:37 - Sheila and Robert to EY, will be laser hazard, opening tables doors
• 22:58 - Jenne and film crew to LVEA

As of 23:00, end of day shift coverage:

• Sheila and Robert at EY
• Jenne and film crew in LVEA
• Nutsinee is on site, and will transition the LVEA to Laser Hazard once the film crew is set up at ISCT1

I forgot to log this action yesterday:

I noticed that the Diode Chiller alarm 'light' was flashing red, intermittently. I went into the chiller room and also noticed that the Xtal chiller was also a little low.

Long story short: I added 200ml to both.

Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 274 seconds. TC B did not register fill. LLCV set back to 21.0% open.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill not completed after 3600 seconds. LLCV set back to 33.0% open.

H1NDS1 locked up on Monday at around 10:51am localtime.  Here what happened to the best of my understanding.

Background

TJ was testing his operator alog summary generation script with a pre-release version of the nds2-client at my request.  The build of the nds2-client was updated to fix some serious performance regressions on the streaming data interface.  When TJ ran his script it ran for a while, then timed out.  At this point H1NDS1 was unresponsive and required a restart of the the nds/daqd services.

Analysis

The logs showed that the same querie was being repeated over and over again.  Followed eventually by a large number of socket closed events.

Reviewing the code it ran into an endless loop.  If the queiry failed it would retry and retry and retry ...  This is what locked up the nds1 server.  The reason that it worked on an older nds2-client (0.12.2) and failed on the newer client, is there was a gap in the data, that the 0.12.2 client just skipped over.  Where the 0.14.x client raised an error saying there was a gap.  This error triggered a retry, which raised and error, which triggered a retry, which ...

The gap was not 'missing' data but was temporarily not-accessible via H1NDS1.  For background there are two sources for the nds1 server to pull from, its in memory buffer (which is fed by a continuous stream of data from the data concentrator) and from the files on disk.  It takes about 70-120s (depending on where the data falls in the frame, io load on the disk system, ...) from the time that the data leaves the data concentrator before it is written to disk.  Each raw frame is 64s long.  It takes roughly 40s to write the frame, so you need about 100s from the time the frame writer gets the first seconds worth of data for a frame until the frame can be read from disk.  The nds1 servers keep a buffer in memory of live data, and read from it where they can.  H1NDS1 is configured to keep 50s of data in memory, which leaves you with a gap of about 55s when you transition from data on disk to data in the in memory buffer.

As a side note, h1nds0 (which guardian uses) has a 100s buffer and does not see a gap unless there is a glitch in the writting the frame.  I beleive that this is how the nds1 servers at LLO are configured.

Testing this Morning

I ran some tests on the production system this morning from roughly 7:00-7:30am localtime.  I was able to reproduce the endless error loop.  I did not see H1NDS1 lock up, which was probably related to seeing connections close right after the request for data (which is what I would expect).  So I am not sure what kept the connections open until the termination of TJs script on Monday.

Recommendations

I recommend the following

• H1NDS1 be configured with a larger buffer, removing the gap.  In this case it was shown to be an issue.
• TJ will fix the error handler to retry a limited number of times.
• The nds2-client will be set to skip gaps (the old behavior at least with nds1) unless you ask for it to be patched in.
  • The behavior in the code he was testing was to abort on gaps.  We are trying to get a consistent behavior of the nds2 client code between nds1 and nds2 servers, and this is one of the areas the servers just behave differently.

Virtually no change in IMC mirror beam spots before to after the vent.

The #3 Mitsubishi cooling unit in the MSR has been repaired (FRS #8105). Both control boards and a resistor have been replaced and it is pumping out cold air. 

• gate valves are closed
• LVEA is laser safe for film crew
• light pipes between the PSL and HAM1 are closed, Main Beam and ALS

Activities: all times UTC

• 14:00 - Jonathan started work on WP6638
• 14:00 - Chris to LVEA to move engine hoist into high bay, briefly opened the inner high bay door
• 15:00 - PeterK transitioned LVEA to laser safe

Robert and I went yesterday and today to the end-Y station to do some work related to line-hunting. 

Yesterday, we shutdown the MAD CITY LABS nano-drive located on top of the laser table in the VEA room and we also shutdown the CNS Clock II located in the computer room (both systems were shutdown for about two hours). Today, at ~10:58 local time, we shutdown the ESD Pressure Interlock for about an hour. 

Pictures of the three systems are attached to this log.

We have been working to stabilize the temperatures, mostly at End Y because that is the station that has the completed FMCS controls. End X should be completed by the end of this week.
During this process several changes have been made to various components such as the face/by-pass damper, cooling valve, chillers etc. Currently the changes have only been made to the End Y station where we were trying to maintain 68 degrees F. 

After several conversations with Jeff K. and some calculations made by Jeff, he has determined that a more suitable temperature for E Y would be ~64 degrees F. I have changed the set point in the E Y VEA to to 64 F. 

I will monitor the temperature closely tonight and hopefully this will have the correct effect on the suspensions.

I also had an alarm on the Mid X supply fan this morning and upon further investigation, I found that someone had turned the fan off at the electrical disconnect. I restored the HOA switch to auto position and started the fan.