I forgot to log this action yesterday:

I noticed that the Diode Chiller alarm 'light' was flashing red, intermittently. I went into the chiller room and also noticed that the Xtal chiller was also a little low.

Long story short: I added 200ml to both.

Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 274 seconds. TC B did not register fill. LLCV set back to 21.0% open.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill not completed after 3600 seconds. LLCV set back to 33.0% open.

H1NDS1 locked up on Monday at around 10:51am localtime.  Here what happened to the best of my understanding.

Background

TJ was testing his operator alog summary generation script with a pre-release version of the nds2-client at my request.  The build of the nds2-client was updated to fix some serious performance regressions on the streaming data interface.  When TJ ran his script it ran for a while, then timed out.  At this point H1NDS1 was unresponsive and required a restart of the the nds/daqd services.

Analysis

The logs showed that the same querie was being repeated over and over again.  Followed eventually by a large number of socket closed events.

Reviewing the code it ran into an endless loop.  If the queiry failed it would retry and retry and retry ...  This is what locked up the nds1 server.  The reason that it worked on an older nds2-client (0.12.2) and failed on the newer client, is there was a gap in the data, that the 0.12.2 client just skipped over.  Where the 0.14.x client raised an error saying there was a gap.  This error triggered a retry, which raised and error, which triggered a retry, which ...

The gap was not 'missing' data but was temporarily not-accessible via H1NDS1.  For background there are two sources for the nds1 server to pull from, its in memory buffer (which is fed by a continuous stream of data from the data concentrator) and from the files on disk.  It takes about 70-120s (depending on where the data falls in the frame, io load on the disk system, ...) from the time that the data leaves the data concentrator before it is written to disk.  Each raw frame is 64s long.  It takes roughly 40s to write the frame, so you need about 100s from the time the frame writer gets the first seconds worth of data for a frame until the frame can be read from disk.  The nds1 servers keep a buffer in memory of live data, and read from it where they can.  H1NDS1 is configured to keep 50s of data in memory, which leaves you with a gap of about 55s when you transition from data on disk to data in the in memory buffer.

As a side note, h1nds0 (which guardian uses) has a 100s buffer and does not see a gap unless there is a glitch in the writting the frame.  I beleive that this is how the nds1 servers at LLO are configured.

Testing this Morning

I ran some tests on the production system this morning from roughly 7:00-7:30am localtime.  I was able to reproduce the endless error loop.  I did not see H1NDS1 lock up, which was probably related to seeing connections close right after the request for data (which is what I would expect).  So I am not sure what kept the connections open until the termination of TJs script on Monday.

Recommendations

I recommend the following

• H1NDS1 be configured with a larger buffer, removing the gap.  In this case it was shown to be an issue.
• TJ will fix the error handler to retry a limited number of times.
• The nds2-client will be set to skip gaps (the old behavior at least with nds1) unless you ask for it to be patched in.
  • The behavior in the code he was testing was to abort on gaps.  We are trying to get a consistent behavior of the nds2 client code between nds1 and nds2 servers, and this is one of the areas the servers just behave differently.

Virtually no change in IMC mirror beam spots before to after the vent.

The #3 Mitsubishi cooling unit in the MSR has been repaired (FRS #8105). Both control boards and a resistor have been replaced and it is pumping out cold air. 

• gate valves are closed
• LVEA is laser safe for film crew
• light pipes between the PSL and HAM1 are closed, Main Beam and ALS

Activities: all times UTC

• 14:00 - Jonathan started work on WP6638
• 14:00 - Chris to LVEA to move engine hoist into high bay, briefly opened the inner high bay door
• 15:00 - PeterK transitioned LVEA to laser safe

Robert and I went yesterday and today to the end-Y station to do some work related to line-hunting. 

Yesterday, we shutdown the MAD CITY LABS nano-drive located on top of the laser table in the VEA room and we also shutdown the CNS Clock II located in the computer room (both systems were shutdown for about two hours). Today, at ~10:58 local time, we shutdown the ESD Pressure Interlock for about an hour. 

Pictures of the three systems are attached to this log.

J. Kissel, B. Gateley, Apollo

Bubba and Apollo have been actively working on the upgrade of the EX HVAC system during the day today and will continue tomorrow, but since we've learn so much from EY (LHO aLOG 36271), I wanted to look at EX in the same light. No need to take any action other than planned thus far.

Unfortunately it looks like, in the midst of the upgrade, EX is going to get even hotter than EY. The PCAL temperature sensors report a 4.1 [deg C] = 7.38 [deg F] change from where we were 5 days ago. As such, the suspensions are in quite a different place than when in observation. We look forward to the completion of the upgrade, but we should be aware that it will take quite some time for the SUS to recover.

Both these cold cathode gauges went out today. I found PT-410's interlock OFF. Neither should have been effected by today's events:  1) corner beckhoff reboot; 2) Robert powering OFF rack at EY.

Machine was responding slowly when I went to update the vacuum site overview.

We have been working to stabilize the temperatures, mostly at End Y because that is the station that has the completed FMCS controls. End X should be completed by the end of this week.
During this process several changes have been made to various components such as the face/by-pass damper, cooling valve, chillers etc. Currently the changes have only been made to the End Y station where we were trying to maintain 68 degrees F. 

After several conversations with Jeff K. and some calculations made by Jeff, he has determined that a more suitable temperature for E Y would be ~64 degrees F. I have changed the set point in the E Y VEA to to 64 F. 

I will monitor the temperature closely tonight and hopefully this will have the correct effect on the suspensions.

I also had an alarm on the Mid X supply fan this morning and upon further investigation, I found that someone had turned the fan off at the electrical disconnect. I restored the HOA switch to auto position and started the fan.         

Around 1:30 pm local time, Gerardo and I opened GV 1,2 slowly by ramping up to 500 rpm, 800 rpm, 1200 rpm, finally 1700 rpm. Note that the gate annuli were pumped out before opening. GV2's volume was sustained by its AIP alone. GV1 needed help from aux turbo cart.

GV 5,7 are LOTO.

All ion pumps are ON and valved out, except IP1 which is OFF because its isolation GV leaks (valve open).

Check list before opening GV 5,7:

1. Valve in IPs
2. Turn IP1 ON
3. Collect RGA scan
4. Pump out GV gate annuli

[Aidan,TJ,Nutsinee]

Following the HWSY lens replacement during last week's mini-vent, we have been working to recover the alignment of both HWS beams. Without the ALS beams, this is accomplished using the more laborious method of injecting the HWS beams into the vacuum system, close to centered on the in-vacuum lens, and swinging SR3 around in PITCH and YAW. The return power on the HWS CCDs is plotted as a function of position and yaw. This technique relies on the ITMs being properly aligned.

HWSX

We received no return beam from ITMX in any portion of the PIT and YAW phase space. We concluded that, given the HEPI lock down and ITMX activity last week, ITMX has not returned to the same alignment that it had before the vent. Without access to the ALS beams or the optical levers, it was difficult to make any progress on HWS alignment until the ITMX alignment is corrected. The HWSX alignment effort is on hold until ITMX is fixed.

HWSY

After locating three regions in SR3 angular phase space that held a total of four return beams, we identified which belonged to the ITMY HR surface by running a ring heater test and observing the expected thermal lens (in agreement with the online simulation).

We had to fold the HWSY layout to lengthen the distance from the last lens to the HWSY CCD such that the latter could be placed at the image plane of ITMY. We added an additional optic and moved the HWS CCD as shown in the attached images. The return beam looked much cleaner and the main BS EQ stops look much sharper (less diffraction).

The current status is that the HWSY camera is running WITHOUT the Hartmann plate installed but WITH the bandpass filter installed. The HWSY CCD is currently attached to the HWSX computer (H1HWSMSR) instead of HWSX. This is a temporary measure until the HWSY computer (H1HWSMSR1) is functioning correctly.

Before:

After:

HWSY return beam - misaligned to show the EQ stops.

WP 6639

The Beckhoff PLC code for h0vacmr was updated to match the change of the IP6 controller from a dualvac to a gamma. This required stopping and restarting the code, so there should be a gap in the data for the vacuum MR channels during this time.

The control room MEDM screens are updated except for the dedicated vacuum computer in the back of the room. I will update that one when the film crew is done.

The minute trend files were renamed to make the following changes:

Old -> New
H0:VAC-MR_IP6_EI166_A_HV_VOLTS -> H0:VAC-MR_IP6_EI166_HV_VOLTS
H0:VAC-MR_IP6_EI166_A_HV_VOLTS_ERROR -> H0:VAC-MR_IP6_EI166_HV_VOLTS_ERROR
H0:VAC-MR_IP6_EI166_A_HV_KVOLTS -> H0:VAC-MR_IP6_EI166_HV_KVOLTS
H0:VAC-MR_IP6_EI166_A_HV_KVOLTS_ERROR -> H0:VAC-MR_IP6_EI166_HV_KVOLTS_ERROR
H0:VAC-MR_IP6_II166_A_IC_VOLTS -> H0:VAC-MR_IP6_II166_IC_VOLTS
H0:VAC-MR_IP6_II166_A_IC_VOLTS_ERROR -> H0:VAC-MR_IP6_II166_IC_VOLTS_ERROR
H0:VAC-MR_IP6_VI166_A_PRESS_TORR -> H0:VAC-MR_IP6_VI166_PRESS_TORR
H0:VAC-MR_IP6_VI166_A_PRESS_TORR_ERROR -> H0:VAC-MR_IP6_VI166_PRESS_TORR_ERROR
H0:VAC-MR_IP6_CS166_A_STATUS -> H0:VAC-MR_IP6_CS166_STATUS

The following channels are no longer available through dataviewer:

H0:VAC-MR_IP6_166_STATUS
H0:VAC-MR_IP6_CS166_A_STATUS
H0:VAC-MR_IP6_CS166_B_STATUS
H0:VAC-MR_IP6_EI166_A_HV_KVOLTS
H0:VAC-MR_IP6_EI166_A_HV_KVOLTS_ERROR
H0:VAC-MR_IP6_EI166_A_HV_VOLTS
H0:VAC-MR_IP6_EI166_A_HV_VOLTS_ERROR
H0:VAC-MR_IP6_EI166_B_HV_KVOLTS
H0:VAC-MR_IP6_EI166_B_HV_KVOLTS_ERROR
H0:VAC-MR_IP6_EI166_B_HV_VOLTS
H0:VAC-MR_IP6_EI166_B_HV_VOLTS_ERROR
H0:VAC-MR_IP6_HS166A_A_START
H0:VAC-MR_IP6_HS166B_A_STOP
H0:VAC-MR_IP6_HS166C_B_START
H0:VAC-MR_IP6_HS166D_B_STOP
H0:VAC-MR_IP6_II166_A_IC_VOLTS
H0:VAC-MR_IP6_II166_A_IC_VOLTS_ERROR
H0:VAC-MR_IP6_II166_B_IC_VOLTS
H0:VAC-MR_IP6_II166_B_IC_VOLTS_ERROR
H0:VAC-MR_IP6_VI166_A_PRESS_TORR
H0:VAC-MR_IP6_VI166_A_PRESS_TORR_ERROR
H0:VAC-MR_IP6_VI166_B_PRESS_TORR
H0:VAC-MR_IP6_VI166_B_PRESS_TORR_ERROR
H0:VAC-MR_IP6_XA166_A_FAULT
H0:VAC-MR_IP6_XA166_A_FAULT_ERROR
H0:VAC-MR_IP6_XA166_B_FAULT
H0:VAC-MR_IP6_XA166_B_FAULT_ERROR

[Jenne, Kiwamu, Vaishali with help from JimW and JeffK]

Continuing the locking effort from yesterday (36197), we managed to get the mode cleaner to lock. Here's a roll down of the events from today:

1. We (Jenne and I) first aligned the MC2 REFL camera because we were almost on the edge of the PD.

2. As this didn't fix the not locking problem, we asked Kiwamu for help and then we looked at a bunch of parameters like filters, gain thresholds, ramp times.  We also looked to check if the suspensions were behaving correctly and then found the mirror which had been turned off. This button (MC2 M2) was not in use at all. Maybe we should have double checked the sdf differences but we know better now.

While we were aligning the MC2 REFL we noticed that MC2 Trans wasn't looking like what it used to. We tried to trace the beam and found that the camera was being illuminated by a ghost beam and not the actual transmission beam of the MC. We found a bright spot by looking into the light pipe and then found that the beam wasn't coming onto the telescope at all. As we couldn't see anything on the viewer card, we turned up the laser power to 10 W and found the beam again in the light pipe only with the IR viewer.

Jenne then tried to gently tap the light pipe (this is the same pipe that had problems yesterday and was fixed) with me looking at the bright spot and it didn't move at all which leads to us believe that the beam might be hitting the edge of the table somewhere.

After hypothesising out loud that this might have been because the tables hadn't returned to the correct positions, we were corrected by JimW who told us that the ISIs return back to their positions on their own.

We then tried to change the axis of the modecleaner in order to redirect the beam in the light pipe but we weren't too successful.

Not having solved this mystery of what happened to MC Trans, we concluded the work for today with a modecleaner that locks at 2 W and 10 W.

Jenne will correct me if I have missed something or used wrong names of mirrors in comments !