Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 344 seconds. LLCV set back to 15.0% open.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 1094 seconds. TC A did not register fill. LLCV set back to 33.0% open.

FAMIS task 6881

Spectrum attached.

model restarts logged for Tue 17/Jan/2017
2017_01_17 12:30 h1dc0
2017_01_17 12:30 h1fw0
2017_01_17 12:30 h1fw2
2017_01_17 12:30 h1nds1
2017_01_17 12:30 h1tw1
2017_01_17 12:32 h1broadcast0
2017_01_17 12:32 h1fw1
2017_01_17 12:32 h1nds0

Maintenance day, DAQ restart for VAC-MR changes and EDCU reconfig

model restarts logged for Mon 16/Jan/2017 - Fri 13/Jan/2017

No restarts reported. cdswiki web pages became unavailable close to midnight Fri 13 due to SAML-DS service disruption. Short term fix was installed Sun 15.

model restarts logged for Thu 12/Jan/2017
2017_01_12 20:57 h1fw0
2017_01_12 20:59 h1fw0
2017_01_12 21:00 h1fw0
2017_01_12 21:04 h1fw0
2017_01_12 21:05 h1fw0
2017_01_12 21:08 h1fw0
2017_01_12 21:11 h1fw0
2017_01_12 21:12 h1fw0
2017_01_12 21:15 h1fw0
2017_01_12 21:16 h1fw0
2017_01_12 21:19 h1fw0
2017_01_12 21:20 h1fw0
2017_01_12 21:23 h1fw0
2017_01_12 21:26 h1fw0
2017_01_12 21:27 h1fw0
2017_01_12 21:30 h1fw0
2017_01_12 21:33 h1fw0
2017_01_12 21:35 h1fw0
2017_01_12 21:37 h1fw0
2017_01_12 21:41 h1fw0
2017_01_12 21:42 h1fw0
2017_01_12 21:44 h1fw0
2017_01_12 21:46 h1fw0
2017_01_12 21:48 h1fw0
2017_01_12 21:50 h1fw0
2017_01_12 21:52 h1fw0
2017_01_12 21:55 h1fw0
2017_01_12 21:57 h1fw0
2017_01_12 21:59 h1fw0
2017_01_12 22:00 h1fw0
2017_01_12 22:03 h1fw0
2017_01_12 22:06 h1fw0
2017_01_12 22:07 h1fw0
2017_01_12 22:10 h1fw0
2017_01_12 22:13 h1fw0
2017_01_12 22:15 h1fw0
2017_01_12 22:17 h1fw0
2017_01_12 22:19 h1fw0
2017_01_12 22:20 h1fw0
2017_01_12 22:23 h1fw0
2017_01_12 22:25 h1fw0
2017_01_12 22:27 h1fw0
2017_01_12 22:30 h1fw0
2017_01_12 22:32 h1fw0
2017_01_12 22:34 h1fw0
2017_01_12 22:36 h1fw0

Unexpected restarts of h1fw0 due to LDAS server room without power for several hours due to temperature issues.

model restarts logged for Wed 11/Jan/2017 No restarts reported

TITLE: 01/18 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.69 μm/s
QUICK SUMMARY: VERY ICY ROADS and parking lot. Be careful.
 

Sno-Valley Refrigeration was on site yesterday to investigate a problem with the #2 chiller condenser fan motor VFD. The problem appears to be with the control module and not the drive. The technician swapped a couple of the modules with other ones in the control panel. Different modules for different circuits. This seemed to have remedied the problem at least for now. I left both chillers running last night just in case there was an issue with the repaired chiller, both seemed to have run fine overnight.

I turned off Chiller 1 this morning and we will see how the other chiller does.  

I am going to heed the warnings of fellow Hanfordites and the weather service and avoid the roads tonight.

Ops Shift Log:01/17/2017, Evening Shift 00:00 – 08:00 (16:00 - 00:00) Time - UTC (PT)

Shift Summary: Have been trying to relock the IFO all shift. The commissioners fixed a couple of problems, which helped. Reran initial alignment and continued attempting to relock. I have not been successful. The lock keeps breaking at various places between LOCKING_ALS and CARM_ON_TR. Watching the Guardian logs while running through the locking sequence has not revealed a smoking gun.

   There is an outstanding winter weather alert in effect until mid-day Wednesday. The forecast is for ice and freezing rain starting around 04:00 (20:00). The ice precipitation forecast is at 49% to over 60% for the rest of the evening. Aviation Radar shows a solid precipitation zone over the Tri-Cities, with a small break before another large precipitation cell moves into the area.

   Reports from commissioners going home have conditions deteriorating, with roads becoming slippery and windscreens icing. There is currently no one other than myself on site. 

   Given these conditions, I have put the IFO into a DOWN State, and abandon the rest of this shift. It is my recommendation to cancel tomorrow mornings Owl shift. I have informed the incoming operator and site management. I have informed LLO of our status.    

Activity Log: Time - UTC (PT)

Tonight Jeff B and Jeff K struggled with ALS DIFF for a while.  The problem was with the way the error conditions are handled in the guardian, I've removed the check of the diff beatnote for now but another day we should think carefully about how we want these checks to be handled. 

The ALS_DIFF beatnote was less than -15dB, which is a fault condition for the DIFF guardian, which caused it to make a jump to DOWN.  This caused the ISC lock guardian to recognize a lockloss.  This is an old bug, which has probably been around for more than 2 years. 

For now I have removed the beatnote check, but there are other checks which have the same error handling, so we should come back to looking at how this is done.

J. Kissel

I've run a full suite of top-to-top transfer functions for all four chains of the ITM quads in order to make a complete and traditional assessment of rubbing, given the latest suspicions of it impacting the IFO performance (see LHO aLOG 33380). 

Though I have not yet processed all the data, I can confirm that ITMX R0 and ITMY M0 chains are definitely rubbing with out the recently applied large vertical offsets. With the offsets applied, ITMY's M0 chain rubbing is alleviated. However, the case is not so clear for ITMX's R0 chain. I attach several preliminary plots to whet your appetite. Here "Old Meas" is the last data set we have for the ITMs which were after the vent in 2014. More details to come tomorrow.

Now -- both the FMCS crew is playing with the LVEA temperature (see LHO aLOG 33373), and the observatory is expecting above-freezing temperature for the first time in weeks in the next few days. So, I expect that the rubbing situation will be totally different every 8 hours or so.

So I'll process this data to it's fullest tomorrow, just to grease the analysis script wheels, but I wouldn't get one's hopes up that (a) I'll have any perfect solutions, or (b) that that solution will be right in the next 24 hours. Also, we must remember that IFO problems come in sevens, so even if we do alleviate the rubbing...

--------
The data lives here:
/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ITMX/SAGM0/Data/
2017-01-17_2134_H1SUSITMX_M0_WhiteNoise_L_0p02to50Hz.xml << No extra offsets and Alignment Sliders ON (Nominal)
2017-01-17_2134_H1SUSITMX_M0_WhiteNoise_P_0p02to50Hz.xml Initial Assessment: CLEAR OF RUBBING.
2017-01-17_2134_H1SUSITMX_M0_WhiteNoise_R_0p02to50Hz.xml
2017-01-17_2134_H1SUSITMX_M0_WhiteNoise_T_0p02to50Hz.xml
2017-01-17_2134_H1SUSITMX_M0_WhiteNoise_V_0p02to50Hz.xml
2017-01-17_2134_H1SUSITMX_M0_WhiteNoise_Y_0p02to50Hz.xml

/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ITMX/SAGR0/Data/
2017-01-17_1931_H1SUSITMX_R0_WhiteNoise_L_0p02to50Hz.xml << Extra V & P Offsets w/ Alignment Sliders ON
2017-01-17_1931_H1SUSITMX_R0_WhiteNoise_P_0p02to50Hz.xml Initial Assessment: Unclear if still rubbing.
2017-01-17_1931_H1SUSITMX_R0_WhiteNoise_R_0p02to50Hz.xml
2017-01-17_1931_H1SUSITMX_R0_WhiteNoise_T_0p02to50Hz.xml
2017-01-17_1931_H1SUSITMX_R0_WhiteNoise_V_0p02to50Hz.xml
2017-01-17_1931_H1SUSITMX_R0_WhiteNoise_Y_0p02to50Hz.xml

2017-01-17_2000_H1SUSITMX_R0_WhiteNoise_L_0p02to50Hz.xml << NO Extra V & P Offsets w/ Alignment Sliders ON
2017-01-17_2000_H1SUSITMX_R0_WhiteNoise_P_0p02to50Hz.xml Initial Assessment: Unclear if rubbing.
2017-01-17_2000_H1SUSITMX_R0_WhiteNoise_R_0p02to50Hz.xml
2017-01-17_2000_H1SUSITMX_R0_WhiteNoise_T_0p02to50Hz.xml
2017-01-17_2000_H1SUSITMX_R0_WhiteNoise_V_0p02to50Hz.xml
2017-01-17_2000_H1SUSITMX_R0_WhiteNoise_Y_0p02to50Hz.xml

/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ITMY/SAGM0/Data/
2017-01-17_2031_H1SUSITMY_M0_Mono_WhiteNoise_L_0p02to50Hz.xml << Extra V Offsets ON w/ alignment sliders ON
2017-01-17_2031_H1SUSITMY_M0_Mono_WhiteNoise_P_0p02to50Hz.xml Initial Assessment: CLEAR OF RUBBING, Extra Offsets GOOD.
2017-01-17_2031_H1SUSITMY_M0_Mono_WhiteNoise_R_0p02to50Hz.xml
2017-01-17_2031_H1SUSITMY_M0_Mono_WhiteNoise_T_0p02to50Hz.xml
2017-01-17_2031_H1SUSITMY_M0_Mono_WhiteNoise_V_0p02to50Hz.xml
2017-01-17_2031_H1SUSITMY_M0_Mono_WhiteNoise_Y_0p02to50Hz.xml

2017-01-17_2131_H1SUSITMY_M0_Mono_WhiteNoise_L_0p02to50Hz.xml << NO Extra V Offsets w/ alignment sliders ON
2017-01-17_2131_H1SUSITMY_M0_Mono_WhiteNoise_P_0p02to50Hz.xml Initial Assessment: DEFINITELY RUBBING. Needs Offsets
2017-01-17_2131_H1SUSITMY_M0_Mono_WhiteNoise_R_0p02to50Hz.xml
2017-01-17_2131_H1SUSITMY_M0_Mono_WhiteNoise_T_0p02to50Hz.xml
2017-01-17_2131_H1SUSITMY_M0_Mono_WhiteNoise_V_0p02to50Hz.xml
2017-01-17_2131_H1SUSITMY_M0_Mono_WhiteNoise_Y_0p02to50Hz.xml

/ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ITMY/SAGR0/Data/
2017-01-17_2038_H1SUSITMY_R0_L_WhiteNoise_0p02to50Hz.xml << No extra offsets, alignment sliders ON (Nominal)
2017-01-17_2038_H1SUSITMY_R0_P_WhiteNoise_0p02to50Hz.xml Initial Assessment: CLEAR OF RUBBING.
2017-01-17_2038_H1SUSITMY_R0_R_WhiteNoise_0p02to50Hz.xml
2017-01-17_2038_H1SUSITMY_R0_T_WhiteNoise_0p02to50Hz.xml
2017-01-17_2038_H1SUSITMY_R0_V_WhiteNoise_0p02to50Hz.xml
2017-01-17_2038_H1SUSITMY_R0_Y_WhiteNoise_0p02to50Hz.xml

Evan G., Jeff K., Heather F.

The time for re-calibration of the End Y Pcal PD calibration has elapsed. In addition, we wanted to determine if the calibration of the EY RX PD has been impacted by the observed--and fixed--clipping (see LHO aLOGs 33108 and 33187). This was checked by making a new round of power calibration measurements. The new report is attached (first attachment).

While there, as preventative maintenance we decided to align the outer beam so that it entered the integrating sphere aperture closer to the center. Attached are a before and after images showing this adjustment. 

At the end station, we thought that the measurements were looking good from the spot checks, but only upon getting back to the corner station and using the full suite of Pcal Matlab analysis scripts did we find an issue. The full-data of the working standard had especially weird behavior for two of the measurements (see attachments, especially pcal_endy_20170117_2.png). Consulting with Darkhan T., he suggested that if any other electrical equipment is plugged into the same outlet as the working standard current amplifier, then there can be added noise or glitches in these measurements.

This is unfortunate, but the report indicates the RX/WS ratio (corrected for optical efficiency) has not changed significantly for either of the Pcal beams. Perhaps this is because added noise is sufficiently averaged such that the impact is small. The measurements are only partially conclusive, but they suggest that the calibration has not been impacted by the clipping issue.

The bottom line is that the measurement is not as clean as we would like, and so we would like to repeat this measurement next week to gain confidence in the calibration.

Ops Shift Transition:01/17/2017, Evening Shift 00:00 – 08:00 (16:00 - 00:00) - UTC (PT)

J. Kissel, B. Weaver

Betsy gathered charge today. I post the plots, 'cause my user account appears to be the only one that can consistently use NDS2 without error. The usual plots attached -- effective bias voltage continues to hover around 0 +/- 10 [V] for all quadrants thanks to the regular bias flipping. 

It was all a dream, I used to read Word-Up magazine...

I reset the circuit beaker and it tripped in a few 10s of seconds, reset again and it tripped after a few minutes -> As the heat trace was off upon arrival, I would expect the RTD resistance to be at a minimum.  So, perhaps, if allowed to be energized for a long enough period, i.e. to get warm enough, the resistance would increase to the point that the CB would stop tripping -> As the CB is a 20amp unit and the trace heat is nominally a few watts/foot, something is amiss -> For the time being, I am using a shop light as a local heat source to keep the LLCV from freezing and will pursue this later during daylight etc...

TITLE: 01/17 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventitive Maintenance
INCOMING OPERATOR: Jeff
SHIFT SUMMARY:  Maintenance day. Some work at EY and a stuck Norco truck at EX kept us from starting to lock on time. I just finished an initial alignment but I an having trouble keeping the arms locked in green now. Handing it off to Jeff.
LOG:

• 16:01 Joe to LVEA to start charging.
• 16:18 Joe out.
• 16:33 Manually broke the lock to allow PCAL crew to do their work at EY. Betsy will start charge measurement for EX.
• 16:39 Joe to the roof to clear snow.
• 16:45 Peter starting WP6436
• 17:10 Christina to EX (holding off going to the VEA till charge meas. is complete), Karen to EY.
• 17:16 Fil, Gerardo to EY
• 17:30 JimW, Robert to EY to install STS table on BRS
• 17:35 Ed to LVEA to work on TCS rack.
• 17:37 Norco trucks at gate, one to EX other to CS(80)
• 17:59 Betsy to LVEA to look at ITM OpLevs
• 18:07 Peter, Jason done
• 18:18 Betsy out.
• 18:32 Marc to EY to deliver parts
• 18:55 Christina leaving EX
• 19:00 PCAL crew leaving EY
• 19:04 JimW, Robert back
• 19:15 Hugh to EX
• 19:20 Robert to LVEA to set up
• 19:26 Ed out
• 19:33 Joe and Robert out
• 19:40 Robert to LVEA
• 1947 Hugh back
• 21:00 Jim W to EY to attempt to settle BRS a bit more
• 21:25 Gerardo, Fil, Marc back
• 21:51 Bubba to EY to check on contractor in chiller yard
• 22:25 Jim W back

(Filiberto, Marc, Gerardo)

Cable repair:

We revisited the cable today, we pulled it in towards the VEA, and did a couple of HiPOT tests, we determined that a possible short could be the connector at the cable end (and it was), after removing some of the old shrink wrap, some carbon like deposits were found on the now exposed connector.

The connector was removed (see attached photo of damage) and reterminated, and once again HiPOT tested, this time it passed.

Controller Mods:

The controller unit was powered down and disconnected to change its output to 5.6 KV from 7.0 KV, did a re-configuration after via the front panel.

Cable was reconnected and the controller powered back up, and pressure is going down.

Under WP 6430.

This morning I got a window to mount the "spare" PEM STS at End Y on the BRSY platform (full, final set up shown in the first image). It didn't go completely as planned however. When Jeff and I pulled the feet off of the STS, we found that the studs for the feet were too short to mount to Krishna's mounting table as planned, they just cleared the bottom of the STS pod 9second image). Per Robert's suggestion, we put the feet back on and just put the pod on the table, where I used the original mounting holes as references to align the STS on the table (third image, I did my best to get each foot lined up with the middle of each hole with the same offset along +X). The table itself is clamped with C-clamps to the BRS platform, and mounted so that the edges of the table are square with the BRS base. I tried to put the STS thermal shell on, but there wasn't enough room between the STS and the vertical part of the BRS. Otherwise clearance around the platform looks okay (last image).

BRSY is out of commission for a little bit, I just got done settling it by hand after my and the PCAL team's work today and it looks like the BRS will be almost out of range until it thermalizes. I'll monitor, but the Evening operator may have to wait to turn it back on. For now, I think using WINDY_NOBRSY in SEI_CONF will be enough to get the IFO back running. The switch to WINDY, when it happens, shouldn't drop us out of observe, but should probably be logged.

Temperature sensors for the HVAC system tend to be on the walls. So when it is really cold outside, I think the temperature at the chambers tends to be higher than usual because the system is trying to maintain temperature near the walls, which, because of the higher gradient, tend to be colder relative to the room than usual. Figures 1 and 2 show a moderately good correlation between range/inability to lock and BSC3 temperature (BSC1 is not that different). The trouble really seems to start when BSC3 warms up above 9810 counts. It seems like a long shot, but it might be worth turning the zone 1A temperature set point down, and perhaps others as well, when it is cold, so that the temperature is not so hot away from the walls. Figure 3 shows that BSC3 tends to be warm when the outside temperature is cold. Eventually it might be good to control the HVAC with sensors near the chambers and not on the walls.