There are a few problems with the way violin damping is managed in the guardian.  We have fixed some but it would be good to have somebody look at the VIOLIN mode damping state to improve it (to reduce the amount of time we go without checking for locklosses, and to use some for loops)

The violin mode damping state takes too long between times that it checks for locklosses.  We had one lockloss earlier today caused by a large violin mode, which unlocked us early in the VIOLIN mode damping state, but the guardian didn't recognize the lockloss for 17 seconds causing things to get quite rung up. Vaishali and I edited the state so that the 10 second sleep is replaced by a timer (which continuously checks for lockloses), but it still goes for about 11 seconds without checking for locklosses. 

Kiwamu found that the adjust damper gain function was adjusting the gain to set the filter output of each mode to 1.5e5, so we reduced the target output to 1.5e4. 

I also saw that the LSC triggering meant that the violin damping loops didn't send huge signals to the suspensions even though the gaurdian took a long time to recognize the lockloss.  However, in the down state we were setting the TRAMP and the gain to 0, which was causing a large signal to go to the optics.  I have removed setting the TRAMP to 0, so it will be the value that it is set to in the VIOLIN mode damping state (10 for most modes).

I opened WP 7005 this evening after consulting Keita in order to keep testing the next release of the nds2-client.  During office hourse I had run into some test failures in the day that I needed to track down when testing on a CDS workstation.  I worked remotely from ~9:30pm - 12:30 localtime on nucws20.  My work did not require accessing the local nds or epics resources, just simulated or recorded data, so there was no impact on the recovery work.

The issues I had ended up being related to the matlab environment setup and test scripts that did not account for matlab 2012b still being used.  After I was able to get a cleaner environment and adjust some of the test scripts for matlab 2012b all the tests passed.

Unless something comes up quickly the code I was testing should be submitted as the next nds2-client release and I will likely file a work permit to install it in the control room next week.

TITLE: 05/26 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: Sheila, Kiwamu, Jeff K. and Vaishali working on recovery. Made it to 26 W.
LOG:

15:18 UTC Jim running low frequency measurements on end Y ISIs.
15:44 UTC Jason to LVEA to look for a crate lid by the H2 PSL enclosure.
15:53 UTC Jason done.
16:35 UTC Jim done measurements and reverting configuration back for locking.
16:39 UTC Kiwamu starting locking attempts.
16:59 UTC Travis driving car to receiving to load material.
16:59 UTC Christina to mid X.
17:24 UTC Aidan working on HWS computer through remote login.
17:42 UTC Dave restarting h1hwsmsr.
18:15 UTC Amber leading school tour through CR.
18:21 UTC Nutsinee to LVEA TCS cabinet.
18:53 UTC Nutsinee out of LVEA.
19:49 UTC Starting initial alignment.
19:54 UTC Dave restarting nds1.
20:02 UTC Dave restarting nds0.
20:21 UTC Initial alignment done.
20:49 UTC Gerardo to end X.
21:08 UTC Kyle to mid Y to retrieve cable.
21:14 UTC Gerardo back.
21:22 UTC Kyle back.
21:51 UTC Travis to LVEA to retrieve tools.
21:59 UTC Travis back.
22:17 UTC Jeff K. taking Nick on tour through LVEA.
22:43 UTC fw0 crashed.
22:46 UTC Jeff K. and Nick back
23:13 UTC fw0 crashed.
23:24 UTC fw0 crashed.
00:22 UTC Starting initial alignment.
01:02 UTC Initial alignment done.

Sheila D., Patrick T.

Reached 26 W at approximately 04:13 UTC. Increased CSOFT gain by 3dB. This seems to have improved the stability. Lines just above 20 Hz are from a2l script. Frequency noise is high above 2kHz. Jitter noise appears better.

V. Adya, S. Dwyer, K. Izumi, J. Kissel, P. Thomas

After doing the same kind of alignment reference adjustment to the POP QPDs -- i.e. moving around PR2, we were able to surpass yesterday's problems and get through the CARM reduction and closing the ARM's SOFT loops. As such, while holding just before the transition to DC READOUT, we captured new green alignment offsets (updating green QPD offsets and green camera offsets). I attach the relevant screenshots showing the deference between the previous values and the old values. Y arm QPD B Yaw moved the most. The changes to the green camera offsets where in the same direction and rought the same value to which Sheila and I adjusted the offset manually last night.

We lost lock due to a dP/dTheta 0.45 Hz oscillation, and are re-running initial alignment, and trying again.

J. Kissel

Having worked with Bubba and the Apollo Crew, we've finally settled on a stable VEA temperature that's roughly where we were before (EY dead on, EX about 0.5 [deg C] / 1 [deg F] colder). However, with the upgrade, we now see a diurnal oscillation in temperature at the 0.25 (EY) 0.5 [deg C] (EX) (0.5 / 1 [deg F], respectively) level. While there are spikes 0.1 [deg C] every ~24 hours that coincide with the peaks of the oscillation, every other temperature sensor around the VEAs report some sort of temperature oscillation that fluctuates more than this.

I attach 20 day trends for each end station, 3 types for each:
(1) Showing the FMCS temperature sensor against the most calibrated / precise sensors in and around the VEA (with the PCAL receiver module being most representative)
    Here, we see that the VEAs (FMCS, PCAL) were stable compared the exterior electronics rooms (EBAY) prior to the upgrade (16 days ago for EY, 9 days ago for EX). Now it appears that the VEA is in-sync (or delayed/anti-corellated) with the EBAYs.
(2) Showing the PCAL receiver module temperature against the vertical position of the three suspensions in the end-stations.
    Just in case you needed proof that changes in temperature are real, and larger than before the upgrade.
(3) Showing all temperature sensors in the VEA.
    More proof that even the worst, uncalibrated temperature sensors have seen the fluctuations over the past 20 days.

We haven't yet recovered the IFO fully, so it's difficult to say yet how our alignment (and other things) are really impacting the duty cycle of the IFO, but we should continue to keep an eye on this. These temperature oscillations are definitely slower than the bandwidth of the ASC loops so we should be able to control for such fluctuation -- T1100595 suggest we have about 500 [urad] of range -- but it's a question of whether we'll need to do initial alignment more often between lock stretches. 

Plus there's always a risk of temperature impacting electronics in some terrible unknown way, but given that the EBAY electronics have been oscillating in temperature since before the upgrade, it may not be any worse than before.

Keita noticed that the PSL power watchdogs were off and alerted me to this.  When we went into the PSL yesterday (1st bullet point), we turned off the power watchdogs; this is standard procedure when working in the PSL enclosure (if something we do briefly effects the output power of the laser to the point it would trigger the watchdog, we don't want it taking the whole system down).  Apparently when we left the enclosure we forgot to turn the watchdogs back on, which then of course defeats the purpose of having power watchdogs in the first place.  Luckily, nothing happened in the last ~26 hours and the laser is still running.  We (Team PSL) need to be more vigilant about this in the future.

I turned them back on at approximately 23:14 UTC (16:14 PDT).

Attached is an RGA scan of the full, or open, LHO vacuum volume prior to isolating and venting the Vertex volume on May 8th (see h0rgacs_SEM_analog_05082017a).  Also included is a scan taken after the Vertex was pumped down and re-combined with the rest of the vacuum volume (see h0rgacs_SEM_analog_05262017a).  

after being super reliable for all of O2, h1fw0 daqd stopped running at 15:42 PDT. The error log suggests it was not able to write to its QFS disk system, and internal buffers filled up. Monit restarted daqd, and it has been running for 20 minutes so far, but second trend frame file writing looks too slow

[Fri May 26 15:41:55 2017] main profiler warning: 1 empty blocks in the buffer
[Fri May 26 15:41:56 2017] main profiler warning: 0 empty blocks in the buffer
[Fri May 26 15:41:57 2017] main profiler warning: 0 empty blocks in the buffer
[Fri May 26 15:41:58 2017] main profiler warning: 0 empty blocks in the buffer
[Fri May 26 15:41:59 2017] main profiler warning: 0 empty blocks in the buffer
....

FRS8182

WP7001

I've started the process of archiving the raw minute trend files from h1tw1 as its SSD-RAID is 87% full.

The first part of this procedure is to archive the minute_raw directory on h1tw1 as minute_raw_1179862578 and create a new empty minute_raw within the five minute quiet period when h1tw1 is not writing minute trends. This permitted the archival to be done without stopping and restarting the raw minute trender.

Then h1nds1 and h1nds0's daqdrc files were temporarily modified to read the past 140 days of minute trends from h1tw1's archive directory. This requires a restart of the daqd processes on these machines:

I verified that last day's minute trends are now being read by h1nds1 from the online and archived minute_raw directories on h1tw1.

The next step in the process is to copy all the files from h1tw1 /trend/minute_raw_1179862578 over to the MSR SATABOY using h1fw1. This is done using a low priority flag, so as not to interfere with h1fw1's frame writing. It is a slow process, taking about 3 days. This allows he wiper script to keep the SATABOY RAID from filling.

Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 17 seconds. TC B did not register fill. LLCV set back to 17.0% open.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 107 seconds. LLCV set back to 38.0% open.

H1 Status:

• 12:03UTC - L1 out of lock, I took H1 to commissioning to move IM1, IM2, IM3 yaw for range
• 12:33UTC - R wave from M5.0 in Indonesia arrives
• 12:47UTC - H1 back in Observe
• as of 12:54UTC - 3-10Hz ground motion still elevated

SITE:

• 12:43UTC - Bubba calls, says the Hanford Fire Department was here at Mid-Y and they say they were inside the building
  • I was not aware of any activity