J. Kissel, C. Whittle

While playing around with actuating on the ITMs to test out Chris's new charge scripts, we noticed that the Voltage monitor for the DC Bias was railed at -32768 for both ITMX and ITMY. Worried that it was us, we trended the channel for the past 12 days. Other than the shenanigans on Tuesday for the timing upgrade, it looks like these monitors have been railed constantly for all 12 of those days, and unless we're up and running in DC Readout, there's no requested bias voltage on ITMX and ITMY's requested voltage is has been changing.

We'll have a look at the electronics when convenient.

Jim reported yesterday that the cold restart of the ISI from the long Tuesday shutdown led to a thermal sag causing an increase vertical drive in turn causing a yaw...Not sure why the horizontal loops would not keep the yaw at minimum unless the heating is also impacting Stage0.  Attached are 48 hours of the ITMY vertical drives, one horizontal drive and the OpLev signals.

Okay, based on the horizontal drive, the loop is dealing with the yaw but of course relative to Stage0.  Is HEPI or Stage0 the source of the actual Yaw or a secondary affect...not sure I can reconcile that.  Also, while V1 and V2 show pretty flatline drive before and since the isolation cycle yesterday afternoon, V3 and the H2 show a continuing trend that was present before the reisolation.

J. Kissel, T. Sadecki, C. Whittle

We found the IFO Up and running this morning when we got in, and it had be up for the last 8 hours. It was stagnant at 2 [W] input power, in DC READOUT TRANSITION. However, I noticed immediately that just about all IFO signals were reporting huge slow oscillations. I attached a screenshot of one of Sheila's strip tools that shows the oscillation was present in the ASC signals for at least 80 minutes, then a trend of the entire lock stretch revealed that there was a power oscillation in the Y arm at similar frequencies. As of writing this aLOG we've lost lock; unclear if it was because of Chris's testing out of ITM charge measurements, or if the oscillation just finally got out of control.

I won't speculate what this might have been, but I wanna capture it for the record and talk with the vanguard in case it has something to do with any of the new SRM alignment scheme (i.e. LHO aLOG 28538).

We've had several locklosses throughout the day due to the 18040 Hz ETMY PI mode. 

This last lock at 25W, I finally found a combination of settings that seemed to be actively damping it according to the OMC DCPDs.  (Using the QPD as the error signal, ETMY_PI_DAMP_MODE4, -60deg +30deg -3000gain).  In the OMC DCPD spectrum, the peak was going down.  According to the RMS monitor of this mode in the DCPDs, I was winning.  However, according to the RMS monitor of this mode in the ETMY QPD, the mode was still going up.  We lost lock because of this.  I don't know of any DQed high freq channels for the QPDs, so we can't look back in time at the spectrum to see if that helps illuminate the situation.

Anyhow, if someone can help me understand how a mode can seem like it's going down according to the OMC DCPDs, but still actually be increasing according to the transmon QPDs and causing a lockloss, that would be great.

Title:  07/20/2016, Day Shift 23:00 – 07:00 (16:00 – 00:00) All times in UTC (PT)
State of H1: IFO locked. Wind is a Light to Gentle Breeze (4 to 12mph). Seismic activity is low, with the exception of End-Y. It is a bit elevated; but should pose no operational difficulties. Microseism is also low.         
Commissioning: Commissioners are commissioning.
Outgoing Operator:  Ed 
 
Activity Log: All Times in UTC (PT)

23:00 (16:00) Take over from Ed
23:41 (16:41) Gerardo – Back from N2 overfill



Title: 07/20/2016, Day Shift 23:00 – 07:00 (16:00 – 00:00) All times in UTC (PT)
Support: Jenne, Sheila, Kiwamu          
Incoming Operator: N/A

Shift Detail Summary: IFO has been up and down as the commissioner's were working through a variety issues and improvements. Did one initial alignment and several relocks. All went well. Environmental conditions remained favorable for commissioning work during the entire shift. There were 3 earthquakes during the last hour of the shift (4.8, 4.8, 5.3mag), which (per Terramon) did not represent much of a threat to the IFO lock. They did pass without any notice. 

Cleared TIM errors: H1IOPSUSEY, H1SUSETMY, H1SUSETMYPI, H1SUETMXPI
Cleared IPC errors: H1IOPSEIEY, H1ISIETMY, H1PEWMEY, H1ALSEY, H1IOPSEIEX, H1PEMEX, H1ALSEX, H1SCEX
Cleared ADC errors: H1IOPSUSEX
     

The goal of yesterdays firmware upgrade was to change the behavior of the blinking LED lights and to allow for additional diagnostics of the problem. In the past the LED typically had three states: (i) Off, (ii) Blinking and (iii) On. This upgrade interchanged the states (ii) and (iii). For a typical link light this means: Off indicates no fiber signal, Blinking indicates a fiber signal is present, and On means the timing signal is received and synchronized. Operating without error now means all LEDs are On.

Furthermore, it is possible with the Altium JTAG port to change the behavior of the LED state while running and without disrupting the timing signal. LEDs which are on can be made to blink in rates of 0.5Hz, 1Hz, 2Hz, 4Hz, etc. This should allow us to isolate the coupling site.

The upgrade was successful for all timing slaves, but failed for the timing master/fanouts. Strangely, it failed due to bogus code in svn on how to calculated the CRC-32 (ethernet) checksum. Baffling. More investigations are ongoing.

Signal for GV15 AIP went red around 18:41 utc, noticed it after lunch, went out to the mid station and it turned out to be a bad extension cord (the wire crimped to the spade slipped out of its crimped joint), replaced the bad cord with another one for now, we will install the permanent one soon.

Note: This is the 3rd cord of this type that has failed in a similar manner, we will be replacing all extension cords of this type in the near future.

Over-filled CP3 with the exhaust bypass valve fully open and the LLCV bypass valve 1/2 turn open.

Flow was noted after 45 minutes, closed LLCV valve, and 3 minutes later the exhaust bypass valve was closed.

I did a trend of the level and noticed that sometime Monday the slope of the consumption changed (it slowed down considerably).  See attached.

I changed the manual setting from 17 up to 25, left such setting there for 2 hours, it quite did not look right so I dropped it to 24.

We closed some alignement loops for SRM using a dither and demodulating POP90 this morning.  These loops seem to have a ugf below 100 mHz, so we may want to increase their gain, but they seem to be working well for now.  They come on in the guardian durring the DRMI ASC and leave them on.  They maintain POP 90 (and the ratio of AS90/POP90) at a reasonable level as we go through the CARM offset reduction, engaging the soft loops, and increasing the power. 

We saw a instability that seemed to be a cross  coupling between SRM and SOFT yaw loops, Jenne increased the gain in the soft yaw loops by a factor of 10 which seems to have taken care of the problem and is now in the guardian. 

As long as this keeps working, operators should no longer have to adjust SRM. 

Hugh, JeffK, JimW

Short version:

• Large actuator drives may cause thermal drifts in the ISI, especially after a "cold" start.
• Things like power outages or long maintenance periods where ISI actuators get no signal for hours should be followed by warmup periods for the ISIs.
• We should damp as soon as possible, then do an isolate / cook / de-isolate / re-isolate cycle. Or just live with a "squirmy" IFO for ~6hrs.
• It's possible the timing stuff yesterday caused the drifts Sheila saw yesterday, but theres no evidence for that yet.

I noticed this afternoon that all of the BSC-ISIs were pushing large DC-ish drives on their St1 vertical actuators (generally about 2000 counts, normal drives are ~100). When I trended the BS and ITMX drives against their Sus oplevs, there seems to be a very clear correlation between large ISI drives and oplev yaw. I think this may be the cause of the drifts that Sheila complained about yesterday. The first attached plot shows the BS 3 vertical drives and BS oplev yaw, second is for the same for ITMX.

One possible cause of this is the actuators heating up the ISI. During the timing kerfluffle yesterday, all of the ISIs sat for ~6 hours with no acuator drive. The control room then recovered all of the platforms and proceeded to start aligning the interferometer, so the ISIs went from a "cold", immediately to an aligned state. Arnaud found a similar effect at LLO,  where large ISI tidal drives caused angular drift during locks, when LLO tried offloading tidal to the ISI ( https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=20222 ).

In the future we should try to at least get the ISIs damped as soon as possible after similar maintenance, or go through a cycle of de-isolating/re-isolating after the ISIs have been running for a while, before trying to relock the IFO. The morning of Jul 11th, ITMX was down for almost 14hrs, but the damping loops were still on, and the actuators were still getting a drive signal. When the ISI was re-isolated, the oplev didn't see any appreciable drift over the next several hours (see third plot). Similarly, sometime before the commissioners start locking tomorrow, all the BSCs should be taken down to damped and then re-isolated.

We also considered that something in the timing system could have been responsible, but were unable to find anything to support that. None of the timing signals we looked at showed anything like the drift we saw in the ISIs. We also saw a similar alignment drift after the June power outage, that would seem to support a thermal drift over a some timing shenanigans.

As Hugh reported ealier today, the hydrolic fluid pressure as read by the HEPI pump controller units dropped yesterday when the timing FPGAs were being reprogrammed. He also reported that this was seen the last time we reprogrammed the FPGAs on Tuesday 19th May 2015.

We are seeing other unrelated EPICS channels showing data issues around the time of timing upgrades. Attached are dataviewer plots (24 hours of minute trends) which show an EY vacuum gauge and the EY-HEPI pump controller's 4th  pressure sensor for 19 May 2015 and 19 July 2016. In both cases as the timing is being upgraded the fluid pressure appears to drop and the vacuum pressure appers to increase.

Investigation continues. We may ask for some EY downtime next Tuesday to see if this is reproducible and to more closely monitor what is changing. At this point we are assuming this is a sensor or read-out error and the vacuum/fluid pressures were not actually changed.

We have been studying the list of blip glitches that Miriam Cabero Müller generated for O1. We noticed that the rate of blip glitches increased dramatically during two time periods, see figure 1. We first checked if the glitch rate might be correlated with outside temperature, in case the blip glitches were produced by beam tube particulate events. The correlation with outside temperature was strong, but, for beam tube particulate, we expected it to be stronger with rate of temperature change than with temperature, and it was not. So we checked relative humidity and found that inside relative humidity was well correlated with outside temperature (and glitch rate), most likely because of the extra heating needed in cold spells.  A plot of the blip glitch rate and RH inside the CS mass storage room is attached.  

While the correlation with inside relative humidity is not better than with outside temperature, we plot inside relative humidity because we can better speculate on reasons for the correlation. Dry conditions may lead to the build up and discharge of static electricity on electronics cooling fans. Alternatively, there may be current leakage paths that are more likely to discharge in bursts when the pathways dry out. While this is, at best, speculation, we set up a magnetometer near the HV line for the EY ESD to monitor for possible small short fluctuations in current that are correlated with blip glitches. At the same time, we suggest that, as risk mitigation, we consider humidifying the experimental areas during cold snaps.

The low-humidity correlated blip glitches may represent a different population of glitches because they have statistically significantly smaller SNRs than the background blip glitches.  We analyzed the distribution of SNR (as reported by pycbc) of the blip glitches during three time periods – segments 1, 2, and a relatively quiet period from October 5 – October 20 (segment 3).  This gave approximately 600 blip glitches for each segment.  Figure 2 is a histogram of these

To determine if these distributions are statistically different, we used the Mann-Whitney U test.   Segments 2 and 3 matched, reporting a one-sided p-value of 0.18.  The distribution in SNR for segment 3 - the low glitch rate times -  did not match either segment 1 or 2, with p-values of 0.0015 and 2.0e-5, respectively.  Thus we can conclude that the distributions of 1 and 2 are statistically significantly different from 3.

We are currently examining the diurnal variations in the rate of these blip glitches, and will post an alog about that soon.

Paul Schale, Robert Schofield, Jordan Palamos

J. Kissel, E. Hall

Checking in on the power delivered to the ITMY compensation plate after the strange drop in TSCY's front-end laser power drop yesterday (see LHO aLOG 28506), it looks like the laser power has recovered, mostly. The power, as measured by the pick-off beam just before the up-periscope into the vacuum system (H1:TCS-ITMY_CO2_LSRPWR_MTR_OUTPUT) is roughly stable at 0.3075 [W], where it used to deliver a really stable 0.312 [W].

I attach two zoom levels of the same 4-day trend.

There's also some weird, 10-min period feature in the *minimum* of the minute-trend, where the reported power drops to 0.16 [W]. Given its periodicity, one might immediate suspect data viewer and data retrieval problems, but one can see in the un-zoomed trend that this half-power drop has been happening since the drop-out yesterday, but tracks the reported laser power even before the delivered power was increased back to nominal.

At 8:09 UTC  (on July 20th) we had a large glitch in interferometer signals and MC3 saturated. This seems like a suspect for some kind of a suspension glitch that would be worth following up on.

Upgrade of Timing Firmware

Daniel, Ansel, Jim, Dave

Most of today was spent upgrading the entire timing system to the new V3 firmware. This did not go as smootly as planned, and took from 9am to 6pm to complete. By the end of the day we had reverted the timing master and the two CER fanouts to the orginal code (the end station fanouts were not upgraded). We did upgrade all the IRIG-B fanouts, all the IOChassis timing slaves, all the comparators and all the RF Amplifiers.

The general order was: stop all front end models and power down all front end computers, upgrade the MSR units, upgrade the CER fanouts, upgrade PSL IO Chassis (h1psl0 was restarted, followed by a DAQ restart), upgrade all CER slaves (at this point the master was reverted to V2), at EY we upgraded IRIG-B and slaves (skipping fanout), at MY we upgraded the PEM IO Chassis, at EX we did the same as EY and at MX the same as MY. 

All remaining front ends were now powered up. The DAQ was running correctly but the NDS were slow to complete their startup. Addiional master work in the MSR required a second round to restarts, at this point comparators which had been skipped were upgraded and the CER fanouts were downgraded. Finally after h1iopsush56 cleared a long negative irig-b error all systems were operational.

During these rounds of upgrades FEC and DAQ were restarted several times.

Addition of Beam Splitter Digital Camera

Richard, Carlos, Jim

An analog camera was replaced with a digital video GIGE-POE camera at the Beam Splitter.

New ASC code

Sheila:

new h1asc code was installed and the DAQ was restarted.

Reconfigured RAID for ldas-h1-frames file system

Dan:

The ldas-h1-frames QFS file system was reconfigured for faster disk access. This is the file system exported by h1ldasgw0 for h1fw0's use. After the system was upgraded, we reconfigured h1fw0 to write all four frame types (science, commissioning, second and minute). As expected, h1fw0 was still unstable at the 10 minute mark, similar to the test when h1fw0 wrote to its own file system. h1fw0 was returned to its science-frames-only configuration.