Whilst observing the IMC trying to lock this morning and its moving the laser frequency around,
it appeared that in the lock acquisition phase as soon as light was resonating in the IMC cavity
the boost gain was engaged.

    I asked TJ to insert a 2 second delay prior to engaging the IMC servo boost gain so that things
had a chance to settle down.  Behaviourally things seemed better with the delay in.



TJ/Jason/Peter

TITLE: 07/22 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Jeff
SHIFT SUMMARY:  After performing initial alignment this morning, the IFO has been locking well all day.  We had a brief wrestling match with the FSS again, but since then, no issues.
LOG:

16:30 FSS oscillations, Peter and Jason looking at it

16:52 Richard to both ends electronics room

17:45 Richard back

22:07 Gerardo to CP3
 

I am working on the Noisebudget Model these days and made a summary of the WFS in use with calibrations:

Calibration (from W to cts) = RF Trans. * ADC * Responsivity * Digital Gain * 10^(Demod. Gain + Whit. Gain)/20

Attached is a plot of the pre-modecleaner heater signals similar to that posted at LLO:
PMC glitches.
No glitches observed here.



Jason/Peter

The OMC was behaving funny, as seen on the trans camera, so I glanced at its control screen.  I noticed that the PZT was super railed, and the "OFF_RESONANCE" guardian state was just making it larger and larger. 

The logic for dealing with this failed, since it checks if the offset is identically equal to 50, it moves the PZT to -50.  I have changed it so that if the offset is larger than 50 it'll reset to -50.

Attached is a 3 day trend - this railing started sometime yesterday.

I made a map which plots my measured locations of the NN array L4C sensors to scale against a LVEA layout drawing from D970308. Each location has a number indicating the channel number of that sensor. The channels are H1:NGN-CS_L4C_Z_#_OUT, where # is the number of the sensor on the map. It's hard to quantify the uncertainties in my position measurements, but the positions look correct by eye. I'll update this map when I place more sensors next week.

The data dropouts observed in second trend data (see alog 28579) are a result of missing data caused by restarts of frame writer 1.  If the raw data is used instead of second trend data, you can see that dropouts are not present, see attached example.  Both nds0 and nds1 are reading the same second trend data, and only frame writer 1 is currently writing second trend data as we explore frame writer instability.

Attached are screenshots for the past 7 days optical ever trends for PIT, YAW, and SUM.

This completes FAMIS request 4685.

The laser tripped around 3:20 this morning.  Suspect it is the "usual" problem.  The crystal
chiller was found off but with its dummy still in place.

    The data drops out looks like an artifact of data acquisition.

    Another observation is that the head temperatures all increase when the flow rates drop.
That is not surprising but the humidity also rises.  It might be that we have a tiny water leak
whose effect on humidity is noticeable when the temperature rises.  We have seen evidence of
tiny leaks before because of some corrosion stains in the laser base plate, however when the
laser is observed over time scales of ~30 minutes, no leak is visible.  The (possible) leak is
small enough that it does not make a noticeable difference in the water level of the chiller.

Jeff B, Kiwamu,

We powered up the input power to 50 W tonight with SRM controlled by the dither alignment. We held the interferometer at 50 W for 30-ish minutes but this was ended because of a PI at 18056 Hz (28259).

During this short period, we moved the PRM pointing (or PRC1) in pitch and confirmed that moving PRM further in the negative direction (by introducing an offset in the error point of PRC1) improved the recycling gain. See the attached. The recycling gain slowly went back to 28 or so at 50 W. We did not get a chance to explore the optimum point yet. POPX started railing when we steered PRM at 40 W on the way to reach 50 W.

Title:  07/21/2016, Day Shift 23:00 – 07:00 (16:00 – 00:00) All times in UTC (PT)
State of H1: IFO locked at DC_READOUT. Wind is Calm to Moderate Breeze (0 to 18mph). Seismic is a bit elevated at the End Stations; but should pose no operational difficulties. Microseism is low.         
Commissioning: Commissioners are commissioning.
Outgoing Operator:  Travis
 
Activity Log: All Times in UTC (PT)

23:00 (16:00) Take over from Travis
23:08 (16:08) Nutsinee – Going to TCS-X Table
23:11 (16:11) Sheila & Jenne – Going to ISCT1 to add a new lens
23:45 (16:45) Sheila & Jenne – Out of LVEA
00:16 (17:16) Jenne – Going to ISCT1 
00:25 (17:25) Jenne – Out of LVEA
00:39 (17:39) Jenne – Going back to ISCT1
01:26 (18:26) Jenne – Out of the LVEA


Title: 07/21/2016, Day Shift 23:00 – 07:00 (16:00 – 00:00) All times in UTC (PT)
Support:  Sheila, Jenne, Kiwamu 
Incoming Operator: N/A

Shift Detail Summary: IFO has been mostly up while the commissioners were working. The lock losses were relativity easily recovered. Had to do some minor touch up of PRMI a couple of times, but nothing too serious. The wind has come up over the last 3 hours. Base winds are in the mid to upper teens with gusts into the mid-30s. Microseism remains low and seismic activity has rung up with the increase in the winds.     

   Added 150ml water to the Crystal chiller. Diode chiller was OK. Noted no new contamination or debris in either chiller filter. 

   Trended the pressures and flows for the Diode and Crystal chillers. The pressures are still slowly trending upward and the flows are trending downward. Both are changing by very small amounts. I have the new (higher flow and less restrictive)filters for the chiller room in-line filters. Plan to swap these at the next maintenance window.   

Just a theory here...But I believe the H2 coil driver needs attention.

H2 Drives off as RZ and X Loops, H2 only directly affects RZ and X.  If it were the loops, of Actuator Drives would respond.  Page 11 of attached show the CD I & V INMONs going wacky impacting the RZ and X(page 4) loops 1 or more seconds before the model responds.

I know this is long winded, you should see the stuff in the text file not preinted here as I went down other rabbit holes.

ITMX ISI Tripped Wednesday July 13 AM from M6.3 EQ in New Zealand. No other platforms tripped.  The extra sensitivity of the ITMX to EQs is not new.  Based on the ground Seismometer STS2-B, it looks like the trip occurred with the arrival of the S-Wave (Shear, not surface.) See page 1 of attachment.  Based on the EPICS records, ST2 tripped first; based on the FIRSTTRIG_LATCH value, the Actuators hit their rail first.

Page 2: 80 seconds with the Actuator drives. The upper right graphs have the horizontals and verticals on the same plot. Given that there is no RX or RY loop closed, it makes sense that the three vertical drives are exactly the same. Leading up to the trip, the H1 and H3 drives appear to be getting a greater request from the loops compared to the corner2 horizontal. The H2 GS13 is parallel to the X arm and so contributes nothing to the Y DOF. The verticals are well behaved. However, a couple of seconds before the EPICS INMON registers the state change (1 to 2,) the H2 drive shoots almost straight to the rail. This delay makes sense with the 8192 saturations required before tripping on this 4k model. The upper left plot zooms out on the H2 drive as it ramps its way up to e7 before the watchdog trip takes it to zero. The H1 and H3 drives continue along until the trip when they start to ring. After the trip the three horizontals behave similarly.  You might believe there is a kink in these other drives a few seconds earlier,...ehhh.

So the behavior of Drive2 may be of interest.  Is this a coil driver problem or does it originate from up stream?

Upstream is the damping, isolation, and feedforward paths. There is no indication that the vertical loop is a problem and feedforward only goes to Z; and, there are no DQ channels for the damping loops—they will have to wait for further upstream examination. This leaves the isolation loops.

Page 3 shows 80 seconds of the Isolation loop inputs. The ITMX and ITMY IN1s are overlain on the same plots. Also shown are the horizontal ground motions; the vertical is uninteresting. The ground motion signals overlain on the isolation plots are scaled and shifted so as to be seen. This shows that both ITMX and ITMY X & Y DOFs are doing about the same and are fairly well correlated with the input ground motion hence the control loops are pretty much the same. I've also verified this looking at the Matlab Controller figures. Around xx8505 gps, the Y ground motion quickly increases, this may be the arrival of the s-wave. The Y loop responds but within several seconds, the response doesn't seem as reasonable and looks to be ringing up and getting behind the curve. The X gets hit with this some 30 seconds later. I don't know if the mostly flatish nature of the RZ loops is or is not remarkable. As well, is the scale of noise/buzz difference between ITMX and Y worth noting? Conversely, the Z DOFs seem to be responding to the same input but certainly drive harder on the ITMX.

So the H2 Drive goes wacky before the others. The X and RZ loops feed the H2 Drive and both those ISOs head off to the outer reaches. The Y loop sees a blip and then gets buzzy/noisy; the Z sees nothing. The H2 Drive is pushing the platform and the X and RZ motion is seen back at the ISO_IN1. It makes sense that the Y DOF would also see something if the H2 actuator is the only one pushing. The push on H2 Drive could produce canceling affects on the H1 and H3 sensors and not push that DOF iso off the page. If the RZ loop was driving it, the response would be the same on all Horz GS13s but they are not responding the same.

On page 4 the Drives and ISO Ins are plotted together for 4 seconds. Clearly the H2 Drive leads the other drives as already established. On the top plot of the ISOs, the X and RZ loops respond dramatically compared to the blip on Y. Further, the ISOs are scaled by 5 to emphasize their behavior before the Drive goes off. The Y loop looks to be trundling along but X and RZ and maybe more clearly RZ changes what it had been doing.

This might suggest the loops are causing this but the fact that only the H2 Drive reacts implies that circuit is more sensitive.

On page 5, only ISOs X and RZ are shown with the H2 Drive. On the top plot, the drive is scaled down to be visible with the zoomed in ISO loops. The RZ loop really seems to be headed somewhere bad before the drive freaks out. Again however, if the loop was the source, the other drives would be reacting similarly and they are not.

On Page 6, one sees the ITMY loops don't respond like the ITMX X and RZ loops--not caused by the common ground motion.

On pages 7 & 8, an interesting beat is seen on the HEPI L4C RZ loops. Otherwise nothing else on the HEPI IPS or L4Cs of either ITM.

Page 9, the GS13s of ITMY are overlain on the ITMX GS13s for 20 seconds showing the motion on the platform is nearly identical on the two ITMs.

On page 10, the horizontal GS13s of ITMX are plotted together (left middle graph) showing the relative magnitudes. This implies only corner 2 is being driven and the others are just seeing the rotation caused by the one corner.

Page 11 plots the H2 CD INMONs, 80 seconds to show the normal before some badness at the trip time.

Page 12 zooms into 10 seconds showing the Coil Driver current and voltage making some really ugly step and excursion one or more seconds before the RZ loop starts to head off.  The model did not tell the coils to do this or it would be seen on the DRIVE.  The RZ loop and the X loop (page 4) are responding to what the coil driver is doing to the platform.  I'm pretty sure this tells us the coil driver is the source of this problem.

Maybe the coil driver is marginal in some way and when the extra demand of dealing with the work of the earthquake motion occurs, something lets go.

We closed some alignement loops for SRM using a dither and demodulating POP90 this morning.  These loops seem to have a ugf below 100 mHz, so we may want to increase their gain, but they seem to be working well for now.  They come on in the guardian durring the DRMI ASC and leave them on.  They maintain POP 90 (and the ratio of AS90/POP90) at a reasonable level as we go through the CARM offset reduction, engaging the soft loops, and increasing the power. 

We saw a instability that seemed to be a cross  coupling between SRM and SOFT yaw loops, Jenne increased the gain in the soft yaw loops by a factor of 10 which seems to have taken care of the problem and is now in the guardian. 

As long as this keeps working, operators should no longer have to adjust SRM. 

Hugh, JeffK, JimW

Short version:

• Large actuator drives may cause thermal drifts in the ISI, especially after a "cold" start.
• Things like power outages or long maintenance periods where ISI actuators get no signal for hours should be followed by warmup periods for the ISIs.
• We should damp as soon as possible, then do an isolate / cook / de-isolate / re-isolate cycle. Or just live with a "squirmy" IFO for ~6hrs.
• It's possible the timing stuff yesterday caused the drifts Sheila saw yesterday, but theres no evidence for that yet.

I noticed this afternoon that all of the BSC-ISIs were pushing large DC-ish drives on their St1 vertical actuators (generally about 2000 counts, normal drives are ~100). When I trended the BS and ITMX drives against their Sus oplevs, there seems to be a very clear correlation between large ISI drives and oplev yaw. I think this may be the cause of the drifts that Sheila complained about yesterday. The first attached plot shows the BS 3 vertical drives and BS oplev yaw, second is for the same for ITMX.

One possible cause of this is the actuators heating up the ISI. During the timing kerfluffle yesterday, all of the ISIs sat for ~6 hours with no acuator drive. The control room then recovered all of the platforms and proceeded to start aligning the interferometer, so the ISIs went from a "cold", immediately to an aligned state. Arnaud found a similar effect at LLO,  where large ISI tidal drives caused angular drift during locks, when LLO tried offloading tidal to the ISI ( https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=20222 ).

In the future we should try to at least get the ISIs damped as soon as possible after similar maintenance, or go through a cycle of de-isolating/re-isolating after the ISIs have been running for a while, before trying to relock the IFO. The morning of Jul 11th, ITMX was down for almost 14hrs, but the damping loops were still on, and the actuators were still getting a drive signal. When the ISI was re-isolated, the oplev didn't see any appreciable drift over the next several hours (see third plot). Similarly, sometime before the commissioners start locking tomorrow, all the BSCs should be taken down to damped and then re-isolated.

We also considered that something in the timing system could have been responsible, but were unable to find anything to support that. None of the timing signals we looked at showed anything like the drift we saw in the ISIs. We also saw a similar alignment drift after the June power outage, that would seem to support a thermal drift over a some timing shenanigans.

Upgrade of Timing Firmware

Daniel, Ansel, Jim, Dave

Most of today was spent upgrading the entire timing system to the new V3 firmware. This did not go as smootly as planned, and took from 9am to 6pm to complete. By the end of the day we had reverted the timing master and the two CER fanouts to the orginal code (the end station fanouts were not upgraded). We did upgrade all the IRIG-B fanouts, all the IOChassis timing slaves, all the comparators and all the RF Amplifiers.

The general order was: stop all front end models and power down all front end computers, upgrade the MSR units, upgrade the CER fanouts, upgrade PSL IO Chassis (h1psl0 was restarted, followed by a DAQ restart), upgrade all CER slaves (at this point the master was reverted to V2), at EY we upgraded IRIG-B and slaves (skipping fanout), at MY we upgraded the PEM IO Chassis, at EX we did the same as EY and at MX the same as MY. 

All remaining front ends were now powered up. The DAQ was running correctly but the NDS were slow to complete their startup. Addiional master work in the MSR required a second round to restarts, at this point comparators which had been skipped were upgraded and the CER fanouts were downgraded. Finally after h1iopsush56 cleared a long negative irig-b error all systems were operational.

During these rounds of upgrades FEC and DAQ were restarted several times.

Addition of Beam Splitter Digital Camera

Richard, Carlos, Jim

An analog camera was replaced with a digital video GIGE-POE camera at the Beam Splitter.

New ASC code

Sheila:

new h1asc code was installed and the DAQ was restarted.

Reconfigured RAID for ldas-h1-frames file system

Dan:

The ldas-h1-frames QFS file system was reconfigured for faster disk access. This is the file system exported by h1ldasgw0 for h1fw0's use. After the system was upgraded, we reconfigured h1fw0 to write all four frame types (science, commissioning, second and minute). As expected, h1fw0 was still unstable at the 10 minute mark, similar to the test when h1fw0 wrote to its own file system. h1fw0 was returned to its science-frames-only configuration.

The shot noise level from last night seems higher (worse) than the O1 level by 6%. Here is the spectrum:

You can see that the red trace (which is the one from the last night) is slightly higher than the (post-) O1 spectrum. The 6% increment was estimated by dividing the two spectra for frequencies above 1200 Hz and taking a median of it.