I reset the H1 PSL FE Watchdog at 15:48 UTC (08:48 PDT).  While garbing down afterwards I noticed the crystal chiller water level was low, so I added 250 mL of water.

J. Kissel, J. Batch
ECR E1500386
WP 5553
II 1126

I've run a make and make-install on the h1calex front-end model changes that were described yesterday (see LHO aLOG 22445). Jim has restarted the model and DAQ for me, at roughly 15:00 UTC, about 10 minutes after the lock loss.

Will post more details once we get MEDM screens up and running.

Title: 10/13 OWL Shift: 07:00-15:00UTC (00:00-08:00PT), all times posted in UTC

State of H1: IFO unlocked - Maintenance

Incoming Operator:  Jim (main), Patrick

Quick Summary: Two lock losses

09:31:23UTC - lock loss

11:39:42UTC - IFO back in Observe, see previous alog for details on relocking

14:43:14UTC - lock loss, Maintenance starts

Attached plot, from the lockloss tool, is zoomed in to the OMC-DCPD_SUM_OUT_DQ channel.  

The plots shows that all of the other channels glitched after the OMC DC channel.  

The channels were chosen by the lockloss tool, so maybe not a complete story yet.

Lock ended at  9:31UTC, and this plot shows 8.5 hours, which all but the very beginning of the lock.

Plots attached show that POP_A_LF_OUT is oscillating between 16200 and 17000 (800 counts) in the last 10 minutes of the previous lock, and is oscillating between 16200 and 16600 (400 counts) in the current lock (channel 1).

PO_A_LF_RF9_Q_ERR was oscillating between 35000 and 55000 (20000 counts), and is now between 55000 and 65000 (10000 counts, channel 2).

Lock loss at about 9:30UTC, and while nothing is broken, many small issues to relaocking.

Bottom Line - I can see the useism effecting the locks; arms in green, arms in red, PRMI, and DRMI, and that's made relocking twitchy.

Details:

- 3 or more times, I had the the PRMI locked and aligned, and when I brought SRM back in, it didn't just break the PRMI, the SRM watchdog tripped

- at least two times, HAM5 ISI also tripped

- during one of those, the BSC2 ISI tripped

- at least 3 times the DRMI locked but not good enough for WFS, and broke before I could manually fix it

- once DRMI locked and WFS did engage, I let it sit for a while, as I took a much needed break, if you must know, and even then it broke on it's way to Engage ASC

- DRMI just relocked and at 11:21UTC, it looks like the IFO might make it back to Low Noise

- at 11:25UTC, IFO is in Engage ASC and looks OK

Looking at ER8 and O1 data with the long transient search pipeline stampas we have noticed an excess of triggers corresponding to 2 narrow frequency lines at 39Hz and 43 Hz (plot 1). This excess of triggers happen at the beginning of H1 relock periods and is so large that it is visible in the spectrogram of h(t) (plot2). After a dozen of minutes, the excess of loud triggers disappear along with the lines. We are seen this problem since the beginning of ER8 but actually it seems that since October 8th, the excitation is smaller at each relock. Plot 3 shows that the long transient search is dominated by these triggers. We will develop a veto against these triggers, but we would be happy to see the pb fixed in the data!

TITLE: "10/08 [EVE Shift]: 23:00-07:00UTC (16:00-00:00 PDT), all times posted in UTC"

STATE Of H1: Observing at ~75 Mpc for the past 5 hours

SUPPORT: Sheila, Jeff K., Kiwamu

SHIFT SUMMARY: Few hiccups during lock acquisition. Sheila found that the CSOFT_Y was running away during DC_READOUT_TRANSITION (I believe) which probably caused a lockloss at DC_READOUT we had earlier. later ISS 2nd loop wouldn't engage. Kiwamu had to do it by hand. Microseism slowly coming down. Seismic activity in EQ band is nominal. Wind ~5mph.

INCOMING OPERATOR: Cheryl

ACTIVITY LOG:

00:51 Locked at NOMINAL_LOW_NOISE

01:00 Robert and Vinney to water pump area to do tamper injection.

01:45 Robert back

01:47 Switched to Undisturbed.

John, Gerardo

16:25 - Spun up LD, Turbo and QDP80
16:50 - LD was Calibrated with cal-leak, checked out.
16:55 - Valved in Turbo to vacuum volume, LD reported He background at 3.2 x 10-8 torr*L/sec, left system pumping.
17:08 - left Y-mid.
18:04 - Returned to Y-Mid to bag the butt welds of cylinder spool roll, 2 of them as half sections, for a total of 4 sections to test.  Also continued to watch LD report..
19:20 - left Y-mid, LD reported background of 0.8 x 10-8 torr*L/sec.
20:25 - We returned to Y-Mid, LD reported a background of 5.3 x 10-9 torr*L/sec.  Using a rate of 1 l/s of He, for 60 seconds dwell, leak hunting started:
    * 3 butt welds of cylinder spool roll were tested, starting closer to the GV-11, no change on background for first 2 welds.  Moved to the 3rd one, there was a signal, slow steady creep. Up until background reached 0.8  x 10-8 torr*L/sec.
    * We paused for some time, gave the background a chance to drop with the VEA doors opened.
    * We moved on to GV-11, since it is near to the butt welds, we checked its conflat and bellows with no change on background, other components had been checked before.
    * Then some of the conflats on the spool piece by GV-10 were also tested, and again no change on the background.
    * We moved to the turbo pump ports and turbo pump gate valve, they were tested with no change on the background.
    * We returned to the butt weld, the 4th section (second butt weld) and tested it, we had a response and the background went up to  0.8 x 10-7 torr*L/sec, the signal was slow to creep up and it took sometime to show, thus creating a bit of confusion.
    * We stopped leak testing because the background was too high, and it is going to take some time for the background to go down.  We valved out the turbo pump from the vacuum volume.

We will continue with leak testing in the following days.


Note: While seen the background go up the turbo pump was isolated 3 times from the vacuum volume via its gate valve, and every time the background signal went down to <10-11 torr*L/sec

We are observing again at ~70 Mpc. Kiwamu had to engage ISS second loop by hand (alog 22449). It is unclear to me if we need to do this by hand after Maintenance day tomorrow. POP_A_LF fluctuates a lot and doens't look happy. Same goes for the control signals (they look unrested on the FOM, if the scale hasn't been changed). This is probably due to high microseism. Wind ~10 mph. Seismic activity in earthquake band looks nominal. Four ETMY saturations since we locked at Nominal Low Noise ~2 hours ago.

I noticed RF45 jumped above the reference at 0-10Hz just now while writing this log. Looks like it's been glitching for a while judging from the DMT Omega but didn't cause ETMY saturation so I didn't look up until now.

Made tamper injections at EY while LLO was having problems 6 to 6:30 local

Nutsinee, Sheila

We ran a2L again, here are the beam position results in meters (horizontal, then vertical):

Sheila, Nutsinee, Kiwamu,

For some reason, the 2nd loop ISS was not able to converge the PID loop at all during the engagement step. In the end, we engaged it by hand.

We will take a close look tomorrow during the maintenance period, but for the night, we will leave it as it is without any fix.

Instruction for manual engagement

Here is an instruction for the operators if the ISS runs into  a difficulty in engaging the 2nd loop:

• If ISS does not engage the 2nd loop for more than 5 minutes in the "ENGAGE_ISS_2ND_LOOP" state, please proceed to the following steps.
• Check the IMC guardian state.
  • If it has fallen back to 'LOCKED' or something equivalent, re-request ''ISS_ON" manually.
  • Make sure that the IMC_LOCK guardian is in the "PREPARE_ISS" state.
• Open up the medm window for the 2nd loop (SITEMAP --> PSL ---> ISS --> SECOND LOOP).
  • Getting to the 2nd loop medm from the generic ISS medm can be tricky for some people, so see the 1st attachment to check where to click.
• In the 2nd loop medm, check how the output is converging. See the 2nd attached.
  • If the output value is bigger than +/- 5 and does not move, please call an expert.
  • If the output value is moving (or often oscillating), then you can manually turn on the loop by pressing a green "ON" button at the top as shown in the 2nd attached screenshot. A rule of thomb is to click the "ON" button when the output value is close to zero.
• After the manual engagement, make sure that the output value becomes smaller, ideally close to zero without any oscillations.
  • Then the guardian will wait for 60 seconds and automatically move onto the "CLOSE_ISS" state. We are done. It is going to automatically take you to the "ISS_ON".
• If the above does not work or break the lock, please call an expert.

Software seem to be unchanged

Looking at the guardian and foton file, we confirmed that there was no change since Sep 9th. So we think this is not due to a softaware change or some sort. It could be some kind of hardware issues, but because of the fact that it runs OK once engaged, probably the hardware is not completely broken either.

We have lost lock twice in a row after turning up the gain of the soft loops, which seemed to cause an oscillation at around 40 mHz. We confirmed this a few times by tunring them on and off, and watching the oscillation come and go.  We have now turned them on one at a time after increasing the power, and things seem OK.  The attached screenshot shows the POP power fluctations durring the locks over the last 5 days. You can see that as the ground motion became large, the control signals from the soft loops become large and so do the  fluctuations of the POP power.  

CHARD yaw also doesn't seem to have enough juice to suppress the error signal with this kind of ground motion.  

J. Kissel for the people actually doing the work: 
S. Dwyer, K. Izumi, N. Kijbunchoo, T. Schaffer

A couple of problems, compounded with the high wind and microseism have made lock acquaisiton difficult today. Kiwmau and Shiela are still working on the last two I mention, and will likely aLOG details later:
- The initial alignment was poor, as identified by a few lock losses that had too low a REFL power too soon so we ran through initial alignment.
- During initial alignment, we found that the X-ARM's IR loop had too low a gain margin (see LHO aLOG 22443), for which a permanent solution will be tried tomorrow.
- Once we finally got to DC READOUT, we found that all of the SOFT loops were difficult at best to close (that's the status now, again Shiela will aLOG more later)
- Also, the ISS 2ND loop has become difficult to turn on (again, current status, Kiwamu's working on diagnosing)

And again, on top of this the microseism has been at ~0.3 [um/s], and wind had gusted up to ~20 [mph] for a few hours in the early afternoon.

Stay tuned.

Here are the observed H1-L1 coherences as of October 12, 2015, as calculated by stochmon:
https://ldas-jobs.ligo.caltech.edu/~thomas.callister/stochmonO1/stochmon.html

Some plots attached.

100 mHz resolution. 1E-5 is the expected level of coherence due to uncorrelated noise.
Frequency, coherence
33.6 Hz, 1E-3
34.6 Hz, 2E-2
35.4 Hz, 1E-3
42.5 Hz, 6E-4 (barely above background, could be a noise fluctuation)
64.0 Hz, 2E-4 (barely above background, could be a noise fluctuation)
74.0 Hz, 5E-4 (barely above background, could be a noise fluctuation)
128.0 Hz, 1E-2
256.0 Hz, 2E-3
331.8 Hz, 3E-4 (barely above background, could be a noise fluctuation)
351.1 Hz, 4E-4 (barely above background, could be a noise fluctuation)
559.9 Hz, 2E-4 (barely above background, could be a noise fluctuation)
736.0 Hz, 3E-3 (23rd harmonic of 32 Hz)
1904.0 Hz, 2E-3 (119th harmonic of 16 Hz)
2160.0 Hz, 4E-4 (135th harmonnic of 16 Hz)
2376.0 Hz, 4E-3 (297th harmonic of 8 Hz)
2768.0 Hz, 7E-3 (173rd harmonic of 16 Hz)
2984.0 Hz, 7E-4 (373rd harmonic of 8 Hz)
3001.2 Hz, 1E-1 (calibration lines)
4928.0 Hz, 3E-3 (77th harmonic of 64 Hz)
5016.0 Hz, 3E-3 (627th harmonic of 8 Hz)
5144.0 Hz, 2E-1 (largest coherence, 643rd harmonic of 8 Hz)
5616.0 Hz, 2E-3 (351st harmonic of 16 Hz)
5624.1 Hz, 6E-3 


0.25 Hz, 4E-6 is the expected level of coherence due to uncorrelated noise.
Frequency, coherence
34.25 to 35.25 Hz, 1E-3
36 to 37 Hz, 1.5E-3
37.25 Hz, 2E-3
42.5 Hz, 2E-4
128.0 Hz, 3E-3
256 Hz, 6E-4
1904.0 Hz, 4E-4 (119th harmonic of 16 Hz)
3552.25 Hz, 2E-4
4163.75 Hz, 5E-3 Hz
4576.0 Hz, 2E-4 (143rd harmonic of 32 Hz)
5016.0 Hz, 7E-4 (627th harmonic of 8 Hz)
5144.0 Hz, 1E-1 (largest coherence, 643rd harmonic of 8 Hz)
5584.5 Hz, 2E-2
5655.5 Hz, 1E-3