TITLE:  10/14 day Shift:  15:00-23:00UTC

STATE of H1:  Low-noise for 4 hours, periodic commissioning during single IFO time

Support:  Usual control room population

Quick Summary:

Quiet, busy shift with lots of noise investigations, H1 was well behaved

Shift Activities:

16:04 lockloss

18:03 LLO has a bounce mode rung up, so we go to commissioning

18:05 Evan turning on BLRMS filters, Richard to EY, Robert to LVEA

19:00 Gerardo to EX, JeffB to LVEA

19:00 Robert to EX for tamping

21:10 JeffB to LVEA

21:00 Evan trying some BLRMS filters

22:30 Robert turning HVAC off, done 22:50

Now that all systems are using OBSERVE.snap for their SDF reference I modified the check_h1_files_svn_status script to scan for OBSERVE.snap file status as well as safe.snap. Here is the current source code SVN status:

david.barker@sysadmin0: check_h1_files_svn_status
 
SVN status of front end code source files...
done (list of files scanned can be found in /tmp/source_files_list.txt)
 
SVN status of filter module files...
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSMC1.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSMC3.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSPRM.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSPR3.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSPR2.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSSR2.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSSRM.txt
M       /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSSR3.txt
M       /opt/rtcds/userapps/release/isc/h1/filterfiles/H1OAF.txt
M       /opt/rtcds/userapps/release/lsc/h1/filterfiles/H1LSC.txt
M       /opt/rtcds/userapps/release/asc/h1/filterfiles/H1ASC.txt
done (list of filter module files scanned can be found in /tmp/full_path_filter_file_list.txt)
 
SVN status of safe.snap files...
done (list of safe.snap files scanned can be found in /tmp/safe_snap_files.txt)
 
SVN status of OBSERVE.snap files...
M       /opt/rtcds/userapps/release/psl/h1/burtfiles/iss/h1psliss_OBSERVE.snap
M       /opt/rtcds/userapps/release/isc/h1/burtfiles/h1oaf_OBSERVE.snap
M       /opt/rtcds/userapps/release/lsc/h1/burtfiles/h1lsc_OBSERVE.snap
M       /opt/rtcds/userapps/release/omc/h1/burtfiles/h1omc_OBSERVE.snap
M       /opt/rtcds/userapps/release/asc/h1/burtfiles/h1asc_OBSERVE.snap
M       /opt/rtcds/userapps/release/asc/h1/burtfiles/h1ascimc_OBSERVE.snap
done (list of observe.snap files scanned can be found in /tmp/observe_snap_files.txt)
 
SVN status of guardian files...
M       /opt/rtcds/userapps/release/isc/h1/guardian/ISC_DRMI.py
M       /opt/rtcds/userapps/release/isc/h1/guardian/ISC_LOCK.py
M       /opt/rtcds/userapps/release/isc/h1/guardian/lscparams.py
done
 
 

Summary of Tuesday's mainteance work:

h1calex model change for hardware injection

Jeff, Jim, Dave: WP5553, ECR1500386

h1calex model was modified to add CW and TINJ hardware injection filter modules. Also ODC channel names were changed from EX to PINJX. Three new channels were added to the science frame (HARDWARE_OUT_DQ and BLIND_OUT_DQ at 16k, ODC_CHANNEL_OUT_DQ at 256Hz)

The DAQ was restarted. Conlog was rescanned to capture the new ODC channel names.

Guardian DIAG_EXC node was modified to permit both calex and calcs excitations while in observation mode.

MSR SATABOY firmware upgrades

Carlos: WP5544

The two Sataboy RAID arrays used by h1fw1 had their controller cards firmware upgraded. Also the one Sataboy used by the DMT system was upgraded. No file system downtime was incurred.

Beckhoff SDF testing

Jonathan, Dave: WP5539

Tested Gentoo version of the SDF on h1build machine as user controls. For initial testing we are only connecting to h1ecatcaplc1. We discovered that this version of the sdf system set all the PLC1 setpoints each time it was restarted, so ECATC1PLC1 was reset several times between 10am and 1pm PDT Tuesday morning. This system was left running overnight for stability testing. We discovered that some string-out records cannot be changed. If we change the string for these records (when we accidentally applied the safe.snap strings on restarts) the PLC immediately (100uS later) reset to an internally defined string.

Long running CDS Server reboots, workstation updates

Carlos:

As part of our twice-yearly preventative maintenance, Carlos patched and rebooted some non-critical servers. CDS workstations were inventoried and updated.

Complete OBSERVE.snap install

Dave: WP5557

The models which were still running with safe.snap as their SDF reference were updated to us OBSERVE.snap. Models in the following systems were modifed: IOP, ODC, SUSAUX, PEM. Their initial OBSERVE.snap files were copied of their safe.snap.

Several systems had static OBSERVE.snap files in their target areas instead of a symbolic link to the userapps area. I copied these files over to their respective userapps areas and check them into SVN. During Wed morning non-locking time, I moved the target OBSERVE.snap file into an archive subdirectory and setup the appropriate symbolic links. I relinked within the 5 second monitor period, so no front end SDF reported a "modified file".

As part of Wednesday's commissioning excercises, we looked at the coupling of input jitter into DARM.

I injected band-limited white noise into IM3 pitch (and then IM3 yaw) until I saw a rise in the noise floor of DARM.

We can use the IM4 QPD as an estimate of the amount of jitter on the interferometer's S port. On the AS port side, we can use the OMC QPDs as an estimate of the AS port jitter, and DCPD sum indicates the amount of S port jitter coupling into DARM.

One thing of note is that the jitter coupling from IM3 to DARM is mostly linear, and more or less flat from 30 to 200 Hz:

• For the pitch excitation, at 100 Hz the IM4 pitch ASD was 3.1×10⁻⁷ ct/rtHz, the measured TF was 1.0 mA/ct, and the observed IM3-induced DARM noise was sqrt[3.7² − 1.7²]×10⁻⁷ mA/rtHz = 3.3×10⁻⁷ mA/rtHz. Numbers hang together.
• For the yaw excitation, at 100 Hz the IM4 yaw ASD was 1.6×10⁻⁷ ct/rtHz, the measured TF was 5 mA/ct, and the observed IM3-induced DARM noise was sqrt[8.2² − 1.7²]×10⁻⁷ mA/rtHz = 8.0×10⁻⁷ mA/rtHz. Numbers hang together.

The upper limit on IM3 jitter that one can place using the IM4 QPD seems to be weak. At 40 Hz, projecting the quiescent level of the IM4 yaw signal to the DCPD sum suggests a jitter noise of 2×10⁻⁷ mA/rtHz, but this is obviously not supported by the (essentially zero) coherence between IM4 yaw and DCPD sum during low-noise lock. Of course, this does not rule out a nonlinear coupling.

As for AS port jitter, the coupling is seen more strongly in OMC QPD B than OMC QPD A.

The plots attached to this elog show the trend of the LHO detector noise over the O1 time span so far. Each plot shows the band-limited RMS of the CAL-DELTAL_EXTERNAL signal, in a selected frequency band. I didn't apply the dewhitening filter. The BLRMS is computed over segments of 60 seconds of data, computing the PSD with 5s long FFTs (Hann window) and averaging. The orange trace shows a smoothed version, averaged over one hour segments. Only time in ANALYSIS_READY have been considered.

The scripts used to compute the BLRMS (python, look at trend_o1_fft_lho.py) are attached and uploaded to the NonNA git repository. The data is then plotted using MATLAB (script attached too).

I think there's a lot of interesting information in those plots. My plan would be to try to correlate the noise variations with IFO and environmental channels. Any suggestion from on-site commissioners is welcome!

Here are my first comments on the trends:

• 10-20 Hz and 20-30 Hz noise can vary by a factor about 2 on average, with a somewhat fast time scale (hours)
• There is a sudden drop of the noie in the 30-40 Hz region on 9/23 UTC. After that the noise stayed lower. I don't recall what happened at that time...
• The "unknown noise" in the 70-100 Hz region is rather non-stationary, varying on a daily basis of some 50-70%. Similar trends in the 140-160 Hz region.
• The noise in the 200-220 Hz region increased on 9/23. It seems to be the same time of the drop in the noise in the 30-40 Hz region... Same thing for 250-260 Hz, but there is clearly more variability here
• The jitter peak bumps (340-360 Hz) are varying quite a lot. As Evan pointed out, they are almost always small at the beginning of the lock, and then they never go back to the same level. Their variation seems quite slow, with a time constant of the order of a day. This could be something thermal in the IFO.
• Noise is the 400-450 Hz and 700-900 Hz regions are reasonably stationary, being shot noise. The 400-450 Hz noise showed an increase, again on 9/23, small but consistent. 

So what happenend on September 23rd around the end of the day, UTC time?

The first attached plot (H1L1DARMresidual.pdf) shows the residual DARM spectrum for H1 and L1, from a recent coincident lock stretch (9-10-2015, starting 16:15:00 UTC). I used the CAL-DELTAL_RESIDUAL channels, and undid the digital whitening to get the channels calibrated in meters at all frequencies. The residual  and external DARM rms values are:

The 'external DARM' is the open loop DARM level (or DARM correction signal), integrated down to 0.05 Hz. The second attached plot (H1L1extDARMcomparison.pdf) shows the external DARM spectra; the higher rms for H1 is mainly due to a higher microseism.

Some things to note:

• L1's residual DARM is 6x smaller than H1, due to more suppression in the 2-5 Hz region. As the 2nd plot shows, there is more DARM excitation in this region on H1 than L1.
• The DARM DC offset for both interferometers is 10 pm, so the residual DARM represents a relative fluctuation of the detected power or photocurrent of 0.1% for L1, 0.6% for H1.
• The violin mode fundamentals are much lower in H1 than L1 - by around 3 orders of magnitude! This is because for H1 the violin mode active damping is left ON during low noise mode, whereas for L1 I believe it is turned OFF. However, this also means that the H1 front end calibration is not accounting for these damping paths.

The 3rd attached plot (H1L1DARMcomparison.pdf) shows the two calibrated DARM spectra (external/open loop) in the band from 20-100 Hz. This plot shows that H1 and L1 are very similar in this band where the noise is unexplained. One suspect for the unexplained noise could be some non-linearity or upconversion in the photodetection. However, since the residual rms fluctuations are 6x higher on H1 than L1, and yet their noise spectra are almost indentical in the 20-100 Hz band, this seems to be ruled out - or at least not supported by this look at the data. More direct tests could (and should) be done, by e.g. changing the DARM DC offset, or intentionally increasing the residual DARM to see if there is an effect in the excess noise band.

Two issues affected the H1 detchar summary pages. The diskcache servers was hung and the detchar home directory needed to be remounted.

Both of these issues are now fixed.

It will take a while for the jobs generating the summary pages to catch up, but they should catch up here sometime later today:

https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20151014/

As Jim had almost relocked the IFO, we had an epics freeze in the gaurdian state RESONANCE.  ISC_LOCK had an epics connection error.

What is the right thing for the operator to do in this situation?

Are these epics freezes becoming more frequent again?

screenshot attached.

Hi State: locked in Low Noise and in Observe for 3+ hours, GRB arrived at 14:14UTC

Help: Kiwamu

Shift Overview:  IFO was down and trying to lock PRMI when I arrived - 3.5 hours to return to Observe.

- DRMI produced a "Split Mode" 

- MICH Dark and SRC alignment corrected DRMI "Split Mode"

- first return to Low Noise had issues with the ISS Second Loop

- IMC_LOCK Guardian had buttons pushed by humans

- only change was ISS Diffracted Power (increased)

Timeline:

- 10:12 to 10:40UTC - IFO in Low Noise Lock with ISS Second Loop issues

- 11:39UTC - IFO in Low Noise Lock and Observe - no ISS issues

Title: Owl Mid-Shift Update, 10:48UTC, 3:49PT

H1 State: relocking and making it to Low Noise but engaging the ISS Second loop is gltiching all of the ISS and the IFO

Help this shift:  Kiwamu on the phone, Kiwamu's alog about engaging the ISS Second Loop manually, Jim's alog about DRMI Split Mode

Details:

07:00UTC - I arrive and IFO is down due to an earthquake.

07:00-08:00 - PRMI locked 3 times, but can't hold due to the earthquake

08:00-09:00 - PRMI locks and I align, then I transition to DRMI and thus the "Split Mode" announcements start, and after multiple "Split Mode" locks I call Kiwamu, he says do a MICH Dark and SRC align

09:20 - MICH Dark and SRC Align alignments are successful, and I take the IFO to Down and Init, and take IFO to full-lock

10:07:41 - Guardian tries to engage the ISS Second loop, I wait, then start the procedure in Kiwamu's alog, and then he called back and we tried unsuccessfully to engage the ISS second loop

10:40:38 - the IFO lost lock due to our work trying to engage the ISS Second Loop - Kiwamu is drining in

11:03 - IFO has not relocked, but I don't know why since I've been writing the alog...

More to come...

John, Bubba, Richard, Gerardo

16:44 Opened 10" valve.  Turbo, LD and QDP80 remained on overnight. Background at 1.1 x 10-08 torr*L/sec.
    ***Left system be for background signal to drop.***
We returned to the center longitudinal seam weld (previously referred as butt weld) on the North side of BT.  Divided the section into 3 equal sections, and bagged them.  Using a rate of 1 L/s of He, for 60 seconds dwell, background at 5.2 x 10-09 torr*L/sec, leak hunting started,:
18:00 Section 1 was sprayed with He, and the other two section were purged with instrument air, we waited for 5 minutes to see a response.  Background did not change.
18:06 Section 2 was sprayed with He, and once again the other two sections were purged with instrument air, no change on background.
18:10 John bags the purge/vent valve.
18:12 Section 3 was sprayed with He, while the other two sections were purged with instrument air.
18:15 Purge/vent valve is sprayed with He, and background value starts going up.  Response was to quick??
18:22 section 3 is purged with instrument air.
18:30 observed max value for background 5.2 x 10-08 torr*L/sec.
19:05 Aux cart connected and pumping on closed purge/vent valve, no change noted on BT pressure.
19:12 left for L U N C H, while Turbo pump pumped He out.

===Activity after lunch===

21:28 Background signal at 5.1 x 10-09 torr*L/sec
21:40 sprayed He at a rate of 1 L/s for 60 seconds on the longitudinal weld seam section 3.
21:43 background signal starts changing, slooooow.
21:50 background signal at 2.1 x 10-08 torr*L/sec
21:50 started purging section 3 with instrument air.
21:51 background at 3.0 x 10-08 torr*L/sec
21:52 background at 4.0 x 10-08 torr*L/sec
21:54 background at 5.0 x 10-08 torr*L/sec
21:56 background at 7.5 x 10-08 torr*L/sec
21:58 background at 9.0 x 10-08 torr*L/sec
22:00 background at 1.0 x 10-07 torr*L/sec
22:03 background at 1.2 x 10-07 torr*L/sec
22:10 background at 1.2 x 10-07 torr*L/sec *
22:28 background at 8.0 x 10-08 torr*L/sec
22:38 Turned off and removed aux cart from purge/vent valve, valve was restored as found, except now space is vented.
22:47 background signal at 4.5 x 10-08 torr*L/sec

At this point we divided section 3 into two sides.
    * East side which is away from the stiffener ring, this one was sprayed first with He, while the West side, close to the stiffener ring, was purged with air, we sprayed as before (1 L/s of He for 60 sec).

22:48 started spraying He on the East side.
23:03 background at 3.1 x 10-08 torr*L/sec, background continues to drop.
23:05 background at 2.9 x 10-08 torr*L/sec
23:05 switched hoses on the 2 sides, now we are going to spray He in the West side and purge with instrument air the East side.
23:05 background at 2.9 x 10-08 torr*L/sec
23:06 background at 2.8 x 10-08 torr*L/sec
23:07 start of He spraying at West side
23:10 background at 2.6 x 10-08 torr*L/sec
23:12 background at 2.8 x 10-08 torr*L/sec  ***
23:14 background at 3.2 x 10-08 torr*L/sec
23:17 background at 4.1 x 10-08 torr*L/sec
23:19 background at 4.8 x 10-08 torr*L/sec
23:20 background at 5.2 x 10-08 torr*L/sec
23:21 background at 5.4 x 10-08 torr*L/sec
23:26 background at 5.9 x 10-08 torr*L/sec
23:27 closed 10" gate valve to isolate turbo pump, LD and QDP80 from vacuum system, but they remain on.

Conclusion,
We think that the leak is somewhere in the small West side of the center longitudinal seam weld, we will continue to test to pinpoint the leak source.

TITLE:  10/13 EVE Shift:  23:00-07:00UTC (16:00-00:00PDT), all times posted in UTC     

STATE of H1:  Locking due to recent EQ.  Waiting for seismic to come down & allow for PRMI alignment tweaks.

Incoming Operator:  Cheryl

Support:  Sheila by phone (about new PRMI guardian)

Quick Summary:

Nice shift with H1 in Observing Mode for 7of 8hrs & then a Russian EQ knocked us out.  

Shift Activities:

• ETMy Saturations:  23:28, 2:06, 
• 23:47 Mid-Y Leak Checking complete for day (WP5526, Worden & Moreno)
• REMINDER:  Aux Cart running at EX, connected to Annulus IP at BSC5.  Connected at 16:20utc (9:20am) & will be kept ON on the order of days  (Moreno)
• New Lock Timer on Nuc0
• See an SR2 SPM diff for it's M1 LOCK P&Y Inputs
• 6:02 H1 earthquake lockloss (noticed I forgot to mark Obs Mode to EQ!  So from 6:02-7:02 it was in "lock acquisition", but should have been earthquake.  I just switched it to earthquake as I wait for PRMI to lock.

Jordan, Sheila

In the summary pages, we can see that something non stationary appeared in DARM from about 80-250 Hz durring the long lock that spanned Oct 11th to 12th, and has stayed around.   Links to the spectra from the 11th  and the 12th.  

HVETO also came up with a lot of glitches in this frequency span starting on the 12th, (here) which were not around before.  These glitches are vetoed by things that seem like they could all be realted to corner statin ground motion: refl, IMC and AS WFS, all kinds of corner station seismic sensors, PEM accelerometers, MC suspensions.  

Although this noise seems to have appeared durring a time when the microseism was high for us, I think it is not directly related. (high microseism started approximately on the 9th, 2 days before this noise appeared and things are quieting down now but we still have the non stationary noise sometimes up to 200 Hz.)  

The blend switching could also seem like a culprit, but the blends were not switched at the begining of the lock in which this noise appeared, and we have been back on the normal (90mHz blends) today but we still have this noise.  We've seen scattering from the OMC with velocities this high before (17264 and 19195).  

Nutsinee and Robert have found that some of the glitches we are having today are due to RF45, but this doesn't seem to be the case on the 12th. 

Betsy, Sheila, Travis, Evan

Something about the following coils appears to be unhealthy: PRM M1 RT&SD, PR3 M1 T1&T2. See attached OSEM sensor spectra.

According to Betsy, this high-frequency junk started around 2015-08-29 15:00:00 Z. On the control side, this junk dominates the rms of the PRM M1 LF drive (it is about 10000 ct).

Probably this is related to the problem that Kiwamu saw last night with PRM M1 DAMP L.