The first attached screenshot is a series of DCPD spectra from the last several months.  When we first turned on the HPO, the noise in DARM did not get any worse.  The noise stayed similar to O1 until July 16th, the lump from 200-1kHz started to appear in late July, when we started a series of changes in alignment and TCS to keep us stable with a decent recycling gain at 50 Watts.  

PMC

Last night, Gabriele pointed out that the noise in DARM is coherent with the PMC HV.  The HV readback had been mostly some white noise (probably ADC noise), until the last few weeks, but has been getting noisier so that some of the things jitter peaks show up in it now (SEcond attached screenshot, colors corespond to the dates in the DCPD spectrum legend.  This may be related to the problem described in LLO alogs 16186 and 15986.  The PMC transmision has been degrading since July, which could be a symptom of misalingment.  Since July, the REFL power has nearly doubled from 22 to 38 Watts, while the transmission has dropped 28%.  The PMC sum has also dropped by 4%, roughly consistent with the 3% drop in the power out of the laser.  Peter and Jason are planning on realigning the PMC in the morning, so it will be interesting to see if we see any difference in the HV readback.  

TCS + PRC alignment:

The other two main changes we have had in this time are changes in the alignment through the PRC, and changes to TCS.  These things were done to improve our recycling gain and stability without watching the noise impact carefully.  In July we were using no offsets in the POP A QPDs.  We started changing the offsets on August 1st, after the lumpy noise first appeared around July 22nd.  We have continued to change them every few weeks since then, but generally moving in the same direction.  

The only TCS change that directly coresponds to the dates when our noise got worse was the reduction of ETMY RH from 0.75 W each on July 22nd, the other main TCS changes happened September 10th.  It would be nice to undo some of these changes before turning off the HPO, even if it means reduing the power to be stable.  

J. Kissel

We're continuing to struggle to even get back to DRMI locking as of last night (LHO aLOG 30614). One suspicion is the sharp feature 27.0977 Hz (using a 0.005 Hz BW ASD) that has suddenly appeared, as shown in Jenne's attached ASD. 

Because we haven't yet measured them, and we're dead in the water until we figure out the problem, I've taken the time to excite and measure all of the HAM2 and HAM3 HSTS and HLTS's highest vertical mode frequencies, modeled to be at 27.3 Hz and 28.1 Hz, respectively. The results are as follows:

 Optic      Sus. Type     V3 Mode Freq (Hz)
  MC1         HSTS            27.387
  MC2         HSTS            27.742
  MC3         HSTS            27.426

  PRM         HSTS            27.594
  PR2         HSTS            27.414

  PR3         HLTS            28.211

None match up with the mystery line. The closest is MC1, at ~0.3 Hz away.

All values were measured with 5 averages at 0.005 Hz BW. I attach screenshots of the DTT and awggui templates for MC3 as an example. The DTT templates have been copied and committed into the Sus repository here:
/ligo/svncommon/SusSVN/sus/trunk/HLTS/H1/PR3/Common/2016-10-18_H1SUSPR3_V3_Mode.xml
/ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/MC1/Common/2016-10-18_H1SUSMC1_V3_Mode.xml
/ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/MC2/Common/2016-10-18_H1SUSMC2_V3_Mode.xml
/ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/MC3/Common/2016-10-18_H1SUSMC3_V3_Mode.xml
/ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/PR2/Common/2016-10-18_H1SUSPR2_V3_Mode.xml
/ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/PRM/Common/2016-10-18_H1SUSPRM_V3_Mode.xml

Some work was started on the HWS camera on ISCTEX today as per WP6253. It wasn't completed but brought to a point where it can be completed realtively easily at the next opportunity. It was returned to it's original configuration and confirmed to still be operational as it was before the work was started.

Rick, Evan G., Travis

As part of the bi-annual PCal maintenance, today we optimized the drive range of the PCal OFS for both end stations.  The procedure for this was:

1) Turn off PCal lines and inject 10 Hz sine wave.

2) Break the OFS lock.

3) Note that the AOM drive is large (~1.5V) with loop open.

4) Adjust the offset in 1V steps to find the maximum OFS PD output.

5) Record max OFS PD output.

6) Close shutter and record minimum OFS PD output.

7) Set offset to half of 95% of max OFS PD output.

8) Find amplitude of injected sine wave that give us the maximum p-p OFS PD voltage.

9) Record magnitudes of carrier and sideband frequencies of the OFS and TX PDs.

Results:

PCal X:

PCal Y:

M. Pirello

Work Permit: #6246 & #6248

Prior to working on this we determined that the analog voltages on the channels in questions are outputting anticipated values.  The immediate problem is with the PowerOK bit for each of these signals.

Investigation into these chassis yeilded the following:

1.  Upon rebooting RF Amp Concentrator #1 on ISC_C3 in the CER (S1103450) the stuck bits on that chassis AMP18M and AMP24M1 were reset.  We are 2/3 of the way done with this task, piece of cake!

2.  Upon rebooting RF Amp Concentrator #2 on ISC_C3 in the CER (S1103451) 3 new bits were stuck for a total of 4 bits,  including the original DIV40M, drat!

3. I tried disconnecting, measuring, and reconnecting each of the 4 stuck signals on the DB25's on the back of the unit.  These voltages look good, no luck resetting the latched bits here.

     a.  DB25#1 = DIV40M = 3.64V 

     b.  DB25#2 = AMP40M = 3.75V

     c.  DB25#3 = DIV10M = 3.61V

     d.  DB25#4 = AMP10M = 3.67V

4.  I then disconnected the DB37 on RF Amp Concentrator #2 (S1103451) and checked each signal coming out of the RF Amp Concentrator.  This output is confusing.  The I couplers inside are referenced to 5V.  An on state should be 5V, off state should be 0V.  All of the PO signals should be either 5V or 0V.  PO11 is 2.5V, PO7 is 2.5V, PO2 is 2.5V.

5.  I then reconnected the DB37 to H1_EtherCAT_Corner_3.  Ten out of twelve of the OK bits remained red.  I recycled power on the RF Amp Concentrator #2 (S1103451) on ISC_3 and again all but 4 of the OK bits were green like before.

6.  I put everything back together, removed the breakout boards, etc.  When I left the CER, the 4 bits were latched, DIV40M, AMP40M, DIV10M, AMP10M.  After noon I checked the OK bits and 10 out of 12 OK bits are red including the original four.  I am relatively sure that the issue is with the RF Amp Concentrator. The power OK signals going into this chassis are good, the power OK signals coming out of it seem to latch up spontaneously and output bad voltages.  Perhaps the 5V regulator is outputting 2.5V?

Recommendation:

Strike DIV40M from WP6248 and close out WP6248 and FRS 6391 because DIV40M is connected to a different chassis. Expand FRS 6059 to encompass RF Amp Concentrator #2 (S1103451) and work to debug the source of the spontaneous latchup/latchdown and bad voltages (including DIV40M).

note that fw2 and tw1 have issued retransmission requests since the last daq restart

Here are the current versions of daqd and nds Jonathan built and installed today:

Note that h1tw0 was upgraded to U12 because it was showing RAID errors, h1tw1 has been kept back with it original Gentoo OS (these machines were the original frame writers)

here is the nds process table (NDS-1 servers run two processes; daqd and nds)

Today, I calculated the volume of piping that feeds the circulation loop for the TCSY laser.  The total volume of water in the piping alone (to and from the laser to the chiller on the mech room mezzanine) to be 36 liters.  Wow, much more than I had assumed!  The chiller reservoir holds 7 liters, for a total of 43L in the system at any given "full" time.

Recall that the system popped a mesh filter and ran the reservoir dry on Sept 28th alog 30041 - at which time only 6L was added to ~fill the reservoir.  At the time, they also noticed that there was air in noisy lines down at the table so there was air pushed through some or all of the piping volume (at least the chiller->laser piping was full of air as they mentioned, so half of the 36L piping would have needed to be refilled).

I've added up the small-ish amounts of water we have been adding daily to keep the chiller reservoir topped off - we have added 10.5L over the last 3 weeks.  With the 6L added Sept 28th, we've added 16.5L to a 43L system so far.  Even assuming there was some water still in the pipes during the Sept 28th "leak", we likley still have a ways to go before we have the system full.

Keep on filling...

(From VE drawings, I estimated ~2880" of piping length round trip chiller-laser-chiller.  The piping ID looks to be 1".)

WP6251 RCG3.2 upgrade

Jonathan, Jim, Dave:

The main work today was in upgrading the front end models and the DAQ systems to RCG3.2. All models were recompiled yesterday evening (see alog 30609). This morning we installed this new code (took 1hr 27mins). New mx code was compiled, new daqd binaries were created for all DAQ systems. The install sequence was:

• stop all daq systems
• kill all models on all front end computers
• start two front end systems (h1psl0 and h1susauxb123), prior to starting the models mx_stream was restarted
• start DAQ h1dc0 data concentrator (bug with EDCU was discovered, initial restart was with old daqd and Jonathan installed the corrected daqd on the second restart)
• start h1tw1 to start minute trending the VAC channels
• start h1nds0 with old daqdrc configuration to keep guardian happy
• start all front end models, again restarting new mx_stream prior to starting  models
• hand front end models over to operator and guardian usage
• start all daq systems except the frame writers and nds1 (at this point they are waiting for single-raw-frame upgrade)

WP6258 DAQ removal of science frame

Jonathan, Greg, Jim, Dave:

All frame writers were configured to no longer write science frames, and rename the old commissioning frames as the new raw frame.

Detailed procedure given in wiki page https://lhocds.ligo-wa.caltech.edu/wiki/MakingTheCommissioningFrameTheNewFullFrame

After this change, there are no C-named frames, only R-named frames. Archived C-Frames are now R-Frames, new frames are what were previously called commissioning frames with R-names in the frames/full directories. What were called science frames are not written, no new frames in frames/science directories.

h1nds0 and h1nds1 were reconfigured to the new R-names and restarted

the wiper scripts on h1ldasgw0 and h1ldasgw1 were changed to not use science frames, and to give the disk these frames used to the full frames instead

WP6237 Remove h1ldasgw2

Jim, Dave

a third QFS/NFS server was install in the MSR during the summer as part of the attempt to fix h1fw0/h1fw1 instability. It offloaded the exporting of the frame directories (in read-only mode) to the NDS servers. It later proved to be a liability when corrupted frame files were served by h1ldasgw2 to both NDS servers.

Today we decommissioned h1ldasgw2 and reconfigured h1ldasgw0, h1ldasgw1 to serve their respective file systems as a read-only export to the NDS machines. h1nds0 and h1nds1 were configured to no longer use h1ldasgw2.

TITLE: 10/17 Eve Shift: 15:00-23:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition

SHIFT SUMMARY: We restarted the world, now initial alignment, fss, iss and pmc are being fun.

WP 6176
WP 6247

Pulled in new power and network cables for the RGA (next to HAM4). Cables need to be terminated.
Cables at EY were terminated.

Each main turbo pump (MTP) has an adjoining box containing (4) ea. 12v 18Ah sealed lead acid (SLA) batteries connected in series that provide 48VDC for pump rotor levitation during loss of AC power.  These only get charged when the turbo controllers are energized.  As preventative maintenance during the prolonged periods of pump inactivity, we would energize the controllers for a few hours during Tuesday's maintenance period.  We now will keep these battery packs centrally located (hallway between AHU1 and AHU2 in the Corner Station Fan room) and on a battery tender.  The trade off is that we now will need to remember to take a battery pack with us when activating an MTP at an outbuilding.  

This morning, Chandra and I weighed an empty BSC LTS container. After weighing that container, we decided to check the weight on a loaded BSC LTS container. The loaded container never lifted off the floor. The crane capacity is only 10,000 lbs. and another 380 lbs. would not have lifted the container.   The results are in the photos here. 

(Kyle, Gerardo)

Today compressors #3, #4, and #5 were all greased and pressure tested.

All compressors passed their compression test, #3 at 120 psi, #4 at 120 psi, and #5 at 130 psi.
Then all compressors and electrical motors were greased.

All compressor assemblies were run tested after service was performed.

Work performed under WP#6257.

(See WPs #6221 and #6255)

Will need to make daily check each morning.  Plan to run through 10/23 inclusive unless too egregious for commissioners

M. Pirello

This FPGA code ( E1200034 ) disconnects the 1pps signal from the 4 internal LED's and grounds them.  All four timing comparators at LHO were updated to v5.

S1107952 in the CER

S1201224 in the MSR

S1201886 at X end

S1201222 at Y end

Work Permit 6256

The medm SYS_CUST_SHUTTER_SUMMARY.adl shows the ALS shutters, but the names are wrong.  Attached is a map of what each shutter did today.

Looking at the coupling of carrier jitter with misaligned RF sidebands in REFL:

Jitter (alog 30237): 10^-6/√Hz (level) to n x 10^-4/√Hz (peaks)
IMC suppression (alog 30124): ~¹⁄200
⇒ at IFO: 5 x 10^-9/√Hz to ⁿ⁄2 x 10^-6/√Hz

Fixed misalignment of RF sidebands: Δα < 0.3
DC power in reflection with unlocked ifo at 50W: REFL_DC^unlocked ~ 300 mW
Error offset in REFL = jitter * REFL_DC^unlocked * Δα
⇒ 5 x 10^-9/√Hz  * 0.3 W * 0.1 ~ 1.5 x 10^-10 W/√Hz (low)
⇒ ⁿ⁄2 x 10^-6/√Hz * 0.3 W * 0.3 ~ ⁿ⁄2 x 10^-7/√Hz (high)

Frequency noise coupling into DARM (alog 29893):
⇒10^-10 m/W at 1kHz (approx. f-slope)

at 1kHz: 10^-20 m to 10^-17 m
at 300 Hz: n x 10^-18 m (high) with periscope peak n ~ 4.

 Possibly after some changes made durring maintence, the lockloss tool stopped working. I can use lockloss plot, but not select. 

sheila.dwyer@opsws4:~/Desktop/Locklosses$ lockloss -c channels_to_look_at_TR_CARM.txt select