Sheila, Terra, Jeff K, Patrick, Nutsinee

Tonight we continued with the PRMI locking difficulties of yesterday.  Nutsinee managed to lock PRMI by lowering the MICH gain and commenting the boost and offloading out of the guardian. 

Patrick tracked down some differences between the "good" PRMI time that Jenne mentioned (Oct 17 16:03 UTC) and now, there was a filter missing in the BS top mass offloading, and two whtening stages (with the anti whitening) were engaged in the middle of the lock yesterday afternoon.  We don't know why this happened, but just undoing it actually makes it impossible to lock at all. 

To match the PRCL loop shape to a reference, I had to add 6 db of gain (PRCL digital gain of 16 rather than 8).   However it looks like we still missing some kind of boost in PRCL (1st screenshot).  The second screenshot shows the MICH loop measured today on the left, before and after the PRCL fix, and an old measurement on the left. 

We have commented out the boost and offloading of MICH in the guardian for PRMI, and nutsinee created  new prmi_mich_gain_als and prmi_prcl_gain_als parameters in lscparams, and we have set them to 0.7 and 16 for now, although the nominal values should be 1.4 and 8.  You will have to reduce the MICH gain by hand to get it lock. 

Could who ever did some measurements that required changing the REFLAIR whitnening gain triple check that whatever they did is not causing us problems in the morning?

Don't have remote monitoring and will need to enter X-end VEA to make measurements sometime this afternoon.

Helped Sheila troubleshoot. Ran through an initial alignment. Still not able to lock DRMI. Peter is in the PSL enclosure aligning the PMC. Strange behaviour with the PSL noise eater (Peter agrees). Reported in oscillation but no seeming effect on ISS, FSS, etc.

11:35 UTC Set request to LOCK_DRMI

Did DRMI catch very very briefly? Went to CHECK_MICH_FRINGES. Aligned BS. Back to LOCK_DRMI. No catches.

11:50 UTC Start initial alignment
11:53 UTC PSL noise eater oscillating. No one else is here and I'n not sure I'm permitted to go in the LVEA alone to fix it. Leaving it until someone comes in. FSS, ISS and PMC are still locked despite this.
12:35 UTC Finished initial alignment

Set to LOCK_DRMI. Still no catches. PSL noise eater is still in oscillation.

13:38 UTC Peter to PSL enclosure to align PMC. Set IFO to down.
13:49 UTC Bubba to mid X to check on supply fan
14:21 UTC Bubba back

TITLE: 10/18 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Initial alignment didn't go so smoothly but we did it eventually. We had issue at INPUT_ALIGN (kept knocking ISS, FSS, and PMC out of lock), MICH_DARK_LOCKED (MICH won't catch the mode -- I had to pre-adjust BS in DOWN before requesting MICH_DARK_LOCKED), and SRC_ALIGN (the signal kept running away instead of converging). We kept losing lock at PRMI. Similar thing happened last night. Sheila mentioned that MICH gain was too high. We lowered the gain by half and managed to kept PRMI locked stabily.

Note: Lockloss tool complained about ASC-AS_A_DC_SUM_OUT_DQ claiming that it doesn't exist (so was dataviewer) so I removed the channel from the lockloss file.

The first attached screenshot is a series of DCPD spectra from the last several months.  When we first turned on the HPO, the noise in DARM did not get any worse.  The noise stayed similar to O1 until July 16th, the lump from 200-1kHz started to appear in late July, when we started a series of changes in alignment and TCS to keep us stable with a decent recycling gain at 50 Watts.  

PMC

Last night, Gabriele pointed out that the noise in DARM is coherent with the PMC HV.  The HV readback had been mostly some white noise (probably ADC noise), until the last few weeks, but has been getting noisier so that some of the things jitter peaks show up in it now (SEcond attached screenshot, colors corespond to the dates in the DCPD spectrum legend.  This may be related to the problem described in LLO alogs 16186 and 15986.  The PMC transmision has been degrading since July, which could be a symptom of misalingment.  Since July, the REFL power has nearly doubled from 22 to 38 Watts, while the transmission has dropped 28%.  The PMC sum has also dropped by 4%, roughly consistent with the 3% drop in the power out of the laser.  Peter and Jason are planning on realigning the PMC in the morning, so it will be interesting to see if we see any difference in the HV readback.  

TCS + PRC alignment:

The other two main changes we have had in this time are changes in the alignment through the PRC, and changes to TCS.  These things were done to improve our recycling gain and stability without watching the noise impact carefully.  In July we were using no offsets in the POP A QPDs.  We started changing the offsets on August 1st, after the lumpy noise first appeared around July 22nd.  We have continued to change them every few weeks since then, but generally moving in the same direction.  

The only TCS change that directly coresponds to the dates when our noise got worse was the reduction of ETMY RH from 0.75 W each on July 22nd, the other main TCS changes happened September 10th.  It would be nice to undo some of these changes before turning off the HPO, even if it means reduing the power to be stable.  

J. Kissel

We're continuing to struggle to even get back to DRMI locking as of last night (LHO aLOG 30614). One suspicion is the sharp feature 27.0977 Hz (using a 0.005 Hz BW ASD) that has suddenly appeared, as shown in Jenne's attached ASD. 

Because we haven't yet measured them, and we're dead in the water until we figure out the problem, I've taken the time to excite and measure all of the HAM2 and HAM3 HSTS and HLTS's highest vertical mode frequencies, modeled to be at 27.3 Hz and 28.1 Hz, respectively. The results are as follows:

 Optic      Sus. Type     V3 Mode Freq (Hz)
  MC1         HSTS            27.387
  MC2         HSTS            27.742
  MC3         HSTS            27.426

  PRM         HSTS            27.594
  PR2         HSTS            27.414

  PR3         HLTS            28.211

None match up with the mystery line. The closest is MC1, at ~0.3 Hz away.

All values were measured with 5 averages at 0.005 Hz BW. I attach screenshots of the DTT and awggui templates for MC3 as an example. The DTT templates have been copied and committed into the Sus repository here:
/ligo/svncommon/SusSVN/sus/trunk/HLTS/H1/PR3/Common/2016-10-18_H1SUSPR3_V3_Mode.xml
/ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/MC1/Common/2016-10-18_H1SUSMC1_V3_Mode.xml
/ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/MC2/Common/2016-10-18_H1SUSMC2_V3_Mode.xml
/ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/MC3/Common/2016-10-18_H1SUSMC3_V3_Mode.xml
/ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/PR2/Common/2016-10-18_H1SUSPR2_V3_Mode.xml
/ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/PRM/Common/2016-10-18_H1SUSPRM_V3_Mode.xml

Some work was started on the HWS camera on ISCTEX today as per WP6253. It wasn't completed but brought to a point where it can be completed realtively easily at the next opportunity. It was returned to it's original configuration and confirmed to still be operational as it was before the work was started.

Rick, Evan G., Travis

As part of the bi-annual PCal maintenance, today we optimized the drive range of the PCal OFS for both end stations.  The procedure for this was:

1) Turn off PCal lines and inject 10 Hz sine wave.

2) Break the OFS lock.

3) Note that the AOM drive is large (~1.5V) with loop open.

4) Adjust the offset in 1V steps to find the maximum OFS PD output.

5) Record max OFS PD output.

6) Close shutter and record minimum OFS PD output.

7) Set offset to half of 95% of max OFS PD output.

8) Find amplitude of injected sine wave that give us the maximum p-p OFS PD voltage.

9) Record magnitudes of carrier and sideband frequencies of the OFS and TX PDs.

Results:

PCal X:

PCal Y:

M. Pirello

Work Permit: #6246 & #6248

Prior to working on this we determined that the analog voltages on the channels in questions are outputting anticipated values.  The immediate problem is with the PowerOK bit for each of these signals.

Investigation into these chassis yeilded the following:

1.  Upon rebooting RF Amp Concentrator #1 on ISC_C3 in the CER (S1103450) the stuck bits on that chassis AMP18M and AMP24M1 were reset.  We are 2/3 of the way done with this task, piece of cake!

2.  Upon rebooting RF Amp Concentrator #2 on ISC_C3 in the CER (S1103451) 3 new bits were stuck for a total of 4 bits,  including the original DIV40M, drat!

3. I tried disconnecting, measuring, and reconnecting each of the 4 stuck signals on the DB25's on the back of the unit.  These voltages look good, no luck resetting the latched bits here.

     a.  DB25#1 = DIV40M = 3.64V 

     b.  DB25#2 = AMP40M = 3.75V

     c.  DB25#3 = DIV10M = 3.61V

     d.  DB25#4 = AMP10M = 3.67V

4.  I then disconnected the DB37 on RF Amp Concentrator #2 (S1103451) and checked each signal coming out of the RF Amp Concentrator.  This output is confusing.  The I couplers inside are referenced to 5V.  An on state should be 5V, off state should be 0V.  All of the PO signals should be either 5V or 0V.  PO11 is 2.5V, PO7 is 2.5V, PO2 is 2.5V.

5.  I then reconnected the DB37 to H1_EtherCAT_Corner_3.  Ten out of twelve of the OK bits remained red.  I recycled power on the RF Amp Concentrator #2 (S1103451) on ISC_3 and again all but 4 of the OK bits were green like before.

6.  I put everything back together, removed the breakout boards, etc.  When I left the CER, the 4 bits were latched, DIV40M, AMP40M, DIV10M, AMP10M.  After noon I checked the OK bits and 10 out of 12 OK bits are red including the original four.  I am relatively sure that the issue is with the RF Amp Concentrator. The power OK signals going into this chassis are good, the power OK signals coming out of it seem to latch up spontaneously and output bad voltages.  Perhaps the 5V regulator is outputting 2.5V?

Recommendation:

Strike DIV40M from WP6248 and close out WP6248 and FRS 6391 because DIV40M is connected to a different chassis. Expand FRS 6059 to encompass RF Amp Concentrator #2 (S1103451) and work to debug the source of the spontaneous latchup/latchdown and bad voltages (including DIV40M).

note that fw2 and tw1 have issued retransmission requests since the last daq restart

Here are the current versions of daqd and nds Jonathan built and installed today:

Note that h1tw0 was upgraded to U12 because it was showing RAID errors, h1tw1 has been kept back with it original Gentoo OS (these machines were the original frame writers)

here is the nds process table (NDS-1 servers run two processes; daqd and nds)

Today, I calculated the volume of piping that feeds the circulation loop for the TCSY laser.  The total volume of water in the piping alone (to and from the laser to the chiller on the mech room mezzanine) to be 36 liters.  Wow, much more than I had assumed!  The chiller reservoir holds 7 liters, for a total of 43L in the system at any given "full" time.

Recall that the system popped a mesh filter and ran the reservoir dry on Sept 28th alog 30041 - at which time only 6L was added to ~fill the reservoir.  At the time, they also noticed that there was air in noisy lines down at the table so there was air pushed through some or all of the piping volume (at least the chiller->laser piping was full of air as they mentioned, so half of the 36L piping would have needed to be refilled).

I've added up the small-ish amounts of water we have been adding daily to keep the chiller reservoir topped off - we have added 10.5L over the last 3 weeks.  With the 6L added Sept 28th, we've added 16.5L to a 43L system so far.  Even assuming there was some water still in the pipes during the Sept 28th "leak", we likley still have a ways to go before we have the system full.

Keep on filling...

(From VE drawings, I estimated ~2880" of piping length round trip chiller-laser-chiller.  The piping ID looks to be 1".)

WP6251 RCG3.2 upgrade

Jonathan, Jim, Dave:

The main work today was in upgrading the front end models and the DAQ systems to RCG3.2. All models were recompiled yesterday evening (see alog 30609). This morning we installed this new code (took 1hr 27mins). New mx code was compiled, new daqd binaries were created for all DAQ systems. The install sequence was:

• stop all daq systems
• kill all models on all front end computers
• start two front end systems (h1psl0 and h1susauxb123), prior to starting the models mx_stream was restarted
• start DAQ h1dc0 data concentrator (bug with EDCU was discovered, initial restart was with old daqd and Jonathan installed the corrected daqd on the second restart)
• start h1tw1 to start minute trending the VAC channels
• start h1nds0 with old daqdrc configuration to keep guardian happy
• start all front end models, again restarting new mx_stream prior to starting  models
• hand front end models over to operator and guardian usage
• start all daq systems except the frame writers and nds1 (at this point they are waiting for single-raw-frame upgrade)

WP6258 DAQ removal of science frame

Jonathan, Greg, Jim, Dave:

All frame writers were configured to no longer write science frames, and rename the old commissioning frames as the new raw frame.

Detailed procedure given in wiki page https://lhocds.ligo-wa.caltech.edu/wiki/MakingTheCommissioningFrameTheNewFullFrame

After this change, there are no C-named frames, only R-named frames. Archived C-Frames are now R-Frames, new frames are what were previously called commissioning frames with R-names in the frames/full directories. What were called science frames are not written, no new frames in frames/science directories.

h1nds0 and h1nds1 were reconfigured to the new R-names and restarted

the wiper scripts on h1ldasgw0 and h1ldasgw1 were changed to not use science frames, and to give the disk these frames used to the full frames instead

WP6237 Remove h1ldasgw2

Jim, Dave

a third QFS/NFS server was install in the MSR during the summer as part of the attempt to fix h1fw0/h1fw1 instability. It offloaded the exporting of the frame directories (in read-only mode) to the NDS servers. It later proved to be a liability when corrupted frame files were served by h1ldasgw2 to both NDS servers.

Today we decommissioned h1ldasgw2 and reconfigured h1ldasgw0, h1ldasgw1 to serve their respective file systems as a read-only export to the NDS machines. h1nds0 and h1nds1 were configured to no longer use h1ldasgw2.

 Possibly after some changes made durring maintence, the lockloss tool stopped working. I can use lockloss plot, but not select. 

sheila.dwyer@opsws4:~/Desktop/Locklosses$ lockloss -c channels_to_look_at_TR_CARM.txt select