M. Pirello

Work Permit: #6246 & #6248

Prior to working on this we determined that the analog voltages on the channels in questions are outputting anticipated values.  The immediate problem is with the PowerOK bit for each of these signals.

Investigation into these chassis yeilded the following:

1.  Upon rebooting RF Amp Concentrator #1 on ISC_C3 in the CER (S1103450) the stuck bits on that chassis AMP18M and AMP24M1 were reset.  We are 2/3 of the way done with this task, piece of cake!

2.  Upon rebooting RF Amp Concentrator #2 on ISC_C3 in the CER (S1103451) 3 new bits were stuck for a total of 4 bits,  including the original DIV40M, drat!

3. I tried disconnecting, measuring, and reconnecting each of the 4 stuck signals on the DB25's on the back of the unit.  These voltages look good, no luck resetting the latched bits here.

     a.  DB25#1 = DIV40M = 3.64V 

     b.  DB25#2 = AMP40M = 3.75V

     c.  DB25#3 = DIV10M = 3.61V

     d.  DB25#4 = AMP10M = 3.67V

4.  I then disconnected the DB37 on RF Amp Concentrator #2 (S1103451) and checked each signal coming out of the RF Amp Concentrator.  This output is confusing.  The I couplers inside are referenced to 5V.  An on state should be 5V, off state should be 0V.  All of the PO signals should be either 5V or 0V.  PO11 is 2.5V, PO7 is 2.5V, PO2 is 2.5V.

5.  I then reconnected the DB37 to H1_EtherCAT_Corner_3.  Ten out of twelve of the OK bits remained red.  I recycled power on the RF Amp Concentrator #2 (S1103451) on ISC_3 and again all but 4 of the OK bits were green like before.

6.  I put everything back together, removed the breakout boards, etc.  When I left the CER, the 4 bits were latched, DIV40M, AMP40M, DIV10M, AMP10M.  After noon I checked the OK bits and 10 out of 12 OK bits are red including the original four.  I am relatively sure that the issue is with the RF Amp Concentrator. The power OK signals going into this chassis are good, the power OK signals coming out of it seem to latch up spontaneously and output bad voltages.  Perhaps the 5V regulator is outputting 2.5V?

Recommendation:

Strike DIV40M from WP6248 and close out WP6248 and FRS 6391 because DIV40M is connected to a different chassis. Expand FRS 6059 to encompass RF Amp Concentrator #2 (S1103451) and work to debug the source of the spontaneous latchup/latchdown and bad voltages (including DIV40M).

note that fw2 and tw1 have issued retransmission requests since the last daq restart

Here are the current versions of daqd and nds Jonathan built and installed today:

Note that h1tw0 was upgraded to U12 because it was showing RAID errors, h1tw1 has been kept back with it original Gentoo OS (these machines were the original frame writers)

here is the nds process table (NDS-1 servers run two processes; daqd and nds)

Today, I calculated the volume of piping that feeds the circulation loop for the TCSY laser.  The total volume of water in the piping alone (to and from the laser to the chiller on the mech room mezzanine) to be 36 liters.  Wow, much more than I had assumed!  The chiller reservoir holds 7 liters, for a total of 43L in the system at any given "full" time.

Recall that the system popped a mesh filter and ran the reservoir dry on Sept 28th alog 30041 - at which time only 6L was added to ~fill the reservoir.  At the time, they also noticed that there was air in noisy lines down at the table so there was air pushed through some or all of the piping volume (at least the chiller->laser piping was full of air as they mentioned, so half of the 36L piping would have needed to be refilled).

I've added up the small-ish amounts of water we have been adding daily to keep the chiller reservoir topped off - we have added 10.5L over the last 3 weeks.  With the 6L added Sept 28th, we've added 16.5L to a 43L system so far.  Even assuming there was some water still in the pipes during the Sept 28th "leak", we likley still have a ways to go before we have the system full.

Keep on filling...

(From VE drawings, I estimated ~2880" of piping length round trip chiller-laser-chiller.  The piping ID looks to be 1".)

WP6251 RCG3.2 upgrade

Jonathan, Jim, Dave:

The main work today was in upgrading the front end models and the DAQ systems to RCG3.2. All models were recompiled yesterday evening (see alog 30609). This morning we installed this new code (took 1hr 27mins). New mx code was compiled, new daqd binaries were created for all DAQ systems. The install sequence was:

• stop all daq systems
• kill all models on all front end computers
• start two front end systems (h1psl0 and h1susauxb123), prior to starting the models mx_stream was restarted
• start DAQ h1dc0 data concentrator (bug with EDCU was discovered, initial restart was with old daqd and Jonathan installed the corrected daqd on the second restart)
• start h1tw1 to start minute trending the VAC channels
• start h1nds0 with old daqdrc configuration to keep guardian happy
• start all front end models, again restarting new mx_stream prior to starting  models
• hand front end models over to operator and guardian usage
• start all daq systems except the frame writers and nds1 (at this point they are waiting for single-raw-frame upgrade)

WP6258 DAQ removal of science frame

Jonathan, Greg, Jim, Dave:

All frame writers were configured to no longer write science frames, and rename the old commissioning frames as the new raw frame.

Detailed procedure given in wiki page https://lhocds.ligo-wa.caltech.edu/wiki/MakingTheCommissioningFrameTheNewFullFrame

After this change, there are no C-named frames, only R-named frames. Archived C-Frames are now R-Frames, new frames are what were previously called commissioning frames with R-names in the frames/full directories. What were called science frames are not written, no new frames in frames/science directories.

h1nds0 and h1nds1 were reconfigured to the new R-names and restarted

the wiper scripts on h1ldasgw0 and h1ldasgw1 were changed to not use science frames, and to give the disk these frames used to the full frames instead

WP6237 Remove h1ldasgw2

Jim, Dave

a third QFS/NFS server was install in the MSR during the summer as part of the attempt to fix h1fw0/h1fw1 instability. It offloaded the exporting of the frame directories (in read-only mode) to the NDS servers. It later proved to be a liability when corrupted frame files were served by h1ldasgw2 to both NDS servers.

Today we decommissioned h1ldasgw2 and reconfigured h1ldasgw0, h1ldasgw1 to serve their respective file systems as a read-only export to the NDS machines. h1nds0 and h1nds1 were configured to no longer use h1ldasgw2.

TITLE: 10/17 Eve Shift: 15:00-23:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition

SHIFT SUMMARY: We restarted the world, now initial alignment, fss, iss and pmc are being fun.

WP 6176
WP 6247

Pulled in new power and network cables for the RGA (next to HAM4). Cables need to be terminated.
Cables at EY were terminated.

Each main turbo pump (MTP) has an adjoining box containing (4) ea. 12v 18Ah sealed lead acid (SLA) batteries connected in series that provide 48VDC for pump rotor levitation during loss of AC power.  These only get charged when the turbo controllers are energized.  As preventative maintenance during the prolonged periods of pump inactivity, we would energize the controllers for a few hours during Tuesday's maintenance period.  We now will keep these battery packs centrally located (hallway between AHU1 and AHU2 in the Corner Station Fan room) and on a battery tender.  The trade off is that we now will need to remember to take a battery pack with us when activating an MTP at an outbuilding.  

This morning, Chandra and I weighed an empty BSC LTS container. After weighing that container, we decided to check the weight on a loaded BSC LTS container. The loaded container never lifted off the floor. The crane capacity is only 10,000 lbs. and another 380 lbs. would not have lifted the container.   The results are in the photos here. 

(Kyle, Gerardo)

Today compressors #3, #4, and #5 were all greased and pressure tested.

All compressors passed their compression test, #3 at 120 psi, #4 at 120 psi, and #5 at 130 psi.
Then all compressors and electrical motors were greased.

All compressor assemblies were run tested after service was performed.

Work performed under WP#6257.

(See WPs #6221 and #6255)

Will need to make daily check each morning.  Plan to run through 10/23 inclusive unless too egregious for commissioners

M. Pirello

This FPGA code ( E1200034 ) disconnects the 1pps signal from the 4 internal LED's and grounds them.  All four timing comparators at LHO were updated to v5.

S1107952 in the CER

S1201224 in the MSR

S1201886 at X end

S1201222 at Y end

Work Permit 6256

Work Permit Number: 6252

The DMT calibration has been updated to gstlal-calibration 1.0.6-2.el7. Because of dependencies GDS was also updated to: gds-2.17.10-2.

John Zweizig has restarted the DMT monitors.
 

The medm SYS_CUST_SHUTTER_SUMMARY.adl shows the ALS shutters, but the names are wrong.  Attached is a map of what each shutter did today.

DCS verified all H1_R frames were copied from the CDS "science" directories into the LDAS archive up to the GPS endTime 1160847040.

LDAS has started copying the new larger H1_R frames from the CDS "full" directories, with the first larger H1_R frame with the "full" channel list starting from 1160854656.

Looking at the coupling of carrier jitter with misaligned RF sidebands in REFL:

Jitter (alog 30237): 10^-6/√Hz (level) to n x 10^-4/√Hz (peaks)
IMC suppression (alog 30124): ~¹⁄200
⇒ at IFO: 5 x 10^-9/√Hz to ⁿ⁄2 x 10^-6/√Hz

Fixed misalignment of RF sidebands: Δα < 0.3
DC power in reflection with unlocked ifo at 50W: REFL_DC^unlocked ~ 300 mW
Error offset in REFL = jitter * REFL_DC^unlocked * Δα
⇒ 5 x 10^-9/√Hz  * 0.3 W * 0.1 ~ 1.5 x 10^-10 W/√Hz (low)
⇒ ⁿ⁄2 x 10^-6/√Hz * 0.3 W * 0.3 ~ ⁿ⁄2 x 10^-7/√Hz (high)

Frequency noise coupling into DARM (alog 29893):
⇒10^-10 m/W at 1kHz (approx. f-slope)

at 1kHz: 10^-20 m to 10^-17 m
at 300 Hz: n x 10^-18 m (high) with periscope peak n ~ 4.

 Possibly after some changes made durring maintence, the lockloss tool stopped working. I can use lockloss plot, but not select. 

sheila.dwyer@opsws4:~/Desktop/Locklosses$ lockloss -c channels_to_look_at_TR_CARM.txt select