I updated the software on all of the Beckhoff vacuum controls computers as per work permit 5875. Carlos and I took the opportunity to change the computer names at the same time. They are now h0vaclx, h0vacly, h0vacmr, h0vacmx, h0vacmy, h0vacex and h0vacey. I have attached the channel names changes that were made in the DAQ. I set the smoothing on all of the CP pump levels to 0.99. A lot of the cold cathode gauges have still not come back on. CP4, CP5 and CP6 are still in the process of recovering.

Kyle, Joe D. 

Today we re-installed the SRS RGA100 analyzer after having replaced its fried filaments and CDEM last Friday.  We leak checked the new joints but won't be able to test the basic functionality until tomorrow.  Assuming that every thing works, it will still be "weeks" before VBOD can process bake loads as the SRS RGAs are impossible to bake out effectively as the result of near zero conductance resulting from poor focus plate design - if anybody has an extra $20K in their budgets, please send it our way.  The newer aLIGO-era RGAs pay for themselves in production up time and minimal man hours to Futz with.

(see also https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=27078) 

Today I inspected the "hang tags" on the N2 and Kr calibration gases at each of the currently installed aLIGO RGAs (Pfeiffer Prisma Plus, Y-end, X-end and Vertex).  It turns out that these are NOS (new old stock) calibration gases that were intended for but never used during the Beam Tube bake back in 1998.  Though I don't specifically remember, I must have decided to use these since they were the correct gases and leak rates needed for the RGAs and were unused, albeit old, and would save having to spend another $8,000.  This looks to have been a bad idea as the factory fill pressures for the N2 bottles were sub-atmospheric, ~500 torr.  This is typical for "crimped capillary" leaks for the mid 10-8 torr*L/sec leak rates and isn't an issue as long as their isolation valves never get opened at pressures > 500 torr, say like at atmosphere.  The problem here is that these sat in a drawer for 15 years along with various other cal-gas bottles and were accessible to anybody wanting to turn a knob.  

It looks very much like the N2 cal-gas used at the Vertex had been opened to atmosphere at some point during its long storage.  I am currently re-baking the Vertex RGA after having exposed it to the N2 bottle last Friday but while both cal-gas isolation valves are open.  It may be that the N2 bottle may need to isolated and tagged-out and we may be stuck with only a Kr leak at the Vertex.  I haven't reviewed the data but this might also explain the poor clean-up of the RGAs at the end stations.  

Jenne told me this afternoon that the laser had tripped.  The crystal chiller indicated an error with
flow sensor 1.  All the flow sensors in the chiller are the vortex style and have no moving parts
and are not likely to fail in the same manner as the old turbine ones.  My guess is that after ~31k
hours of service, it's time to start scheduling a time where we switch the chiller out for the spare
and send this unit back to Termotek for servicing - who recommend that the servicing be done annually.

    This is the second time in about 4 days that the laser has tripped because of this problem.

h1pemey

Dave:

I modified h1pemey to receive the muxed OMC DCPD demodulated channels from h1omc, and demux them into 8 dolphin channels. These will be received by the h1susetmypi model in a later mod. The h1pemey model now has the same mods as were applied to h1pemex last week.

Vacuum Controls

Carlos, Patrick, Dave:

Patrick made new vacuum controls changes today. Some LLCV channels changed names, and new INI files were generated. I copied the running minute trend files from old-name to new-name on both h1tw0 and h1tw1. I created the new edcu file, which was renamed from H0EDCU_VE.ini to H0EDCU_VAC.ini

Target files, autoburts and snap files are being renamed from h0veXX to h0vacXX.

Beckhoff computers were renamed from h0veXX to h0vacXX. The NTP settings were verified, but it looks like the clocks remain unsynced.

Slow Controls Beckhoff Computers

Carlos, Jim, Dave:

The NTP settings on h1ecat[c1,ex,ey] were changed from name based to IP address. This caused a resync of the time on these machines (were 4 seconds out).

DAQ Restart

Dave:

DAQ restart to incorporate the h1pemey new model and the VAC new ini.

Ryan, Carlos, Jim, Dave:

The new ZFS file system on h1fs0 which is NFS exported as /opt/rtcds became 97% full recently. This is a fast (SSD) but relatively small (1TB) file system.  The reason for being fast is to prevent front end epics freez-ups which we saw last year when the disk was busy syncing to its backup machine.

With this being a snapshotted ZFS file system, freeing up space is not as easy as removing and/or compressing files. First all the snapshots must be destroyed to free up the file blocks and allow file modifications.

Process on both h1fs0 (primary) and h1fs1 (backup) was:

 1 destroy all snapshots

 2 remove all old adl files in the rcg generated dirs /opt/rtcds/lho/h1/medm/model-name/

 3 tar and compress old build areas in /opt/rtcds/lho/h1/rtbuild*

 4 delete all non-rcg generated target_archive dirs (mostly from 2011-2012)

 5 delete all rcg generated target_archive dirs with the exception of the most recent four dirs

2&3  freed up only a modest amount of disk space, they were mainly to clean up clogged directories.

4&5: prior to doing this we verified these old target archives were backed up to tape. Stop 5 freed the lion's share of disk space, about 600GB.

The end result is the amount of free space went from 53GB to 685GB, disk usage went down from 97% to 28%.

At the end of the procedure, a full ZFS-snapshot sync of h1fs0 to h1fs1 was done manually, and the hourly sync was turned back on.

Transition Summary:

Title:  05/10/2016, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT)
State of H1: IFO unlocked. Work on PSL is ongoing. Tuesday Maintenance day  
Commissioning: 
Outgoing Operator: N/A
 
Activity Log: All Times in UTC (PT)

14:00 (06:00) Peter – In the H1 PSL enclosure
14:30 (07:30) Jeff – Replace network cable to PSL dust monitor #102
15:00 (08:00) Start of Tuesday maintenance day
15:00 (08:00) Christina & Karen – Cleaning in the LVEA
15:14 (08:14) Sprague – on site to remove pest control boxes
15:30 (08:30) Hugh – Restarting HAM models (WP #5874)
15:48 (08:48) Filiberto, Leslie, & Manny – Going to pull cables for the access system
15:49 (08:49) Hugh – Finished with model restarts
15:54 (08:54) N2 delivery at both end stations
16:00 (09:00) Patrick – Update Beckhoff code for vacuum controls computer (WP #5875)
16:04 (09:04) Kyle – Going to both end stations VEAs. 
16:10 (09:10) Jim B – Restarting H0EPICS to apply security patch.
16:10 (09:10) Bubba – Replacing faulty check valve air comp at Mid-X (WP #5876)
16:15 09:15) Peter – Out of the H1 PSL enclosure
16:17 (09:17) Filiberto – Power off Vac rack LX, & LY to check grounds
16:35 (09:35) Christina & Karen – Cleaning at End-X
16:35 (09:35) Chandra – Going to Mid-X to work on CP4
16:36 (09:36) Kiwamu – Going to HAM6 area to check S/Ns on electronics
16:41 (09:41) Kiwamu – Out of LVEA 
16:43 (09:43) Jim B. – Finished with H0EPICS restart
16:45 (09:45) Cintas – On site to service rugs and garb
16:50 (09:50) Ed – Going to End-Y to terminate cables in the VEA
16:59 (09:59) Bubba – Back from Mid-X
17:05 (10:05) Christina & Karen – Finished at End-X and are going to End-Y
17:06 (10:06) Kyle – Back from the End Stations
17:10 (10:10) Filiberto – Out of the LVEA
17:18 (10:18) Hugh – Going to both End Stations to check HEPI pumps
17:45 (10:45) Hugh – Finished with HEPI checks at End Stations 
17:50 (10:50) Hugh – Checking CS HEPI pumps
17:58 (10:58) Hugh – Finished with CS HEPI pump checks
18:04 (11:04) Nutsinee – Going to LVEA to take pictures near HAM4
18:13 (11:13) Nutsinee – Out of the LVEA
18:44 (11:44) Vendor on site to deliver drinks
18:54 (11:54) Gerardo – Going to End-Y VEA to measure for EitherCat cables
19:29 (12:29) Gerardo – Back from End Stations
19:49 (12:49) Manny & Leslie – Going into the LVEA to brackets for access sys on IO cabinets
20:27 (13:27) Bubba – Going into LVEA to look at N2 LTS system
20:28 (13:28) Gerardo – Going to Mid-Y to shim CP3 and CP4
20:58 (13:58) Bubba – Out of the LVEA
21:04 (14:04) Manny & Leslie – Finished in the LVEA
21:11 (14:11) Sheila & Jeff – Adjusted the ALS wrong polarization from 43% to 7%
22:10 (15:10) Sheila – Going to End-X
22:20 (15:20) Manny & Leslie – Going to both End Stations to install access controls bracket
22:39 (15:39) Chandra – Going into the LVEA to the walkover bridge
22:45 (15:45) Chandra – Out of the LVEA
22:48 (15:48) Chandra – Going to both end stations to make measurements
22:48 (15:48) Gerardo – Going into the LVEA – climbing around HAM6

End of Shift Summary:

Title: 05/10/2016, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT)
Support: 
Incoming Operator: Cheryl 

Shift Detail Summary: Maintenance Tuesday. Work continues on recovery of the PSL.   

Attached are measurements of the frequency stabilisation taken from this morning.
Noise spectra and the transfer functions are plotted for common gains of 20 dB and 22 dB,
and fast gains of 22 dB and 8 dB respectively.

    At lower frequencies the noise is lower with the common gain at 20 dB.  However this
results in a slight decrease in the unity gain frequency.  With these settings the UGF
is ~450 kHz.  With the common gain at 22 dB, the UGF is ~650 kHz.  Anecdotally it seems
that the frequency servo is more glitchy with the common gain at 22 dB.

    Improving the cross-over by looking at the spectrum and flattening it and adjusting
the fast gain, improves the flatness of the spectrum but then increasing the common gain
does not push the noise floor as low as the two cases illustrated.

Related log(s): 21131

Summary. - Today, I have updated the first anti-whitening filters of the OMC DCPDs in the OMC digital front end model in order to make them more accurately match the actual analog circuits. The attached show screenshots of the foton filters before and after the update.

Impact. - The new OMC DCPD responses are different from what they used to be by 1% at 10 Hz in magnitude and almost no change at 100Hz and above. The O2 DARM model must take this update into account. Note that even though the qualitative behavior of the mismatched anti-whitenings is similar to the large bias we have been seeing in the DARM response (for example 24569), they do not explain the bias (which is about 10% at 10 Hz in magnitude).

DCPD balancing. - According to my calculation, the balancing should be good at a 0.1% level without introducing an artificial imbalance in the OMC front end. So I removed the existing artificitial imbalace (-0.3%) and update the OBSERVE and DOWN SDFs. However, the imbalance should be experimentally double-checked and possibly re-adjusted.

I have removed the power stabilisation photodiode box from the table for
testing/debugging.  Please do not attempt to engage the power stabilisation.

The h0epics computer was patched and restarted.  Dust IOC's for the diode room, lab, lvea, psl, ex, and ey were restarted.  The dewpoint monitor IOC was restarted.  The protocol file for the psl IOC was updated to handle three digit ID numbers more reliably.

WP 5874

SEI update summarized in T1600090. This includes :

o GS13 signals storing rate changed. ECR E1600077

o STS signals at the senscor channel point not recorded anymore -> ECR E1600077

o blend glitch-> ECR E1500456

o 10 secs delay on coil driver status before triggering watchdog-> ECR E1600076

o remove user dackill-> ECR E1600042

o Cumulative saturation count->ECR E1500325

Thanks to Arnaud for this round up.

Required updating the isihammaster.mdl, guardian isi manager.py and the HAM ISI Overview medm.

See attached for full log.

State of H1: DRMI locked!  Then DRMI lost lock, FSS went into oscillation

H1 Configuration:

• ISS Refracted Power at 15.5% and stable
• FSS Fast Gain at -8dB (per alog 17875 / lowest PZT MON) and I turned the locking loop off (seemed like the right thing to do, given the PZT MON)

Initial Alignment / Locking: 

• Summary: Initial Alignment is complete, but not without trouble; unknown, alignment, Guardian(?)
• Jenne and Sheila finished locking the X arm in IR, but I couldn't get the PRM to lock, I don't know why
• I did another Initial Alignment and spent a lot of time trying to get the X arm to lock in IR again, and eventually punted with raising the gain to 0.1
• SRC locked a couple times, but WFS killed the lock, and I saw SRCL_OUT was railed
• ... and oh, SRM was misaligned in pitch by 800urad, which I would assume is it's actual misaligned position, though it thought it was aligned
• even after realigning SRC, locking it was kicking SRM so hard again it wouldn't lock, I had to take Guardian to the IS_LOCK, run Init and Down, and then go back to Initial Alignment, then take ALIGN_IFO Down and back to SRC_LOCK, then SRC locked and made it through SRC_ALIGN_OFFLOADED
• on to Full Lock, and DRMI locked, but I didn't catch it in time to fix the small misalignment, so DRMI lost lock, that killed the IMC, that killed the FSS, and the FSS is still recovering
• FSS went into oscillation earlier in my shift, and then again when DRMI broke lock, PZT MON is high, I admit I pushed some buttons (Common Gain, Fast Gain, AUTOLOCK on/off) that I think should not have prolonged this situation, however, PZT MON is currently at 9V and slowly dropping, and I have the Fast Gain set at -8dB, after reading Peter's alog 17875.

Activities:

• Chandra - VAC - working with Cryo Pumps (cryo pump 4 was temporarily in alarm)
• Sheila, Jenne - recovery/alignment/relocking H1
• DaveB - DAC - nds1 had to be restarted, glitch in data from 5/9 23:59UTC to 5/10 00:05UTC

Other H1 activities / configuration:

• alignment check/recovery IMs, RMs, OMs, PR2, Y arm quads, X arm quads, TMS X/Y
• dark offsets were addressed
• ETMX BRS X direction is enabled, Y and Z are disabled, Z is turned off, Y has zero gain
• ETMY BRS Y direction is enabled, X and Z are disabled, Z is turned off, X has zero gain

CP4 is back on scale and in PID mode. 

Experimented with PID loops using CP2. 

Sheila, Jenne, Cheryl

Xarm IR locking was difficult again today, and we made two observations. 

We have a more aggresive low pass filter in MC2 M3 than we really need, CLP300.  We locked the arm twice sucsesfully with this filter off and CLP500 on instead. screenshots are attached with the digital gain set to 0.15.  We didn't add this to the guardian yet, but I think its a good idea (third screenshot attached shows the trial configuration).

We are in no danger of saturating M3, but M2 saturates when we are trying to acquire.  This might be the root of some of our troubles accquiring the xarm.   

I just pushed a minor upgrade to guardian to fix a small issue with the guardlog client when following node logs.  The new version is r1542.

The client was overly buffering stream data from the server, which was causing data from the stream to not be output in a timely manner.  This issue should be fixed in the version I just pushed.

As always make sure you have a fresh session to get the new version:

jameson.rollins@operator1:~ 0$ which guardlog
/ligo/apps/linux-x86_64/guardian-1542/bin/guardlog
jameson.rollins@operator1:~ 0$