Over the past three days we have been slowly increasing GN2 heater power on CP4 via new variac. Attached is plot since Wed. with variac settings. Today I opened the LN2 draw valve all the way (2-1/8 turns) at Dewar to increase flow but as we know from last time, we bottom out and need the help of the pressure build circuit. I opened that valve 1/2 turn and slowly the Dewar head pressure increased from 12 psig to 16 psig, but I noticed a hissing and vapor coming out of the capped pipe where the LN2 delivery truck connects to. My guess is either the top or bottom fill valves is leaking. I closed the pressure build valve for the weekend. The temperature dip at the end of the plot is likely due to intermediate increased flow and is now recovering. Will monitor over weekend.

Spent the past two days with SAES Getters reps to commission the new high vacuum NEG pump (HV1600-10 NexTorr) installed on BSC6 at EY. We activated the pump yesterday and ran it nominally at 180C overnight. This morning we valved it into the main volume. Attached is ~2day plot. The pressure improved with the NEG+IP valved in. We were scanning the RGA during the transition of valving the pump in today and saw no change in partial pressures immediately. Over hours H2 fell by ~30%. I will attach RGA scans next week.

We tested the 50 m cable this afternoon with the power supply in the adjacent mechanical room. It powered up ok but I found it faulted when I went back later to check on it and the temperature dropped from 180C to 46C. It seemed to turn back on OK. Will check it one more time before I leave. It can run cold, but won't be as effective. The error message read:  Global: 202 - Main Line Wrong Voltage. Touch this bar to clean the alarm.

We also activated the CapaciTorr NEG pump on BSC6 while we were out there, but left it valved out. Before activating we pumped out with cart the Ar, etc gasses that built up over months during the EY vent, which caused the pressure to creep to e-4 Torr.

Plot legend: PT-428 is hot cathode (HC) IG on HV NEG housing; PT-410B is CC on BSC10; PT425 is HC on BSC6.

Gerardo, Patrick

Wanted to only restart the screenshots for the vacuum system but ran into trouble and ended up restarting all of them. Kept getting 'Xvfb failed to start' when attempting to run h0_start, but not when running the others. Deleted /tmp/.X99-lock and /tmp/.X6-lock after searching the web for answers, but this did not help. Upon Gerardo's suggestion we compared the scripts. h1_cds_psl_start, h1_sei_start, and h1_sus_start had 'exec xvfb-run -a', but h0start only had 'exec xvfb-run'. I copied h0start to h0start.original_20_apr_2018 and added '-a' to h0start. This allowed it to run, but now all of the screenshots have black rectangles over them.

Nutsinee, Sheila

This morning Nutsinee and I measured the beam profile arriving in HAM6 from the squeezer, before OM1.  

Then we improved the alignment of the 532 beam into the OPO, the power in the 1st order mode is now about 5% of the power in the 00 mode.  Nutsinee saved scan data for the 532 beam.  

Then we placed an aperture on the far side on HAM6 in the seed path, and removed the translation stage lens from the translation stage.  We measured the beam profile in a few locations after ZM1, which seems to be agreeing with the model that Dan and Thomas have.  We repalced the lens and checked that the beam still comes back to HAM6 on the OMC side. 

Lastly we measured the distance from the beam diverter to the LVEA wall is 137.5 inches. 

More details about the measurements are coming soon.

Checked while Georgia and I were working on EFM. No contact was visible and it seemed to swing freely.

After finally resolving the seg fault issues in guardian (report in full to follow), guardian has been upgraded site-wide and moved to the new production guardctrl host, "h1guardian1".

We've been monitoring the system for the last couple of days and it seems to be working nominally and not showing signs of excess load on the guardian machine.  We have seen no seg faults with this new version.

new guardian version: 1.2.0

Other than the fix for the seg faults, which turned out to be because pcaspy is not thread safe, there aren't too many changes in this version that users should notice.  The main new features have to do with the node management interface:

• Ability to use NodeManager to request states of subortinates without setting them to MANAGED mode (i.e. without issuing a set_managed command).  This enables a "loose" management whereby one node can request a state of another node, without putting it in to MANAGED mode, and without having to deal with stalls on jump.
• Allow for mulitiple NodeManager objects.  This is useful if you want to do a mix of loose and micro management.
• Warnings if formatting is incorrect (e.g. mixing spaces and tabs)

Other than that there are just various bug fixes and minor improvements.

new guardian machine, node process supervision

The new production guardian machine is h1guardian1, which is running Debian 9 with all needed software installed from packages from CDSSoft.

Guardian process supervision on h1guardian1 is now handled by systemd.  In particular, it's handled via systemd --user under the "guardian" user account.

The load on this machine seems pretty good:

The above is while running all 124 of the H1 nodes.  We'll be monitoring this to make sure load average stays below 100.

Note: this is mostly of no consequence to users, as they'll continue to interact with the supervision system via the guardctrl interface, which has been updated to work with the new system...

new guardctrl interface

The new systemd supervision infrastructure required a new version of guardctrl, which is now installed on all workstations.  It works mostly the same as the old version, with some slight changes in some of the subcommands:

jameson.rollins@zotws6:~ 0$ guardctrl -h
usage: guardctrl [-h] [-d] ...

Guardian daemon supervision control interface.

Control guardian daemon processes managed by the site systemd
supervision system.

optional arguments:
  -h, --help   show this help message and exit
  -d, --debug  print debug information to stderr

Commands:
    help       show command help
    list       list nodes and node subervision state
    status     print node service status
    enable     enable nodes
    start      start nodes
    restart    restart nodes
    stop       stop nodes
    disable    disable nodes
    log        view node logs

Add '-h' after individual commands for command help.

Node names may be specified with wildcard/globbing, e.g. 'SUS_*'.
A single '*' will act on all configured nodes (where appropriate).
jameson.rollins@zotws6:~ 0$

Known issues:

• Searching through old logs is currently not working.  This will be fixed in a new minor release soon.
• Logs sometimes take a few seconds to show up.  Still investigating how to make log retrieval snappier.

code archiving temporarily disabled

Guardian node code arching has been temporarily disabled until we're fully confident in the new system.  This is because of ownership of the code archives that will need to be moved to the new guardian user.  If things are still looking good by next week we will enable the code archives then.

old guardctrl host h1guardian0 still available in case of emergency

If for any reason there are problems with the new setup, we can easily restore everything to the old configuration on h1guardian0.  The new guardctrl interface is backwards compatible with the old h1guardian0 host.

The procedure to restore a single node to the old host would be something like this:

$ guardctrl stop NODE_NAME
$ GUARDCTRL_HOST=h1guardian0 guardctrl start NODE_NAME

To do the same for all nodes, replace "NODE_NAME" with "'*'".

NOTE: if all nodes are moved back to the old host, the CDS admins would need to update the DNS record for "h1guardian" to point to "h1guardian0" instead of "h1guardian1".

Prior to shift:
Verbal alarms is crashed
Jamie working on guardian

14:47 UTC Jeff B. to end X to reset dust monitor
15:28 UTC Jeff B. back from end X
15:37 UTC Let Advanced Protection Services (APS) through gate
15:48 UTC Peter to PSL enclosure
15:59 UTC Let visitor through gate to see Chandra
16:04 UTC Let Advanced Protection Services (APS) through gate
16:38 UTC Robert to end X to change a setting on electric field injection
16:43 UTC Sheila and Nutsinee to HAM6 to run a beam scan on the squeezer beam
17:05 UTC Robert back from end X
17:12 UTC Greg G. to LVEA
17:24 UTC Robert and Georgia to end X, inchamber work on EFI, check cyropump baffles
17:30 UTC Ed to optics lab, LVEA for property inventory
17:37 UTC Gerardo to LVEA to retrieve power supply to take to end Y
18:03 UTC Peter done in PSL enclosure
18:55 UTC Terry to HAM5 to give equipment to Sheila
18:55 UTC Gerardo to end Y to check on status of HV NEG pump bake
18:59 UTC Robert and Georgia back from end X
19:06 UTC Ken to end X to work on the ceiling above the outside door
20:02 UTC Ken done at end X
20:06 UTC Marc and Elizabeth to LVEA to look for equipment for property inventory
20:14 UTC Marc and Elizabeth done
20:14 UTC Let visitor through gate to see Chandra
20:24 UTC Ed and Elizabeth to end Y for property inventory
20:49 UTC Ed and Elizabeth back from end Y
20:55 UTC Nutsinee back to HAM6
20:56 UTC Elizabeth to LVEA to look for coil of wires
20:58 UTC Corey and Hugh to HAM5 area to prep for vent next week
21:09 UTC Elizabeth back from the LVEA
21:09 UTC Sheila back to HAM6
22:25 UTC Hugh back
22:47 UTC Corey back

I have just completed a full round of model code compilation. This is just a "make" not a "make install" so no target/DAQ/GDS files have been modified.

Before the builds I backed up the H1.ipc file and emptied the file. After the build was completed, I reverted the original H1.ipc file back in case we restart any systems this weekend.

I wrote a script to compare the new DAQ-INI files with what are currently being used. Of the 107 models, 11 have different INI files. This indicates their code has been changed since the last H1 build (calcs, susauxb123, susauxex, susauxey,susauxh34, susetmx, susetmy, susitmx, susitmy, susitmpi, susprocpi)

I'm reviewing these models to see what has changed.

As per Chandra's request:

CP4_TE253A Rate-Of-Change channel has been removed from alarms.

CP4_TE253A (low,high) levels have been changed from (0,150) to (100,200) degC

As per Ryan's request, Kyle's contact email address has been changed to his new caltech.edu one.

h1susauxb123 and h1pemmx front end systems are both running RCG3.4.2/Gentoo3.0.8 with no current issues.

The problem of the models not starting automatically on reboot has been resolved.

There have been some occasional  DAQ issues seen when h1susauxb123 models were started/stopped. Specifically: sometimes when the models are stopped the DAQ data from h1seih16 were glitched (running start_streamers.sh on susaux123 clears this). During some starts of the code the DAQ status for the models h1susopo and h1ascimc were flashing between 0x0 and 0x2000. This was when the susaux models were not starting correctly, has not been seen since.

One surprise, after recompiling h1susauxb123 the resulting DAQ-INI file was different. The file h1susauxb123.mdl has not been modified since 2015, so I suspect some of the common mdl files used by this model have been modified recently.

PeterK, JasonO, RickS

Yesterday afternoon and evening we realigned the beam path from the 70-W amplifier to the PMC, addressing the discrepancy in optical component mounting heights - 10 cm vs. 4".  All components for mounting lenses (lens mounts, rails, etc.) and the new Faraday Isolator located between the Front End and the 70-W amplifier were designed for a fixed optical height of 10 cm.  Almost all mirror mounting components, the 70-W amplifier, and the PMC were designed for a fixed optical height of 4".  The difference of ~1.6 mm is significant.

After the first turning mirror downstream of the 70-W amplifier and up to the last two turning mirrors (with PicoMotors) directing the beam into the PMC, we set the beam height at 10 cm.  Thus, the beams are obviously below center on all turning mirrors.

We removed the high-power AOM for the ISS and spent a few hours tweaking the modematching to the PMC.

Currently, the PMC visibility is about 75%.  We suspect that the alignment through the 70-W amplifier needs to be adjusted.

We left the system with the new PMC running at full power (about 70 W) and the PMC output power directed to the water-cooled beam dump directly downstream of the PMC.

We are modifying some of our alignment flags, designed for 4", to the 10 cm beam height.

[Craig Niko Georgia]

Conclusions:

    - We have taken electric field spectra with the foil door off, foil door on, and a foil house built around the EFM. Spectra coming soon.
    - We also applied a DC voltage to the calibration plates and looked at the decay time at the output of the EFM, which should give us a confirmation of the EFM resistor (should be 1 TOhm)


We went back to end-X to take more electric field meter (EFM) data. We wanted to find out whether the low frequency slope shown in our previous log post is due to sensor noise or fields within the chamber. We were also interested in confirming the dependence of the 60 Hz and 30 Hz peaks on the presence of the foil door. We made a foil house for the EFM which attached to the rod, and were careful not to ground the sensor plates (picture 1). The soft door cover was on for all these measurements. These three spectra will be posted shortly.

We also used a voltage calibrator to apply a DC voltage to the calibration plates, and measure the decay time of the voltage out of the EFM. The reason we checked this was to confirm the 1e12 Ohm resistor. The time constant of the RC circuit formed by the capacitance of the calibration-sensor plate (~7.6 pF), and the 1e12 Ohm resistor should be 7.6 seconds.
    - Note that we were using larger spacers between the sensor plates and calibration plates as we (embarrassingly) lost track of the old ones, and so cleaned up and used the spacers from the first EFM, the capacitances were measured to be 7.8 pF from calibration plate to +X sensor plate, and 7.5 pF from calibration plate to -X sensor plate.
    - We applied 3V DC continously to the calibration plates.
    - The decay is shown in the 2nd photo, the two traces are the positive and negative x outputs (generally we do a differential measurement, this is not a very fancy scope). The decay time looks to me to be about 12 seconds (though it’s hard to tell since I can’t tell when exactly the voltage was applied, Crag can correct me here...), so perhaps our resistance is larger than expected.