I just noticed that the h1guardian0 machine is in need of some attention. It currently is using all its RAM and is swapping from disk, so presumably it is now performance limited.
The memory usage started increasing around 17:30 PDT Thursday 22nd, and got steadily worse. Swapping started at 03:30 PDT Friday 23rd and has been doing so ever since then.
We will most probably reboot during tomorrow's maintenance.
controls@h1guardian0:~ 1$ free -g
total used free shared buffers cached
Mem: 11 11 0 0 0 0
-/+ buffers/cache: 11 0
Swap: 11 5 6