The primary frame writer h1fw0 is not able to write frames. It looks like an Oracle QFS file system issue, the NFS mounted file system is not accessible. The QFS writer h1ldasgw0 is pingable, but I cannot ssh onto it remotely. If anyone is going to the site today, please call me and perhaps we can try power cycling the machine.
The secondary frame writer continues to write frames OK. If this machine goes down we will no longer be trending vacuum controls signals so I would like to resolve this before Monday if possible.
It looks like the system started failing at 5pm Wed. Our DAQ seems to know about holidays.
h1fw0 shutdown:
The h1fw0 file system is actually on-again-off-again, and I've seen times when the two frames written by fw0 and fw1 are not identical. Since we know fw0 is having issues, we can assume that its frames are the incorrect ones. Unfortunately, as the primary writer, h1fw0's frames are preferentially archived by LDAS over h1fw1. I have stopped the daqd regeneration process, h1fw0 has now stopped running and it not being restarted. This will allow LDAS to fully switch over archiving h1fw1's frames.