Just prior to this evening's DAQ restart at 21:04, h1fw1 froze up again with disk issues at 21:01. Similar to this morning's problem, the file system appears to exist but files cannot be fully formed.
I tried at remote reset of h1ldasgw1, but that did not proceed. Chris went into the MSR and looked at h1ldasgw1's console, it had gotten stuck in the shutdown sequence with disk issues (not surprisingly). He power cycled the Sun box using the front panel button and it did reboot. I was able to log in remotely but could not get the QFS/SAMFS file system to mount. I got the error
root@h1ldasgw1:~# mount /cds-h1-frames
mount_samfs: Configuring file system
mount_samfs: Enabling the sam-fsd service.
mount_samfs: Adding service tags.
mount_samfs: Filesystem "cds-h1-frames" not found.: No such file or directory
so unfortunately we will have to limp along with just h1fw0 for this evening. Since h1nds1 is the default NDS, for archived data the control room will have to use h1nds0 instead.
later remount attempt gave more information
root@h1ldasgw1:~# mount /cds-h1-frames
mount_samfs: Configuring file system
Device /dev/dsk/c0t6000402003F466D66444B7EB00000000d0s0 not found.
Device /dev/dsk/c0t6000402003F466D66140BDCD00000000d0s0 not found.
Device /dev/dsk/c0t6000402003F466D66140BDDB00000000d0s0 not found.
Problem in mcf file /etc/opt/SUNWsamfs/mcf for filesystem cds-h1-frames
sam-fsd: Problem with file system devices.