J. Kissel, S. Ballmer, R. Mittleman Executive summary: The seismic's HAM 2&3 front end was unable to drive output signals this weekend -- unclear whether it was related to the CDS maintenance. I have now restored its functionality with a soft-reboot of all front-end processes on the computer. The HAM-ISIs have been restored to level 3 isolation, HEPIs have been left with no drive requested but watchdogs are happy and master switch is on. Details below. ---- Details ---- Stefan informed me that Rich was having trouble getting signals out of the h1seih23 front end (both HAM2 / HAM3 ISIs and HPIs), and after a few unsuccessful trouble shooting attempts gave up. The symptoms were strange -- all levels of watchdogs were untripped (including the IOP model) and master switches were on, but no signals were getting out to the DAC (as reported by, e.g. H1:FEC-53_DAC_OUTPUT_1_7 type channels). The only metric of the failure mode was on the CDS overview screen the "WD" bit in the CDS State Word (the eighth bit [the 128 bit] of H1:FEC-53_STATE_WORD) for the H1IOPSEIH23 was red (and only the IOP, none of the user models' WD bits were red), and the GDS_TP screen for the IOP model showed the timing (H1:FEC-53_TIME_ERR) was red and claimed NO SYNC. After trying a few iterations of soft-rebooting things without success, the gave up. They knew that there was a proper song-and-dance, correct order, to soft-booting but they admitted to not remembering what it was and not being able to find any documentation describing it. I came in this morning, and did the proper procedure, and all is now functioning properly. ---- The proper front end soft-reboot procedure ---- I'm sure it's documented some where, but being able to find it... [in words] (1) log into the front end, (2) kill all the user models (order doesn't matter), (3) restart the IOP model, and (4) restart the user models (order doesn't matter). [what the terminal looks like] controls@opsws3:~ 0$ ssh h1seih23 # Step (1) controls@h1seih23 ~ 0$ killh1isiham2 # Step (2) controls@h1seih23 ~ 0$ killh1isiham3 # | controls@h1seih23 ~ 0$ killh1hpiham3 # | controls@h1seih23 ~ 0$ killh1hpiham2 # v controls@h1seih23 ~ 0$ starth1iopseih23 # Step (3) h1iopseih23epics H1 IOC Server started Burt restored /opt/rtcds/lho/h1/target/h1iopseih23/h1iopseih23epics/burt/safe.snap Old : H1:FEC-53_BURT_RESTORE 1 New : H1:FEC-53_BURT_RESTORE 1 * WARNING: awgtpman_iop has already been started. controls@h1seih23 ~ 0$ starth1hpiham2 # Start of Step (4) h1hpiham2epics: no process found h1hpiham2epics H1 IOC Server started Burt restored /opt/rtcds/lho/h1/target/h1hpiham2/h1hpiham2epics/burt/safe.snap Old : H1:FEC-54_BURT_RESTORE 1 New : H1:FEC-54_BURT_RESTORE 1 * Starting h1hpiham2 awgtpman ... [ !! ] controls@h1seih23 ~ 0$ starth1hpiham3 h1hpiham3epics: no process found h1hpiham3epics H1 IOC Server started Burt restored /opt/rtcds/lho/h1/target/h1hpiham3/h1hpiham3epics/burt/safe.snap Old : H1:FEC-55_BURT_RESTORE 1 New : H1:FEC-55_BURT_RESTORE 1 * Starting h1hpiham3 awgtpman ... [ !! ] controls@h1seih23 ~ 0$ starth1isiham2 h1isiham2epics: no process found h1isiham2epics H1 IOC Server started Burt restored /opt/rtcds/lho/h1/target/h1isiham2/h1isiham2epics/burt/safe.snap Old : H1:FEC-56_BURT_RESTORE 1 New : H1:FEC-56_BURT_RESTORE 1 * Starting h1isiham2 awgtpman ... [ !! ] controls@h1seih23 ~ 0$ starth1isiham3 h1isiham3epics: no process found h1isiham3epics H1 IOC Server started Burt restored /opt/rtcds/lho/h1/target/h1isiham3/h1isiham3epics/burt/safe.snap Old : H1:FEC-57_BURT_RESTORE 1 New : H1:FEC-57_BURT_RESTORE 1 * Starting h1isiham3 awgtpman ... [ !! ] controls@h1seih23 ~ 0$ # End of Step (4) After the turning on the front end processes, I - was delighted to see the safe.snaps were reasonably up to date - Turned on the HAM2-ISI and HAM3-ISIs damping loops and restored to Level 3 isolation - left HAM2 and HAM3 HEPIs as untripped and master switches on, but no requested drive since I figured Rich was just going to immediately play with it once he got in.