J. Kissel

I've solved the loose wires in the software problem indicated a bit ago in in LHO aLOG 15571, with the sledge hammer solution of restarting all of the front end processes. Unfortunately this is now quite an arduous process. Details of exactly what I did are below, but the problem is now solved.

This is another data point on Integration Issue 931, which remains open, because these errors happen only once every month or so -- enough time for other CDS fires to overwhelm this one. 

Curiously, it took two starts of the IOP process in order to get the AWG light to behave. Another mystery.

--------
Restart process

(1) Turn off all sensor correction, because this is not yet under guardian control. Set H1:ISI-*_ST1_SENSCOR_GND_STS_?_MATCH_GAINs to 0, and turn off OUTPUT switch for good measure
(2) Request Manager SEI GUARDIANs to take the chambers to OFFLINE (noting the state from whence they came. Beam Splitter is ISOLATED_DAMPED, ITMs are FULLY ISOLATED)
(3) Save any alignment offsets on the SUS for the current state they're in, if GUARDIAN is warning you they're not saved
(4) Request the SUS GUARDIANs to take the SUS to SAFE (noting the state from whence they came AND the EPICs values of the alignment offsets)
(5) Take the opportunity to grab a updated safe.snap file, and commit to userapps repo.
jeffrey.kissel@operator1 ]$ makeSafeBackup sus h1susbs
(6) Log into the h1susb123 front end as controls
(7) Kill all USER front-end processes
controls@h1susb123 / 0$ killh1susitmx
controls@h1susb123 / 0$ killh1susitmy
controls@h1susb123 / 0$ killh1susbs  

(8) Restart IOP front process, make sure to have GDS_TP screen open, to hit the BURT button once EPICs comes back
controls@h1susb123 / 0$ starth1iopsusb123 
h1iopsusb123epics H1 IOC Server started
Burt restored /opt/rtcds/lho/h1/target/h1iopsusb123/h1iopsusb123epics/burt/safe.snap
Old : H1:FEC-28_BURT_RESTORE         1
New : H1:FEC-28_BURT_RESTORE         1
 * WARNING:  awgtpman_iop has already been started.
controls@h1susb123 / 0$
(9) Restart USER front-end processes, make sure to have GDS_TP screen open, to hit the BURT button once EPICs comes back
controls@h1susb123 / 0$ starth1susitmx    
h1susitmxepics: no process found
h1susitmxepics H1 IOC Server started
Burt restored /opt/rtcds/lho/h1/target/h1susitmx/h1susitmxepics/burt/safe.snap
Old : H1:FEC-29_BURT_RESTORE         1
New : H1:FEC-29_BURT_RESTORE         1
 * Starting h1susitmx awgtpman ...                                                                                                                                                                [ !! ]
controls@h1susb123 / 0$
(rinse and repeat for the other two SUS)
(10) Reset all watchdogs on SEI, SUS, and IOP (there's about 30).  << --- It's only at this step d'you know if you've actually fixed the problem
(11) Request SUS GUARDIANs to return to original state (whether it be aligned or misaligned, make sure alignment offsets have been restored properly)
(12) Request SEI Manager GUARDIANs to return to original state
(13) Go through CDS OVERVIEW Screens and hit "DIAG RESET" in the upper left corner of any GDS_TP screens which show an IPC Error in the CDS STATE WORD. The IPCs glitch on start up, and the error latches, so it just needs to be cleared. If it goes red again after a few seconds, then there's *actually* a problem.
(14) Turn sensor correction back ON by first turning ON the MATCH OUTPUT, and then ramping up the gain back to 1.0, with a ramp time of something like 10 to 15 seconds. 



I've turned on the following sensor correction configuration configuration for all three BSC chambers:
- Ryan's IIR filters, SC-rdr, FM2 for X & Y with a gain of 1.0. Only the BS has a "MATCH" filter (just a gain of 1.03), which I've left ON as I'd found it.
- Rich's IIR filters, Mitt_SC, FM6 for Z with a gain of 1.0.

For the record, both the ISI and HEPI Sensor Correction Screens still suck:
- The filters that we actually use in the GND sensor correction are not visible on the sensor correction overview
- The links to sub-screens are ambiguous locations on the SC overview. Thus, one can pull up different screens -- that don't look different at first glance -- depending on where you click
- In the HEPI sensor correction overview, there's no link for the full MATCH filter bank, so you have no control over / knowledge of important things like the ramp time
- Remove ST0 to ST1 HEPI L4C Feed forward path, it's currently not used and there's scientific motivation (pending) that it should never be used.