To speed up the restart we have created an SSH key-pair between the operator station and h1oaf0, and written a script to kill and restart the models in the correct sequence.
1. log into operator0 as user ops (password in shared secrets)
ssh ops@operator0
2. run script
./restart_h1oaf0_models.bsh
When the models are running again, at the bottom of the CDS Overview MEDM press the buttons "!Press All Diag Reset Buttons" and "!DAQ Clear Accumulated CRC" to green up the board.
If problem persists, we will do more hardware investigations tomorrow
[Terra, Jenne, Cheryl, Nutsinee]
When the OAF model requires a restart during a full 25W lock, the parametric instabilities lose their damping during the reboot (since the damping stuff lives on that computer). Unfortunately, the PIs ring up badly enough during the short time it takes to boot that we lose lock shortly after the computer goes down for a restart. This is the 2nd time today that we've seen this. Despite Cheryl's efforts when this happened during the daytime, and Nutsinee's efforts just now, the modes are just too high for the damping to work.
So. Obviously the CDS team is already working on figuring out why the oaf computer is so unhappy, but I just wanted to highlight that even if we reboot before the TCS trips, we still lose lock due to this hardware problem.
Previously the PI gain SDF setpoints were set to zero since guardian turns on gains at DC READOUT. Both model restarts today happened at NLN, so when the PI model restarted it was setting the damping gains to zero (since we weren't moving through DC READOUT), so I've changed the setpoints to the final gain settings. Correct PI gains should now turn on even with a restart in NLN. Note that Jenne's comment above still stands; this past restart Nutsinee and I immediately set gain settings back upon restart but the modes were still too rung up as Jenne said.