As Patrick reported, the Guardians for HAM2 & 3 ISI were reporting some issue; TJ is looking into that more deeply now. Regarding the Guardian update I did on 24 November, that update was only for common files and the code had been exercised on most platforms as of 1 Dec. It had not been exercised on HAM 2 3 4 6 BS or ITMX before this morning according to my notes.
This morning I decided to restart the guardian for HAMs 2 & 3 and the error condition went away. Still don't know why there was the error condition.
Next, both the ISIs were tripping during the Isolation process. It looked like this was occurring as the ISI was switching the GS13 back to high gain. I disabled the Sensor Correction thinking the high microseism might be the problem but tripping was still occurring. So I disabled the guardians, SEI and ISI, and brought the ISI to Isolation using the commands scripts. This way the GS13s could be kept in low gain. I was then able the switch the GS13 back to high gain after giving the platforms a few moments after Isolation was completed.
I tested the guardian this morning on HAM5 and it worked fine so while the first reported error gives us reason to suspect, the fundamental performance of the guardian seems to function. Again, while there are runtime parameter files unique for each ISI (although they are generally the same,) the guardian are all running the same common code. Isolation is the same on a HAM and the two stages of the BSC.
My first theory would be that the position offsets on HAMs 2 & 3 are larger than those for the other platforms. This may cause the GS13s to be noisier after the completion of Isolation just as the GS13s are switched causing the trip. However, HAM6 verticals are about 5 to 10 times that of all the others and the HAM2 & 3 horizontals are maybe only a little larger than the HAM5 drives. So, that may be the issue but it looks to be suttle. Does the payload contribute with the high microseism and the GS13 switch to trip the platform? Maybe except the HAM3 payload is much different from HAM5 and is much different from HAM2. What about the alignment wrt IFO? The HAM2 & 3 platforms, both HEPI and ISI are rotated compared to the other HAMs. What about FeedForward? HAM2 & 3 are getting X & RY FF from HEPI as is HAM6 but HAM6 GS13s aren't switched like HAM 2 & 3. HAM4 & 5 are getting their FF signals from the ISI Stage0 L4Cs.
So, could the alignment of HAM2 & 3 make them more vulnerable to the incoming high microseism? The BLRMS would not suggest this and the arrangement of the arms (NW & SW) would indicate the coastal noise woould be the same for the arms.
So the FF may be the issue. The guardian update also turns on the FF but this has proven to be a non issue in turn on transients but the microseism environment may be contributing.
The above instructions have changed, only in that the name of the state to request is now PRMI_LOCKED:
If you find that DRMI won't lock and seems to be missalinged (you see a lot of higher order modes on the AS camera, POP90 and POP18 flashes are low) you may want to lock PRMI to adjust alignment manually:
The transition from PRMI to DRMI only works sometimes, so this is probably best used only when the alignment is bad for now. (Its not a reliable way to speed up DRMI locking).