That story agrees with my observation on Friday night when I started looking at the FS after 11:15 PM.

Sheila reported (79092) that the Fast Shutter reopened only after ~50ms or so. It seems that the low voltage drive to keep the shutter closed was not working. 1st attachment shows the last time that happened at around 23:10 local time. Daniel points out that the shutter was in an error state at that time but that was after Ryan power cycled the FS driver. We don't know exactly what kind of state the fast shutter was in here.

The next time the HV firing was tested was at 23:23 local time (2nd attachment), the shutter was kept shut (i.e. low voltage thing was somehow working) but there are two things to note.

1. H1:ASC-AS_B_NSUM_OUT_DQ was about 25 or so, i.e. larger than the dark offset threshold of +-15. The FS guardian go to failed state.
2. Bouncing was worse than I've ever seen. It bounced multiple times with 40 to 50ms period and eventually settles to the "closed" posotion. (Before Friday I only saw single bounce and that was it.)

The last FS test I've done was 00:12 local time on Jul/13 when the error was cleared, with the smaller power than nominal (3rd attachment). Bouncing was as bad but the power coming to HAM6 was smaller (see the trigger power at the top left). AS_B_NSUM was somewhat smaller (more like 10).

The reason why AS_B_NSUM is worse is because I reduced the analog DC gain by a factor of 10 and compensated for that by digital gain. The effect of analog offset as well as ADC/electronics noise are 10x worse than AS_A. I adjusted the dark offset while IMC was unlocked but we can probably increase the threshold to 30 or so if it continues to bother us.

Bouncing behavior might be more serious as it could mean that the beam was close to the end of the travel of the FS mirror (and it was bad on Friday because of funny alignment), or low voltage drive was somehow still funny. I suspect the former.

Here is my attempt at recreating the (rough) timeline of events related to the fast shutter Friday night (all times in UTC):

• 04:55:41 - While locking, ISC_LOCK gets to CHECK_AS_SHUTTERS and has the FAST_SHUTTER Guardian go to its TEST_SHUTTER state
  • The fast shutter test runs. Shutter closes, but due to improper dark offsets in the AS WFS DC NSUM channels (noted by Daniel above), Guardian thinks the test failed
    • Guardian sets AS_PROTECTION error code
    • Guardian reports "Fast shutter failed tests!! Downrange light does not disapear properly!" and jumps to SHUTTER_FAILURE
• 04:55:43 - The fast shutter shows its error code
• 04:55:46 - Fast shutter error code clears
• ~05:20 - I power cycle the fast shutter driver chassis
• 05:21:46 - Shutter opens
• 05:28:00 - Next shutter close test; only closes for 65ms
• ~05:45 - I power cycle the shutter logic chassis
  • No change in behavior is seen
• 06:10 - There have been several attempts at running the shutter test at this point, all of them the shutter only was closed for 60-70ms at a time
• 06:20 - The shutter finally closes and stays closed
• 06:53 - Keita fixes the dark offset on AS WFS
• 07:10 - Keita uses the Guardian to test the shutter; it works and stays open
• 07:12 - Beckhoff errors clear after another full cycle using the Guardian

Seems like ASC-AS_B_DC gain was probably a red herring, important thing is that the beam got uglier/bigger at some point, therefore a part of the beam was not blocked by the fast shutter.

First attachment shows when ASC-AS_B_DC gain switch was flipped from x10 to x1 on Tuesday. You can see that the Fast Shutter has been firing OK until Friday evening.

The rest of the plots show the FAST Shutter test done by the Beckhoff at three different points in time, i.e. the last test before my AS_B_DC change ("happy" Monday July/08 ~16:37 UTC or 9:37 local), the first one after my AS_B_DC change ("happy" July/11 4:43 UTC or Wed July/10 21:43 local time), and the first time the FAST shutter went into the error mode ("sad" Jul/13 4:56 UTC or Fri Jul/12 21:56 local time). The last one is when Sheila and Ryan started having problem.

Important points are:

1. The two "happy" plots are quite similar. Especially, even though I see bouncing action, the power measured by both AS_A and AS_B more or less settle to the minimum value. Ratio of closed/open is about 230ppm in both cases. This is compatible with the specification of Fast Shutter mirror (which is <1000ppm).
2. The "sad" plot looks nothing like the happy ones. Right after the shutter was closed the light level went down to less than 1000ppm, but bounced back and eventually settled down to ~6500ppm. You can also see that the light level when FS was open (~2200 cts) is about 50% of what it used to be in happy plots.
3. ASC-AS_C was pretty close to center in all of the three plots.

From these, my conclusion is that the beam position on the fast shutter mirror was pretty much the same in all three tests, but the beam was pretty ugly for the "sad" plot as was witnessed by many of us on AS port camera. Because of this a part of the lobe was missed by the Fast Shutter. Centering an ugly beam on AS_C might have complicated the matter.

Later, when I forced the test with much lower power, the error was cleared because even though the ugly beam was still there the power went lower than the "shutter closed" threshold of the guardian.

I don't fully understand who did what when during the time the shutter was in the error state (it includes people pressing buttons and power cycling the driver and then pressing buttons again, and I certainly pressed buttons too).

Looking at this, and since Daniel agrees that the Fast Shutter has been working fine, my only concerns about locking the IFO are:

1. The beam should be properly attenuated by the fast shutter. Not thousands of ppm, but more like ~230ppm.
2. The beam should not clip on ASC-AS_C. If it's not clipped there, the chances of thing getting clipped downstream is very small.