16:45 UTC Sheila, Filiberto, Vern, Nutsinee to end Y to investigate problem with Beckhoff. Looks like some of the channels froze around 09/15 5:45 UTC. May be coincident with start of 2 Hz comb in spectrum.
Does this have any interaction with the Pcal? We've seen something go very wrong there at the time near where the 2 Hz glitches started (we'll alog that shortly). There's also something wrong with some ETMY M0 OSEMs. Detchar is happy to have this investigated at the cost of locked time. This problem ruins the data and it's probably unanalyzable.
The PCAL team is investigating.
17:37 UTC Vern, Nutsinee, Filiberto and Sheila back. Brought Beckhoff chassis with them. Vern says they found a broken Beckhoff terminal.
18:11 UTC Filiberto, Sheila, Vern replacing end station 2 Ethercat chassis at end Y with spare 18:31 UTC Error remains in Beckhoff. Team at end Y is restarting Beckhoff computer. I put ISC_LOCK guardian to manual and down prior.
18:48 UTC Hugh burtrestored h1ecaty1plc1, h1ecaty1plc2, h1ecaty1plc3 to 09/14/22:10 as requested by team at end Y.
19:32 UTC Filiberto restarted Beckhoff vacuum gauges, going to turn back on high voltage for ESD
Strange DC level shift in EY microphone channels, especially EBAY racks (see ch3). But not low frequency mic.
As soon as the Beckhof froze, EY microphone signals DC level went down, and after Vern/Sheila/Elli/Fil restarted Beckhof the DC level came back close to original. Why? Is the ground level of ebay area pulled by something else controlled by Beckhof (or Beckhof chassis itself)?