Today has been a rough day for the input HAM chambers. This morning we found that a channel went dead in expansion chassis (see Richards alog for the resolution), and troubleshooting involved a lot of restarts of SEIH23. This afternoon, after that problem was sorted, Evan found that the SEI MEDM screens for HAM2&3 were frozen. Dave and JimB should be posting a log about the resolution of that issue.
It would be very good if Detchar could at do some comparisons of the H3 IPS on HAM3 HEPI with other sensors to see if the failure that took out this chamber this morning (killing commissioning efforts until ~now) gave us any warning. I attach some dataviewer trends of the IPS blend ins, and I can kind of convince myself that the horizontal loops all look a little noisier 18 hrs ago versus now. Not necessarily true, I haven't looked in any detail and I'm just guessing based on the max/min being noisier 18 hours ago.
Plots like this, for the last couple of weeks. This is a spectra from 23 hours ago before the trip at 01:00 local today. Maybe also look at impacts on the ISI and/or MC2 PR2?
I've attached spectra of H2 and H3 spaced by three hours. There's a clear indication that H3 is going bad on Apr 2 between 18 and 21 UTC. The next spectra pin the start time to between 19:20 and 19:25 UTC (I think that's noon local time). The time series, which starts at 19:15, shows that the problem may come in the form of bursts of noise. Update: I've added an hour-long time series of the channel, high-passed at 10 Hz. The problems start 12 minutes in. It looks like there are bursts of noise, as well as maybe an increase in the overall noise level.
According to the summary pages, this had an impact on the ISI and on the optic motion. The ISI spectrogram and optic motion spectrogram show an increase in noise right around 19:20 UTC. If we need a monitor for this sort of problem, a BLRMS from 30 to 100 Hz would probably work. The sensor is just flat white noise there, until the signal goes bad. Attached is a 12 hour BLRMS showing the onset of the problem.
Sadly, this doesn't show up in the BLRMS signals that we currently have on HEPI. The HPI-HAM3_BLRMS_X_30_100 channel is the BLRMS of the L4C cartesian signal in the 30 to 100 hz band and it doesn't show the sensor going bad. Attached plot is the same 12 hour window that Andy plots just above, and the problem is not apparent. Mo channels, mo problems.
