Adrian, Camilla, Gabriele, Fil, Richard
While trying to track down the source of the 4 Hz bumps in DARM, Richard noticed the cosmic ray photomultiplier tube (PMT) power supply voltage display numbers were jumping around. When he reset the dial, we noticed the 4 Hz oscillations in H1:PEM-ADC_5_29_2K_OUT_DQ disappeared for a few minutes. We unplugged the power supply to the PMTs at 19:05:00 UTC and have not seen 4 Hz features in the ADC channel or DARM since. Plots of the change in ADC channel timeseries behavior when Richard first touched the power supply knobs as well as the ADC channel behavior after the power supply was unplugged are attached. A spectrum comparing the ADC power supply plugged in (blue) and unplugged (red) is attached. We had about 40 minutes to watch DARM before some commissioning tasks started and we didn't see any more bumps. Another plot is forthcoming for this...
(edit for spelling...)
We need to collect more data to be sure, but for now it looks like the 4.05 Hz bumps in DARM are gone
This is awesome and fingers crossed that it fixes things.
Just a quick note that as we try to figure out how this coupled, it seems that H1:PEM-CS_ADC_5_29_2K_OUT_DQ might not be our best witness channel because while today it saw the 1.35Hz and 4.05Hz before the CR PS unplug, it did not see those peaks during some past times when n*4.05Hz bumps were seen in h(t) such as 4/12, 4/14, and 4/16.
Some of the other PEM channels such as H1:PEM-CS_ACC_LVEAFLOOR_HAM6_Z_DQ do seem to be reliable witnesses, having seen the 1.35Hz and 4.05Hz during h(t) bump times on those previous days as well as today before CR PS unplug, but not after unplug.
There was another instance of the 4 Hz bumps at 20:50 UTC. The attached plot shows OMC-DCPD_SUM, with blue as a reference, green dotted an example of the glitch a few days ago, and orange the one now. It's complicated a little by the PCal line at 24 Hz; it could be notched if we wanted a better plot. The bump at 20 Hz does look weaker but the rest are similar to before.
Most likely one of the photomultiplier tubes has developed a minor short and is going through a charge/discharge cycle. Not surprising after ~20 years! It is probably worth testing if the discharge is in the PMTs or the HV power supply by disconnecting the HV cables from the power supply and powering it back up. Either the fluctuations in the display will stay or not.
It looks like there are fewer of those 4.05Hz bump / glitches, but the problem is still with us, some of them quite loud.
Power was restored to the PMTs around 11:00:00 local on May 16, 2023, as we believe the 4 Hz glitching is coming from the the cryyobaffle. All four displays worked when I turned the power supply back on again.