I was notified that the h(t) pipeline were hung up (thanks to Patrick Brockill) at about 18:20UTC. I found that the primary (h1dmt0) pipeline was producing data, but with a latency ~5 hours. The redundant pipeline (h1dmt2) was not producing output frames. I restarted both at about 18:25 (h1dmt0) and 10:29 (h1dmt2). Both pipelines seem to be producing data now.

HofT data gap. Path is /gds-h1/dmt/frames/hoft/H1/{gps-time}/. PST times.

Mar  4 22:59 H-H1_DMT_C00-1172732480-4.gwf

Mar  5 10:32 H-H1_DMT_C00-1172774052-4.gwf

DMT monitors are back after Cheryl rebooted h1broadcast0. I verified that no full frames were lost from h1fw0 or h1fw1 overnight. We did lose HofT frames when the DMT broadcaster was down. Keith is testing redundant broadcasters at LLO to prevent a problem like this.

Redundant GDS broadcasting now working at LLO to one DMT machine.  Need to install network switch(es), additional Ethernet cards for redundancy on the redundant DMT machines..

That's interesting.  When I checked the make-up air on Wednesday (after Travis first witnessed this temp drop) it was definitely running at 20%.  Did you check the make-up air fan in the acoustic closet (East of the PSL enclosure) to confirm it was running at 100% (it's very loud and drowns out the power supply in the closet when at 100%)?  There have been times when the PSL environmental controls indicate the fan is 100%, but when the make-up air fan itself is checked it is running at 20%.  There seems to be some odd link in the evironmental controls software where it shows 100% on all fans if one set of fans (i.e. the anteroom or laser room HEPA fans) was set to 100%.  This also works the other way, where the HEPA fans will sometimes show 20% (they are either 100% or off) after the make-up air has been set to 20%.  Essentially, the environmental controls appear to hold the last user set fan speed that then follows from fan screen to fan screen, but appears to not be applied until the user does so.  I don't know if this is how the system is supposed to behave, but I have seen this behavior on more than one occasion.

  Received alerts on PSL AC/Temp at 21:29 & 21:59 UTC. Spoke with Jason. Dropped out of Observing (LLO down) to go into the LVEA and check Makeup Air fans and AC controls. 
   (1). Makeup Air fan is running at LOW SPEED. It is NOT running at 100%.
   (2). The AC Controls - Makeup Air fan speed at 100%. I reset it to 20%. Verified that the Makeup Air fan was still at low speed. 

  Trended the five temperature sensors located in the PSL Laser enclosure for the past 30 days. Results are posted below. 

  There were 8 low temperature event starting on 03/01 at around 20:02 UTC. These events are consistent across the five sensors. The Dust Monitor looks a bit odd but this is due to the 1 minute sample taken every 10 minutes. There is some time lag but reasonably good coherence none the less. There is less coherence between the temperatures in the PSL and in the LVEA.      

(John W, Gerardo M)

There appears to be something that is turning on, then off, maybe something triggered by a thermostat.  Noise can be observed on the accelerometers on the PSL table.  See attachments, all plots are 5 days long.  On first plot the signal for the accelerometer (color blue) was modified to match the temperature signal.  Other plot no mods done.

• WFS A DC PIT - good, then temperature drops, seems to be recovering, lockloss
• PSL temperature drops 2degF in 14 minutes
• EOM power drops, Periscope power drops, PZTs move by 100 in pitch and 40 in yaw

The picture is more likely to be the pre-modecleaner reflection, not the reference cavity transmission.
If it really is the reference cavity transmission, then we have a big alignment and mode matching
problem.

    The picture seems to indicate that the horizontal alignment has changed.

• Noise is clearly visible on the HAM2 ISS second loop qpd, and the X arm Trans A qpd.
• Also, 11 hours ago when this happened, there was a realitively small increase in noise, and it seemed to recover, and then the noise got worse again, and H1 lost lock.
• In this lock the small increase in noise has come and gone, and H1 just glittched in range down to about 10Mpc.

• taking a closer look at REFL_DC and the temperature, REFL recovers part or all of the way after the bursts of noise, at least in a time series, so lock loss may not be related

Here's a couple of trends around the relevant time.  The first attachment shows all 4 PSL Enclosure laser room temperature sensors.  The channels labeled *_ACS_* and *_ACN_* are the sensors at the output of the enclosure AC units, which are turned off during normal operation; the channels labeled *_TBLN_* and *_TBLS_* are the sensors attached to either end of the PSL table.  The second attachment shows the south AC sensor (*_ACS_*) and the TPD signal from the FSS RefCav; I used the RefCav TPD as this has shown to be one of the more temperature sensitive alignment in the enclosure.  As can be seen, as soon as the temperature begins to dip, the TPD signal becomes more jagged and begins to drop.  This is consistent with a real change in temperature in the PSL enclosure.  We are currently mystified by this behavior, as the AC units were both off during this time.

Similar behavior was reported by Travis yesterday, and again both AC units were off during the temperature dip.  At this time we have no explanation for this behavior.  Investigations continue and will update as we know more.

There is a fan in the LVEA return air plenum which continually pushes air into the PSL enclosure even when the AC units are off. This fan is behind the east wall next to the PSL. It would be worthwhile to find out what the temperatures are doing in that space. This return air space is much larger than the supply ducting to the LVEA and therefore the flow velocities are low and there may be air currents driven by convection. Also, winds will have the effect of pressurizing one side of our building with respect to the other side - this may also drive air currents in the return air plenum. Is there a correlation with wind conditions?

I do not know of any temperature sensors in the plenum.

It looks like something was changed on March 1st at about 20:00 UTC. PSL-ENV temp trends show a marked difference in behavior since then, see attached trends. It would be good to figure out what has happened because the temperature dips correlate with extra motion in the IMC, and we have lock losses that follow the temperature dips. Maybe not causal, but it looks like something is not nominal.