The strip charts were stopping and starting periodically as evidenced by flat lines in the signal plots. Restarting the computer seems to have fixed it for now.
Out of observing briefly while Jeff B. went to end Y. Otherwise no issues.
O1 days 62,63
model restarts logged for Thu 19/Nov/2015 No restarts reported
model restarts logged for Wed 18/Nov/2015 No restarts reported
Jeff B. has been given permission to drive to end Y to take pictures in the receiving area and hallway to the VEA.
TITLE: 11/20 [DAY Shift]: 16:00-00:00 UTC (08:00-16:00 PDT), all times posted in UTC STATE Of H1: Observing @ ~ 80 MPc. OUTGOING OPERATOR: Jeff B. QUICK SUMMARY: From the cameras the lights are off in the LVEA, PSL enclosure, end X, end Y and mid X. I can not tell if they are off at mid Y. 0.03 - 0.1 Hz seismic band is between ~ 0.01 and 0.04 um/s. 0.1 - 0.3 Hz seismic band is between ~ 0.1 and 0.4 um/s. Winds are between ~ 0 and 5 mph. ITMX, ITMY and BS ISI blends are on 45mHz. EMTX X and ETMY Y ISI blends are on 45mHz. ETMX Y and ETMY X ISI blends are on Quite_90.
Activity Log: All Times in UTC (PT) 08:00 (00:00) Take over from Ed 08:23 (00:23) GRB – LHO locked in Observing mode, LLO down – Ignored alert 08:45 (00:45) Lockloss – Unknown - ITM-Y & ETM-Y saturations at time of lockloss. 09:23 (01:23) Relocked in Observing mode 11:15 (03:15) Seismic_FOM_STS display crashed (Seismic DMT on nuc5) - Restarting 15:00 (07:00) Tumbleweed bailing on X-Arm near CS – Then will go to LSB 16:00 (08:00) Turn over to Patrick End of Shift Summary: Title: 11/20/2015, Evening Shift 08:00 – 16:00 (00:00 – 08:00) All times in UTC (PT) Support: None needed Incoming Operator: Patrick Shift Summary: Lockloss cause unknown. Relocked with no problems. IFO back into Observing mode. Seismic FOM (top display nuc5) crashed. Restarted the nuc. Waiting for Kerberos ticket to renew (at 12:00 (04:00), and will try to restart. Good Owl shift. IFO in Observing mode for the past 6.75 hours, with a 76Mpc range. Three ETM-Y saturations during the second half of the shift. Wind is calm to light (0-4mph) Seismic activity is low, microseism centered around 0.35um/s.
Previously I had noted a slow oscillation in the output power of the front end laser. The plot AmplifierPower.png partly shows the oscillation but unfortunately the fluctuation seen is the one dominated by the humidity change in the Laser Room. However the fluctuations are more obvious in the pump diode power plot (DiodePower.png). The question is then what is driving these fluctuations? DiodeTEC.png shows the voltage applied to the Peltier cell to cool the pump diodes. The fluctuation is clearly seen here. It is also evident in the pump diode voltage (DiodeVoltage.png) and current (DiodeCurrent.png). It is possible that we have a small oscillation in the output of the front end laser's two power supplies. Or as Richard pointed out to me, it's possible that the two power supplies are beating against each other. The oscillation is not evident in the pre-modecleaner reflected light signal, and so is probably just a curiosity at the moment.
For information's sake, the switching frequency of the pump diode power supplies is over 200 kHz. The other power supply in the diode box has a switching frequency of 50 kHz.
Turned on the power watchdog for the laser, since it wasn't reset when the power glitch happened the other day. 13:14-something UTC.
Had one lockloss just after Ed relocked at the end of the Evening shift. Reason unknown. Was able to relock without issues and IFO has been in Observing mode for the past 3.5 hours. Power is 22.2W, range is 79Mpc. Environmental conditions are good, with wind at a light breeze (4-7mph) and seismic activity quiet. Three ETM-Y saturations since relocking. The Seismic part of the DMT session crashed. Rebooted nuc5 and working on restarting FOMs.
Received GRB alert at 08:23 (00:23) - LHO in Observing for only 19 minutes before alert. LLO still recovering from earlier lockloss. Ignored alert.
Title: 11/20/2015, Evening Shift 08:00 – 16:00 (00:00 – 08:00) All times in UTC (PT) State of H1: 08:04 (16:00), The IFO locked at NOMINAL_LOW_NOISE, 22.2w, 77Mpc. Outgoing Operator: Ed Quick Summary: IFO has been in Observing mode for 4 minutes. Ed has just recovered from an earthquake related lockloss. Environmental conditions are good – wind is a light breeze (4-7mph), seismic activity is below 0.1 um/s, and microseism is centered around 0.3 um/s.
TITLE: Nov 18 DAY Shift 00:00-08:00UTC (04:00-00:00 PDT), all times posted in UTC
STATE Of H1: SUPPORT:
LOCK DURATION: ≈ 4.75
INCOMING OPERATOR: Jeff B
END-OF-SHIFT SUMMARY: Lockloss; 5.5Mag - Japanese bounce mode. First attempt at re-locking made it to DARM_WFS. Second attempt required manual DIFF VCO adjustment. Stopped at CARM_ON_TR to wait for ASC to converge - ETMX Saturations. Third attempt: All three times got good DRMI lock. I’m gonna let this one cook for a bit. EQ band still not really below .1µm/s yet. Dropped slightly below .1µm so proceeded….no luck. Small aftershock is coming. SUCCESS! Handing off to Jeff.
ACTIVITY LOG:
03:15 Tour group into the control room
05:00 Playing with cdsutils audio just to hear some sounds I got some Verbal Alarms:
TRANS Injection running (Nov 20 05:05:00 UTC)
TRANS Injection inactive (Nov 20 05:05:25 UTC)
TRANS Injection running (Nov 20 05:08:00 UTC)
TRANS Injection inactive (Nov 20 05:08:24 UTC)
Hmmmm…
06:10 Lockloss - Earthquake; EQ bands rise to 1µm/s. No significant movement in µSeism
07:44 Jeff B on site. Still no lock and small aftershock is inbound.
08:04 Intention Bit set to Undisturbed
08:04UTC!
06:10UTC
| ifo | P-phase Arrival Time | S-phase Arrival Time | R-wave Arrival Time | R-Wave Velocity (micro m/s) | EQ Distance (km) | GPS P-phase Arrival Time | GPS S-phase Arrival Time | GPS R-wave Arrival Time |
|---|---|---|---|---|---|---|---|---|
| H1 | 21:43:39 PST | 21:52:58 PST | 22:12:12 PST | 0.979172 | 8475.152 | 1132033436.5 | 1132033995.0 | 1132035149.8 |
| L1 | 23:45:52 CST | 23:53:45 CST | 00:26:46 CST | 0.605742 | 11533.17 | 1132033569.5 | 1132034042.5 | 1132036023.5 |
| G1 | 06:44:57 CET | 06:53:23 CET | 07:20:16 CET | 0.741625 | 10168.32 | 1132033514.3 | 1132034020.2 | 1132035633.5 |
| V1 | 06:45:25 CET | 06:53:34 CET | 07:23:32 CET | 0.668707 | 10852.45 | 1132033542.3 | 1132034031.2 |
1132035829.0 |
Guardian is notifying me that AS beam shutter isn't closed.Is it supposed to be?
MID-SHIFT SUMMARY: IFO has been locked and observing for 4hrs40min @ 80.7Mpc. Winds are calm. EQ bands haven’t changed. µSei is just touching .5µm/s. There was a tour groub of Cub Scouts in the control room at 03:15UTC. Other than a couple of big ETMY saturations, everything has been running smoothly.
This entry is to document a wandering line that is sometimes present between ~600hz to ~670hz in H1. This line is annoying for some burst searches (I know it shows up in the GRB-triggered X-pipeline search, and I understand it may show up in other burst searches as well). For two-detector searches, triggers caused by this line are generally caught by coherence vetos. Also, the folks running the pipeline are able to manually cut triggers from these frequencies in post-processing. So, while this is not extremely urgent, my hope is that someone will see this and have an idea of how to easily get rid of this line.
The easiest place to see this line is in the FScan plots on the detchar summary page. For example, the following is an FScan spectogram from Nov 10, you can see this line trending from 670 Hz down to 655 Hz over the course of ~14 hours (the x-axes on these plots are quite annoying): https://ldas-jobs.ligo-wa.caltech.edu/~pulsar/fscan/H1_DUAL_ARM/H1_DUAL_ARM_HANN/H1_DUAL_ARM_HANN/fscans_2015_11_10_06_00_01_PST_Tue/H1_GDS-CALIB_STRAIN/spec_620.00_720.00_H1_1131112818_1131199218.png
The 2nd harmonic shows up as well: https://ldas-jobs.ligo-wa.caltech.edu/~pulsar/fscan/H1_DUAL_ARM/H1_DUAL_ARM_HANN/H1_DUAL_ARM_HANN/fscans_2015_11_10_06_00_01_PST_Tue/H1_GDS-CALIB_STRAIN/spec_1320.00_1420.00_H1_1131112818_1131199218.png
Usually it varies more erratically: https://ldas-jobs.ligo-wa.caltech.edu/~pulsar/fscan/H1_DUAL_ARM/H1_DUAL_ARM_HANN/H1_DUAL_ARM_HANN/fscans_2015_11_08_06_00_01_PST_Sun/H1_GDS-CALIB_STRAIN/spec_620.00_720.00_H1_1130940019_1131026418.png
Sometimes, it goes away entirely: https://ldas-jobs.ligo-wa.caltech.edu/~pulsar/fscan/H1_DUAL_ARM/H1_DUAL_ARM_HANN/H1_DUAL_ARM_HANN/fscans_2015_11_16_06_00_02_PST_Mon/H1_GDS-CALIB_STRAIN/spec_620.00_720.00_H1_1131631219_1131717619.png
This line shows up in the other ifo degrees of freedom (perhaps unsurprisingly):
I had an idea to check some PSL vibration and PWR channels. There was a moment when I thought the PSL microphone was seeing the line (compare these times: STRAIN vs MIC), but I think it was a coincidence as the line in MIC stays constant even as the wandering line wanders. Nothing in the following channels:
As a follow-up to Hartmut's question about the DARM rms, here are spectra of the in- and out-of-loop DARM sensors before and after transitioning to DC readout (at 2 W input power). The spectra are calibrated into RIN.
The left-hand plots show the sensors when DARM is controlled via AS45Q. Above 10 Hz or so, we see the sensing noise of AS45Q impressed onto the DARM loop. From 2 Hz to 8 Hz or so, and below 0.5 Hz, both sensors appear to see some DARM displacement noise. Between 0.5 Hz and 2 Hz, the sensors are not seeing the same signal. If the AS45Q signal in this region is indeed DARM displacement noise (rather than some kind of loop-suppressed sensing noise), this would seem to indicate that the sensing noise of the DC readout chain is higher than the intrinsic DARM displacement noise by a factor of a few in this region.
The right-hand plots show the sensors when DARM is controlled via DCPD sum. During the transition, an additional boost is engaged (two complex poles at 2 Hz, two complex zeros at 8 Hz), so this makes it tricky to directly compare these plots with the left-hand plots. Probably it is best to try to get an equivalent set of plots for the intermediate case (DARM controlled by AS45Q, boost engaged).
Attached is the same plot as before, but this time both configurations have the same OLTF (boost on).
Times:
(Borja, Vinny Roma)
This is a continuation of the work started yesterday here. Today, during maintenance, we worked all morning on the hunting of the 60Hz glitch noise and we can now confirm that the issue was identified and solved.
At 2015-11-17 17:10 (UTC) we arrived at the EndY station. We noticed an aircon unit outside of the building (although different model to that reported at Livingston) also used for cooling old clean rooms and no longer in use. We were sure that it was not running at the times that we observed the 60Hz bursts. We also noticed a fridge ON as we came in...more on this later.
We carried portable magnetometers similar to the ones used at the sites but plugged into oscilloscopes for portability. The area we concentrated most of our noise hunting was the electronics bay (EBAY) as from previous measurements we noticed that the bursts were stronger at the magnetometers located there (MAG_SUSRACK and MAG_SEISRACK) in comparison with MAG_VEA (see attached figure 'Comparison_MAGs_QUAD_SUM.png'). Looking at the spikes in more detail (see 'Zoom-spikes_Mag_VEA_and_EBAYs.png') we observe that while the spikes in MAG_VEA has a frequency of 60Hz, the spikes on MAG_EBAY_SUS and MAG__EBAY_SEI has double the frequency. This seems to be cause to a non-linear response of the transducer to the magnetic field, stronger in MAG_SEI than in MAG_SUS as both are identical sensors we assume the magnetic field from the spike is stronger at the SEI magnetometer.
Another clue that pointed to EBAY as the area of interest is the coherent plot attached of the MIC_EBAY and MIC_VEA_MINUSY with MAG_EBAY_SEI, we can clearly see correlations at 60Hz and harmonics being always stronger at MIC_EBAY. Notice however that we were never able to hear the burst so we assume the microphones pick them up electromagnetically.
In order to confirm that the bursts were actually real signals (instead of rack related issues) we swapped the axes of both magnetometers on EBAY as we observed they had different signal strenght. The change in the observed signal strength after the swap was compatible with the axes changes. Notice that we undid these changes after the morning work, so now is all back to normal.
Then we moved the portable magnetometer around the EBAY racks and noticed no strong magnetic noise anywhere with the exception of the 'PEM Endevco Power supply' which powers the accelerometers. The magnetic field around this box was very strong and MAG_EBAY_SEI is not far away from it. We also noticed that this was the only device connected to the wall AC power supply (see attached pictures) and this is also the case anywhere this PEM power supply is used.
We attach a time plot of EY_MAG_EBAY_SEI during the whole morning working period and we can see several things:
1) The time interval between bursts is much shorter and less regular than before (this was also observed previously when work was done at the End station). Compare attached plots from yesterday night ('Latest-60Hz_Bursts', very regular 85minutes separation between spikes) and today ('Morning_60Hz_timeplot_MAG', totally irregular with as short separation as 3 minutes).
2) The burst structure is different than the one previously related to 60Hz glitch noise (see here). For instance see the red circled area. During this time the vacuum cleaner was on near EBAY.
At this point we realized human activity with electric devices plugged to the wall at the station was involved with the generation of 60 Hz bursts although with a different signature to the bursts we knew and came to hunt.
Suddently for almost an hour (between hours 1.7 and 2.5 in plot 'Morning_60Hz_timeplot_MAG') we saw nothing. Then the burst became more spaced so after a while we tried to reproduce the vacuum cleaner burst signature by switching it on. The vacum cleaner was in the same room as the fridge and we noticed that the fridge was now turned OFF (we later learned that John and Bubba turned it OFF).
Then everything started to make sense...the fridge compresor only needs to be on when the temperature inside the fridge drops below a threshold which it can happen every 1.5 to 2 hours or longer depending on the environment temperature and quality of the fridge insulation. Notice that the interval between burst was shorter in summer than current months. Then the compresor is usually on for a few tens of minutes until the temperature is winthing desired range and then the compresor turns off. So in order to confirm the fridge as the cause of our 60Hz burst and glitches we tested turning it ON and we saw a burst (circled green on the previous plot at hour 3.5). And as we turned it OFF then the 60Hz burst dissapeared.
It appears that the fridge was ON the whole O1, this will no longer happen. But notice that any device drawing current from the mains seem to generate 60Hz bursts at least picked up by the magnetometers in EBAY, so soon we thought that maybe this is relared with the only device in that room that is plugged to the mains and that has a considerable Magnetic contamination...the 'PEM Endevco Power supply'.
So after lunch we went back to EndY station (arriving at UTC 23:07:00) with the intention of checking if unplugging the PEM Power Supply from the wall would be enough for the EBAY magnetometers not seeing the current draw by the fridge as this was turned on and off on 1 minute intervals for 3 times. For comparison we did the same test before with the Power supply still plugged and turned on. Unfotunatelly we see no difference between these two cases on MAG_EBAY_SEI as per attached plot 'Checking_PEM_Power_Supply_Coupling.png' magenta circle is with PEM Power Supply ON and brown is with the Power supply OFF. Interestingly however we can see a small spike at about UTC (23:34:00) when the turned off the Power supply and at 23:55:00 when we turned it back on.
Notice the spikes at the beginning correspond to our arrival to End station probably due to switching ON the shoe cleaner at the entrance and the desktop computer at EBAY.
As a follow up from yesterday entry. I attach a time plot of MAG_EBAY_SEIRACK at EndY for the last 19 hours after yesterday fix of the 60Hz bursts. We can see that the regular 60Hz burst are no longer happening. The only spike in that 19 hours period took place at UTC 2015-11-18 16:41:30 which is 8:41:30 am local time which agrees perfectly with the time at which several people went into the building to look at some ESD tripping related issues. Therefore as expected the spike is related to current draw at the building due to human activity.
A final follow up on the FIX of the 60Hz glitches.
Now that LHO has been looked for quite some time I decided to compare Omicron triggers spectrograms before and after the fix. The evidence is clear that the regular 60Hz gliches are now gone.
