TITLE: 11/20 [DAY Shift]: 16:00-24:00 UTC (08:00-16:00 PDT), all times posted in UTC STATE Of H1: Locked at NLN SHIFT SUMMARY: Out of observing while Jeff B. went to end Y. Lost lock when Keita and Evan removed the RF AM monitor coupler from the EOM driver cable. Switched blends to Quite_90. Had to move TMS X and Y to optimize green. Had to move BS and PRM to lock on DRMI. Lost lock on SWITCH_TO_QPDS. Locked on NLN. SUPPORT: Jenne, Evan, Keita, Jim W. INCOMING OPERATOR: TJ ACTIVITY LOG: 16:12 Jeff B. to end Y to take pictures in receiving area and hallway to VEA, out of observing 16:18 Chris and Joe to X arm beam tube enclosure 16:39 Jeff B. back, back to observing 17:24 Bubba reports that the landscapers are bailing tumble weeds around the highbay 17:32 Landscapers truck driving back toward kitchen 18:20 Gerardo to mid Y to pick up power supply and then to X28 18:23 Karen to mechanical building to get coffee cups 18:36 Fire department through gate, notified Richard 18:38 Kyle to X28 18:38 Fire department to mid X to check on RFAR Fire department back (forgot to note time, before 19:39) 19:54 Joe back, Chris on his way back 20:12 Kyle and Gerardo back 21:32 Evan and Keita into the H1 PSL enclosure to remove the RF AM monitor coupler from the EOM driver cable, out of observing 21:33 Kyle and Gerardo back to X28 21:38 Joe and Chris back to X arm beam tube enclosure 22:12 Evan and Keita done Put ISI blends on Quite_90 Had to move TMS X and TMS Y to increase green power in both arms even though WFS were engaged Went to LOCK_PRMI and misaligned PRM. Aligned the BS on flashes. Realigned PRMI and was able to lock PRMI. Moved mostly PRM to increase power buildup. Requested DRMI_LOCKED. Transition from LOCK_PRMI to DRMI_LOCKED failed. Eventually DRMI locked. 23:13 Joe and Chris back 23:38 Kyle and Gerardo back
Observing @ 00:19 UTC
TITLE: "11/20 EVE Shift: 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC"
STATE Of H1: Nominal Low Noise, need to go through checks before Observing.
OUTGOING OPERATOR: Patrick
QUICK SUMMARY: Wind low, seismic low, blends on 90s
Because the microseism is down to a level similar to what we saw over the summer, I asked Patrick to put the ISI's in the more wind tolerant 90 mhz blends. Wind is also low right now, but it can come up suddenly, while microseism takes days to rise. We also need more data about what conditions necessitate switching. After, Evan and Keita finished in the PSL, Patrick started relocking but was having troubles with the green. Fiddling the TMS alignments fixed it, I looked to see if the ISI positions had changed. They haven't moved but I found something interesting.
Most of the ISIs all showed a relatively smooth switch from the 45 to 90 blends except for ETMY. ITMY is the first plot and is pretty consistent with all the other chambers, except ETMY. Patrick switched the X&Y blends at the same time , in the middle of the time span I grabbed in the plot. You can kind of tell because the ITMY location mon range gets a little smaller and there are fewer low frequency swings on X&Y, due to the lower gain peaking at higher frequency of the 90 blend. The other DOFs don't seem to see anything.
ETMY however shows a huge 30 micron shift in the Y direction (second plot), and is visible in all DOFs. This chamber was only running the 45mhz blend in the Y DOF, so only the Y blend got switched on this chamber, at about 21:48 UTC today. No idea why it should be any different from the other chambers, although BrianL did note a while ago that the ETMY STS is poorly centered (maybe relevant because the STS is used for sensor correction, so gets summed with the CPS signal before the blend). It would be good to get some time to see if this is repeatable and truly limited to ETMY or not.
For some discussion about how to fix this, see the SEI log, entry 887. https://alog.ligo-la.caltech.edu/SEI/index.php?callRep=887 short answer - 1) SEI team needs to update the blend-switch code to wait longer during the switch process so that the transients can settle down. 2) don't try to change the blend filters while the Tidal-offload is pushing hard on HEPI - wait at least 200 seconds after it finishes. 3) SEI team needs to have less tilt on HEPI when it gets moved by the tidal offload.
The lock for the double doors on the upper section of the staging building is completely disabled. It will be addressed next week.
This report will include this week and last week. We were able to clean the joints, install metal strips and caulk on 360 meters of enclosure.
Kyle, Gerardo Today we tested the functionality of the ion pump mounted at X2-8 using a local controller and cable -> pump works as expected -> We also changed the oil and filter on the diesel generator and finished installing the heat tapes etc. on the ion pump-connecting vacuum hardware -> We will start the 72 hr. bake out of the new ion pump Monday morning. We will need access to BT port X2-8 every 10 hrs. or so between 0900 Monday morning thru Wednesday afternoon to refuel the generator.
Keita, Evan
We went in and removed the couplers between the EOM driver and the EOM (see 23472), as well as the extra phase delay cable on the ISC rack which was being used to match the coupler-induced delay.
Removing the couplers got rid of the extra noise on the 45 MHz OOL readback (see attachment).
Strangely, putting the couplers on the spare driver in the CER (which is terminated with a 50 Ω resistor, rather than an EOM) does not produce this extra noise (see attachment).
We have turned the spare driver back on and reconnected it to the digital system.
Performed a run spanning the time period from 1129593617 (Oct 23 2015 00:00:00 UTC) to 1131828544 (Nov 17 2015 20:48:47 UTC)
The following parameters were used for this run:
Only two injections were found to be scheduled during this period:
However, neither of these injections were found to actually occur within the examined time period.
There were no normal injections that occurred within the examined period. There were a number CAL-INJ resets, all single-IFO with the same curious pattern to the frame flags as denoted in the immediately prior HWInj report (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23485). There was a single-IFO burst injection in H1
BURST 1129673117.000 (H1)
with the only anomaly being that it is UNSCHEDULED. There was a single UNKNOWN injection in L1
UNKNOWN 1129673117.000 (L1)
This injection was confirmed to occur only within the CAL-INJ channel of RAW frames with the TRANSIENT bit indicating an injection but none of the specific injection-type bits, CBC, BURST, DETCHAR, or STOCHASTIC, indicating an injection. It was confirmed to not occur in ODC HOFT, ODC RDS, ODC RAW, or GDS HOFT frames.
It is interesting that the single-IFO injections BURST 1129673117.000 (H1) and UNKNOWN 1129673117.000 (L1) are found to occur at the exact same time but within different IFOs. Even further, the fact the BURST injection in H1 is perfectly normal, other than being UNSCHEDULED, while the UNKNOWN injection in L1, in addition to being UNSCHEDULED, is inconsistent. HWInjReport currently does not check the HW bit or the CW bit in the CAL-INJ channel for RAW frames for the presence of injections; however, upon examining these bits directly, it was found that while the HW bit indicated an injection, along with the TRANSIENT bit, the CW bit did not indicate an injection (my first thought was that this may be a CW injection only occurring in L1). This means this was a truly UNKNOWN, UNSCHEDULED injection occurring only with L1. Given both injections occur at the exact same time and the L1 injection is a truly UNKNOWN type (practically a ghost injection, by the bits), it is likely this was intended to be a normal network injection; however either user-error or a catastrophic software error corrupted the proper setting of the bits.
There was an additional UNKNOWN injection that occurred at 1131397646 in both H1 and L1 (as noted by Peter Shawhan) with a signature similar to the one reported here. However, for some reason HWInjReport missed it. I will have to investigate why HWInjReport missed that particular injection, but, unlike the last session of bug-fixing, I am not going to cease production of reports. It is my estimation that HWInjReport is currently operating with reasonable confidence to find and report most true anomalies. If reporting errors or omissions are found, then those errors should be correctable and a new report on the relevant time period can be generated.
I didn't see any follow-up on this but I did some checks: The injection at 1129673117 is the stochastic hardware injection that Joe and Chris did. This was out of science mode. The injection at 1131397646 should be the injection in the schedule file from line: 1131397639 1 1.0 coherentbbh0_1126259455_ This CBC injection was not logged properly for the searches because tinj was setting CAL-INJ_TINJ_TYPE instead of CAL-PINJX_TINJ_TYPE. So I would guess that was the problem that you're seeing here. The issues have since been fixed though.
Now that I finally have my diagnostic tool working (it's a ripped-down version of HWInjReport called HWInjCheck that simply returns raw output from FrBitmaskTransitions to allow rapid verification of anomalies found in HWInjReport), I re-examined the missing UNKNOWN injection at 1131397646. Firstly, HWInjReport can only identify injections as UNKNOWN if they occur in the CAL-INJ. This is because, as far as I can determine, only the CAL-INJ channel has an independent excitation bit that indicates whether a transient injection has occurred, regardless of type (as well as an independent HW bit indicating the occurrence of an injection of any type). This does not seem to be true for ODC-MASTER and GDS-CALIB channels. So, in CAL-INJ, if the TRANSIENT bit is "off" (indicating the presence of a transient injection) but the CBC, BURST, DETCHAR, and STOCHASTIC bits, which indicate the type of injection, are "on", then it is clear we have an UNKNOWN type transient injection occurring. However, for ODC-MASTER and GDS-CALIB, if the type bits are not "off", there is nothing to indicate excitation of a transient injection with an unspecified type. If 1131397646 occurs with an unspecified type in the ODC-MASTER or GDS-CALIB channel but not in the CAL-INJ channel, then HWInjReport will completely miss the injection as it won't show up at all in any of the bits and channels that HWInjReport checks.
Using HWInjCheck to check the output from FrBitmaskTransitions, I have not, yet, found any indication of the occurrence of 1131397646, that is, there is no indication of excitation in the excitation bits. I will check again more carefully, but, at this point, it does seem HWInjReport is working properly; this is an expected missing injection. To be clear, there may still have actually been an excitation, there just doesn't seem to be any record of it in the bits and channels that HWInjReport checks.
SudarshanK, RickS
Evan and Keita are done. Attempting to relock. ISI blends are on Quite_90 per Jim's request.
While trying to understand and deal with the Beckhoff SDF diffs in OBSERVE mode, I stumbled on a weird channel that toggled states around 4pm local last Wed Nov 11 (01 UTC WED 11th). (Note, this occured well before the windy troublesome locking period when we saw a power glitch on Tuesd Wed 17th.) If the VCO Frequency Servo is set to "Int" (see the center of the medm pic attached below), do we care what this EXTFREQUENCYOFFSET channel is doing? And where is the medm for this EXTFREQUENCYOFFSET channel anyways? - Even with commissioners, we couldn't immediately find it... And, and, what's with the glitch in the middle of the 20 day trend on Wed 11th?
Not sure but it might be related to this entry. https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23293 Perhaps the wrong button was pushed? Or if there were problems encountered re-locking the input modecleaner after installation of the 45 MHz RFAM hardware, this may have been flipped to see if the input modecleaner would lock.
Attached are the updated charge trends with this weeks newest data. So far so good...
Evan and Keita are going into the H1 PSL enclosure to remove the RF AM monitor coupler from the EOM driver cable. This will take us out of lock.
The strip charts were stopping and starting periodically as evidenced by flat lines in the signal plots. Restarting the computer seems to have fixed it for now.
Out of observing briefly while Jeff B. went to end Y. Otherwise no issues.
O1 days 62,63
model restarts logged for Thu 19/Nov/2015 No restarts reported
model restarts logged for Wed 18/Nov/2015 No restarts reported
Previously I had noted a slow oscillation in the output power of the front end laser. The plot AmplifierPower.png partly shows the oscillation but unfortunately the fluctuation seen is the one dominated by the humidity change in the Laser Room. However the fluctuations are more obvious in the pump diode power plot (DiodePower.png). The question is then what is driving these fluctuations? DiodeTEC.png shows the voltage applied to the Peltier cell to cool the pump diodes. The fluctuation is clearly seen here. It is also evident in the pump diode voltage (DiodeVoltage.png) and current (DiodeCurrent.png). It is possible that we have a small oscillation in the output of the front end laser's two power supplies. Or as Richard pointed out to me, it's possible that the two power supplies are beating against each other. The oscillation is not evident in the pre-modecleaner reflected light signal, and so is probably just a curiosity at the moment.
For information's sake, the switching frequency of the pump diode power supplies is over 200 kHz. The other power supply in the diode box has a switching frequency of 50 kHz.
As a follow-up to Hartmut's question about the DARM rms, here are spectra of the in- and out-of-loop DARM sensors before and after transitioning to DC readout (at 2 W input power). The spectra are calibrated into RIN.
The left-hand plots show the sensors when DARM is controlled via AS45Q. Above 10 Hz or so, we see the sensing noise of AS45Q impressed onto the DARM loop. From 2 Hz to 8 Hz or so, and below 0.5 Hz, both sensors appear to see some DARM displacement noise. Between 0.5 Hz and 2 Hz, the sensors are not seeing the same signal. If the AS45Q signal in this region is indeed DARM displacement noise (rather than some kind of loop-suppressed sensing noise), this would seem to indicate that the sensing noise of the DC readout chain is higher than the intrinsic DARM displacement noise by a factor of a few in this region.
The right-hand plots show the sensors when DARM is controlled via DCPD sum. During the transition, an additional boost is engaged (two complex poles at 2 Hz, two complex zeros at 8 Hz), so this makes it tricky to directly compare these plots with the left-hand plots. Probably it is best to try to get an equivalent set of plots for the intermediate case (DARM controlled by AS45Q, boost engaged).
Attached is the same plot as before, but this time both configurations have the same OLTF (boost on).
Times:
(Borja, Vinny Roma)
This is a continuation of the work started yesterday here. Today, during maintenance, we worked all morning on the hunting of the 60Hz glitch noise and we can now confirm that the issue was identified and solved.
At 2015-11-17 17:10 (UTC) we arrived at the EndY station. We noticed an aircon unit outside of the building (although different model to that reported at Livingston) also used for cooling old clean rooms and no longer in use. We were sure that it was not running at the times that we observed the 60Hz bursts. We also noticed a fridge ON as we came in...more on this later.
We carried portable magnetometers similar to the ones used at the sites but plugged into oscilloscopes for portability. The area we concentrated most of our noise hunting was the electronics bay (EBAY) as from previous measurements we noticed that the bursts were stronger at the magnetometers located there (MAG_SUSRACK and MAG_SEISRACK) in comparison with MAG_VEA (see attached figure 'Comparison_MAGs_QUAD_SUM.png'). Looking at the spikes in more detail (see 'Zoom-spikes_Mag_VEA_and_EBAYs.png') we observe that while the spikes in MAG_VEA has a frequency of 60Hz, the spikes on MAG_EBAY_SUS and MAG__EBAY_SEI has double the frequency. This seems to be cause to a non-linear response of the transducer to the magnetic field, stronger in MAG_SEI than in MAG_SUS as both are identical sensors we assume the magnetic field from the spike is stronger at the SEI magnetometer.
Another clue that pointed to EBAY as the area of interest is the coherent plot attached of the MIC_EBAY and MIC_VEA_MINUSY with MAG_EBAY_SEI, we can clearly see correlations at 60Hz and harmonics being always stronger at MIC_EBAY. Notice however that we were never able to hear the burst so we assume the microphones pick them up electromagnetically.
In order to confirm that the bursts were actually real signals (instead of rack related issues) we swapped the axes of both magnetometers on EBAY as we observed they had different signal strenght. The change in the observed signal strength after the swap was compatible with the axes changes. Notice that we undid these changes after the morning work, so now is all back to normal.
Then we moved the portable magnetometer around the EBAY racks and noticed no strong magnetic noise anywhere with the exception of the 'PEM Endevco Power supply' which powers the accelerometers. The magnetic field around this box was very strong and MAG_EBAY_SEI is not far away from it. We also noticed that this was the only device connected to the wall AC power supply (see attached pictures) and this is also the case anywhere this PEM power supply is used.
We attach a time plot of EY_MAG_EBAY_SEI during the whole morning working period and we can see several things:
1) The time interval between bursts is much shorter and less regular than before (this was also observed previously when work was done at the End station). Compare attached plots from yesterday night ('Latest-60Hz_Bursts', very regular 85minutes separation between spikes) and today ('Morning_60Hz_timeplot_MAG', totally irregular with as short separation as 3 minutes).
2) The burst structure is different than the one previously related to 60Hz glitch noise (see here). For instance see the red circled area. During this time the vacuum cleaner was on near EBAY.
At this point we realized human activity with electric devices plugged to the wall at the station was involved with the generation of 60 Hz bursts although with a different signature to the bursts we knew and came to hunt.
Suddently for almost an hour (between hours 1.7 and 2.5 in plot 'Morning_60Hz_timeplot_MAG') we saw nothing. Then the burst became more spaced so after a while we tried to reproduce the vacuum cleaner burst signature by switching it on. The vacum cleaner was in the same room as the fridge and we noticed that the fridge was now turned OFF (we later learned that John and Bubba turned it OFF).
Then everything started to make sense...the fridge compresor only needs to be on when the temperature inside the fridge drops below a threshold which it can happen every 1.5 to 2 hours or longer depending on the environment temperature and quality of the fridge insulation. Notice that the interval between burst was shorter in summer than current months. Then the compresor is usually on for a few tens of minutes until the temperature is winthing desired range and then the compresor turns off. So in order to confirm the fridge as the cause of our 60Hz burst and glitches we tested turning it ON and we saw a burst (circled green on the previous plot at hour 3.5). And as we turned it OFF then the 60Hz burst dissapeared.
It appears that the fridge was ON the whole O1, this will no longer happen. But notice that any device drawing current from the mains seem to generate 60Hz bursts at least picked up by the magnetometers in EBAY, so soon we thought that maybe this is relared with the only device in that room that is plugged to the mains and that has a considerable Magnetic contamination...the 'PEM Endevco Power supply'.
So after lunch we went back to EndY station (arriving at UTC 23:07:00) with the intention of checking if unplugging the PEM Power Supply from the wall would be enough for the EBAY magnetometers not seeing the current draw by the fridge as this was turned on and off on 1 minute intervals for 3 times. For comparison we did the same test before with the Power supply still plugged and turned on. Unfotunatelly we see no difference between these two cases on MAG_EBAY_SEI as per attached plot 'Checking_PEM_Power_Supply_Coupling.png' magenta circle is with PEM Power Supply ON and brown is with the Power supply OFF. Interestingly however we can see a small spike at about UTC (23:34:00) when the turned off the Power supply and at 23:55:00 when we turned it back on.
Notice the spikes at the beginning correspond to our arrival to End station probably due to switching ON the shoe cleaner at the entrance and the desktop computer at EBAY.
As a follow up from yesterday entry. I attach a time plot of MAG_EBAY_SEIRACK at EndY for the last 19 hours after yesterday fix of the 60Hz bursts. We can see that the regular 60Hz burst are no longer happening. The only spike in that 19 hours period took place at UTC 2015-11-18 16:41:30 which is 8:41:30 am local time which agrees perfectly with the time at which several people went into the building to look at some ESD tripping related issues. Therefore as expected the spike is related to current draw at the building due to human activity.
A final follow up on the FIX of the 60Hz glitches.
Now that LHO has been looked for quite some time I decided to compare Omicron triggers spectrograms before and after the fix. The evidence is clear that the regular 60Hz gliches are now gone.