Mike suggested I should try again when the wind dies down to 0 mph. Wind speed is now ~10 mph ar EY. Without this kind of useism the ifo should have been locked without a problem.
The wind has died below 20mph but useism is getting higher and higher. ALS has become more stable that the wind is low but I still couldn't get pass DRMI locking.
I just had an uncessful PRMI to DRMI transition. ISC_DRMI Guardian stalled at DRMI_LOCK_WAIT for a really long time. I just went ahead and INIT the ISC_LOCK Guardian and going to DRMI from there.
Wind has already died down below 20 mph. I don't think this is wind problem anymore.......... Redoing initial alignment.
Just for fun, since the microseism is very high for LHO, I looked at the ocean wave activity near the Pacific Northwest coast. The map is a nice bright red, close to the coast, meaning 12+ m wave height.
RichM suggested we get the 45mHz Blends running on the Z axis for the ETMs.
After checking that our current version is in SVN, I saved a backup copy and then copied the blends from the X or Y dof with foton. Attached are Trends with the transition from Quite_90s to the 45mHz Z blends fairly obvious. Also see that there is pretty much no effect in RZ from the switch as evidenced on the optical lever. If our Z to Rz subtraction was not too good, the lower blends might reinject tilt and show up on the oplev or some other IFO signal. So far so good.
It probably should happen on the CS platforms as well.
I committed the ETM filter txt files to the SVN revision 12264.
I stopped the bake this morning -> The pump is still noticeably warm and will be for hours but I went ahead and removed the turbo hardware and "fired it up" this afternoon -> It is now valved-in to the BT as is the X2-8 ion pump but both of their controllers still need to be mounted in the end stations and their HV cables will need to be cut-to-length as both have a few hundred feet excess length!
Ed, Sheila
Today we are having useism above 1um/sec, almost 2um/sec in the z direction, and wind speeds mostly between 15-35 mph with a few gusts up to 40 mph.
This morning Ed collected many examples of ALS locklosses, and together we looked at several of them.
We found at least 3 different mechanisms-
X arm VCO runs out of range ( H1:ALS-X_REFL_CTRL_OUT_DQ, which is calibrated in um, reaches around 6 um) An example is shown in the first attached screenshot. We found 4 example of this 19:05:22 UTC, 19:07:25, 19:44:09 and 19:48:12)
DIFF tried to lock when the PLL is moving through its linear range rather quickly, ETMX L3 saturates before the offloading to L1 can relieve it. (2nd screenshot, 19:01:12 and 20:30)
Lockloss while CM board gain is ramping up during COMM locking, large glitches from CM board saturate MC2 M2. Only one example, 19:45:22
We have done a few things to try to mitigate the situation. The most important probably was increasing the X arm PDH offloading to the suspension, H1:LSC-X_ARM_CTRL_UGF from 0.02 to 0.08. This matches the ugf used for the Y arm, which has not been dropping lock as often as the X arm.
I also reduced the wait time that the COMM guardian uses to check if the PLL is locked from 5 seconds to 2, this means less waiting for it to lock. Lastly I changed the step in the ramp of the CM board gain durring the engaging of ALS COMM from 1dB steps to 3dB steps.
I've reverted the gain stepping back to 1dB since it didn't seem like an improvement after several locking attempts.
TITLE: Dec 10 DAY Shift 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC
STATE Of H1: Observing
SUPPORT: Sheila, Hugh,
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY:
A combination of very high microseism (≥1µm/s) and high winds, for most of the day, made it less than possible for re-locking.
Sheila did a lot of work on COMM gain stepping during less than desirable conditions
Hugh changed a Z axis Stage 1 blend at EX on Rich Mittleman’s advice
Looks like an IA is in order for the next shift.
ACTIVITY LOG:
16:43 Out of Observing while Livingston investigates noise. Robert is using the time to do injections.
17:09 Floor covering folks are on site. (OSB carpeting).
17:15 another carpet installer on site
17:27 Robert is going to wait for the carpet installers to unload pallets behind the corner station with the forklift before starting his injections.
17:31 Hugh out to LVEA/HAM1 area to investigate some audible HEPI noise that Robert reported.
17:35 Lockloss
18:40 Kyle out to Y-2-8
19:48 Kyle back from Y-2-8
21:47 Chris driving to MX to work in a fan room.
22:48 Kyle out of EY mech room and going to Y-2-8
23:07 kyle driving back to corner to get keys.
23:23 Kyle driving back down to Y-2-8.
The DCS C01 hoft frames are now available up to 1133178880 == Dec 03 2015 11:54:23 UTC.
See these alogs for further information:
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23966 https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=23263
MID-SHIFT SUMMARY:
After the lock loss that occurred at 17:35UTC by permitted incursions to the LVEA, the wind speeds shot up to the 35mph+ range and have been there since. µSeism is very high ( pushing on 2µm/s). I have been working with Sheila getting more acquainted qith the LockLoss tools. We aren’t missing out on coincidental time with Livingston as they are still investigating their noise /range trouble. EQ bands have even increased from ≈.09 to about .2 µm/s.
16:43UTC. This is with approval from the run coordinator and in sync with LLO doing a noise investigation that is affecting their range.
TITLE: Dec 10 DAY Shift 08:00-16:00UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Observing
OUTGOING OPERATOR: Patrick
QUICK SUMMARY: Looking good! Locked for 10:53:45. 80Mpc
TITLE: 12/10 [OWL Shift]: 08:00-16:00 UTC (00:00-08:00 PDT), all times posted in UTC STATE Of H1: Observing @ ~ 80 MPc SHIFT SUMMARY: Quiet shift. Winds have remained low. Microseism has trended back up above the 90 percentile. Rode through 5.7 magnitude earthquake in South Sandwich Islands. INCOMING OPERATOR: Ed ACTIVITY LOG: 08:24:59 - 08:25:11 UTC Accidentally out of observing 09:10:08 UTC Out of observing while Kyle fills generator at Y2-8 09:38 UTC Kyle back 09:41:27 UTC Back to observing 5.7 151km E of Bristol Island, South Sandwich Islands 2015-12-10 12:05:15 UTC 10.0 km deep
I have set the pump speed at 40 Hz on the corner station chilled water pump #3, as requested by Robert Schofield.
Following the installation of the PBS on the HWS table to measure the polarization state of the 1064nm light leaking out of the BS_AR surface, I took a quick look at the data from the detectors.
Looking at a 5 hour time series around a couple of lock acquisitions, we see 20-30mV on the S detector vs about 97mV on the P-detector. The X-arm transmission is shown as a guide for when the IFO is locked. Vertical axis is the voltage from the detectors.
Next, if we normalize the time series by the median values of each series, we can see much more noise on the S-detector (some of which is caused by the higher relative electronics noise) but significantly higher low-frequency variations during lock - for example, the amount of S-polarization drops by 30% in the second lock-stretch.
Lastly, the time just around the second lock-loss is shown in detail. The S-polarization oscillation is almost exactly out of phase with the oscillation in the arm and in the P-polarization (which is a strong indication for polarization rotation in the interferometer beam). I need to do some further analysis, but the fact that the SUM of the S+P still shows a strong oscillation indicatest that this is not polarization rotation in the HWS optics.
09:10 UTC Kyle is driving to Y2-8 to add fuel to the generator. I'm not sure if we have been taking it out of observing for this, so I am playing it safe.
09:38 UTC Kyle back 09:41 UTC Back to observing
For future reference Mike L. told me that taking it out of observing was not necessary.
The mystery ~650Hz noise reported here and here also shows up in the PEM rf antenna (in both 9MHz and 45MHz located in CER and LVEA). Further investigation revealed that this peak shows up in the PEM antenna during lock aquisition at the start of the DC Readout Transition step (if it appears at all; it's not present in every lock--more on this later). At the start of this step the ALS COMM VCO is parked at some value.
To determine whether this VCO could be responsible for the ~650Hz noise, the frequency readback of the VCO was compared to the frequency of the mystery peak in the PEM antenna. Attached (figure 1) is a plot of H1:ALS-C-COMM_VCO_FREQUENCY timeseries on top and spectrogram of the PEM 45MHz LVEA antenna on the bottom. The frequency of the peak seems to track with the VCO frequency if you take into account the fact that the VCO frequency readback is digitized into steps of 8Hz (does anyone know why / can we fix this?).
Also, there appears to be 2 different values where the VCO can be parked. Figure 2 has similar plots to figure 1, over a 28hr stretch which contained multiple locks where the peak was sometimes present. In locks where the peak was present, the VCO was set to ~78.7873MHz. For locks where the peak is not there the VCO was set to ~78.7944MHz. These values correspond with two different values of H1:ALS-C_COMM_VCO_TUNEOFS : ~-1.39 and ~1.25, respectively.
To test this, we tried moving the COMM VCO TUNE OFS slider with the IFO locked (before continuing to NLN / Observing). While initially it looked like the peak in the PEM rf channel moved as the slider was moved, the lock broke before we could conclusively tell. The lockloss occurred right as Sheila was moving the slider. We don't know why this should cause a lockloss, so this is a subject for further investigations (it was windy and ground motion-y at the time so it could have been a coincidence).
Also included (figure 3) is a plot of the VCO frequency (again, 8Hz digitization) and the CER Rack temperature. More data is needed, but it looks like the freqency trends down after the temperature rises.
Finally, there is still the question as to why this is showing up in the 9MHz and 45MHz channels (and, ultimately, DARM). As a first check, I compared 9.100230 MHz and harmonics to 78.7873 MHz and harmonics to see if a beat would show up within 600 Hz. Out to 10 harmonics of the VCO frequency the closest they came to each other was 200 kHz--still a mystery.
Jordan, Robert, Sheila
8Hz is the single precision rounding, so somewhere somebody is casting the number to single. Bekhof code?
VCO frequency is about 80MHz, and 8e7=(1+fractional)*2^26 (fractional is about 0.1921, but that's not important).
Single precision number uses 23 bits for fractional. For any number in [2^26, 2^27) range, the least significant bit is 2^-23 * 2^26 = 8.
EPICS values are fine, so this is a problem of DAQ/NDS/DV.
Channels are double-precision in the front end, but stored as single-precision in the frames. Maybe Jordan was getting this data from the frames/NDS2, rather than live, so that's why there's this quantization error?
Wind has been below 10 mph so I tried again. Lockloss at SWITCH_TO_QPDS. The furthest I've been all evening.