Kyle, Gerardo New ion pump had shut down with "excess arcing" error shortly after its initial start yesterday -> Restarted this morning and seems normal now -OK
RichM pointed me to look at the RZ drive as a way to confirm that the Z 45mHz blend is not coupling T240 RZ into the platform.
The attached 24 hour trend at ETMX suggests the nice reduction of inertial Z motion on Stage2 GS13s does not come at the expense of increase drive to the RZ of Stage1. Our z-RZ subtraction is doing well. The Blend switch points are shown too.
TITLE: Dec 11 DAY Shift 08:00-16:00UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Environment - µSeism extremely high
OUTGOING OPERATOR: Travis - short shift
QUICK SUMMARY: …..it’s Friday.
Title: 12/11 Owl Shift 8:00-16:00 UTC (0:00-8:00 PST). All times in UTC.
State of H1: Unlocked due to microseism
Shift Summary: After Nutsinee's frustrating shift, I have had similar luck. I made it as far as ENGAGE_ASC_PART3 twice, but while waiting there for the ASC loops to do their deeds, we lost lock. Talked to Mike and he gave the OK to throw in the towel for the night. Microseism is still above 1 um/s and winds are still 5-10 mph.
Incoming operator: Ed
Activity log: Attempting to lock all night.
No luck with locking so far. Made it as far as DRMI_ON_POP once in several attempts. Microseisem continues its upward trend above 1 um/s. Winds have calmed from earlier but still ~10 mph.
TITLE: 12/10 EVE Shift 00:00-08:00UTC (16:00-00:00 PST), all times posted in UTC
STATE Of H1: Lock Acquisition/Environment
SUPPORT: Sheila, Jenne
Incoming Ops: Travis (failed to reach him so he's already on site)
END-OF-SHIFT SUMMARY: I've been trying to lock most of the evening. The shift started off with high wind (20-40 mph) then died down towards the end of shift (now ~10mph and below). Useism has been increasing (now at ~1.5 um/s). Locking was unsuccessful even with wind below 10mph.
Activity:
00:19 Kyle driving back from Y28
Mike suggested I should try again when the wind dies down to 0 mph. Wind speed is now ~10 mph ar EY. Without this kind of useism the ifo should have been locked without a problem.
Wind has been below 10 mph so I tried again. Lockloss at SWITCH_TO_QPDS. The furthest I've been all evening.
The wind has died below 20mph but useism is getting higher and higher. ALS has become more stable that the wind is low but I still couldn't get pass DRMI locking.
I just had an uncessful PRMI to DRMI transition. ISC_DRMI Guardian stalled at DRMI_LOCK_WAIT for a really long time. I just went ahead and INIT the ISC_LOCK Guardian and going to DRMI from there.
Wind has already died down below 20 mph. I don't think this is wind problem anymore.......... Redoing initial alignment.
Just for fun, since the microseism is very high for LHO, I looked at the ocean wave activity near the Pacific Northwest coast. The map is a nice bright red, close to the coast, meaning 12+ m wave height.
RichM suggested we get the 45mHz Blends running on the Z axis for the ETMs.
After checking that our current version is in SVN, I saved a backup copy and then copied the blends from the X or Y dof with foton. Attached are Trends with the transition from Quite_90s to the 45mHz Z blends fairly obvious. Also see that there is pretty much no effect in RZ from the switch as evidenced on the optical lever. If our Z to Rz subtraction was not too good, the lower blends might reinject tilt and show up on the oplev or some other IFO signal. So far so good.
It probably should happen on the CS platforms as well.
I committed the ETM filter txt files to the SVN revision 12264.
I stopped the bake this morning -> The pump is still noticeably warm and will be for hours but I went ahead and removed the turbo hardware and "fired it up" this afternoon -> It is now valved-in to the BT as is the X2-8 ion pump but both of their controllers still need to be mounted in the end stations and their HV cables will need to be cut-to-length as both have a few hundred feet excess length!
Ed, Sheila
Today we are having useism above 1um/sec, almost 2um/sec in the z direction, and wind speeds mostly between 15-35 mph with a few gusts up to 40 mph.
This morning Ed collected many examples of ALS locklosses, and together we looked at several of them.
We found at least 3 different mechanisms-
X arm VCO runs out of range ( H1:ALS-X_REFL_CTRL_OUT_DQ, which is calibrated in um, reaches around 6 um) An example is shown in the first attached screenshot. We found 4 example of this 19:05:22 UTC, 19:07:25, 19:44:09 and 19:48:12)
DIFF tried to lock when the PLL is moving through its linear range rather quickly, ETMX L3 saturates before the offloading to L1 can relieve it. (2nd screenshot, 19:01:12 and 20:30)
Lockloss while CM board gain is ramping up during COMM locking, large glitches from CM board saturate MC2 M2. Only one example, 19:45:22
We have done a few things to try to mitigate the situation. The most important probably was increasing the X arm PDH offloading to the suspension, H1:LSC-X_ARM_CTRL_UGF from 0.02 to 0.08. This matches the ugf used for the Y arm, which has not been dropping lock as often as the X arm.
I also reduced the wait time that the COMM guardian uses to check if the PLL is locked from 5 seconds to 2, this means less waiting for it to lock. Lastly I changed the step in the ramp of the CM board gain durring the engaging of ALS COMM from 1dB steps to 3dB steps.
I've reverted the gain stepping back to 1dB since it didn't seem like an improvement after several locking attempts.
TITLE: Dec 10 DAY Shift 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC
STATE Of H1: Observing
SUPPORT: Sheila, Hugh,
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY:
A combination of very high microseism (≥1µm/s) and high winds, for most of the day, made it less than possible for re-locking.
Sheila did a lot of work on COMM gain stepping during less than desirable conditions
Hugh changed a Z axis Stage 1 blend at EX on Rich Mittleman’s advice
Looks like an IA is in order for the next shift.
ACTIVITY LOG:
16:43 Out of Observing while Livingston investigates noise. Robert is using the time to do injections.
17:09 Floor covering folks are on site. (OSB carpeting).
17:15 another carpet installer on site
17:27 Robert is going to wait for the carpet installers to unload pallets behind the corner station with the forklift before starting his injections.
17:31 Hugh out to LVEA/HAM1 area to investigate some audible HEPI noise that Robert reported.
17:35 Lockloss
18:40 Kyle out to Y-2-8
19:48 Kyle back from Y-2-8
21:47 Chris driving to MX to work in a fan room.
22:48 Kyle out of EY mech room and going to Y-2-8
23:07 kyle driving back to corner to get keys.
23:23 Kyle driving back down to Y-2-8.
The DCS C01 hoft frames are now available up to 1133178880 == Dec 03 2015 11:54:23 UTC.
See these alogs for further information:
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23966 https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=23263
MID-SHIFT SUMMARY:
After the lock loss that occurred at 17:35UTC by permitted incursions to the LVEA, the wind speeds shot up to the 35mph+ range and have been there since. µSeism is very high ( pushing on 2µm/s). I have been working with Sheila getting more acquainted qith the LockLoss tools. We aren’t missing out on coincidental time with Livingston as they are still investigating their noise /range trouble. EQ bands have even increased from ≈.09 to about .2 µm/s.
The mystery ~650Hz noise reported here and here also shows up in the PEM rf antenna (in both 9MHz and 45MHz located in CER and LVEA). Further investigation revealed that this peak shows up in the PEM antenna during lock aquisition at the start of the DC Readout Transition step (if it appears at all; it's not present in every lock--more on this later). At the start of this step the ALS COMM VCO is parked at some value.
To determine whether this VCO could be responsible for the ~650Hz noise, the frequency readback of the VCO was compared to the frequency of the mystery peak in the PEM antenna. Attached (figure 1) is a plot of H1:ALS-C-COMM_VCO_FREQUENCY timeseries on top and spectrogram of the PEM 45MHz LVEA antenna on the bottom. The frequency of the peak seems to track with the VCO frequency if you take into account the fact that the VCO frequency readback is digitized into steps of 8Hz (does anyone know why / can we fix this?).
Also, there appears to be 2 different values where the VCO can be parked. Figure 2 has similar plots to figure 1, over a 28hr stretch which contained multiple locks where the peak was sometimes present. In locks where the peak was present, the VCO was set to ~78.7873MHz. For locks where the peak is not there the VCO was set to ~78.7944MHz. These values correspond with two different values of H1:ALS-C_COMM_VCO_TUNEOFS : ~-1.39 and ~1.25, respectively.
To test this, we tried moving the COMM VCO TUNE OFS slider with the IFO locked (before continuing to NLN / Observing). While initially it looked like the peak in the PEM rf channel moved as the slider was moved, the lock broke before we could conclusively tell. The lockloss occurred right as Sheila was moving the slider. We don't know why this should cause a lockloss, so this is a subject for further investigations (it was windy and ground motion-y at the time so it could have been a coincidence).
Also included (figure 3) is a plot of the VCO frequency (again, 8Hz digitization) and the CER Rack temperature. More data is needed, but it looks like the freqency trends down after the temperature rises.
Finally, there is still the question as to why this is showing up in the 9MHz and 45MHz channels (and, ultimately, DARM). As a first check, I compared 9.100230 MHz and harmonics to 78.7873 MHz and harmonics to see if a beat would show up within 600 Hz. Out to 10 harmonics of the VCO frequency the closest they came to each other was 200 kHz--still a mystery.
Jordan, Robert, Sheila
8Hz is the single precision rounding, so somewhere somebody is casting the number to single. Bekhof code?
VCO frequency is about 80MHz, and 8e7=(1+fractional)*2^26 (fractional is about 0.1921, but that's not important).
Single precision number uses 23 bits for fractional. For any number in [2^26, 2^27) range, the least significant bit is 2^-23 * 2^26 = 8.
EPICS values are fine, so this is a problem of DAQ/NDS/DV.
Channels are double-precision in the front end, but stored as single-precision in the frames. Maybe Jordan was getting this data from the frames/NDS2, rather than live, so that's why there's this quantization error?
