Firstly, some Jason induced phenomenon:
Chiller :
Shifter: Adrian Helmling-Cornell
Fellow: Vlad
DQ Shift Summary:
The fix for the missing observing time on January 9th is as follows (from Robert Bruntz):
Also, the link to my shift, as I didn't originally include it in my aLog.
TITLE: 01/13 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 120Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_CONF state: USEISM_WINDY
Wind: 12mph Gusts, 8mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.49 μm/s
QUICK SUMMARY:
I have nothhing to add to Corey's parting summary regarding the current state of the IFO and site at this time
TCS-X chiller water level is good. Did not add any water. TCS-Y chiller was down to 9.0. Added 125ml to bring it up to 10.1. Everything else looks good. Closing FAMIS #11526
TITLE: 01/13 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:
After the windy weekend, I forgot what it was like to have a shift like today! H1's been locked for 18.5hrs, and for most of the shift hovering at around 118.5Mpc. And now the day starts with addressing: tumbleweeds!
Microseism is still at the 90th percentile, so SEI_CONF hasn't been touched (it is in USEISM_WINDY).
LOG:
Laser Status:
Front End Power is 32.32W (should be around 30 W)
70W Output Power is 70.37W
Front End Watch is GREEN
70W Watch is GREEN
PMC:
It has been locked 11 days, 16 hr 57 minutes (should be days/weeks)
Reflected power = 11.51Watts
Transmitted power = 52.38Watts
PowerSum = 63.89Watts.
FSS:
It has been locked for 0 days 15 hr and 15 min (should be days/weeks)
TPD[V] = 5.252V (min 0.9V)
ISS:
The diffracted power is around 2.5%
Last saturation event was 0 days 15 hours and 15 minutes ago (should be days/weeks)
The following CPS is listed as over threshold (all others OK & all spectra attached):
TITLE: 01/13 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
SEI_CONF state: USEISM_WINDY
Wind: 8mph Gusts, 5mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.51 μm/s
Walked in and there were no winds for a change! Microseism is holding at the 90th percentile.
QUICK SUMMARY:
TITLE: 01/13 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: locked all shift, useism is hovering around 0.5um/s, winds are below 20mph
LOG:
Violin damping: EY modes 1 and 6 have higher gain
STATE of H1: Observing at 115Mpc
CURRENT ENVIRONMENT:
SEI_CONF state: USEISM_WINDY
Wind: 24mph Gusts, 20mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.50 μm/s
I looked at the very noisy period over the weekend associated with the high microseismic peak to double check that the noise was consistent with https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=54298, and that the R0 path we are hoping to incoporate into models on Tuesday would likely help. The figure shows that the arch frequency spacing in DARM pretty well predicts the relative velocity of L2 and R2. The hypothesis is that the scattering path is between the ESD traces on the RM and the HR coating on the TM, with the arch frequency spacing given by the TM-RM relative velocity and the arch stacking in frequency due to multiple cycles on this path with 90% power loss each cycle. R0 tracking would reduce the TM-RM relative motion and appears to have helped at LLO https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=50897 .
Hi Robert,
Figure 1 below is an overlay of the same data you showed, but the spectrogram made with an omega scan, and the fringe frequency calculated from SUS_ETMX_L2_WIT_L_DQ using f=N*abs(2 v(t)/lambda) where v(t) is the velocity (derivative of that channel). The outcome agrees with what you saw, the fringe predictions line up, and specifically, the multiples of N=3,4,6,7 are visible in the h(t) data.
This was run on the hanford LDG cluster using the following commands:
conda activate /home/detchar/.conda/envs/ligo-summary-3.7
python -m gwdetchar.scattering -i H1 -m 3,4,6,7 -d 10 -t 10 -c viridis 1262793685
This scattering functionality was written into gwdetchar by Alex Urban.
Figures 2 and 3 shows the same data, but overlaying the fringe prediction that the Python code produces on the two different spectrogram types available on ldvw (scaled to fit axes).
Cheryl, Jenne, Rahul
On Jan 8, 2020 we had a lockloss due to rung up violin modes. When the operators attempted to re-lock the interferometer, we suffered another lock-loss (at Guardian state 430, Engage_ASC_FOR_FULL_IFO). Hence, we analyzed the typical noise floor during lock acquisition and then tried to define a threshold before attempting full IFO lock. Figure attached below shows the DARM spectrum for 08 Jan 2020, focussed on the 1st and 2nd harmonics for the violin modes. In the plot (Darm.jpg, during the 2nd attempt for re-locking the IFO), the noise floor at Guardian state 430 is shown in red line (UTC 00:02:32). As one can see, several of the modes are rung up, ranging from 10^-15 m/sqrtHz to 10^-14m/sqrtHz (while the typical rms of the floor was around 10^-17m/sqrtHz). The IFO remained at this state for 4 hours, during which the rung up violin modes were damped (1st harmonic for this case). Later, several of the peaks subsided, except for a couple of them, following which a full IFO lock was achieved as shown in the ndscope.
Based on this experience we can set some guidance for the violin mode noise floor during locking attempts. Once Guardian state reaches PREP_ASC_FOR_FULL_IFO (429), at this point if no more than 1-2 modes are above 10^-15m/sqrtHz, while the others are below 10^-16m/sqrtHz then the operators can proceed to the next Guardian state. If at all the rung up modes are required to be damped then during this time the Guardian can remain at the same state.
Rahul-
We should discuss this more in person before issuing guidance to operators, to avoid causing or spreading confusion. In this case there is already guidance available, which I believe should be more reliable than what you outline above. Perhaps we just need to spread the word better.
Operators-
There is a dtt template that Jeff Kissel made several years ago which is linked on the violin medm screen. If operators run it they can check if any modes are rung up enough to saturate the AS_C QPD, and as the text in the template says, if the rms on that QPD is low enough that we aren't saturating it, we know that it is safe to engage the ASC loops which use that QPD.
It seems as though part of the dtt template that Jeff made for checking if the AS_C QPD is saturating has been deleted at some point in the editing that has happened. Jeff K notes in his alog that he put them in the svn so we should be able to recover: 37921
There's a need for two types of templates for the violins, one to use when they are very rung up to assess the success of moving past Engage_ASC_FOR_FULL_IFO, and one to use in low noise to evaluate individual peaks, or groups of peaks, that are ringing up during the lock, or modes that we are damping for the first time.
This alog has one example of the first type of template, and as an example of the second type, I've updated the 2nd harmonic template with the current H1 spectra, I've also been keeping it up to date as we damp more modes, and I've modified it to look all violin frequencies.
TITLE: 01/13 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
SEI_CONF state: USEISM_WINDY
Wind: 28mph Gusts, 19mph 5min avg
Primary useism: 0.07 μm/s
Secondary useism: 0.60 μm/s
QUICK SUMMARY: locked in Observe
TITLE: 01/12 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
LOG:
21:11 H1 back to Observing
H1 had a 4hr lock and this was with sustained winds near 20mph, and then there was a random lockloss.
Once again, FSS would not lock and the PSL_FSS said it was due to "FSS Oscillating". (most recent alog of this I see is from June 2018)
I did the same thing I did earlier in the shift for this: toggled FSS autolocker, took ISC_LOCK to INIT/DOWN, took IMC_LOCK to INIT/LOCKING (& DOWN). Then I took PSL_FSS to INIT/DOWN (then FSS locked). This time it took 8min vs 20min after the last lock.
I marked this down time as CORRECTIVE MAINTENANCE since this isn't normal. (although Ed has just walked in, he mentioned addressing this before and Jason mentioned to him that there was a special way to "toggle" the FSS autolocker....so don't turn it off and wait a while. Just turn off and then on....but I'm pretty sure I tried that and other variants.)
Anyway, FSS during my shift wasn't happy. Am curious to know if this is a trend.
Not only do you have to toggle the autolocker off/on in rapid succession (click OFF then immediately click ON), when you perform the toggle is important as well. If it's having trouble relocking it will get to a point where it wants to grab, but for some reason can't (you'll see the 00 flashes on the PSL quad display, upper-left quadrant). You want to toggle the autolocker during this state, when it should be grabbing a 00 mode but isn't. If the autolocker is running through its temp search (seen by a triangular shape to the NPRO crystal temp graph on the FSS MEDM) then toggling the autolocker will do absolutely nothing.
There have been a few instances of the FSS RefCav having issues relocking in the last few days (a couple on Friday, and the two reported by Corey). This seems to be a growing trend, but oddly enough we very rarely have problems getting the RefCav to lock when we perform work on it. I suspect there may be issues with the archaic FSS RefCav autolocker code; it was written in one of the variants of C many years ago and hasn't been touched since. I'll formalize this in a Wiki page, but here are some pointers if the autolocker continues to have problems (in no particular order):
I also had the FFS not locking on Tuesday (alog 54329), the wind was very high at this time though I don't know if that could have any effect.
How would you suggest pausing the PSL_FFS code? Will taking it to INIT do this as it is Managed by ISC_LOCK.
Thanks for the notes about this, Jason---this helps alot! It was the middle of the night, and I totally did not want to phone you for something like this. I did think I tried different types of toggling of the Autolocker, but I definitely did not correlate toggling with looking at the RefCav flashing modes.
Camilla: (Correct me if I'm wrong Jason) One can PAUSE the PSL_FSS node (and any node) by going to the node's Main Screen & for the "OP" pull-down, click that and select PAUSE (other choices are STOP & EXEC). I imagine we can do that even if PSL_FSS is MANAGED?
Looking at a trend of the wind speeds shows that the end stations are consistently below the other buildings. This is obviously dependent on the direction of the wind, but this looks very promising.
The attachment shows the last ~15 hours with the end station traces (blue and purple) consistently below the other buildings. Zooming out got messy, but you get the idea with this shot.
It's my opinion that comparative anemometers on either side of each windscreen would be a more valid assessment. Typically the variance of the wind speeds from the CS to either unprotected end (and mids) can show fairly large differences. Keeping the H1PEM_WINDSPEED MEDM screen open is part of my regular screen setup (I stack Obs Mode/Wind Speed and Range)
I've been looking at the early data for the impact of the wind fence, and looking from 1st January to 8th January there's on average a ~5mph discrepancy between CS wind speeds and EX and EY (looking at CS wind speed - EX/EY wind speeds), and 2-3mph looking at data from 12th December onwards (I'm looking at max minute trends since that's what I used to see the impact of gusts of wind), compared to discrepancies of 0-1mph on average for O3a and for before the wind fence was completed.