TITLE: 03/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 107Mpc
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
Wind: 8mph Gusts, 5mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.26 μm/s
QUICK SUMMARY: Great long lock - a violin mode ringing up at 500Hz
Ops Shift Log: 03/25/2019, Owl Shift 07:00 – 15:00 (00:00 - 08:00) Time - UTC (PT)
Violin mode-9 (ETM-X)is ringing up, and am unable to damp it. Guardian message is to turn off gain, however the gain on the filter bank is 0.000. When I try to change the gain, Guardian (I assume) is overriding my changes and putting it back to zero.
Looking at the Guardian VIOLIN_MODE.STATE screen, there is a state to turn off the DAMPING for a suspension, but not the gain.
Could find no information in the WIKIs or in the aLOG that addresses this situation. Leaving the IFO in Observation mode and hoping a commissioner comes in before it breaks lock.
IFO has been locked in Observing for the past 7.75 hours. The input power is 35.2w. The IFO has been a bit glitchy all night, but has remained locked. The RF9 and RF45 plots are bouncing around, which may be influencing the glitches. Environmental conditions are good. No outstanding concerns or issues at this time.
TITLE: 03/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ed
SHIFT SUMMARY:
Started out rough, but have been in OBSERVING for most of the last 4hrs.
LOG:
No oplevs are out of nominal, but ITMx Pit is approaching -10 counts.
attached are photos of the Guardian Nodes which I took action on.
Does ISC_LOCK not take SQZ_MANAGER to its nominal state anymore? Or did ISC_LOCK requested SQZ_MANAGER nominal state and failed? What's in Inject_squeezing state in the ISC_LOCK?
If ISC_LOCK requested SQZ_MANAGER to go to nominal state but failed for got stuck, can I have more details of how it fails next time this happens? Including guardian log, screenshots, would be helpful for improving squeezer automation.
ISC_LOCK takes the SQZ _Manager to its nominal state during normal locking, but with what happened tonight (would we call this the "Squeezer losing lock" or something else?), the ISC_LOCK & ALIGN_IFO were transitioned to "NOT OK" states, when the Squeezer did what it did.
Luckily I was able to get the SQZ Guardian nodes mentioned above to their nominal state. Otherwise, ISC_LOCK & ALIGN_IFO would have remained "Not OK" and would prevent OBSERVING. Alternatively, if one is comfortable with changing the ISC_LOCK & ALIGN_IFO scripts, I reckon there must be a way to break this remaining connection they have to the squeezer.
Anyway, when I looked at SQZ_MANAGER and SQZ_LO_LR, they were not actively doing anything according to their logs. SQZ_LO_LR was down and staying there. After I (1) tried getting it to its nominal state, and then (2) getting SQZ_MANAGER to its nominal state, then ISC_LOCK & ALIGN_IFO were back to being OK.
Since this was easy to recover from, maybe it's good to have it go down in this way? Mainly because recovery was fairly straightforward? And we also get a heads up that Squeezer's state has changed.
Oh, and good idea about posting more info. Attached is screenshot showing all the logs of pertinent guardian nodes from this event. Now that I look at them, it looks like ISC_LOCK & ALIGN_IFO probably did not take us out...but they point the way to SQZ since they give the messages when they were listed as "not ok" (but their logs don't show anything until the very end when I go back to Observing).
I reckon it's the SDF diffs from the squeezer which took us out of OBSERVING.
Hope this helps!
Oh, and the other thing from the screenshots is that you can see ISC_LOCK definitely does not do anything to the SQZ_MANAGER when this happened. (but ISC_LOCK & align_ifo were listed as NOT OK....and this would not have let us return to Observing until we fixed their situation with the SQZ or edited their scripts).
H1 has been finicky this evening. One lock stretch was 17min and then lockloss.
Currently, we are NLN & quickly went through SDF diffs so we can immediately get to OBSERVING. Doing this with Squeezing, AWG_LINES guardian node now has IDLE as its Nominal setting. Range is hovering between 105-110Mpc. For this lock, mainly let Guardian take us all the way up with no pauses.
Sheila, Corey, Cheryl
We have been struggling with oscillations around 4.2 Hz which show up in our DARM, CHARD and CSOFT signals for the last several days. Motivated by that, I checked on the L2A decoupling for ETMX. It seems that the L2A is actually increasing the coupling to both pitch and yaw above 2.5 Hz, but the DARM loop is unstable at 8 Hz without these L2A filters on. We haven't fixed anything yet, but we probably do need to spend some time to make length to angle decoupling measurements this week. We haven't see a 4.2 oscillation all day, that might be because Daniel changed the whitening on ASC REFL A 45.
Background:
This is a similar frequency to the problems that were mitigated by increasing the relative gain between the PUM and ESD stages in the DARM loop in late Febuary 47164. In the last week the spot position was moved on ETMX (P2L gain went from 4 to 5) to get an improvement in the power recycling gain, 47609. There have also been several changes to the pum coil driver states, we've reduced the cut offs in CHARD P 47750 to make that loop more stable, and there have been changes to the offloading gains in the DARM actuator 47784 See the two attached PDF to see what happens to the DARM loop with the PUM gain reduction people tried on Friday, there is nearly a crossover instability at ~9 Hz, although it doesn't seem that the loop was actually unstable.
L2A checks:
After we lost lock because of some ADS errors, I took a bit of time to check the ETMX length to angle decoupling for L1 +L2. The first attachment shows the response of the optical levers to L2 length drive with the decoupling filters on and off. The decoupling does it's job at 0.5 Hz, but above 2 Hz both the pitch and yaw decoupling are actually increasing the cross coupling. (The measurements were made with the same excitation level, with the decoupling off there is no coherence above ~2.5 Hz, but with the decoupling on the coherence is high). I also tried this measurement with the L2Y decoupling off and the L2P on, the results were what one would expect. Hang redid the fitting of these decoupling filters for the new suspension, 42692 based on measurements Jeff K posted in 42662. The coherence of several of those measurements petters out above ~2.5 Hz, so it's not surprising that the decoupling isn't good. I attempted to make some measurements that would have coherence to higher frequencies, but didn't get good results. We might need to take the time to do swept sine measurements.
For L1 the situation is different, the L2P decoupling doesn't have much of an impact at all. L2Y reduces a peak at 1.37 Hz. (second attached screenshot) There is an alog that describes the L1 L2P filter here: 43150 and in comments. The coherence of these measurements isn't good above about 2.5 Hz. For L2Y we have not used a fit to the data that Jeff collected, we've just been using a DC gain since January: 46236
Impact of L2A on locking:
We tried locking several times without L2 L2A engaged, and had a DARM instability at 8 Hz ring up within a minute or 2 of transitioning DARM control back to EX with the low noise ESD each time. I reverted the changes to the LOCK gains, but we still see the 8Hz instability unless the L2 L2A decoupling is back on.
Configurations we tried:
Other notes from today:
TITLE: 03/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
Wind: 12mph Gusts, 9mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.25 μm/s
QUICK SUMMARY:
H1 continues to be looked at.
useism on its way down. (yay!)
Saw ALSx polarization flash on DIAG_MAIN because it is up near 20% again. Talked with Sheila and she said this can be adjusted if we have H1 at NLN (or any state not involving ALS/Green)....but it would probably knock us out of Observing, so we don't want to do it then. I might try to address this again.
Earlier in the day, around 19:20 UTC, there was an odd excursion of the anthropogenic noise in 3-10Hz, and also seen in 10-30Hz, only at the Corner Station, increasing, then falling off quickly, I looked for a herd of Elk, but saw none. Tagging DetChar. Snapshot of nuc5 showing the event is attached.
TITLE: 03/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Corey
SHIFT SUMMARY: locked most of the shift for commissioning by Sheila, Daniel, and Nutsinee
LOG:
The main thing I did here was adding SQZ_MANAGER to EXCLUDE_NODES[ ] so we can go into Observing without looking at what squeezer is doing. This is a temporary solution while we work on improving squeezer automation reliability. We haven't had a lot of experience running squeezer long term. Please continue to report any squeezer locking issue you encounter.
I also did the following:
Remove dead nodes (I don't know why this guardian is not complaining about the dead nodes) -- SQZ_FREQ, SQZ_BEATNOTE, SQZ_LOCK, and all the old SQZ guardian nodes used for the old H1 locking scheme.
Added new sqz nodes that ended with "LR"
And of course, added SQZ_MANAGER
Sheila, Daniel, Nutsinee, and I are here.
Don't use the old A2L script that was reachable through the SUS A2L_screen. When the new dither alignment system was installed in H1, the old A2L script became a hazard.
The old script changes many more channels than guardian does during reacquisition of lock, so the old A2L alters the state of H1 in a way that guardian can't recover.
Sheila found the output matrix of the dither align loops were all zeros, except for one entry to the BS, aolng with other missing/wrong entries.
Sheila is recovering the system by hand using SDF.
I am removing the access to the A2L script from the SUS A2L_screen.
Old medm and new medm images attached.
Old A2L script id still here, .../trunk/isc/common/scripts/decoup/a2l_min_LHO.py.
- Sheila, Cheryl
Issue 1: The polarization from the PSL fiber has changed which results in too much power on the broadband PD used for the squeezer beat note. At too much power this PD saturates and squashes the beat note which was found at -14dBm (should be -1dBm). The SQZ-FIBER _TRANS_PD_POWER should be between 10 and 30µW. New limits have been set. If the laser locking complains with the error message "Fiber Trans PD Error", the waveplates names "SQZ PSL FIBR HWP/QWP/HWP" should be adjusted to bring the normalized power back to 1, or roughly 20µW.
Attached a screenshot of where you will find this error.
In the future if people encounter squeezer locking issue please attached following screenshots to the alog so we have a glimpse of what's going on. The main SQZ screen can be access via SITEMAP >> SQZ Overview
