As per Daniel's request I changed ITMX value from .5 to .4 and ITMY value from 1.4 to 1.3.
Don't really know what to do at this point. 5 nodes not ok...! SDF diff accepted. 4 Squeezer nodes not ok and not doing anything....etc.
@ 12:37UTC 87.25Mpc. After a long struggle by Daniel to get the squeezer locked and running. Livingston is down and we are coincidental with Virgo
Initial Alignment done after a few failed locking attempts following a successful lock that allowed Craig to perform a freq sweep measurement.
11:19 H1 back to NLN.
It also looks like the squeezer isn't on. Daniel is working on it.
Commissioners reset the PCal-X because it was elevated. THe IFO had been Observing for 1:11 minute according to GWI Stats. This knocked the Observing bit out and it was immediatly reset. There was no Observation Mode setting changed for this.
Anamaria, Georgia, Danny, Corey, Craig Locking has been a problem since we keep saturating our ETMX PUM drive. The problem is a combination of this oft-referenced 4.1 Hz ringing somewhere in our pitch ASC (from the control signal, CHARD P is the most likely culprit), in combination with high microseism finding it's way into the PUM length drive. Together, these two effects ask too much of the PUM stage and causes locklosses. Similar to last night, we've sacrificed range for stability to get more observing time for ER14. We decreased L2 LOCK gain from 23 to 15, increased L1 LOCK gain from 1.06 to 1.16, and left all quad coil drivers in the "1" high noise state rather than the "3" low noise state. These changes are in guardian. We have not tried putting the coil drivers into low noise with the gain changes. We updated and checked the calibration, it seems fine from 25 Hz up, so the high noise is real. The first attachment is the PCAL BB injection. The second is the ETMX PUM Coil MASTER OUT channels before changing the suspension hierarchy around. The third is the same plot after. It is clear our coils were saturating before, and now are not coming close to their rails. More thought required on why our LSC microseism is showing up so strongly in PUM.
Looking at the sensemon range FOM, it seemed like we were losing ~10 Mpc when we locked with lower ETMX L2 gain and higher range on our coil drivers.
I had a look at the DARM spectrum during a few locks over the last few days, it seems like in locks with low ETMX L2 gain there is additional broadband noise in DARM from 28 - 80 Hz. There's at least one lock with the coil drivers in their high range state (state 1) , and nominal L2 gain, and the noise in DARM is lower (brown trace).
We had a lockloss at 1237370491 due to the 4.2 Hz ringup again, even after we changed the ETMX_L1_LOCK and ETMX_L2_LOCK gains. Looking at the drive to ETMX L2 (bottom row, second-from-left in attached plot), our gain changes are doing their job - the low frequency drive to L2 is small - but the 4.2 Hz is still too big (most noticeable in CHARD_P, the pale green trace, but dominating all the arm ASC DOFs). Also of note is that this signal shows up at twice double the frequency in the OMC DCPD SUM channel (second-from-top row, leftmost plot).
This noise is very interesting, I wonder if we can chop between state 1 and 3 in low noise. It looks very similar to the L1 "sunday noise" situation, where noise would come up in this frequency band in random locks. This noise has not appeared again since one of the ITM coil drivers was switched out.
With the higher microseism it seems L2 gets too much length drive and saturates the DAC; the addition of the angular cancellation in the "spot position" through A2L filters on top of that makes the DAC saturate easier but I don't think we can afford very different spot positions. This is why we have to be in state 1 right now. But we should rethink the DARM filter so L2 offloaded properly at low frequencies.
Furthermore, it also seems that the 4.2Hz is worse when ETMX is in state 1 versus state 3, which might be related to how accurate the analog compensation filters are in the digital path. Regardless, Craig thinks it might be coming from the spring change in the DARM plant once we added SR3 heater so maybe it's in line with Sheila's previous fix and some retuning of that filter would help.
I had a closer look at the two times Georgia listed where the L2 LOCK gain was changed for the same coil driver state. The difference in the noise is just because the MICH feedforward became mis-tuned when the L2 gain was changed. The attachment shows that the MICH coherence was high in the frequency band where we saw excess noise when the gain was lowered.
TITLE: 03/23 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 88Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
Wind: 12mph Gusts, 9mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.42 μm/s
QUICK SUMMARY:
Corey updated me on waiting for dither lines to reduce back to reference during POWER_30W, before moving on to NLN.
TITLE: 03/22 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Ed
SHIFT SUMMARY:
useism slowly increasing over last day & winds picked up at the beginning of the shift (w/ gusts just under 40mph).
Locking Note: wait for a long time at Power_30W for ADS to converge to the level of the noise floor
LOG:
Attaching screenshot of the squeezer differences which came up while we were trying to transition to observing. We unmonitored them since presumably they are being touched by a guardian.
Sharan Banagiri, Anamaria Effler, Philippe Nguyen, Kara Merfeld, Robert Schofield
19:30 - 21:30 UTC
23:30 - 01:00 UTC
2:06 - 2:50 UTC
Over the course of eight hours, during the periods when the interferometer was locked, we made shaker injections and follow up acoustic injections at EY, and some magnetic injections at the corner station.
Have made it to NOMINAL LOW NOISE 3 times this shift, but only for short amounts of times and we are sort of blaming our winds (which have gusted up to 40mph). Will continue trying! (well, after lunch break for me.)
Today I updated the RH guardian. This post is to provide a summary on how to use it:
The default state is the NOMINAL state. This state acts as if there is no guardian present so you should have the ability to manually change the power to each of the segments.
Making a change with the inverse RH filter
Over the last day and a half the gain of the POP_X Q3 quadrant was not scaled correctly when we went through the reduce 45 modulation depth state. 47759 We don't know how much this was contributing to locking difficulties over the last day and a half.
The guardian stopped setting this gain yesterday, but didn't make any obvious error messages. I was alarmed that we didn't get any errors, and filed a bug report 1137
Jamie called the control room and asked us to check if this was using the fast_ezca script that some of our guardians use, it was and this was allowing the guardian continue to move through the state even though some values which could be important were not written.
I've expunged fast_ezca from ISC_LOCK, and hope we can delete it completely soon.
Once we removed the fast ezca, the guardian didn't keep moving along as though nothing was wrong, but instead stopped and turned maroon. We stopped the node, and then went back to exec, and it was able to connect to the channel.
Dave and Jonathan are investigating why the guardian was unable to connect: 47773
On Robert's request, I've created an MEDM which shows a summary of the PEM filter modules which are driving the GDS 18bit DAC outputs. The MEDM shows all three locations (4 DAC channels at each end station, 7 in the corner).
For this application, the state of; no filter control modifications and zero output counts is nominal. Any deviation from this state is flagged RED.
The screen has been linked to the SITEMAP as the last entry in the PEM pulldown.
The 'related display' button on the upper right opens the full FILTER.adl MEDM for the filter module. This small overview is read-only, you need the full screen to make changes.
Jamie, Sheila, Dave, Jonathan,
ISC_LOCK was not connecting to H1:ASC-POP_X_RF_Q3_GAIN. From h1guardian1 was was able to connect to this channel with no problems when running caget manually in a login shell. Sheila forced a reconnection of this node's EPICS channels and was able to proceed past this point.
We noticed that htop on h1guardian1 was looking very different to restart times. After a reboot all 40 cores were evenly loaded at about 50%. Now many are in the 70-90% range, and this shifts constantly over the cores.
Jonathan installed his system diagnostics EPICS IOC on the guardian machine this afternoon. We will monitor this via StripTool over the next few days, and add these channels to the DAQ next Tuesday. I would also like to reboot h1guardian1 next tuesday so we can monitor the core usage evolution.
htop core usage shown
The squeezer did not engage when we got to NLN tonight. The 79.4MHz VCO was locking to 79.2MHz, and the guardian told us to adjust TUNE OFS. We disabled the servo and moved the slider to 79,4MHz. We could not enable the frequency servo without it driving back down to 79.2MHz. We tried to continue locking with the servo disabled and manually setting to 79.4, however, SQZ OPO guardian is now saying it "can't lock the OPO, check pump light on SQZT6". We are unsure what is going on.
The solution to this issue was to ensure that the "Ramp Enable" switch in the TTFSS servo page was switched to "off".
I think this is the same issue as Friday night when TTFSS seemed to have jumped straight to "Acquire" without engaging the loop. The 79MHz message came up because when TTFSS was unlocked and the SQZ laser was not following the IMC VCO (the SQZ laser needs to follow the main IFO laser for the squeeze angle loop to close, the 3MHz one). The whole guardian automation was written assuming that TTFSS works reliably enough and it doesn't touch any of these TTFSS channels. TTFSS is all Beckhoff controlled.
Found myself stuck with this error tonight as well. Don't know how exactly I solved it but it seemed to be a combination of switching the error signal on the Slow frequency servo to "On" and possibly waiting at "Adjust Frequency" state for the SQZ_LO_LR guardian.
I have put together a few python functions which allow for briefly spawning multiple threads to switch many filters at (roughy) the same time. The idea here is NOT to provide synchronous filter switching, but rather to speed up Guardian transitions which change the state of many filter modules (or more generally, write many channels).
The new code is in:
userapps/release/isc/h1/guardian/
fast_ezca.py - new functions for writing, switching, and generally doing things quickly
test_write_many.py - test functions for multi-thread writing
test_switch_many.py - test functions for multi-thread switching
test_do_many.py - test functions for multi-thread compound actions
and it is being used in the ISC_library function slow_offload_fast. There is a single-thread version of this function in ISC_library in case of trouble: slow_offload_many. The only caller is gen_OFFLOAD_ALIGNMENT_MANY in ISC_GEN_STATES, so go there if you need to switch this out.
This allows the guardian to move on without setting a setting, and can cause problems because settings can be wrong and the user has no clues.
I want to delete this completely.
Didn't want to leave the Observation Mode in Observing so I left it in Lock Acquisition.