The violin duet is still playing. Evan is trying some new filters soon. Should know more in a couple, but currently we still can't go to low noise. Otherwise, winds are not bad, but useism is still high.
Evan, Jim W, Dave:
h1nds1 locked up this afternoon at 14:36PDT. Jim manually reset the computer around 19:00PDT, but monit failed to autostart the daqd process. I logged in as root and restarted monit, whereupon it correctly started daqd. We have seen this "monit at reboot time" failure before.
This initial problem occured at 14:36 PDT, a swapper page allocation failure. Details attached.
20 s oscillations seen in EX UIM drive and dHard pitch. Cause not immediately clear.
Again ca 04:45:00 Z.
In both cases DARM was being controlled with EX.
Now that we are in the nominal low noise configuration, big excursions are seen on the EY UIM instead of the EX UIM.
As Corey and Kiwmau noted, ETMY has 2 violin mode harmonics fairly close to each other (1008.45 and 1008.49). ETMY MODE3 has been set up to damp the higher frequency one with a broad bandpass, a phase of +60 and a gain of +100 set in the guardian (FM1,3,4,10, total phase at 1008 is 82 degrees). Cheryl looked at the DARM spectrum over some of the long locks recently, and it seems that with these settings the lower frequency one 1008.45 has slowly rung up.
The phase shifting filter modules also had 6 dB of gain, which cause some unintended saturations, so I edited them to have 0 dB of gain at 1008.45 Hz. We addded two stop bands to the filter bank, one for 1008.493 and one for 1008.45 Hz.
After some confusion, we have been able to slowly damp this with a positive gain and FM1,2,4,5 (the new notch) ,and 10 on. This means a total phase of -75 degrees. This is not ringing up any other modes that we can see so far.
So for now:
The damping of both of these modes are commented out in the guardian, so it will only be damped if you engage one or the other by hand.
To damp 1008.49Hz use FM1 (broad bandpass at 1010), 3 (+60 degrees phase),4 (100dB), FM6 (to notch the 1008.45 mode), FM10 to notch 1009.6 mode and a positive gain
To damp 1008.45 use FM1,FM2 (-60 degrees) FM4, FM5 (to notch 1008.49Hz) and FM10 and a positive gain.
We can make two different filters to damp these two modes simulateously in the future.
This configuration causes the mode at 1009.03 Hz to ring up. Note that this mode already has its own dedicated damping FM (MODE7).
I added a stopband for this frequency in the MODE3 filter module. So far both modes (1008.45 Hz and 1009.03 Hz) are damping simultaneously now.
Damping the 1008.45 Hz mode with MODE3 was very slow going, so I implemented a separate damping loop using MODE9 and the following settings: FM1, FM2, FM4, FM9, positive gain, and length drive to the PUM (instead of pitch). This seems to be somewhat faster than before, but still much slower than some of our other damping loops. Anyway, the ADC counts for the DCPDs are topping out around 25000 ct in full lock with 1 stage of whitening (the limit is 32000 ct), so this is enough to proceed to nominal low noise. The mode should continue to damp down if we can maintain lock with these new damping settings.
To keep the 1008.49 Hz mode from ringing up again, I have turned on MODE3 with the settings that Sheila described above: FM1, FM3, FM4, FM6, FM9, FM10, positive gain, and pitch drive to the PUM.
As before, these settings will NOT engage automatically during lock acquisition. However, since we accepted these settings in SDF, either the loops will need to be engaged by hand, or new SDF settings accepted.
The Violin Mode Table has been updated. From what I have in my note, ETMY MODE3 filter has always been turned on by Guardian. I have the same question as Dan, what changes?
Title: Ops Day Shift: 15:00-23:00UTC, 8:00-16:00PT, all times in UTC
H1 State: locked in ENGAGE_ISS_2ND_LOOP
Help: Sheila and Evan
Overview: H1 lost lock after violin 2nd harmonics rung up about 14 hours ago. Kiwamu worked on it, then Sheila, and now Sheila and Evan.
Details: Violin 2nd harmonic at 1008.45 has been ringing up for a few days, and last night caused a lock loss. There is a nearby peak at 1008.49.
Day was spent working on filter gain and phase to lower peak - time consuming, and within the last 30 minutes, the peak is now coming down in amplitude. Details in Sheila's alog.
ISI blend filters changed to 45mHz, which helped, given the useism.
No other issues to report.
Elli was here and used about 30 minutes of unlocked IFO time to complete her measurement. She work from about 17:07-17:38UTC.
Evan, Sheila, Cheryl
We switched blend filters from 90 to 45 mHz in the last several minutes. There were large glitches in green arm transmissions when I switched ETMX X direction and ETMY Y direction. It seems like ETMY direction switched something got misalinged (the Y arm green transmission dropped). Cheryl adjusted transmon Y to recover the build up.
If anyone from seismic is interested in figuring out why switching blends causes glitches, I started switching around 18:50 UTC and finished by 19:01 UTC on October 25.
The microseims BLRMS is around 0.5 um/sec and the wind had been gusting up nearly up to 20mph over the last hour but is dying down now. ALthough the arms got misalinged, the build ups are much more stable with 45 mHz blends in these conditions.
Ops Day Shift - 15:00-23:00UTC, 8:00-12:00PT, all times listed are UTC
H1: cannot get back to Low noise due to violin second harmonics around 1008Hz
Current Plans:
- Elli on her way in to do her measurement - takes about 30 minutes and she needs an unlocke IFO
- Sheila on her way in - she had me comment out the line that engages ETMY L2 MODE3 in the ISC Guardian, but suspects that that won't be enough to get back to Low Noise, so is driving in
Changes:
- ETMY L2 MODE3 in the ISC Guardian commented out, and I left a comment in the file that I did this today
Current state of ground motion:
- useism coming up again and upper data points are touching 1 on the seismic plot
- also some anthropogenic noise starting about one hour ago
The ROLL modes are quite bad, and I believe prevent IFO from getting to ENGAGE ASC
- Only time IFO has made it ENGAGE ASC this morning was when I turned on ROLL mode damping early
It should be possible to construct two narrow filters to damp the violin modes - each perhaps 20mHz wide, centered on the line you want to damp. Filters this narrow should be able to damp one line without ringing up the other. From there it's just a matter of finding the right phase. Don't be too timid with the gain - i think using 30% of the DAC range is perfectly safe, once you are confident your settings won't ring anything else up. The higher harmonics need lots of gain.
In the past these modes have damped very easily. The settings have been stable for months - I wonder what changed?
TITLE: 10/25 OWL Shift: 07:00-15:00UTC (00:00-08:00PDT), all times posted in UTC
STATE of H1: ETMy violin mode issues. H1 currently down.
Incoming Operator: Cheryl
Support: Kiwamu (drove onto site)
Quick Summary: ETMy violin mode (2nd harmonic) has been ringing up over last 2 days.
Shift Activities:
Handing off H1 to Cheryl. H1 should be able to get up to Engage ASC3 or DC Read out and then one can continue work on damping ETMy violin. Cheryl will contact Sheila when she gets a chance. She has experience working on damping these modes, but will give Sheila a call this morning.
Notable changes:
anti whitening (FM1) OFF for OMC DCPDs (A&B)....as Kiwamu notes.
ETMy MODE3 currently at gain of 50 (100 is what used to be nominal, but this rings up ETMy).
State: Out of Observation Mode and at DC_READOUT for Violin Mode work.
As noted earlier, we had rung up violin modes. Kiwamu is now on site. We are pausing Guardian so we can address the ETMy 2nd order violin mode. (was not sure what to mark this as with Observatory Mode....used Preventative Maintenance at first, but then switched to Corrective Maintenance. We will stay here until we are able to bring down these huge violin modes.
No success so far.
It seems that there are two modes close to each other separated by roughly 46 mHz according to the beatnote I see in the damping signal.
Having +120 deg with a gain of -4500 allowed us to damp one of them while the other rung up. The nominal gain setting seems to bring us to some kind of equilibrium state where the two modes neither ring up nor down, perhaps it is very slowly ringing up.
Note: at the beginning I trended a channel which was red in the ODC Overview (H1:0MC-DCPD_A_INMON) & over the last day, these dcpds have been noticeably getting big during the last two locks (starting at 10/24 11:00 utc). See attached plot for these PDs getting rung up (presumeably due to the ETMy violin).
At 9:25, started getting OMC DCPD Saturation alarms. Not sure what to do here. Can only see issue on the ODC overview & I'm getting Verbal alarms every few minutes. The range is also taking a dive.
I don't have the rf45 template up because nuc5 had issues. And I don't know where to find the dtt session for it. So not sure if the issue is related to that.
The DMT Viewer sessions (for seismic trends) on NUC5 had been closing out the last few days. The NUC5 instructions say this is probably due to memory loss, so rebooted the computer by clicking the button on the nuc computer.
Unfortunately, I didn't obviously see the computer come up on the TVs. Luckily Jim was here and tinkered with the TVs and saw that the TV switched to a different input source. So, had to switch it to HDMI (also had to switch the upper TV, which required a ladder).
I couldn't log in to the computer, so I had to pull a keyboard and mouse from another computer & logged in. (after this I could share the screen remotely).
I opened up the seismic DMT sessions. I wasn't sure how to open the rf45 session (Cheryl posted the link to that file...but do I need to ssh into a different session to run this?).
Final note: I could not get the mouse onto the bottom TV screen.
After a few hours noticed odd display on the low frequency seismic changes. Then I looked at the date for this session and it was for some time in August! Not sure why this was the case. I just hit RUN. Closed the session out and hit RUN again, and this time it was live with today's date.
Now can see that over the last 12hrs the useism has been trending up noticealby.
Chris B, Joe B After LLO had locked again, Joe and I took the opportunity to perform the coherent stochastic injection. CW injections were off at both sites. And the intent bit was off at both sites. For the LLO couterpart aLog entry see: LLO aLog 21999. Waveform: The waveforms injected were: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/stoch/Waveform/SBER8V3.txt Injection: At H1 Chris performed the injection with the command: awgstream H1:CAL-INJ_TRANSIENT_EXC 16384 SBER8V3_H1.txt 1.0 1129673117 -d -d > log_stoch.txt I've attached the log. As we were doing the injection we noticed a range drop of ~10%. IMPORTANT ACTION ITEM: The end of the stochastic waveform was not tapered. So when the injection ended it introduced a large transient into the ETMY. Robot voice was activated. This needs to be fixed. The beginning was properly tapered.
It looks like no one set the CAL-INJ_TINJ_TYPE EPICS channel prior to running awgstream. At LHO it happened to be equal to 2, so this stochastic injection was logged in H1 ODC bits and in the DQ segment database as a burst injection. At LLO it happened to be equal to 0, so this stochastic injection did not flip any of the type-specific bits in L1:CAL-INJ_ODC, although it did still flip the TRANSIENT ODC bit. So, this stochastic injection should be represented in the segment database with the ODC-INJECTION_TRANSIENT flag, but not with the ODC-INJECTION_STOCHASTIC flag.
