I've added the Beckhoff SDF diffs as a matrix in the lower left hand corner of the CDS Overview MEDM.
Guardian will not permit observation mode if any of these diffs are non-zero. Similar to the SDF diff display for the front end models, a non-zero will be displayed with a red circular border around the number.
Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 20 seconds. TC B did not register fill. LLCV set back to 17.0% open. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 87 seconds. TC A did not register fill. LLCV set back to 37.0% open.
Dave, Jim W.
We restarted all the seismon code which runs on h1hwinj2. The problem we are investigating is an empy H1O2 directory even though earthquakes with mag>4.0 have occurred.
looks like we are still having issues. USGS code has not reported on a 5.2 at 13:50UTC, it did report a 5.2 at 18:50UTC, which is in our 5 hour window but traveltimes did not report it. Investigation continues.
I have cleaned out the ProductClient/data/listener_storage/origin directory to only files created in the past two days. This directory was getting very full and causing 'ls' to fail.
Users may have noticed that when working on cdswiki you have had to press the "I am LIGO" identification button many times. On Ryan's suggestion we extended the timeout of this identification from 600 to 86400 seconds. The shibd and apache2 daemons were restarted on cdswiki.
Verbal has been crashing at 00:00:03UTC some days and not others. I went through the code and found that it came from the Lock_Logging.py module that records lock data and puts it in a text file. From here I had to track down exactly where and what was going on, but I eventually found the issue and my stupid mistake. I wrote it a long time ago, so that maybe makes me feel a bit better....maybe not.
A good reminder to all the python users out there, because I have fallen into this trap more times than I would like to admit.
>>> a = [1,2,3,4]
>>> b = a
>>> b.append(5)
>>> print b
[1, 2, 3, 4, 5]
>>> print a
[1, 2, 3, 4, 5]
Unlike normal variable assignment, when assigning lists, Python will make both assignments point to the same list ( a -> [1,2,3,4] <- b ). So if you change list a, then list b will also change, and vis-versa.
The Solution: use list()
>>> a = [1,2,3,4]
>>> b = list(a)
>>> b.append(5)
>>> print b
[1, 2, 3, 4, 5]
>>> print a
[1, 2, 3, 4]
Hopefully my dumb mistakes can help prevent someone else making the same ones.
TITLE: 01/25 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC STATE of H1: Observing at 64Mpc INCOMING OPERATOR: Ed SHIFT SUMMARY: Very quiet night. I restarted video1 to run the updated vacuum overview medm screen. H1IOPASC0 reported a timing error, Ed has reset it. LOG: No events logged.
At some point in the night I noticed that the HAM6 HEPI guardian was reporting SPM diffs. Since I was told at one point that this feature of guardian is not used, I have ignored it.
Thanks for noticing Patrick. This is related to the Guardian Pringle loop changes TJ and I did yesterday. I am surprised the HPI Guardian is complaining as we changed the guardian and used the guardian to reisolate the platform. The SPM diffs should clear safely with an INIT but maybe best to wait until a lock loss to do that. The INIT should not trip the platform but I'd like to understand why there are SPM diffs. Maybe if the platform is reisolated using the SEI Guardian rather than the HPI. I've asked the operator to let me know if there is a lock loss so this can be addressed: when the light is green the trap is clean!
The IFO Broke Lock ~2220utc, HAM6 ISI tripped too on Actuators (Shutter Fire.) Ran Guardian INIT on HPI, again, and then on SEI Guardian but these did not clear the SPM diffs. Untripped the ISI and the SEI Manager brought the ISI back up. SPM diffs remain. Executed INIT again on SEI but this did not clear the SPM diffs. Took SEI down to READY (deisolating ISI and HEPI.) SEI Guardian then brough HPI and ISI back to Isolated and SPM diffs are gone. Good things these aren't in Observing bit.
15:59UTC Diag reset. I expect it will be back.
Yesterday, all of the AXIVANE Supply Fans on site were lubricated per FAMIS. The lubrication schedule is quarterly for the fans and semi-annual for the motors. At the out buildings, i.e. mid stations and end stations, we typically only run 1 fan, so when either of the above mentioned lube schedules are performed I rotate the run schedule on the fans at those buildings, I have completed that rotation this morning.
Have remained in observing. I restarted video1 to run the updated vacuum overview medm screen. H1IOPASC0 has a timing error. No other issues to report.
TITLE: 01/25 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC STATE of H1: Observing at 65Mpc OUTGOING OPERATOR: Nutsinee CURRENT ENVIRONMENT: Wind: 0mph Gusts, 0mph 5min avg Primary useism: 0.06 μm/s Secondary useism: 0.37 μm/s QUICK SUMMARY: No issues to report.
TITLE: 01/25 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 66Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Small violin mode commissioning during the beginning of shift. Otherwise locked and observe for the most part. See alog33608 and comments therein.
LOG:
00:55 Out of Observe to load violin filters
01:07 Out of Observe to test violin 1kHz settings
02:02 Back to Observe. The new settings of 1kHz damping were left on. Guardian will also take care of turning them on the next time we loses lock.
05:32 Lockloss. Unclear how. Relocking wasn't much of a trouble.
06:36 Back to Observe.
I was tasked by Sheila to find out which of the 1st violin mode harmonics are consistently rung up. We were hoping to see just a few modes that like to ring up so we can have the guardian damp them automatically. However, pretty much everything in 1kHz rings up over time (also true back in O1). Some of the lines seem to be ringing down on their own and some don't. Because we only have 2-3 free violin filter banks per test mass I chose to damp some of the lines that are high and don't ring down on their own. The lucky frequencies that will be damp automatically by the guardian are:
[ITMX]
992.42Hz -- MODE9: FM8(BP), FM2(-60deg), FM4(100dB), -10gain
994.27Hz -- MODE10: FM5(BP), FM2(-60deg), FM4(100dB), -10gain
[ITMY]
994.65Hz -- MODE9: FM8(BP), FM2(-60deg), FM4(100dB), +10gain
998.81Hz -- MODE10: FM8(BP), FM2(-60deg), FM4(100dB) -10gain
[ETMX]
1003.67, 1003.78, 1003.91 -- MODE10: FM1(BBP), FM2(-60deg), FM4(100dB), +30gain -- Luckily the phases work out such that I can make a broader bp filter that covers all three lines. I wish to make more of these for other test masses but they weren't as straight forward.
1004.54 -- MODE9: FM9(BP), FM2(-60deg), FM4(100dB), +10gain
[ETMY]
1009.44, 1009.49 -- MODE1: FM6(BBP), FM4(100dB) +15gain (these two lines are generally the highest of 1k violin amplitude. I don't want to crank up to the gain too much. Just let them damp slowly, safely, and surely)
With permission from Keita I took the IFO out of observing briefly to test all the settings (LLO was down), wrote them in the ISC_GEN_STATES.py, and accepted the differences in the SDFs. I haven't hit the load button yet. I hit load button after a lockloss.
Note: BBP = Broad Band Pass filter (not that broad, actually, just broader than usual)
All the settings were tested for 30+ minutes and will be monitored until the end of shift. I also updated the violin mode table.
Notice that the damping filter of the 1009Hz lines may have an effect on the 2kHz glitch line reported here.
7 hours later the 1009.44Hz and 1009.49Hz are continued to be damped (I gave them a very small gain given their amplitudes). Other modes seem to be done damping.
Borja: I hope the "effect" is a positive one!
Fil called and reported that one of the supplies was tripped upon his arrival.
18:30 - 19:00 - UTC 1/24/2017
Found one of the HV ESD power supplies alarming and screen blacked out. Tried power cycling the unit, but would instantly trip. Removed adapter card from back of power supply and re-seated. Unit was powered on, and set to "Recall 1" settings. Enabled ouput, screen showed 430V.
I turned off Supply Fan 2 at E Y since the weather is warming up and temps seem to be stabilizing (sorta).
We are seeing glitches that come in sets and look like stacks of harmonics in the most recent lock. It looks like they can be explained as the non-linear upconversion of some 1 kHz violin modes, based on the spacing between glitches. We think that these glitches happen when the violin modes get high enough to run into some non-linearity in the sensing. I used the derivative of OMC-DCPD_SUM, since it seems like these glitches should not care about the DC value and would most likely be some kind of slew-rate limit. The 1 kHz violin modes dominate the RMS, in particular a pair at 1009.44 and 1009.487 Hz. The beat frequency between these gives a period of about 21 seconds, which is the spacing between bursts of glitches. The glitches occur when the amplitude of the DCPD derivative is highest. The amplitude has a period of 21 seconds because of the two 1 kHz violin modes. Comparing to the previous lock, when these glitches were not present, the amplitude of these two modes is no higher. But there are a number of other modes near 1 kHz and several of those are substantially higher. So they may have pushed the amplitude into the nonlinear region. Attached is a PDF showing the glitches as they appear on the summary page (they are most clearly seen at 2 kHz in Omicron), and the comparison of the 1 kHz spectrum with the previous lock which did not have these glitches. The second page shows the bursts of glitches compared to the amplitude of the DCPD derivative.
If the interferometer is up I will spend some time damping them tonight.
It is important to notice that this 2kHz glitch line has been appearing and dissapearing quite irregularly in the past, but when it is present the associated Omicron glitches are of high SNR. In fact the last time this line showed up was all the way back to 29-30th November:
* 2kHz glitch line started to show on first 29th Nov lock
* 2kHz glitch line disappears on the 30th Nov
Originally I thought that the 2kHz glitch line could have been related to PCALX roaming calibration lines, based on Evan's alog on PCALX roaming calibration line frequency changes. The 2kHz glitch line seem to start as soon as the detector locked after the PCALX calibration line at 1001.3Hz was activated on UTC 2016-11-30 17:16:00, and then the glitch line disappeared around the time the cal line at 2001.3Hz was moved to 2501Hz at 2016-11-30 22:07:00. The fact that the time coincidence was not precise made me believe that the time coincidence may have been casual. It can now be confirmed that it must be unrelated because not such PCALX roaming line was not set at 2001.3Hz during the time of the current appearence of the 2kHz glitch line.
The obvious question is if the November 2kHz high SNR glitch line shows a similar 21 second spacing between bursts of glitches. The answer is yes ,as seen next during the dissapearance of the 2kHz glitch line on the 30th Nov (attached are the original images from which this image was made):
A zoom around the beginning of the above spectrum shows the ~21secs periodicity of the features:
A closer look to the 2nd harmonic violin modes for 30 mins during the time of the 2kHz glitch line (in blue) and 30 mins after the glitch line dissapears (in red) shows that only few violin modes were higher during the time of the 2kHz glitch line:
There are two cases when the blue lines are higher than the red:
* At about 1003.7Hz:
* At about 1009_4Hz:
It is clear that only the pair at around 1009.45Hz would beat with a periodicity of about 20 seconds. And in this case while the lower frequency violin mode of the pair does not change much in amplitude however it is the higher frequency violin mode of the pair which increases by 30.
I have also compared the 1009.45Hz pair peaks amplitude for 4 different cases, two of them correspond to times when the 2kHz glitch line was present, and 2 other (dashed lines) to cases when the glitch line was not present. It shows how the higher frequency line has to be high enough to cause the nonlinearity for the 2kHz glitch to be present:
Nutsinee has just now created a damping filter for this pair of violin modes, so hopefully this will be enough to avoid growth of this peaks to the point of causing appearance of the 2kHz glitch line but only time will tell.
A small note for the operation purpose: Borja's 2kHz glitches thresholds on 1009.44Hz and 1009.49Hz correspond to
8e-13 and 6.5e-13 m/sqrt(Hz) in the (dtt calibrated) CAL-DELTAL_EXTERNAL_DQ channel in November
and
7e-13 and 4.4e-13 m/sqrt(Hz) in January.
Now that the guardian is turning the damping on, these two modes should be well controlled. But if something bad happened and the modes ring up I would suggest operators to take some time to damp the mode when they get close to 1e-12 on DARM FOM.