O1 Day 41
model restarts logged for Wed 28/Oct/2015
2015_10_28 13:36 h1sysecaty1plc3sdf
restart of Beckhoff SDF node as part of testing (done when H1 out of observation mode)
TITLE: 10/29 [EVE Shift]: 23:00-07:00 UTC (16:00-00:00 PDT), all times posted in UTC STATE Of H1: Observing @ ~79 MPc OUTGOING OPERATOR: Nutsinee QUICK SUMMARY: Lights appear off in the LVEA, PSL enclosure, end X, end Y and mid X. I can not tell from the camera if they are off at mid Y. Winds are less than 15 mph. ISI blends are at 45 mHz. Earthquake seismic band is between 0.01 and 0.1 um/s. Microseism is between 0.1 and 0.3 um/s. Just reached NLN and went to Observing. The ISS second loop engaged on its own just after I got setup to engage it manually. Attached are the SDF differences I reverted (ASC DHARD loop gains and EMTY violin damping filter gains).
Forgot to run a2l.
TITLE: "10/29 [DAY Shift]: 15:00-23:00UTC (08:00-16:00 PDT), all times posted in UTC"
STATE Of H1: Lock Acquisition
SUPPORT: Evan
SHIFT SUMMARY: The ifo was locked for the most part. Few ETMY saturation alerts that seems to associate with DHARD Y amplitude increase during high wind. Kyle done with HAM3 ion pump work. Now relocking. Wind ~10mph. useism ~0.2um/s. Jeff also have asked operator to change the blend from 45 mHz to 90 mHz next time the ifo lose lock.
INCOMING OPERATOR: Patrick
ACTIVITY LOG:
15:24 Joe driving down the x arm to work on the beam tube
17:08 Joe was back, and going out again.
18:46 Joe back
20:08 Kyle to HAM3
Joe to X arm again
20:18 Kyle out. He said he did climb on HAM3. The ifo didn't lose lock.
Evan driving ETMY to find 3rd harmonics violin modes
21:00 Evan done. Back to Observing.
22:14 Joe back for the day, but Chris is still out there.
Kyle to LVEA. Switched to Commissioning.
22:38 Kyle out.
Evan wanted to do more measurement.
22:39 Lockloss.
23:00 The ifo is half way to NLN. Handing off to Patrick.
I had energized the ion pump before lunch (time?) such that the ion pump had been pumping in parallel with the pump cart for a few hours -> New ion pump is OK.
(Evan Hall, Borja Sorazu)
This is a preliminary entry on the identification of 3rd violin mode harmonics with test masses (figures and further analysis will be entered later on). This information will be used to estimate mechanical losses.
Dates and times of injections: 2015-10-28 from 20:42:30 to 20:45:00 --> Injection on ITMX
2015-10-29 from 20:20:00 to 20:32:00 --> Injection on ITMY
2015-10-29 from 20:37:30 to 20:45:00 --> Injection on ETMX
2015-10-29 from 20:53:00 to 21:00:00 --> Injection on ETMY
Injection description: Fed back, in phase, DARM to penultimate mass coil drivers in pitch after bandpassing it between 1450 to 1500Hz.
Frequency modes identification (preliminary):
ITMX:
1463.09Hz, increased by factor 4
1467.96Hz, increased by factor 4
1462.31Hz, decreased by factor 1.3
ITMY:
1461.41Hz, increased by factor 4.6
1462.03Hz, decreased by factor 4
1463.09Hz, increased slightly
1470.38Hz, decreased slightly
1470. 83Hz, increased by factor 2
1472.5Hz, decreased by factor 9
ETMX:
1463.09Hz, decreased slightly by factor 1.077
1471.93Hz, increased by factor 6.6
1474.08Hz, increased by factor 4
1475.25Hz, increased by factor 9
1478. 17Hz, increased by factor 4
ETMY:
1482.6Hz, increased by factor 1.75
1484.67Hz, increased by factor of PI
Future measurement: We will do a similar injection with DARM fedback 90 degrees out of phase.
The filters we used do do the feedback are scattered somewhat randomly, based on what SFMs were not in use for damping:
As Borja said, each of these FMs is a bandpass from 1450 Hz to 1500 Hz. They have 120 dB of gain, and we use an additional 100 dB in another filter module.
I've tracked the alignment shifts in IM2 over 7 days, and it's shifted four times.
The largest shifts for IM2 are in pitch:
The other IMs are also shifting, and I haven't looked at every event for all IMs, but for the power outage event, IM2 pitch was the largest shift at +71urad, IM1 and IM3 both had shifts in pitch of about -20urad, PRM yaw shifted -9urad, and all other HAM2 optics (IM4, MC1, MC3, PR3) pitch and yaw have shifts of 5urad or less.
The only possible conclusion is that IM2 has a problem, and likely also IM1 and IM3 have the same problem but to a lesser degree.
I noticed the LVEA average temperature started to fluctuate about half a degree after power outage on the 20th. The deeper drop only occur once every two days around 16:00-17:00 UTC (9-10 am PDT) follow by a smaller drop (~.2 degrees) that also occur every two days around the same time. Is this related to the on going heater problem at the corner station? What's special about 9-10 am? If this is the response to the temperature change outside why don't I see the rise in LVEA temperature when the outside temperature cools down in the evening?
The cooling trend is unrelated to the power outage - only coincidence. Not coincidence that it is October going on to November!
The coldest part of the morning is just before sunrise and there is some lag before the building can respond. A similar lag occurs at nightfall - the building, as well as the outdoor air and ground, have warmed all day and it takes some time for all this to cool.
Here are the zone average temperatures and a zone map of the LVEA, as well as some outside temperatures.
You can see that some zones are holding well while a few are not. The zones which are the furthest away from the air handler are also more exposed to outside conditions. The west wing of the LVEA has 3 exterior walls while the other areas do not have this same exposure. Zones 4 and 5 are relatively flat - these are the input and output areas of the IFO.
We are suffering for several reasons - some heaters are broken, and the reduced air flows required for quiet running increase the lag when responding to control signals. We may have to increase air flow to get a more uniform response in the LVEA. We can do this by turning on our 4th fan (preferred) or by increasing flow rates of the three which are running.
TITLE: "10/29 [DAY Shift]: 15:00-23:00UTC (08:00-16:00 PDT), all times posted in UTC"
STATE Of H1: Observing at ~ 80Mpc
OUTGOING OPERATOR: Ed
Had a bit of a trouble with PRMI. I trended some of the optics witness and oplev channels and BS remained relatively unchanged after the lockloss but PRM was kicked. So I touched only PRM pitch and yaw to get PRMI to lock while watching POP90 and POP18. I accepted SDF changes for the second harmonics ETMY violin mode damping gain and the new ASC DHARD gains that were introduced to fix the loop instability. I was out of Observing shortly to fix the mistake I made with the SDF (confusion between revert and accept).
Some DRMI and PRMI info for Sheila:
I requested at 15:11:07 UTC. After waiting at least 10 minutes I adjusted PRM to maximize POP90 and 18. Once POP90 flashed close to 40 counts I let Guardian took over. And after waiting another 5 minutes PRMI locked successfully. After I maximize POP90 and 18 during the PRMI locked I re-requested LOCK_DRMI_1F and it locked almost imediately.
This trip appears to have occured twice with the lock dropping on the first one. The second was about 20 minutes later. The watchdog registers tconvert 1130165885 Oct 29 2015 14:57:48 UTC. The guardian first detected a problem at:
2015-10-29T14:38:52.53819 ISI_ITMY_ST2 [HIGH_ISOLATED.enter]
2015-10-29T14:57:49.03752 ISI_ITMY_ST2 [HIGH_ISOLATED.run] USERMSG: SETPOINT CHANGES. see SPM DIFFS for differences
After the WD detected a problem so it doesn't look like the guardian should be suspected.
Looking through the Coil Driver BIO channels, the Stage2 V3 CD status is the only element that registers a change. The first attached 1 hour plot shows the BIO signals, the lock loss, and the stage3 coil driver outputs. I don't see the V3 response any different than the V1 or V2 so nothing clear there. The shift to zero is expected as the DC position is reset to zero offset at each Isolation. The second 10 day plot shows this drive reset to zero each time and then slowly drift away from zero during the Isolated period.
Next I'll look through the model for meaning and why.
Here are Stage2 Coil Driver Voltage and Current spectra from 3 & 1/2 hours and 10 minutes ago. Reference traces are before the trip. Again, no obvious problem revealed in the current traces.
Looks to me like the coil driver sent a bad status which tripped the watchdog. Hopefully others can weigh in on this too.
First is the V3 BIO Status (what tripped the WD) and the OUTF Output fro V3. Little hard to tell who/what happened first. The X axis is zoomed in, the status is bad for less than 1 second. As we understand it, this requires a button reset to unlatch. No one was in the CER at the time.
On the second plot here is a zoom into full data. There are the Coil Driver Monitor channels and the WD state. I've scaled the Current and the WD amplitude to better compare times. I've also zoomed into the X axis--this is less than 1 second of data. It sure looks like the Watchdog state change leads Coil Driver Changes. Hope to get some more info from Caltech EE.
TITLE: Oct 29 OWL Shift 7:00-15:00UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Observing
SUPPORT: N/A
LOCK DURATION: Entire shift
INCOMING OPERATOR: Nutsinee
END-OF-SHIFT SUMMARY:
IFO still locked at 78Mpc. There was a slight downtrend in range to ~65Mpc. Wind Calm. Sei and µSei remain the same. GraceDB External Notification script failing/recovering with frequent regularity. Lockloss at 14:34. ITMY Stage2 Coil Driver BIO.
SUS E_T_M_Y saturating (Oct 29 10:47:01 UTC)
SUS E_T_M_Y saturating (Oct 29 10:47:04 UTC)
SUS E_T_M_Y saturating (Oct 29 12:00:10 UTC)
SUS E_T_M_Y saturating (Oct 29 12:01:37 UTC)
SUS E_T_M_Y saturating (Oct 29 12:01:40 UTC)
ACTIVITY LOG:
ISI ITMY stage 2 watch dog tripped (Oct 29 14:34:47 UTC)
DRMI Unlocked (Oct 29 14:34:47 UTC)
Intention Bit: Commissioning (Oct 29 14:34:47 UTC)
ISI ITMY stage 2 watch dog tripped (Oct 29 14:57:49 UTC)
ISI ITMY stage 2 watch dog tripped (Oct 29 14:57:49 UTC)
I stopped in DC readout and Evan and I went into the LVEA to look for the flashing light that Kyle reported (alog 22931). We think we may have located it somewhere under HAM4 but could not determine the source without crawling under the output beam tube. So we left it for later investigation. I also found a phone near the PSL enclosure with the line still connected. The power was unplugged but I unplugged the line from the wall as well. I also unplugged the extension cord from the wall that the phone had been unplugged from. We also heard what sounded like a drop of water falling in a cave. The light was on in the CER and we turned it off on the way out. We noted that some of the light from the CER leaks into the LVEA through an opening in the wall for a cable tray. We were in there from around 00:51 UTC to 01:14 UTC. Jim and Dave turned off the wireless access point that was turned on for Kyle. (alog 22931) I had to engage the ISS second loop by hand. I had to restart the GraceDB query script. Jenne ran the a2l script. I accepted in SDF the alignment offsets that Cheryl made to MC2. (alog 22933) (see first attached) I reverted the SDF differences for the H1:SUS-ITMY_L2_DAMP_MODE8 and H1:SUS-ITMX_L2_DAMP_MODE5 filter modules (see attached). Jenne and I claim this is alright because the gain on these is 0. Could someone tell us if they changed these and why? Unfortunately LLO went down from a power glitch just as we came back. We went into observing at 02:01 UTC. The range is around ~ 79 MPc.
Also note that a pump cart is running by the east door of HAM3.
It seems to me like somebody was making a filter for violin mode third harmonics. Dave also reported that sombody was making changes to the filter module the same day.
Plot 1: HAM3 rotation in Z, MC2 ptich and yaw, PR2 pitch and yaw, OMC DC shoing lock loss.
HAM3 rotation in Z is the biggest rotation signal.
MC2 shifts -13urad in pitch, and -28urad in yaw
PR2shakes but comes back to within 1urad of where it started.
Plot 2: All IMC optics and Recovery
MC1 and MC3 see a change at lock loss, and then some motion at relocking the IMC, but are unchnged, which we knew since they're in HAM2.
MC2 needed to move +50urad in pitch and an additional -12urad in yaw to recover the IMC with good power at MC2 TRANS.
Overall MC2 was moved +37urad in pitch and -40urad in yaw.
The fact that MC1 and MC3 positions are unchanged after recovering the IMC shows that the change of MC2 alignment, while big, did moved MC2 to match the previous MC1 and MC3 alignments.
Plol 3: HAM3 HEPI and ISI signals that I believe show that their alignment didn't change, though I'm not completely confident that I got the right channels to show this.
Those are good SEI channels to look at. All but the HEPI Pringles are servo'd to DC so something would be wrong if they weren't the same. There is some hysterisis in the HEPI though as it is overconstrained. Those 'extra' dofs are the pringles and are not servo'd at DC hence the trend seen in the HP channel. This is expected to have very little impact on the final ISI table position.
Due to the violin mode problem on 10/25, Sheila has asked me to investigate when this mode really started to rung up. The first plot attached shows that the amplitude of 1008.45Hz were consistant the day before the power glitch and three hours before power glitch (the small difference you see is within the mode normal fluctuation range). The second plot shows the 1008.45 Hz got rung up by an order of magnitude during the first lock acquired after the power glitch just like others. Because this mode didn't have a damping filter at the time, ideally the amplitude should have stayed where it was. However, the final plot shows that the amplitude became worse as time progress while other modes were either stable or being damped until it caused the problem on October 25th. Could anything that happened during the power lost caused the mode to change its phase as it seems to be slowly rung up by ETMY MODE3 that's been existing since before O1? Note that this violin mode had never rung up before. The investigation continues.
To ensure that the 1008.45Hz line hasn't been slowly ringing up all this time, I've looked back at the asd amplitude of this mode until October 1st. First plot attached shows the amplitude/sqrt(Hz) versus frequency of this particular mode, one plot per day. The second plot shows log amplitude versus time. I only plotted one data point per day (10:00:00 UTC if data available, or any time where the BNS range was stable and the ifo was locked for at least an hour). The last data point is today (10/28 02:00:00 UTC). This mode has been fluctuating between 1e-22 and 1e-21 since the begining of the month (10/01) up until 10/20. You can see clearly that the amplitude begins to rise above its nominal on 10/21 after the power outage on 10/20 and continues to grow exponentially until it started to cause problems on 10/25. Indicates that the amplitude grow was causing by a positive feedback, which Sheila found it to be ETMY MODE3.
To conclude this study: This mode wasn't, and hasn't been ringing up before October 20th. Why it started to ring up after power outage is unclear. I can't think of anything else but something must have changed to cause this mode to change phase during the power outage.
Should we worry...?
Was there a significant temperature excursion during the power outage?
Yes.
I've attached the plot of average temperature in the VEAs. After the power outage LVEA average temperature had three big dips of about half degree. Average temperature at EY seems to fluxtuate more often and EX had couple of large drops.
Which turns out to be just a coincidence with the power outage according to John.
