Title: 11/21 Day Shift 16:00-24:00 UTC (8:00-16:00 PST). All times in UTC.
State of H1: Locking
Shift Summary: Locked almost my entire shift. Lockloss at 23:55 UTC likely due to 2 EQs with similar arrival times.
Incoming operator: Patrick
Activity log: None
Locked in Observing for the past ~18 hours. There have been 12 ETMy saturations since the start of my shift, but none seems to correlate with any RF45 glitching as none has been seen.
The psinject process running on h1hwinj1 have stopped several times recently. The monit program on this machine then restarts the process and reports this to sysadmins. The latest stop-start was at 09:34 PST this morning, and I confirmed that the CW excitation was absent for 34 seconds at this time.
Here is a list of all the restarts for the month of November as reported by monit. Some may have been due to maintenance, and H1 may not have been in observation mode for every event.
Sat, 21 Nov 2015 09:34:39
Sat, 21 Nov 2015 00:45:22
Fri, 20 Nov 2015 16:10:05
Wed, 18 Nov 2015 13:58:44
Wed, 18 Nov 2015 05:00:26
Tue, 17 Nov 2015 03:05:18
Tue, 17 Nov 2015 00:01:10
Sat, 14 Nov 2015 07:05:26
Fri, 13 Nov 2015 00:00:34
Sun, 01 Nov 2015 00:00:22
we are investigating the reason why psinject is stopping (FRS4019) https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=4019
I talked with Jeff and Travis and we decided to increment heat as we are running out of range in the FMCS controls. Outdoor temps have fallen to ~18F overnight.
In the LVEA - incremented HC1B from 9 to 10 ma (control signal) at 8:16 PST
EX - incremented HC3 from 10-11 ma at 8:18
EY - incremented HC3 from 10-11 ma at 8:19.
Edit - incremented EX HC3 from 11- 12ma at 8:33 for some additional heat as response was not sufficient.
Activity Log: All Times in UTC (PT) 08:00 (00:00) Take over from TJ 11:44 (03:44) ETM-Y and large spike in RF45 13:21 (05:21) End-X Low Temp Alert (3 more ending at 13:52 (05:52)) 15:55 (07:55) CS_LVEA_REHEAT_4_DEGF alarm 16:00 (08:00) Turn over to Travis End of Shift Summary: Title: 11/21/2015, Evening Shift 08:00 – 16:00 (00:00 – 08:00) All times in UTC (PT) Support: None needed Incoming Operator: Travis Shift Summary: IFO in Observing mode for past 13.5 hours. Range has been on average 80Mpc during this time. Although a bit on the cold side, environmental factors remain favorable with wind and seismic activity promoting good observing conditions. Exceptions: Several ETM-Y saturations. One at 11:44 (03:44) had a large spike in RF45 occur at the same time. There were 4 low temperature alerts at End-X. At 15:55 (07:55) CS_LVEA_REHEAT_4_DEGF alarm. Spoke with John W., he is going to increase the heat to corner and End –X.
O1 day 64
model restarts logged for Fri 20/Nov/2015 No restarts reported
13:21 & 13:28 & 13:37 (05:21 & 05:28 & 05:37) Receiving low temperature alerts at End-X. The outside air temperature at End-X is hovering around 19F, so perhaps this is not unexpected. Fan 1 Mixed Air Temp is below that of Fan 2 Mixed Air Temp by a couple of degrees. General room air is averaging to 66.96F. Do not see a smoking gun, but do not have a good idea what the "normal" VEA temperature should be on this coldest morning of fall. Posted is a snapshot of the FMCS screen for End-X.
A good first half of the Owl shift. IFO in Observing mode for the past 9.75 hours, with a range around 80Mpc. Seismic is coming down from a mag 6.1 EQ in Indonesia, and is currently around 0.01um/s. Microseism is centered around 0.3um/s. Wind is still a calm to gentle breeze (0-7mph). There have been 10 ETM-Y saturation events thus far. At 11:45 (03:45) there was an ETM-Y saturation associated with sharp spiked glitch in RF45.
Title: 11/21/2015, Evening Shift 08:00 – 16:00 (00:00 – 08:00) All times in UTC (PT) State of H1: 08:00 (16:00), The IFO locked at NOMINAL_LOW_NOISE, 22.2w, 80Mpc. Outgoing Operator: TJ Quick Summary: IFO has been in Observing mode for the past 6 hours. Environmental conditions are good – wind is a calm to light breeze (0-7mph) seismic activity is well below 0.1 um/s and microseism is centered around 0.3 um/s.
TITLE: 11/21 EVE Shift: 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC
STATE Of H1: Observing for 5.5hrs at 79Mpc
SHIFT SUMMARY: Lost lock once, still not sure why, but other than that it was smooth sailing.
INCOMING OPERATOR: Jeff B
ACTIVITY LOG:
Observing at 02:32 UTC
Locking DRMI took around 35min. It would start to catch it for a second and then lose it. This happened a handful of times before I went to just lock PRMI. PRMI locked very quickly and needed minimal adjustment. Went back to DRMI and it was the same story, and alignment looked good. Eventually it locked and stayed lock.
There was an odd SDF difference before I went back to Observing. H1:LSC-TRIG_MTRX_2_8 was 1.0 but was saved at 0.0. This was the ASAIR_B_RF18 leading to MICH_TRIG. Trending it showed that it has been 0 for the last few days. I didn't see anything in the alog. So I just put it back to 0. Not sure why it would have suddenly changed...
Lockloss at 01:10 UTC
Cause: Unknown (for now)
TITLE: 11/20 [DAY Shift]: 16:00-24:00 UTC (08:00-16:00 PDT), all times posted in UTC STATE Of H1: Locked at NLN SHIFT SUMMARY: Out of observing while Jeff B. went to end Y. Lost lock when Keita and Evan removed the RF AM monitor coupler from the EOM driver cable. Switched blends to Quite_90. Had to move TMS X and Y to optimize green. Had to move BS and PRM to lock on DRMI. Lost lock on SWITCH_TO_QPDS. Locked on NLN. SUPPORT: Jenne, Evan, Keita, Jim W. INCOMING OPERATOR: TJ ACTIVITY LOG: 16:12 Jeff B. to end Y to take pictures in receiving area and hallway to VEA, out of observing 16:18 Chris and Joe to X arm beam tube enclosure 16:39 Jeff B. back, back to observing 17:24 Bubba reports that the landscapers are bailing tumble weeds around the highbay 17:32 Landscapers truck driving back toward kitchen 18:20 Gerardo to mid Y to pick up power supply and then to X28 18:23 Karen to mechanical building to get coffee cups 18:36 Fire department through gate, notified Richard 18:38 Kyle to X28 18:38 Fire department to mid X to check on RFAR Fire department back (forgot to note time, before 19:39) 19:54 Joe back, Chris on his way back 20:12 Kyle and Gerardo back 21:32 Evan and Keita into the H1 PSL enclosure to remove the RF AM monitor coupler from the EOM driver cable, out of observing 21:33 Kyle and Gerardo back to X28 21:38 Joe and Chris back to X arm beam tube enclosure 22:12 Evan and Keita done Put ISI blends on Quite_90 Had to move TMS X and TMS Y to increase green power in both arms even though WFS were engaged Went to LOCK_PRMI and misaligned PRM. Aligned the BS on flashes. Realigned PRMI and was able to lock PRMI. Moved mostly PRM to increase power buildup. Requested DRMI_LOCKED. Transition from LOCK_PRMI to DRMI_LOCKED failed. Eventually DRMI locked. 23:13 Joe and Chris back 23:38 Kyle and Gerardo back
Observing @ 00:19 UTC
TITLE: "11/20 EVE Shift: 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC"
STATE Of H1: Nominal Low Noise, need to go through checks before Observing.
OUTGOING OPERATOR: Patrick
QUICK SUMMARY: Wind low, seismic low, blends on 90s
Because the microseism is down to a level similar to what we saw over the summer, I asked Patrick to put the ISI's in the more wind tolerant 90 mhz blends. Wind is also low right now, but it can come up suddenly, while microseism takes days to rise. We also need more data about what conditions necessitate switching. After, Evan and Keita finished in the PSL, Patrick started relocking but was having troubles with the green. Fiddling the TMS alignments fixed it, I looked to see if the ISI positions had changed. They haven't moved but I found something interesting.
Most of the ISIs all showed a relatively smooth switch from the 45 to 90 blends except for ETMY. ITMY is the first plot and is pretty consistent with all the other chambers, except ETMY. Patrick switched the X&Y blends at the same time , in the middle of the time span I grabbed in the plot. You can kind of tell because the ITMY location mon range gets a little smaller and there are fewer low frequency swings on X&Y, due to the lower gain peaking at higher frequency of the 90 blend. The other DOFs don't seem to see anything.
ETMY however shows a huge 30 micron shift in the Y direction (second plot), and is visible in all DOFs. This chamber was only running the 45mhz blend in the Y DOF, so only the Y blend got switched on this chamber, at about 21:48 UTC today. No idea why it should be any different from the other chambers, although BrianL did note a while ago that the ETMY STS is poorly centered (maybe relevant because the STS is used for sensor correction, so gets summed with the CPS signal before the blend). It would be good to get some time to see if this is repeatable and truly limited to ETMY or not.
For some discussion about how to fix this, see the SEI log, entry 887. https://alog.ligo-la.caltech.edu/SEI/index.php?callRep=887 short answer - 1) SEI team needs to update the blend-switch code to wait longer during the switch process so that the transients can settle down. 2) don't try to change the blend filters while the Tidal-offload is pushing hard on HEPI - wait at least 200 seconds after it finishes. 3) SEI team needs to have less tilt on HEPI when it gets moved by the tidal offload.
While trying to understand and deal with the Beckhoff SDF diffs in OBSERVE mode, I stumbled on a weird channel that toggled states around 4pm local last Wed Nov 11 (01 UTC WED 11th). (Note, this occured well before the windy troublesome locking period when we saw a power glitch on Tuesd Wed 17th.) If the VCO Frequency Servo is set to "Int" (see the center of the medm pic attached below), do we care what this EXTFREQUENCYOFFSET channel is doing? And where is the medm for this EXTFREQUENCYOFFSET channel anyways? - Even with commissioners, we couldn't immediately find it... And, and, what's with the glitch in the middle of the 20 day trend on Wed 11th?
Not sure but it might be related to this entry. https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=23293 Perhaps the wrong button was pushed? Or if there were problems encountered re-locking the input modecleaner after installation of the 45 MHz RFAM hardware, this may have been flipped to see if the input modecleaner would lock.
As a follow-up to Hartmut's question about the DARM rms, here are spectra of the in- and out-of-loop DARM sensors before and after transitioning to DC readout (at 2 W input power). The spectra are calibrated into RIN.
The left-hand plots show the sensors when DARM is controlled via AS45Q. Above 10 Hz or so, we see the sensing noise of AS45Q impressed onto the DARM loop. From 2 Hz to 8 Hz or so, and below 0.5 Hz, both sensors appear to see some DARM displacement noise. Between 0.5 Hz and 2 Hz, the sensors are not seeing the same signal. If the AS45Q signal in this region is indeed DARM displacement noise (rather than some kind of loop-suppressed sensing noise), this would seem to indicate that the sensing noise of the DC readout chain is higher than the intrinsic DARM displacement noise by a factor of a few in this region.
The right-hand plots show the sensors when DARM is controlled via DCPD sum. During the transition, an additional boost is engaged (two complex poles at 2 Hz, two complex zeros at 8 Hz), so this makes it tricky to directly compare these plots with the left-hand plots. Probably it is best to try to get an equivalent set of plots for the intermediate case (DARM controlled by AS45Q, boost engaged).
Attached is the same plot as before, but this time both configurations have the same OLTF (boost on).
Times:
Attached are plots of the ETMs moving in verticalwith the temperature swing.
And the corner station BS and ITM suses...