TITLE: 08/01 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 52Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
Wind: 8mph Gusts, 6mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.10 μm/s
QUICK SUMMARY: locked and in Observe
TITLE: 08/01 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 53Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Mostly quiet
LOG:
0:10 NLN, with violin mode damping left off, per Sheila's log
0:50 IFO went to commissioning for no reason I could find. IFO guardian only showed some spm diffs on SEI ETMX(that had been there all along), but there were no SDF diffs. No one had touched anything.
Switched "carrier off" on the 2 ifr units at around four in the afternoon.
J. Kissel
Operators have been stifled with junk (bad/absent calibrations / wrong sensors / poor plotting) templates for violin modes that have been linked to the violin mode MEDM screen:
/opt/rtcds/userapps/release/isc/h1/scripts/
V_mode_hunting_NK_2nd.xml
V_mode_hunting_NK_3rd.xml
V_mode_hunting_NK.xml
for quite some time. I took the time this evening to
- Changed the start frequency of the template to be 0 Hz (where they had been set to 490 / 900 / 1400, i.e. the lower band of interested for each template). Maybe this is an old commissioner's tale, but I recall that if you don't set the start frequency to zero, then your ASD results get distorted.
- Restored the calibration of DELTAL_EXTERNAL so that it (a) is correct, and (b) matches the front wall. This was done by loading in the transfer function calibration from
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O2/H1/Scripts/ControlRoomCalib/caldeltal_calib.txt
- Added the OMC DCPDs to all templates, added advice on what the results need to be in order to use DC READOUT.
- Where possible (i.e. only on in the 500 Hz fundamental template), I plotted ASC_AS_C raw segments to check those for saturation. Added similar advice about results in order to use ENGAGE_SRC_ASC.
The templates have also no been committed to the svn, where they hadn't been before.
We have left the violin mode damping off for one lock, with the intention of measuring violin mode Qs to compare to what was measured in 23383. We had the damping off starting at 00:05:00 UTC, and will leave it off for the rest of this lock.
ETMY MODE 7 rang up, and possibly ETMY MODE 2. If this data is unusable, we can try this again but leave all the damping on for ETMY.
Following an e-mail exchange between Jeff K and Brett Shapiro about alog 37847, Daniel, Thomas Vo and I tried to follow Brett suggestion to swing ITMY and see if we could reverse the hysterisis.
First we took the ISI to isolated damped, then set the top mass damping gains to 0. Daniel applied a 0.3 Hz excitation to M0 OPTICALIGN_P until the top mass was swinging with an amplitude of more than 200 urad according to the test mass osems (the sift we saw was about 600 urad). He slowly ramped this down over 5 minutes, we let the suspension swing for a few more minutes. We don't see any shift in the pitch of the optic before and after this test.
The second attachment shows the same channels during the EQ. The motion durring the EQ was much larger than what we applied with this test, so it might be that we need to try swinging harder.
Here is the suggestion from Brett's e-mail:
If you imagine that the hysteresis is coming from dislocations in the wires all moving to one side of the wire or the other (front or back), then letting the quad swing and ring down slowly causes the dislocations to spread out evenly, washing the hysteresis away. But if the quad tips a lot, but rings down very fast, like from an earthquake causing it to bang hard into the stops, then the dislocations may pile up on one side, causing a static pitch. So the hysteresis plot I attached is really a worst case.
A simple thing to do to check if this hysteresis thing is the culprit is to get the pendulum swinging in pitch, with an amplitude at least as large as the offset you see (if possible). Then let it ring down without damping. If the offset goes away or gets smaller, then this was it. If not, then it must be something else.
A minor correction on what I said, which Sheila already caught, is that you would want the swing to be at least as big as the motion caused by the earthquake, not the remaining offset. If this causes the pendulum to bang into the stops, than it may not be feasible to do a proper swing.
Changed high frequency Pcal line at ENDX to 3501.3 Hz and reduced the amplitude from 40,000 cts to 35000 cts.
I've reduced the gain of CSOFT pit by 3dB, to reduce the gain peaking at 2.8 Hz, which was seen to cause some glitches (see comments and alog 37705)
The RMS control signal is reduced by about a factor of 2. The attached OLTF was measrued before the gain reduction.
I've updated the FRS Ticket 8591 to reflect these changes. @DetChar -- let us know if this helps!
1334 - 1357 hrs. local time This completes WP #7095
at around 18:00:17 UTC today both ALS lasers turned off, within seconds of each other. Just now (around 20:10 UTC) I caused a lockloss. Heading to end stations to restart.
J. Kissel, for S. Dwyer, T. Vo, and J. Bartlett
Recovery Note / Instructions:
- The lasers did fail at the same time (within a second).
- Drove to end stations (Thomas to one, Sheila to the other), walked into X/YVEA and hit the "reset" button on the laser controller (Thomas cycled the interlock key out of good practice, but Sheila did not, as it was not needed), laser turned on without problems.
- Once starting lock-acquisition again, found that arms would not lock on green. Chased the Beckhoff errors on the ALS screen to find that the X arm PLL was not locking.
- Upon turn on, one has a 50/50 chance of the ALS laser frequency (H1:ALS-X_LASER_HEAD_CRYSTALFREQUENCY) being on the "wrong side" of the PSL laser frequency, making the PLL unable to to lock the beatnote (H1:ALS-X_FIBR_LOCK_BEAT_FREQUENCY) at 40 MHz. We were unlucky on ALS X. To fix:
- hit "disable" on the PLL locking (H1:ALS-X_FIBR_LOCK_LOGIC_ENABLE),
- manually move around the crystal frequency until the beat note is within 10s of MHz of 40 MHz,
- enable to the PLL.
If it doesn't work, you're likely on the wrong side of the PSL frequency.
- Try bringing the beat note to zero and "come up from the other direction" with the crystal temperature, as the beat note as recorded is an absolute value.
- enable PLL again.
Still no evidence for why the lasers turned off, likely won't investigate.
FRS Ticket 8645 corresponds to this issue.
Jeff B, Sheila, Jim W by phone
I came in to do some commissioning. Jeff B was starting to work on violin 1st harmonics while I planned to some ESD measurements so we took the IFO out of observing. Jeff B made a modest increase in the violin damping gain the moment before the IFO unlocked and both end station ISIs and HPI's tripped.
This seems to be a coincidence, not some sort of violin mode/pump controller conspiracy. The attached plot shows EY pump pressure goes to not OK about one second before EX pump pressure goes to not OK, which was about one second before Jeff changed the gain for ETMX MODE10 (with a 10 second ramp). I am not sure if we can trust the timing of the pump controller channels relative to the front ends.
With advice from Jim W on the phone, and using instructions from 37359 we are bringing back the pump controllers.
Opened (and will shortly close) FRS Ticket 8644 to add statistics to the common failure mode.
On 2017-07-27, during the lock loss around 2 UTC, I noticed a sudden increase in kappa_tst from about 1.05 to 1.07. See the summary page for that day: https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20170727/cal/time_varying_factors/ It is likely correlated with the end reaction mass being moved at that time: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=37800 LLO observed a similar issue, reported here: https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=31108
Attaching plot for easier reference. Links to aLOGs: Documentation of the reaction mass move: LHO aLOG 37800 Documentation that LLO has seen this as well: LHO aLOG 31108
Plots showing changes in both kappa_tst and ETMY Pitch (at the same time).
Sheila, with many conversations with Keita, Jeff K, and others
We have several reasons to believe that something changed in our suspensions durring the Montana EQ. (See alog from Beverly Berger and Josh Smith, (37775) which got us started looking at this, and Cheyl's log about the large triples, (37674). We are still looking at some of the data from alignment sensors, but here arer some things we can say:
There is more to be done checked on here, for example, checking if this has happened at any other times during the run (I checked one large EQ, I see no shifts like this), checking yaw (which has much smaller shifts than pitch), checking the triples, and looking at the alignments of the reaction chains. Can we interpret the information we have to make a gues aout what might have changed in the suspension? Wire slipping or some kind of damage to the prisms are some things we have been thinking about.
Hysteresis is a possibility here. We discovered on the LASTI quad during one of the early builds that these suspensions have significant hysteresis in pitch. That is, if you tip the stages a given amount, they will not come back all the way, leaving you with a pitch offset. The attached plot shows a measurement of this effect from LASTI, showing a pretty standard looking hysteresis plot. We learned that you can 'undo' any offsets by getting the pendulum swinging, and letting it slowly damp itself. The slower the ringdown, the closer it returns to its nominal 'equilibrium' position.
The offsets you see here don't look any bigger than what we saw there, though granted we were trying to measure the effect, so pushed it pretty far. Then again, we didn't have any major earthquakes either.
There were many documents written to investigate what we saw at LASTI. Mark Barton's document, T0900103, includes a list of most if not all of them.
So it could be that this earthquake induced some hysteresis offset, or perhaps there was an offset already and the swinging motion from the earthquake removed it. Anyway, try swinging the pendulum in pitch with some large *but safe* amplitude, and you should return to the nominal 'equilibrium' position, if it isn't already there.
Looking at Beverly log (37775) that shows DC changes in the pitch offset across the earthquake time. Are the changes in the pitch in the lower masses compatible with the reported change at the top mass?
Here are some additional plots, for those who are interested in what happens to the osems between the reaction mass and the top mass. I also have plots that show torque applied to the reaction mass vs measured pitch, these aren't very useful because we don't change the torque applied to the reaction mass, but they do show that there were similar shifts in the reaction chain. In order to interpret the data from the L1 and L2 osems we will need to account for shifts in the reaction chain.
Posting a jpeg version of the LASTI hysteresis plot above, since the pdf was causing issues.
Also, here is a summary of the procedure I used to make the plot way back in 2008:
"These data points are separate pushes and releases. The procedure was to put the top mass on its stops with the rest of the chain suspended. Note, the quad was on the test stand outside the chamber at the time. Then the top mass stops were used to tip the top mass some amount in pitch. The angle of the test mass, with the top mass still tipped, was measured with either an autocollimator or optical lever. That test mass angle is the X axis in the plot, called ‘Input Pitch’. Then the top mass was released slowly to avoid oscillation, and the test mass pitch angle was recorded again. That value is the Y axis, called ‘Output pitch’. This process was repeated for successively larger and larger Input pitch values, until I was afraid to tip the suspension any more. I then started to tip the suspension in the other direction until I was again afraid to tip the suspension any more. And finally, to close the hysteresis loop, I repeated some of the data points along the original tipping direction."
J. Kissel I'm behind on my documentation as I slow process all the data that I'm collecting these days. This aLOG is to document that on this past Tuesday (2017-07-25) I took standard top-to-top mass transfer functions for the Triple SUS (BS, HLTS, and HSTS; 10 SUS in total), as I've done for the QUADs (see LHO aLOG 37689 and associated comments). I saw no evidence of rubbing during the act of measurement, but I'd like to confirm with a thorough comparison. As such, I'll post comparisons against previous measurements, other suspensions, and the appropriate model in due time. This leaves: 3 doubles, 9 singles. Data is stored and committed here: /ligo/svncommon/SusSVN/sus/trunk/BSFM/H1/BS/SAGM1/Data/ 2017-07-25_1501_H1SUSBS_M1_WhiteNoise_L_0p01to50Hz.xml 2017-07-25_1501_H1SUSBS_M1_WhiteNoise_P_0p01to50Hz.xml 2017-07-25_1501_H1SUSBS_M1_WhiteNoise_R_0p01to50Hz.xml 2017-07-25_1501_H1SUSBS_M1_WhiteNoise_T_0p01to50Hz.xml 2017-07-25_1501_H1SUSBS_M1_WhiteNoise_V_0p01to50Hz.xml 2017-07-25_1501_H1SUSBS_M1_WhiteNoise_Y_0p01to50Hz.xml /ligo/svncommon/SusSVN/sus/trunk/HLTS/H1/PR3/SAGM1/Data/ 2017-07-25_1507_H1SUSPR3_WhiteNoise_L_0p01to50Hz.xml 2017-07-25_1507_H1SUSPR3_WhiteNoise_P_0p01to50Hz.xml 2017-07-25_1507_H1SUSPR3_WhiteNoise_R_0p01to50Hz.xml 2017-07-25_1507_H1SUSPR3_WhiteNoise_T_0p01to50Hz.xml 2017-07-25_1507_H1SUSPR3_WhiteNoise_V_0p01to50Hz.xml 2017-07-25_1507_H1SUSPR3_WhiteNoise_Y_0p01to50Hz.xml /ligo/svncommon/SusSVN/sus/trunk/HLTS/H1/SR3/SAGM1/Data/ 2017-07-25_H1SUSSR3_M1_WhiteNoise_L_0p01to50Hz.xml 2017-07-25_H1SUSSR3_M1_WhiteNoise_P_0p01to50Hz.xml 2017-07-25_H1SUSSR3_M1_WhiteNoise_R_0p01to50Hz.xml 2017-07-25_H1SUSSR3_M1_WhiteNoise_T_0p01to50Hz.xml 2017-07-25_H1SUSSR3_M1_WhiteNoise_V_0p01to50Hz.xml 2017-07-25_H1SUSSR3_M1_WhiteNoise_Y_0p01to50Hz.xml /ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/ PR2/SAGM1/Data/2017-07-25_1607_H1SUSPR2_M1_WhiteNoise_L_0p01to50Hz.xml PR2/SAGM1/Data/2017-07-25_1607_H1SUSPR2_M1_WhiteNoise_P_0p01to50Hz.xml PR2/SAGM1/Data/2017-07-25_1607_H1SUSPR2_M1_WhiteNoise_R_0p01to50Hz.xml PR2/SAGM1/Data/2017-07-25_1607_H1SUSPR2_M1_WhiteNoise_T_0p01to50Hz.xml PR2/SAGM1/Data/2017-07-25_1607_H1SUSPR2_M1_WhiteNoise_V_0p01to50Hz.xml PR2/SAGM1/Data/2017-07-25_1607_H1SUSPR2_M1_WhiteNoise_Y_0p01to50Hz.xml PRM/SAGM1/Data/2017-07-25_1607_H1SUSPRM_M1_WhiteNoise_L_0p03to50Hz.xml PRM/SAGM1/Data/2017-07-25_1607_H1SUSPRM_M1_WhiteNoise_P_0p01to50Hz.xml PRM/SAGM1/Data/2017-07-25_1607_H1SUSPRM_M1_WhiteNoise_R_0p01to50Hz.xml PRM/SAGM1/Data/2017-07-25_1607_H1SUSPRM_M1_WhiteNoise_T_0p01to50Hz.xml PRM/SAGM1/Data/2017-07-25_1607_H1SUSPRM_M1_WhiteNoise_V_0p01to50Hz.xml PRM/SAGM1/Data/2017-07-25_1607_H1SUSPRM_M1_WhiteNoise_Y_0p01to50Hz.xml SR2/SAGM1/Data/2017-07-25_1715_H1SUSSR2_M1_WhiteNoise_L_0p01to50Hz.xml SR2/SAGM1/Data/2017-07-25_1715_H1SUSSR2_M1_WhiteNoise_P_0p01to50Hz.xml SR2/SAGM1/Data/2017-07-25_1715_H1SUSSR2_M1_WhiteNoise_R_0p01to50Hz.xml SR2/SAGM1/Data/2017-07-25_1715_H1SUSSR2_M1_WhiteNoise_T_0p01to50Hz.xml SR2/SAGM1/Data/2017-07-25_1715_H1SUSSR2_M1_WhiteNoise_V_0p01to50Hz.xml SR2/SAGM1/Data/2017-07-25_1715_H1SUSSR2_M1_WhiteNoise_Y_0p01to50Hz.xml SRM/SAGM1/Data/2017-07-25_1814_H1SUSSRM_M1_WhiteNoise_L_0p01to50Hz.xml SRM/SAGM1/Data/2017-07-25_1814_H1SUSSRM_M1_WhiteNoise_P_0p01to50Hz.xml SRM/SAGM1/Data/2017-07-25_1814_H1SUSSRM_M1_WhiteNoise_R_0p01to50Hz.xml SRM/SAGM1/Data/2017-07-25_1814_H1SUSSRM_M1_WhiteNoise_T_0p01to50Hz.xml SRM/SAGM1/Data/2017-07-25_1814_H1SUSSRM_M1_WhiteNoise_V_0p01to50Hz.xml SRM/SAGM1/Data/2017-07-25_1814_H1SUSSRM_M1_WhiteNoise_Y_0p01to50Hz.xml
More detailed plots of BS, compared against previous measurements and model. We see perfect agreement with model and previous measurement, so this SUS is definitely clear of rubbing.
More detailed plots if PR3 and SR3. Both are clear of rubbing. The new measurements agree with old measurements of the same suspension, the model, and other suspensions of its type. PR3's L2L transfer function shows "extra" unmodeled resonances that were not there before, but they line up directly with the Y modes. This is likely that, during the measurement the Y modes got rung up, and the power is so large that it surpasses the balance the of the sensors, so they're not subtracted well. I can confirm that these frequencies are incoherent with the excitation, and we've seen such inconsequential cross coupling before. Nothing about which to be alarmed.
More detailed plots of PRM, SRM, and SR2 compared against previous measurements and model. We see good agreement with model and previous measurement, so these SUS are clear of rubbing. There is a subtle drop in response scale factor for all of these suspensions (and in retrospect it's seen on the other SUS types too), and I suspect this is a result of the OSEMs LEDs slowly loosing current over the series of measurements, see attached 4 year trends.
While PR2 shows all resonances are in the right place, there is a suspicious drop in scale for the L and Y DOFs with respect to prior measurements. However, this is the first measurement where we've measured the response with the nominal alignment offsets needed to run the IFO (!!). These DOFs (L and Y) have the LF and RT OSEM sensor / actuators in common (see E1100109 for top mass OSEM layout), so I checked the OSEM sensors, an indeed the LF OSEM sensor is on the very edge of its range at ~1400 [ct] out of 32000 (or 15000 [ct] if it were perfectly centered). I'll confirm that the suspension is free and OK tomorrow by retaking the measurements at a variety of alignment offsets. I really do suspect we're OK, and the measurement is just pushing the OSEM flag past its "closed light" voltage and the excitation is becoming non-linear, therefore reducing the linear response. I attach the transfer function data and a 4 year trend of the LF and RT OSEM values to show that we've been operating like this for years, and there's been no significan change after the Jul 6th EQ.
I'd forgotten to post about the OMCS data I took on 2017-07-25 as well.
The data lives here:
/ligo/svncommon/SusSVN/sus/trunk/OMCS/H1/OMC/SAGM1/Data/
2017-07-25_1812_H1SUSOMC_M1_WhiteNoise_L_0p02to50Hz.xml
2017-07-25_1812_H1SUSOMC_M1_WhiteNoise_P_0p02to50Hz.xml
2017-07-25_1812_H1SUSOMC_M1_WhiteNoise_R_0p02to50Hz.xml
2017-07-25_1812_H1SUSOMC_M1_WhiteNoise_T_0p02to50Hz.xml
2017-07-25_1812_H1SUSOMC_M1_WhiteNoise_V_0p02to50Hz.xml
2017-07-25_1812_H1SUSOMC_M1_WhiteNoise_Y_0p02to50Hz.xml
Detailed plots now attached, and they show that OMC is clear of rubbing; the data looks as it has for past few years, and what difference we see between LHO and LLO are the lower-stage Pitch modes which are arbitrarily influence by ISC electronics cabling running down the chain (as we see for the reaction masses on the QUADs).
Guardian DIAG_SDF suggests a difference showed up momentarily on SUSITMX
From DIAG_SDF log file:
2017-08-01T00:50:00.82615 DIAG_SDF [RUN_TESTS.run] USERMSG 0: DIFFS: susitmx: 1
Dataviewer second trend show the difference was only in effect for 3 seconds (00:50:02 - 00:50:04). Conlog did not report any settings changes at this time.
susitmx just threw us out of observe again at 0:08 UTC . DIAG_SDF logs one sdf diff, but I didn't catch it. susitmx guardian log shows nothing.
I'm running a python script on zotws6 which will print the name of the first channel in the difference list if the number of SUSITMX SDF diffs becomes non-zero.