thomas.shaffer@LIGO.ORG - posted 11:15, Friday 10 January 2020 - last comment - 15:56, Friday 10 January 2020(54407)
Lock Loss 1912 UTC
Seemed very fast, no obvious reasons.
Seemed very fast, no obvious reasons.
At 10:49 PST the daqd process on h1nds1 restarted itself. This is the default NDS for the workstations.
dmesg reports daqd segfaulted, and then monit restarted the process.
[6728271.848460] dqprod used greatest stack depth: 2928 bytes left
[6728344.956938] dqepics[32031]: segfault at 0 ip (null) sp 00007f7840439fb8 error 14 in daqd-3.2-r4285[400000+8d000]
[6729234.107738] dql8088[32209]: segfault at 8 ip 000000000043acdf sp 00007f5ec3fd0180 error 4 in daqd-3.2-r4285[400000+8d000]
the log file does not show any unusual data requests around this time.
If you are reading this in Calibration, and not sure what it means for the IFO and not some hand wavy "temperature dependence study" sence:
Pcal is only ever calibrated at DC frequency (0 Hz).
Having the wrong feedback capacitance introduces frequency dependant response change. Simply: With increasing frequency, the response change will have some non unity magnitude, and is unstable after some frequency.
To understand that change, the system needs to be modelled in a SPICE package.
We will be out for a short bit while the vac team goes into the EY VEA.
WP8510
Back to Observing at 1808 UTC
TITLE: 01/10 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 56Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
H1 has been locked 29+hrs & we just rode through an "unannounced" EQ from Mexico.
Tumbleweed status:
LOG:
TITLE: 01/10 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_CONF state: EARTH_QUAKE
Wind: 13mph Gusts, 10mph 5min avg
Primary useism: 0.10 μm/s
Secondary useism: 0.45 μm/s
QUICK SUMMARY: Sudden earthquake from mexico, but is seems to have past. 29 hour lock.
Just switched SEI_CONF back to WINDY. The ENV_STAT transitioned back to CALM as well.
TITLE: 01/10 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 6mph Gusts, 4mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.35 μm/s
Slight step up in microseism in the last 12hrs (by a factor of ~2, but still under 90th percentile).
QUICK SUMMARY:
TITLE: 01/10 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 120Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Quiet Shift
LOG:
Nothing to report. Environment is quiet, mostly no one here.
TITLE: 01/09 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: Locked for 13 hours, we stepped out of Observing for ~30min for calibration.
LOG:
J. Kissel
No explorative measurements today -- just a quick, standard, 30 minute check-in with the sensing function. Data and analysis to come.
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/
2020-01-09_H1_DARM_OLGTF_LF_SS_5to1100Hz_15min.xml
2020-01-09_H1_PCALY2DARMTF_LF_SS_5t1100Hz_10min.xml
2020-01-09_H1_PCALY2DARMTF_BB_3min.xml
Plan to install a new front-end calibration model on Monday, 01/13, so this data set will be processed and included in the presentation materials of that data.
Among the improvements will be to
- rid the DELTAL_EXTERNAL / PCAL transfer function (which is representative of the systematic error in the calibration) of the sharp feature at 150 Hz due to previously poor models of the UIM dynamics -- see G2000037. (~3% improvment at 150 Hz, 0.5 % improvement between 10 - 100 Hz.)
- update the sensing function to better match the current 38W IFO (instead of the 2019-09-09 model, which used data from the O3A 36W IFO). (~0.5 % improvement above ~30 Hz)
- update to correctly model the O3 ETMX test mass' mass (we've been using the O2 test mass' mass since ... well O2, even though we've replaced the mass) (~0.1% improvement between 20 Hz and 100 Hz.)
S. Dwyer, J. Driggers, J. Kissel, T. Shaffer We resolved why we've been "having" to recently offload the TMS TEST P and Y offset value (which is currently used for MISALIGNMENT offsets in the SUS gaurdian, INCREASES_FLASHES of the ALS gaurdians, and MOVE_SPOTS in the ISC_LOCK guardian) to the OPTICALIGN offsets upon the arrival in to NOMINAL_LOW_NOISE: (a) you don't *want* to do this offloading [I, Jeff, suggested we needed to do this in error], (b) these TEST offsets get used by enough guardians that we should just un-monitor them in the SDF system. We've confirmed that the *important* values of these offsets (i.e. the ones that must remain static and don't change between lock stretches) -- the MISALIGNMENT offsets -- are stored in the lsc_params file and forced in place in the DOWN state (and the INCREASES_FLASHES state restores the values it started with). Thus, there should now longer be anything showing up as an SDF DIFF upon arrival in to nominal low noise.
We will return in around 30min.
Back to Observing at 2136 UTC
Camilla, TJ, Dave:
The violin mode EPICS IOC was updated to also serve the MAXGAIN values (and VIOLIN_DAMPER.py was modified accordingly to write the max_gains).
The details of the gains are displayed in a new MEDM window H1SUS_VOILIN_MODE_GAINS.adl. This can be opened from the VIOLIN_ALL_MONITORS window via a 'Gains' related display button in the lower right.
The detailed gain window highlights those modes whose gains fall outside of the range defined by NOMINAL and MAX. In this case, a MAX=0 is assumed to have not been set and is ignored.
Lockloss from a 6.3M EQ in Russia (not far from Alaska) knocked us out of lock fast from EQ mode. 0.03-0.1Hz band nearly touching 20μm/s. Waiting for the ground to calm a little before attempting relocking.
So it looks like Increase_Flashes has been restoring the TMS TEST bank offsets properly, but ISC_LOCK changes them in ENGAGE_SOFT_LOOPS and MOVE_SPOTS and does not restore the test bank offsets.
Attachment 1 - Showing Incr. Flashes restoring and turning off the TMS offsets in the last lock acquisition.
Attachment 2 - ISC_LOCK moving the TMS and not restoring the offsets.
I'll look next to see why this has changed.
TITLE: 01/09 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Camilla
SHIFT SUMMARY: Violins were a problem early on, quiet after locking
LOG:
When I came, TJ had the IFO sitting while he and Cheryl worked on violins.
5:00 We got to NLN and Jeff and Cheryl had to do the TMS offload again
From 5:00 on we were locked and the IFO was in an Observation ready state, but I thought I had heard Jeff set the Observation bit, somehow I never caught that never actually happened until Camilla came in. Don't know how that happened. So there are about 3 hours here where the IFO was in a good state, just nobody had pushed the button.
Flagging this for DetChar.
I didn't get a chance to look into this much, but it was anther fast lockloss. This time it showed up a bit more in LSC_POP_A_LF_OUT, and as usual the OMC DCPDs saw it just before.