[JenneD, SudarshanK]
We lowered the ASC DHARD Pitch Gain to see if we can actually stay locked with smaller gain. We lowered the gain by a factor of 3 (from -30 to -10). The interferometer happily stayed locked but we didnot notice any significant change in the DARM spectrum (Attachment 1). During this time, the purge air/Kobelco was turned on as part of Tuesday maintenance, which might elevate the DARM noise floor but should be similar throughout our study.
We also made noise injections at these two configurations via DHARD_P_EXC. The sensor corrections was turned off during one of those injections but we didnot see much difference between the two.
At next comissioning opportunity, we will probably try to measure sensing function at the lower gain value because we expect the sensing function to change (move towards "normal") with lower gain.
Rick, Camilla, Dave:
The H1PSLFSS.ini file was hand edited to uncomment the four requested channels and set their data rates to 256Hz.
The new DAQ was loaded onto h1pslfss at 08:05 PDT by pressing the "DAQ LOAD" button on the GDS-TP MEDM.
At the next optimum time, the DAQ data concentrator was restarted (08:06 PDT). This did not go well, monit on h1dc0 did not restart the daqd process. I logged into h1dc0 as root, stopped and then started monit, which in turn started daqd. This meant the DAQ was down for an additional 2 minutes than expected.
I ran my new script to restart all the FOM plots which are NDS clients (ndscope and DTT).
To verify the new channels have been added:
on the PSLFSS GDS_TP MEDM the number of fast channels increased from 4 to 8, the DAQ data rate increased from 293 to 297 kB (+4 because 1024 single prec float data points added per second)
I ran ndscope to view the 4 new channels.
TJ confirms the FOMS are back up and running. I'll add an MEDM button to run my restart script.
TITLE: 03/10 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Camilla
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 3mph Gusts, 2mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.25 μm/s
QUICK SUMMARY: Brief bit of commissioning before the start of maintenance.
TITLE: 03/10 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 119Mpc
INCOMING OPERATOR: Camilla
SHIFT SUMMARY: Some PEM commisioning, not much else
LOG:
2:07 Out of Observe for some Robert measurements on ETMX ISI & HEPI
2:47 Back to Observe
WP8563 add four 256Hz DAQ channels
Rick, Camilla, Dave:
I have hand edited H1PSLFSS.ini to uncomment and set datarate=256 for the following four channels:
H1:PSL-FSS_FAST_RAMP_INJECTION_IN1_DQ
H1:PSL-FSS_NPRO_TEMP_IN1_DQ
H1:PSL-FSS_NPRO_TEMP_OUT_DQ
H1:PSL-FSS_TEMP_SEARCH_LOOP_IN1_DQ
This is purely a DAQ reconfiguration with no associated model change, allowing us to load the new configuration into h1pslfss by just pressing the 'DAQ LOAD' button on the GDS_TP medm, and then restarting the DAQ.
If all goes well we will not need to restart the h1pslfss model, however if it becomes necessary to do this it is important that we do not compile or install h1pslfss because model changes have been made which we do not want to be installing at this time.
I opened a very old dtt, and did not check nor recall that it was set up for an injection, and then I hit start, and for approximately 1 second there was an excitation on H1:SUS-IM2_M1_TEST_L_EXC.
Attached are two plots. The first shows the EXCMON signal, the length, pitch, and yaw DAMP signals, and the MASTER_OUT signals for each OSEM. The second plot shows the EXCMON signal, the length, pitch, and yaw DAMP signals, and the OSEMINF signals. There was an effect on IM2, seen in the Master_OUT and OSEMINF swignals, and an unknown effect on H1.
The IFO should automatically be taken out of observe when an injection happens. Reading the alog above, it sounds like you are saying that this didn't happen. Was the intent bit really observe while this injection was happening?
The IFO was not in OBSERVE at the time of the excitation (1=True for the 'OK' channels):

The IFO node didn't go OK until about 45 seconds later; The DIAG_EXC node that monitors AWG injections was already not OK:
2020-03-09_20:15:15.145648Z DIAG_EXC [RUN_TESTS.run] USERMSG 0: EXC: susim excitation!
2020-03-09_20:15:15.220819Z IFO [READY.run] USERMSG 0: DIAG_EXC: has notification
2020-03-09_20:15:15.291508Z IFO [READY.run] USERMSG 1: waiting for node: DIAG_EXC
2020-03-09_20:15:15.333495Z IFO JUMP target: WAITING_FOR_NODES
2020-03-09_20:15:15.334183Z IFO [READY.exit]
2020-03-09_20:15:15.334183Z IFO STALLED
2020-03-09_20:15:15.400427Z IFO JUMP: READY->WAITING_FOR_NODES
2020-03-09_20:15:15.401093Z IFO calculating path: WAITING_FOR_NODES->OBSERVE
2020-03-09_20:15:15.401093Z IFO new target: READY
2020-03-09_20:15:15.402705Z IFO executing state: WAITING_FOR_NODES (20)
2020-03-09_20:15:15.471219Z IFO [WAITING_FOR_NODES.run] USERMSG 0: DIAG_EXC: has notification
2020-03-09_20:15:15.474856Z IFO [WAITING_FOR_NODES.run] USERMSG 1: waiting for node: DIAG_EXC
2020-03-09_20:15:56.341314Z IFO JUMP target: READY
2020-03-09_20:15:56.341912Z IFO [WAITING_FOR_NODES.exit]
2020-03-09_20:15:56.341912Z IFO STALLED
2020-03-09_20:15:56.388676Z IFO JUMP: WAITING_FOR_NODES->READY
2020-03-09_20:15:56.391380Z IFO calculating path: READY->OBSERVE
2020-03-09_20:15:56.391380Z IFO new target: OBSERVE
2020-03-09_20:15:56.391380Z IFO executing state: READY (50)
2020-03-09_20:16:36.396095Z IFO REQUEST: OBSERVE
2020-03-09_20:16:36.397782Z IFO STALL cleared
2020-03-09_20:16:36.400184Z IFO calculating path: READY->OBSERVE
2020-03-09_20:16:36.448505Z IFO EDGE: READY->OBSERVE
2020-03-09_20:16:36.449190Z IFO calculating path: OBSERVE->OBSERVE
2020-03-09_20:16:36.449967Z IFO executing state: OBSERVE (100)
TITLE: 03/09 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Jim
SHIFT SUMMARY:
Started off shift with H1 making it to NLN. Then there was the weekly Monday calibration measurements, some commissioning, and mostly observing today. There was some tumbleweed work as well. Tour will be visiting
LOG:
I checked that the nominal (and max) gains for ETMX mode 13 and 16 is 30 (in the lscparams). However the gain which was applied was 40. I believe this to be a typo while feeding the gain or did you/Cheryl deliberately tried higher gain for damping them?
As noted in my log, I saw modes ringing up again at around the end of my shift, so I took the gains to 40 to see if this would take violins back down. This is what I handed off to Jim last night.
nothing notable aside from the mysterious stepping DCHILFLOW.
J. Kissel
Without having the chance to assess how much we're improving the overall uncertainty budget, I've gathered another round of all sensing and actuation function measurements today in hopes to push the number of measurements, N, in to the "diminishing returns" region of 1/sqrt(N) improvements. Measurement and analysis to come in the fullness of time.
For now, the templates and exported data for today's measurements live here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/
2020-03-09_H1_PCALY2DARMTF_LF_SS_5t1100Hz_10min.xml
2020-03-09_H1_DARM_OLGTF_LF_SS_5to1100Hz_15min.xml
2020-03-09_H1_PCALY2DARMTF_BB_3min.xml << started at 2020-03-09 18:00:33 UTC for later processing against GDS/DCS h(t) products.
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOActuationTFs/
2020-03-09_H1SUSETMX_L3_iEXC2DARM_12min.xml
2020-03-09_H1SUSETMX_L3_PCAL2DARM_6min.xml
2020-03-09_H1SUSETMX_L2_iEXC2DARM_12min.xml
2020-03-09_H1SUSETMX_L2_PCAL2DARM_6min.xml
2020-03-09_H1SUSETMX_L1_iEXC2DARM_8min.xml
2020-03-09_H1SUSETMX_L1_PCAL2DARM_5min.xml
General/Overview:
Posting Schedules for upcoming work post-O3 in the Control Room (Betsy)
Donuts for weeklies
LVEA
Out Buildings
TITLE: 03/09 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 1mph Gusts, 0mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.21 μm/s
QUICK SUMMARY:
Hand-off from TJ & continued to let H1 go. It's been down 2hrs.
Tumbleweed baling started rougly 30min ago.
It tripped when we reached the RESONANCE state in ISC_LOCK. This trip happened later in the reacquisition from what we have seen in the past.
I also found SEI_DIFF stalled in the DOWN state, I'm a bit confused how it got there, but I'll look into it later.
"I also found SEI_DIFF stalled in the DOWN state, I'm a bit confused how it got there, but I'll look into it later." Could it have anything to do with me hitting the big red button? On a related note, SEI_CONF would not let me switch from WINDY to EARTH_QUAKE for the second earthquake from Canada last night.
Because SEI_CONF is now managed by SEI_ENV, I don't know if you can change SEI_CONF manually like we did in the past. Probably the proper way to force the earthquake controls on is to move SEI_ENV (the manager node) to EARTHQUAKE, not SEI_CONF (the subordinate node). This is something that hasn't been communicated clearly since we started the earthquake automation. This maybe also affects the function of the big red button, I haven't checked, but Patrick's earlier alog made it sound like we didn't trip too many platforms.
Looking at the trends for the second earthquake last night, there is some strange behavior from SEI_ENV. See attached image, SEI_ENV is top left, peakmon is top right, SEI_CONF is bottom left. Before the earthquake, the guardian is rapidly switching between the SEISMON_ALERT state and the CALM state. Peakmon crosses the 400 nm/s threshold to transition SEI_CONF to EARTHQUAKE from SEISMON_ALERT, but SEI_ENV doesn't do that in a normal way. Eventually, SEI_ENV tries to go to EARTHQUAKE, but is again switching rapidly between EARTHQUAKE and CALM, but well after peakmon crosses 1000 nm/s, which is the threshold that we use for the "ground only" test in SEI_ENV. Even then, SEI_CONF never goes to EARTHQUAKE in an normal way. Not sure what is going on, this all seems very wrong.
The SEI_ENV log is filled with a bunch of loops of this:
2020-03-09_06:40:00.638948Z SEI_ENV [CALM.enter]
2020-03-09_06:40:00.703947Z SEI_ENV [CALM.run] Peakmon high, no seismon alert, jumping to EARTHQUAKE
2020-03-09_06:40:00.760043Z SEI_ENV JUMP target: EARTHQUAKE
2020-03-09_06:40:00.760702Z SEI_ENV [CALM.exit]
2020-03-09_06:40:00.817673Z SEI_ENV JUMP: CALM->EARTHQUAKE
2020-03-09_06:40:00.818388Z SEI_ENV calculating path: EARTHQUAKE->CALM
2020-03-09_06:40:00.818388Z SEI_ENV new target: CALM
2020-03-09_06:40:00.818388Z SEI_ENV GOTO REDIRECT
2020-03-09_06:40:00.819659Z SEI_ENV REDIRECT requested, timeout in 1.000 seconds
2020-03-09_06:40:00.820828Z SEI_ENV REDIRECT caught
2020-03-09_06:40:00.821624Z SEI_ENV [EARTHQUAKE.redirect]
2020-03-09_06:40:00.883430Z SEI_ENV EDGE: EARTHQUAKE->CALM
2020-03-09_06:40:00.884153Z SEI_ENV calculating path: CALM->CALM
2020-03-09_06:40:00.888961Z SEI_ENV executing state: CALM (10)
2020-03-09_06:40:00.889582Z SEI_ENV [CALM.enter]
I can't find where it must have been doing the same with the SEISMON_ALERT state, must be too many pages back in the log.
None of this should affect the cps diff, but the ISI trips may have been related. The SEI_DIFF guardian turns off the cps diff if any ISIs trip or if the chamber guardian isn't nominal, maybe that caused a problem and crashed the node?