Ops Shift Transition: 09/12/2019, Eve Shift 23:00 – 07:00 (16:00-00:00) - UTC (PT)
State of H1: Locked
Intent Bit: Commissioning
Weather: 0-15 mph wind
Primary 0.03 – 0.1Hz: 0.01 um/s
Secondary 0.1 – 0.3Hz: 0.1 um/s
Outgoing Operator: Jim
Quick Summary: Locked and Observing for 18 hours.
TITLE: 09/12 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
INCOMING OPERATOR: Niko
SHIFT SUMMARY:
LOG:
17:00 Vanessa to MX
20:00 Kyle to MX
Following the replacement of the last two mac-mini's with nuc's yesterday, today I wrote new startup scripts for these machines (nuc24 replaces video4 and nuc26 replaces video5).
Main differences are: I abandoned the 'post-it' note labeling of the video windows, I could not get this to work consistently on Deb9. I have resized all windows to cover the whole display, i.e. there is no desktop peeking through.
I've found that in order to start the digital video streams, I have had to put in a large delay before the startup script gets going. Otherwise the early video windows do not get started, but non-video applications (e.g. the reservation window) which are started first do get started.
BTW: I have installed ndscope on all the debian9 wall display machines.
I've added some graphical displays for the BLRMS for the BRS to the BRS overviews, they are the four bars in the lower right of the attached screenshot. They are labeled by bin, but for clarity from left to right they are: .03-.1hz, .1-.3hz, .3-1hz and 1-3hz bands. When BRSX was buggy, these were useful channels for finding the problem, so I wanted to make them easy to find and see the status of.
The bars all go from 0-20 nrad/s rms velocity (I think that's right). Coincidentally, for the .03-.1 hz band 20 nrad/s is about the tilt we see with around 40mph winds and this is about the scale of the .3-1hz and 1-3hz noise we had when BRSX had guests. So, when the left-most bar is all (or nearly) blue (I haven't figured out how to get the bar to change color, to say, red), locking will be difficult and when the right two bars are full it likely means the BRS is noisy. When they are all mostly dark grey, things are quiet.
The first attached screenshot is a time machined view to when peak winds were about 40mph, you can see the low frequency bins have a lot going on, the higher not so much. The second image is from an hour or two earlier today when winds were pretty low, not much to see.
I have produced DCS filters that can be used for C01 frame production after the recent calibration model change noted in LHO aLOG 51784. Attached are several plots testing the accuracy of these filters. The first four plots are Bode plots of the filters, comparing them to the model. These all indicate normal, acceptable levels of agreement with the model. The next 3 plots show comparisons of the frequency domain model to each filter's actual impact on real data, i.e., they compare the tranfer function (filter output / filter input) to the model. The last plot compares the frequency-domain response function to the total effect of all the filters, that is, DCS-CALIB_STRAIN(f) / DARM_ERR(f).
The filters can be found in the calibration SVN here:
aligocalibration/trunk/Runs/O3/GDSFilters/H1DCS_C01_1252083618.npz
The script used to generate these filters is here:
aligocalibration/trunk/Runs/O3/H1/Scripts/TDfilters/H1_run_td_filters_C01_1252083618.sh
The filters were made using SVN revision 8411.
All these checks indicate that these filters are good for use.
Adrian, Gavin, Dripta, Ethan On Tuesday this week, X-End calibration of the PCal system was performed. The procedure is attached, as well as a photograph of the spot position on the RX PD integrating sphere aperture. The spot position looks similar to the previous measurement (alog 50215) with maybe some slight drift to the left. This will be monitored through the RX PD channel and checked again during the next measurement at EX. Analysis of the EX calibration confirms that the measurement is consistent with the general trend of the calibration parameters at this endstation.
TITLE: 09/12 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY:
Much nicer shift tonight! H1 is humming along at 10+hrs of Observing & we almost have 2hrs of Triple Coincidence.
Smooth running with H1 at over 6hrs of Lock/Observing (and we are now back to Triple Coincidence as of 7min ago *knock on wood*). Quiet conditions with temps in the low-mid 50s.
(Oh, and Pokey has just been sighted. Haven't seen this porcupine in weeks...he's feasting on leaves near the Staging Parking Lot.)
TITLE: 09/12 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 111Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 9mph Gusts, 8mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.09 μm/s
QUICK SUMMARY:
H1 Observing for 2.5hrs & got a rundown of how Niko's locking work went during his shift.
TITLE: 09/11 Eve Shift 23:00 – 07:00 (16:00-00:00), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Lockloss early shift, possibly from ADS being buried. Had issues with OM saturations while acquiring DRMI, managed to get to NLN with Sheila’s help. Locked and Observing for 2 hours.
LOG:
23:00 (16:00) Start of shift
23:51 (16:51) Calibration/commissioning complete, going to Observing
02:10 (19:10) Lockloss a few seconds after EX glitch. I think this was another example of the ADS getting buried.
02:33 (19:33) Unable to lock PRMI, going to initial alignment
03:03 (20:03) Initial alignment complete, re-locking
03:33 (20:33) Issues with OM1 and OM2 saturations during ACQUIRE_DRMI, trying a MICH_DARK_LOCKED initial alignment
03:50 (20:50) Alignment complete, re-locking
05:03 (22:03) Locked at NLN with Sheila’s help. Going into Observing.
07:00 (00:00) End of shift
J. Kissel More details to come later, but in order to continue the investigation of a potential ~1% level systematic error between H1 PCALX and H1 PCALY (see IIET Ticket 13577), I spent today's commissioning time parasitically tuning the amplitude and phase of a "cancelling" line, in which the same amplitude of PCALX and PCALY line is injected at slightly different phase, so as to leak a little bit of DARM motion into DELTAL_EXTERNAL at exactly 1153.1 Hz. As a result of the tuning choices, the residual amount of DARM at that frequency I've left in DELTAL External is a factor of 4 above the noise floor in an ASD with frequency resolution of 0.02 Hz. This yields residual coherence between each PCAL and DELTAL EXTERNAL of 0.9, which should be plenty enough SNR to track ~1% or better level changes and discrepancies between these two PCALs. Note, also, that these extra lines do not stress the limits of actuation range for either PCAL -- I was running these lines while all normal calibration lines and CW injections were running, including the PCALX high frequency roaming line, which happens to currently be at its highest frequency and loudest requested drive in the long duration sweep. The first observation segment with this line installed is Sep 11 2019 23:51:24 UTC, and the intent is to leave it in indefinitely or at least for the foreseeable future. The settings corresponding to this additional line have been accepted in to the SDF system. The lines have been installed in the 9th oscillator, H1:CAL-PCAL[X,Y]_PCALOSC9_OSC, with individual amplitudes installed in such a way that the excitation results in equal *displacement* (to within 0.1%) as reported by the calibrated receiver photodiode signals, aka the RX PDs. The excitations requested of the DAC are not calibrated, and thus the funny differing amplitudes *requested* of 5000 ct on PCALX and 3619 ct on PCALY. The -15 deg phase installed on PCALY was the result of the above mentioned tuning.
J. Kissel Reconciling safe.snap SDFs this morning reveals that I forgot to to accept the values in the PCAL EY (H1CALEY) safe. The values have now been accepted!
I have completed patching and rebooting the DMT production computers while H1 was down for commissioning this afternoon.
(This work was started yesterday during maintenance, but was stopped when it was realized DMT generation of the redundante hoft stream, when restarted, would not write .gwf files until it could get iDQ info from LDAS. And LDAS was down for maintenance yesterday too. The DMT primary hoft stream was not touched yesterday and there was no interuption with low-latency hoft going to CIT, exept briefly during the reboot of the DMT during this commissioning time today.)
Everything on the DMT is working now, and this completes WP 8345.
Ops Shift Transition: 09/11/2019, Eve Shift 23:00 – 07:00 (16:00-00:00) - UTC (PT)
State of H1: Locked
Intent Bit: Commissioning
Weather: 0-20 mph wind
Primary 0.03 – 0.1Hz: 0.01 um/s
Secondary 0.1 – 0.3Hz: 0.1 um/s
Outgoing Operator: Ed
Quick Summary: Calibration/commissioning ongoing. Wind is up slightly.
TITLE: 09/11 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Niko
SHIFT SUMMARY:
Handing off to Niko
LOG:
15:33 Chris and Tyler to MX to move scffolding (TOO while diagnosing locking issue
16:51 Chris and Tyler are done
17:15 Vanessa to MX
17:15 Kyle to MX
20:00 H1 to Commissioning - in coordination with LLO doing some commissioning work.
I added 100ml to the Xtal chiller
Sheila, Ed
Summary: I've moved the gain decrease for INP1P to the LOWNOISE ASC state, so that we have more gain for this loop while powering up, and in DRMI I added checks on the DC centering loops before engaging ASC in DRMI or PRMI. I think that the change to INP1 wil help aviod some of the locklosses from INCREASE POWER, but haven't done anything to address the other locklosses.
DRMI centering: Ed had one lockloss where the DRMI ASC came on even though the AS centering loops were saturating. I added check for the DC centering loops in both PREP_PRMI_ASC and PREP_DRMI_ASC, so that now the guardian should not move on if the OMs are saturating. If this happens, operators will have to clear the saturations before the guardian will move on.
ASC locklosses:
Last night Corey had several locklosses that seemed to be ASC related, in ENGAGE_ASC, ENGAGE_SOFTLOOPS, or INCREASE POWER. Several of these seem to be similar to the plot attached to 51868
I made a few unsucsesful attempts to increase the gain of INP1P durring ENGAGE_ASC. Even with no low passes engaged, INP1 P will ring up an instability around 1 Hz (0.8-1.1 Hz) with just a 6 dB increase in gain, with the gain settings that it has at ENGAGEASC (-30dB, low pass, integrator in suspension). So we haven't made any changes to ENGAGE_ASC for now.
engage soft loops: One of Corey's locklosses happened right as the ETM dither loops came on, CHARD Y error signal drifted away from 0. Another happened when the SRC2 gains are ramped down after the integrator. There is also 5Hz oscillation in DHARD P which seems to happen as we move the spots towards their final position, for about 20 seconds, and causes saturations of the OMs and OMC. I didn't see that cause any locklosses, although we might be able to move through this faster by increasing the gains of the ADS loops before we move the spots. (currently they are increased after the spots are moved).
We did all the steps in ENGAGE_ASC and SOFT LOOPS manually, leaving the gain of INP1 P high (it is normally reduced by 20dB at the end of the soft loops. When doing this manually I also had the low passes off for INP1. The first attached screenshot is one of Corey's locklosses from yesterday during increase power 1252146916 where the INP1 P error signal was far from its lock point, and the second screenshot shows our power incease today with higher gain in INP1P. I've changed the guardian to use this higher INP1 P gain during the power increase.
Accepted 27 CALCS diffs. Most are accuracy rounding and some aren't,
Fil and I went to EX to look inside the BRSX enclosure in prep for the work next month. While we were there I touched up the centering of the BRS. While we had the box open, we must have disturbed the cable into the interface box inside, because when we got back to the corner station the BRS wasn't damping down. I went back down to look at it and it took a little to realize there is not retention on the ethercat cable into the interface box, so it's really easy to knock loose. It immediately started damping down when I pushed the ethernet cable into the interface chassis.
The BRS is still calming down, so we shouldn't use it for a little while yet. The operator should watch the the BRS health ndscope and wait for the BRSX RY OUT to look more like the BRSY RX OUT signal, i.e. the blue trace on the middle timeseries looks like the yellow(?) trace on that same plot.
Opened FRS Ticket 13570 to track that H1 BRS EX ethernet connector at interface box needs to be repaired.
Is it safe to transition from WINDY NO BRSX back to regular WINDY while in NOMINAL LOW NOISE? (Or do we have to wait until we are not locked to make this transition?)
The transition is handled by the SEI_CONF guardian, and is very similar to making the transition to the earthquake state, and does not affect the observation bit. If anything, putting the BRSX back into the loop should make the IFO quieter, assuming the BRS has calmed down.
Looking at minute trends of the sensor correction outputs for EX and EY for the last day, we could have transitioned back about 5pm local last night, first plot.
This is inspite of the fact that the BRSX was still not totally settled, the DC position is still slowly settling down, second plot.
And, the BRSX RY out still had something like a 5000 nrad offset (third image), it was moving slowly enough to not affect the sensor correction signal.
Ah, My Bad.
Could this be the source of my woes with locking after the h1boot1 recovery?
J. Kissel, E. Merilh
Just before a big EQ in Alaska/Canada took us out of maintenance recovery lock acquisition attempts, we had 2 lock losses at / around the start of the CARM reduction sequence.
Grabbing some plots from the lockloss tool (though the web server appears to be down): it appears as though there's some positive feedback that triggers around this step, ringing up and causing the locklosses. At the moment, I only have POP_A_LF, and the arm powers in red (ASC-[X,Y]_TR_NSUM) showing this symptom, but it's worth looking in to, and I will. Help would be appreciated, though.
Plots are of the three lock losses, attached in chronological order:
UTC time GPS Time
2019-09-10 10:47:42 1252147680
2019-09-10 20:00:11 1252180829
2019-09-10 20:26:58 1252182436
Looking at the guardian, code history a little, it seems that we used to engage REFLBIAS FM4 before reducing the gain in the ALS path, which would give the TR_CARM loop a little bit more phase margin around 90 Hz, which might avoid locklosses like this (it seems like this is a case where the optical gain for TR_CARM was low, which might be alignment related.)
The alogs I can find about this anti-boost are 43190 (some errors where made in these states when the guardian was re-written, this alog includes debugging that), and 36686
For now I have changed the CARM to TR state so that it is engaging FM4n (the anti-boost) before ramping down the ALS analog gain, restoring it to the way it used to work. This seems to have worked, you can compare the two ndscopes from before and after the change.
Ed and Jeff K also reported that when they tried to request PRMI (while DRMI was trying to acquire), the guardian didn't change state. This was because there was a timer which would increment if DRMI triggered momentarily but didn't actually lock, and if that counter was incremented the guardian would not check for a change in the target state. That is working now.
Trying to analyze CARM_TO_TR locklosses, based on the channels listed by Sheila (/ligo/home/sheila.dwyer/Desktop/Locklosses/ lockloss -c channels_to_look_at_TR_CARM.txt select). I am attaching screenshots of the last 3 locklosses which falls under this category. This is still ongoing, as there are several things which I am trying to understand. The following 2 channels (H1:ASC-X_TR_B-NSUM_OUT_DQ, ASC-Y_TR_B-NSUM_OUT_DQ) rings up 1 second before lockloss.
YES THIS WILL DROP THE FRONT-END COMPUTED BNS RANGE, because we're improving the accuracy of the real-time pipeline in order to better matrch the current IFO (more power than when it was last updated, no detuning, good spot positions, SRCL offset, accounting for new UIM boost, etc.) Should be done in an hour or three. When we come back in to OBSERVATION READY, the new model will be in place. See details of the model in LHO aLOG 51784, and references there in. The decision point to change the model is documented in LHO aLOG 51782, which has yet more references and explanation.
INSTALLATION COMPLETE. As predicted, front-end BNS range dropped by ~5 Mpc, and now in better agreement with GDS produced BNS range. First OBSERVATION READY SEGMENT with this new 2019-09-09 model installed started at GPS Time 1252088806 (Sep 09 2019 18:26:28 UTC; Sep 09 2019 11:26:28 PDT).
Reference Model values at calibration line frequencies (aka "EPICs records") for this model update have been generated and installed.
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/CALCS_FE/
epicsrecords_model-H1_20190909_created-20190909.txt
In doing so, I've created the script
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/CALCS_FE/
python3.5 createEPICS_for_20190909.py rev 8397.
and modified it and computeDARM.py to be able to write the EPICs records to file separately from actually pushing it to the front-end.
/ligo/svncommon/CalSVN/aligocalibration/trunk/Common/pyDARM/src/
computeDARM.py rev 8399
I've used the broadband injection made just after the model update to do a comparison of calibration accuracy with different levels of time-dependent corrections applied, using the new model. This is similar to the analysis done in LHO aLOG 51794. The attached plot has 4 versions of offline calibration, each with different TDCF corrections applied, as indicated in the plot legend. The 5th (red) curve is from the online GDS data (C00). The previous conclusions are still supported by this data: 1) We should not be correcting for SRC time dependence; and 2) We should not be applying the imaginary parts of the actuation kappas. Note the large errors caused by applying corrections for SRC time dependence. This may be due to the fact that the model value of f_s is zero, causing numerical instability when applying time-dependent corrections.
This plot, as well as text files with the data, are available in the calibration SVN here:
aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/DCS_BB_plots
The values of the TDCFs used by the GDS and DCS calibration pipelines to produce the calibrated data during this broadband injection were:
For the C00 data (this can be seen on the summary page for that day, just after 18 UTC):
For the DCS data (very similar, but not quite identical):