- Took over from TJ at 23:00 (16:00). Commissioners/Ops working on relocking IFO until 02:00 (19:00). - Set Obs Mode to commissioning – Commissioners working while LLO is down. - After some initial locking problems, the IFO has been stable locked at 22.9W with approximately 65Mpc. The shift is running smoothly. - I added 125ml water the crystal chiller.
I've been investigating the range drops associated with loud (SNR > 1000) glitches.
During ER7, these were thought to be due to particles falling through the beam (see alogs on "dust glitches" 20276, 20354, 20355, 20328, 20395, 20484).
In ER8, we are still seeing loud glitches associated with range drops, however now there are also saturations on SUS ETMY (see alogs 19939, 19947, 20071, 20612, and almost every Ops summary since August 17).
Every Omicron trigger with SNR > 1000 (and most of those with SNR 100 – 1000 as well) is simultaneous with a range drop and an ETMY saturation during the 92 hours from September 3 - 6.
By contrast, none of the SNR > 1000 triggers from June 9 - 11 are associated with ETMY saturations.
On the other hand, OmegaScans and timeseries plots of the ER7 and ER8 glitches look similar, suggesting that all the glitches may be related.
So, if there are two distinct glitch classes, the "dust glitches" appear to have stopped occurring.
Or, if all the glitches have the same cause, then something seems to have changed to make ETMY more sensitive to them.
Has anything in the control system changed in a way that could lead to larger signals appearing on the ETMs for the same ‘glitch stimulus’?
See plots attached, and PreO1WorstGlitches page for more details.
Yes, since ER7 we have turned on analog low-pass filtering on the EY ESD. For any signal above 50 Hz, we drive the DAC 500 times harder than before in order to overcome the effect of this filter.
Restarted DCS Disk2Disk and diffFrames which copies data from CDS to LDAS and checks for diffs between the science frames from the two framewriters. The restart was between 16:28 PDT and 17:19 PDT today (08 Sept. 2015). This was to fix a minor bug found in these scripts where they did not close a file used internally by the scripts to check its own run status. This change should have no impact on CDS.
I added 'H1:OMC-READOUT_ERR_GAIN' to the exclude list, since it sometimes generates 'nan' values and stops Conlog. I then regenerated the channel list and set Conlog to use it. + H1:OMC-FPGA_DTONE_GAIN + H1:OMC-FPGA_DTONE_LIMIT + H1:OMC-FPGA_DTONE_OFFSET + H1:OMC-FPGA_DTONE_RSET + H1:OMC-FPGA_DTONE_SW1S + H1:OMC-FPGA_DTONE_SW2S + H1:OMC-FPGA_DTONE_SWSTAT + H1:OMC-FPGA_DTONE_TRAMP - H1:OMC-READOUT_ERR_GAIN inserted 8 pv names deleted 1 pv names
Collected OBSERVE.snap files for use in full observing configuration monitoring.
With the SDF happy (green) and Guardian Nominal (Robust Isolated) and green, I made an OBSERVE.snap file with SDF_SAVE screen
choosing EPICS DB TO FILE & SAVE AS == OBSERVE
This saves the current values of all switches and maintains the monitor/not monitor bit into the target area, e.g. /opt/rtcds/lho/h1/target/h1hpietmy/h1hpietmyepics/burt as OBSERVE.snap
This file is then copied to the svn: /opt/rtcds/userapps/release/hpi/h1/burtfiles as h1hpietmy_OBSERVE.snap and added/commited as needed.
The OBSERVE.snap in the target area is now deleted and a soft link is created for OBSERVE.snap to point to the svn copy:
ln -s /opt/rtcds/userapps/release/hpi/h1/burtfiles/h1hpietmy_OBSERVE.snap OBSERVE.snap
Finally, the SDF_RESTORE screen is used to select the OBSERVE.snap softlink and loaded with the LOAD TABLE button.
Now, for the HEPIs for example, the not monitored channels dealt with by the guardian will be a different value from the safe.snap but, the not monitored channels are still not monitored so the SDF remains green and happy. And if the HEPI platform trips, it will still be happy and green because, the not monitored channels are still not monitored.
What's the use of all this you say? Okay, I say, go to the SDF_TABLE screen and switch the MONITOR SELECT choice to ALL (vs MASK.) Now, the not monitored channel bit flag is ignored and all records are monitored and differences (ISO filters when the platform is tripped for example) will show in the DIFF list until guardian has the platform back to nominal.
Notice too that the SDF_OVERVIEW has the pink light indicating monitor ALL is set. This should stay this way unless Guardian is having trouble reisolating the platform and then the operator may want to reenable the bit mask to make more evident any switches that guardian isn't touching more apparent.
But rather than rely on selecting ALL in the SDF_MON_ALL selection, I would suggest you actual set the monitor bit to True for all channels in the OBSERVE.snap. That way we don't have to do a two-step select process to activate it, and we can indicate if there are special channels that we don't monitor, for whatever reason.
Yes Jameson. That is why I selectied the ALL button allowing all channels to be monitored.
Hugh, I think the OBSERVE snaps should have the montor bit set for all channels. In some sense that's the whole point of having separate OBSERVE files to be used in this way, that we use them to define setpoints against which every channel should be moniotred.
Tasks completed (Time in PST)
HAM1 Grouting (800-1220)
Crane TCS Chiller (-1220)
EX replace annulus ion pump (1012-1451)
Duotone Frame change (-1211)
CDS Frame Writers (831-1300)
UPS bypass repair (828-841)
PSL laser WD on Bechoff comuter (819-821)
PZT per LPF swap (1231-1321)
HWS alignment (1315-1409)
GDS channels added to DAQ (1300)
ETMX charge measurements (945-1033)
ETMY Coil Driver measurement (950-1150)
Duotone: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=21222
After installation, I confirmed that H1:CAL-PCALX_FPGA_DTONE_IN1_DQ, H1:CAL-PCALY_FPGA_DTONE_IN1_DQ and H1:OMC-FPGA_DTONE_IN1_DQ are with "acquire=3" line in H1CALEX.ini, HACALEY.ini and H1OMC.ini, respectively.
Log:
Handing off to Jeff B trying to lock DRMI_1F.
Longer maintenance Day, but no huge surprises.
Pump cart running next to BSC9 until further notice
Note, when the EXC bit in the CALCS CDS overview is in alarm, we tend to open the screen CAL_INJ_CONTROL to attempt to diagnose - This shows a big red light for some ODC Channel OK Latch, leading us to misdiagnose what is actually in alarm. We have 2 operational problems:
1) If generically, there is a red light on the CDS screen - where do you go? Normally, we follow the logical medm and are able to get to the bottom of the red status via logical nested reds. This is not the case for the CALCS screen - the CDS H1:FEC-117_STATE_WORD bit is RED on the H1CALCS line of the overview screen, yet this bit is nowhere on the CALCS screen.
So, where does the info come from for specifically the EXC bit of the H1CALCS state word, such that we can do something about it?
2) Someone should rework the CAL_INJ_CONTROL.adl so that it doesn't cause us to misdiagnose actual reds. Currently, the HARDWARE INJECTIONS are out of configuration (outstanding issue to still be sorted) and yet, there is NO INDICATION of that on the CAL_INJ_CONTROL screen... Also, the CW injection appears to be off, but there is no "red alarm" on the screen.
BTW, the HARDWARE INJ appear to be off. They dropped around 7pm local time last night (20 hours ago).
The hardware injection folks should comment, but I spoke to Duncan Macleod at LLO last week about redesigning that screen. I told him not to use red for things that don't actually indicate errors, or conditions that need to be acted upon. I think he was working on the screen, so maybe it can be updated at LHO soon.
The EXC bit in the CALCS is expressly excluded from the DIAG_EXC checks, because excitations in the CALCS model are used for hardware injections.
Our confusion with the Cal EXC was that we were expecting an excitation to exist, but it didn't. The CDS overview screen that Betsy posted is modified to show red for the CALCS EXC when there *is not* an excitation, whereas it shows red for every other model if there *is* an excitation. But, we were having trouble on the CAL CS overview screen determining if there was actually a problem.
We have large glitches when we switch the coil driver state for PRM in every example that I've looked at. This causes locklosses, about 5 over the weekend.
One thing that we can do to make this less painfull is to move the switch earlier in the locking process (for example, we can try doing this right after DRMI locks rather than waiting until DRMI on POP).
A real solution might be to change the front end model so that we can switch each coil separately.
We could try tuning the delay, as described for the LLO BS 16295
We didn't get a chance to look at the delay, but we have made a few gaurdian changes that should help mitigate this situation.
We are now increasing the PRM and SRM offlaoding after we transition to 3F before starting the CARM offset reduction. This gives us a bit more headroom when we siwtch the coil dirver. We also moved the coildriver switching sooner (it is now in ISC_DRMI in the LOCKED_3F state) so that we don't waste as much time if this breaks the lock.
Kiwamu, Sheila, Evan
Durring the calibration activities last week Kiwamu noticed that there was upconversion of the drive from ETMY. Specifically when he drove in the L3 LOCK filter, he saw a second harmonic. When he drove each stage individually, through the test filter bank with a similar amplitide he saw no evidence for upconversion. This suggests that the upconversion might come about by driving mutliple stages, or through the length 2 angle paths.
We went back through the data and looked at the relative heights of the peaks. We also calculated a ratio of the amplitude of the second harmonic to the square of the amplitude at the fundamental :
ASD(2f) = alpha * (ASD(1f))^2
| drive Frequency (Hz_ | amplitude at drive frequency (m/rt Hz) | amplitude at second harmonic (m/rt Hz) | alpha (1/m) |
| 4.98 | 5.8e-14 | 2e-15 | 5.9e11 |
| 5.9 | 5.6e-14 | 9.8e-16 | 3.1e11 |
| 6.4 | 3e-14 | 2.8e-16 | 3.1e11 |
| 10 | 3.6e-15 | 2.8e-18 | 2.2e11 |
If we ignore other frequencies which could mix and only consider the second harmonics of DARM control, we would expect this upconversion to be something like a factor of 10 below our measured noise at 20 Hz, and near the measured noise at 10 Hz.
The next step is probably to identify which stages contribute to this upconversion, for example it seems probable that this is only noticeable for frequencies where L2 get a significant fraction of the DARM control signal.
We made some injections into ETMY to check when exactly this upconversion shows up. It is much smaller today than it was durring Kiwmau's measurement.
An injectionat 5 Hz (500 counts in ISCINF) that increased the DARM noise there by a factor of 67 produced a peak at 10 Hz that is a factor of 2.6 above the noise floor. We also injected in L2 L2L (which will bypass the L2P and L2Y filters) to produce a simlar peak, and got a similarly low level of noise at 10 Hz.
It may be that part of the problem durring Kiwamu's measurement was that some ASC loops were accidentaly off.
While Jeff and Darkhan have been trying to get the actuactor coefficients right for calibration, I worked on an orthogonal task which is to check out the latest optical gain of DARM.
Summary points are:
The plots below show the measured optical gain measured by Pcal Y with the loop suppression taken out by measuring the DARM supression within the same lock stretch.
I used the data from Aug 28 and 29th (alog 21190 and alog 21023 respectively). The parameters were estimated by the fitting function of LISO. I have limited the frequency range of the fitting to be avove 30 Hz because the measurement does not seem to obey physics. I will metion this in the next paragraph. The cavity pole was at around 330 Hz which claims a bit lower frequency than what Evan indendently estimated from the nominal Pcal lines (alog 21210). Not sure why at this point.
One thing we have to pay attentin is a peculiar behavior of the magnitude at low frequencies -- they tend to respond lesser by 20-30 % at most while the phase does not show any evidence of extra poles or zeros. I think that this behavior has been consistently seen since ER7. For example, several DARM open loops from ER7 show very similar behavior (see open loop plots from alog 18769). Also, a recent DARM open loop measurement (see the plot from alog 20819). Keita suggested makeing another DARM open loop measurement with a smaller amplitude, for example by a factor of two at a cost of longer integation time in order to detemine whethre if this is associated with some kind of undesired nonlinearity, saturation or some sort.
I did a similar fit that Shivaraj did at LLO (alog # 20146), to determine the time delay between PCAL RX and and the DARM_ERR. Both signal chain have one each of IOP (65 KHz), USER model (16 Khz) and AA filter between them. The expected time delay between the PCAL and DARM_ERR as shown in the diagram below should be about 13.2 us in total. I used the Optical gain as 1.16e+6 from the alog above and fitted for cavity pole and time delay. I got cavity pole estimate of 324 Hz, close to what Kimamu got from his fitting and time delay of 21 us. This is 7.8 us more than what we expected from the model.
Due to a crazy big offset of -0.5 in Y_TR_B_PIT (for SOFT modes sensing), Y IR QPDB is almost always railing a bit in 24W operation, and Y IR QPDA is not too far.
Next time IFO drops out of lock, somebody needs to lower the whitening gain by 3dB and set a new dark offset for each quadrant.
These whitening gains are controlled by the ISC_LOCK guardian. We already lower them by 6 dB in the DRMI_ON_POP state (which produces the momentary fake jump in arm power that everyone asks about), so it sounds like we should be lowering them by 9 dB instead.
[Shamefully, we don't change the dark offsets when we change the whitening in this step.]
Shame.
DRMI_ON_POP now turns down whitening gain from 18 dB to 9 dB.
Eric, Cheryl Following advice from D Shoemaker et al., we have disabled the LIMIT on the HWINJ filter bank. The change was made at approximately GPS = 1125361473. The LLO LIMIT was turned off earlier today: https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=20249
[TJ, Jenne]
TJ pointed out to me that the Cal Hardware Injection ODC bit is red, and I tracked down where it's coming from. The ODC is upset that the CAL-INJ_HARDWARE limiter was turned off. I don't think that this has been preventing us from going to Observing mode, since the limiter was turned off on Thursday, but if it does (particularly if some updates have been made during maintenence day that tie this bit into our "OK" status) I will turn the limiter back on until the ODC can be updated to know that the limit switch should be off.
I have sent EricT an email asking for him to help figure out the ODC part of this.
Just to be clear, there should be NO ODC CHECKS INVOLVED in the IFO READY bit. ODC is only used as transport of the bits, and none of the checks being done by ODC affect IFO READY. The only thing that should be going in to IFO READY now is the GRD IFO top node. In other words, this ODC issue should not have been preventing you from going to Observing mode.
Note, when the EXC bit in the CALCS CDS overview is in alarm, we tend to open the screen CAL_INJ_CONTROL to attempt to diagnose - This shows a big red light for some ODC Channel OK Latch, leading us to misdiagnose what is actually in alarm. We have 2 operational problems:
1) If generically, there is a red light on the CDS screen - where do you go? Normally, we follow the logical medm and are able to get to the bottom of the red status via logical nested reds. This is not the case for the CALCS screen - the CDS H1:FEC-117_STATE_WORD bit is RED on the H1CALCS line of the overview screen, yet this bit is nowhere on the CALCS screen.
So, where does the info come from for specifically the EXC bit of the H1CALCS state word, such that we can do something about it?
2) Someone should rework the CAL_INJ_CONTROL.adl so that it doesn't cause us to misdiagnose actual reds. Currently, the HARDWARE INJECTIONS are out of configuration (outstanding issue to still be sorted) and yet, there is NO INDICATION of that on the CAL_INJ_CONTROL screen... Also, the CW injection appears to be off, but there is no "red alarm" on the screen.
BTW, the HARDWARE INJ appear to be off. They dropped around 7pm local time last night (20 hours ago).
C. Cahillane, D. Tuyenbayev I have updated the strain uncertainty calculator to use Darkhan's latest ER8 model along with ER8 data.
The ER8 model: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/DARMOLGTFs/H1DARMOLGTFmodel_ER8.m
The ER8 params: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/DARMOLGTFs/H1DARMparams_1124597103.m
The ER8 uncertainty calc: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/S7/Common/MatlabTools/strainUncertaintyER8.m
Again, there are slight discrepancies between Plot 1 and Plot 7. To reiterate, Plot 1 uses the full O1 calibration method and real data, while Plot 7 uses the Inverse Response TF with no data:
Plot 1:
dh = DARM_ERR / C_true + A_true * DARM_CTRL
------------------------------------
DARM_ERR / C_0 + A_0 * DARM_CTRL
Plot 7:
dh = (1 + G_true) / C_true
----------------------
(1 + G_0) / C_0
This time, I believe the discrepancies are a result of the as-of-yet incomplete ER8 model. I will rerun this test when Kiwamu, Jeff, and Darkhan have all finished updating the ER8 model.
Note that the ugly spikes in uncertainty have disappeared from Plots 1-6 (as opposed to aLOG 21065).
This is because now I do not have to interpolate my DARM_ERR and DARM_CTRL FFT frequency vector thanks to Darkhan's clever ER8 model functions par.{A,C,D,G}.getFreqResp_total(freq) where freq is a frequency vector I define. This is possible because each portion of the ER8 model is an LTI object. Thanks Darkhan.
I have updated these plots for the newly updated ER8 Calibration Model at GPS time 1125747803 (Sep 08 2015 11:43:06 UTC). Now I am getting better agreement between the data-based strain error plot (Plot 1) and the response function error plot (Plot 7). I am now getting a conspicuous spike in error at 37.3 Hz in Plots 1-6, which is exactly where the X_CTRL calibration line is. I cannot be sure why this line is accounting for massive errors in the strain calculation with only 10% changes in the kappa_tst, etc...