After the earth settled back down, I had no trouble relocking the IFO. The OMC issue persists but our range has been stable. I also had to clear SDF again after relocking, but I imagine this will be resolved when the transition to the Observe.snap state is completed.
12:49 locked in Observing Mode.
8:08 locked Low Noise
8:36 set to Observing after clearing a bunch of SDF diffs (Betsy had told me to set them to Not Monitored since they are still in the process of creating the observe.snap). Sheila helped clear some that couldn't be Not Monitored due to the 'too many decimal places' issue.
9:30 I noticed that the OMC had a red CFC block on the CDS overview. The overflow was incrementing rapidly. I tried to reset it anyways, but to no avail. I then tried calling several people, none answered. Finally I called Corey to let him know about the situation. Eventually, I got a couple of calls back suggesting that I let it ride since we were already in Observing. Looking at a few trends, I saw that this began during the locking sequence. Perhaps the CDS overview had a red block in it, but I failed to notice (at least it didn't prevent me from going to Observing).
10:30 lockloss due to 5.9 EQ in Alaska.
During the ~2.5 hours lock stretch, our range has been steadily trending downward, perhaps related to the OMC issue (?).
Summary
EPICS values used for calculation of DARM time-varying parameters (DCC T1500377) in GDS pipeline and with PCALMON data were updated on 10-Sep-2015, 01:03:38 PDT (10-Sep-2015, 08:03:38 UTC) using H1 DARM model for ER8/O1. We will analyze DARM time-varying parameters (sometimes generalized as "kappas") calculated in GDS pipeline in GDS frames from nominal low-noise lock stretches after this change.
Previously values written in these EPICS records were calculated based on ER7 model (see LHO alog 20452), now we used ER8/O1 model with parameter file "H1DARMparams_1124954397.m", the summary of this param file residuals were given in LHO alog 21318. There is an additional phase of 136.7 degrees that was fudged into H1:CAL-CS_TDEP_REF_INVA_CLGRATIO_TST_(REAL|IMAG) that should provide kappa_tst with a reasonable phase value (ideally the phase of the kappa_tst should be 0), this was done so that we can test overall performance of GDS pipeline when calculating kappas.
These are not the final values for ER8/O1, currently H1 DARM OLGTF model is under testing using different DARM OLGTF and PCAL2DARMTF measurements and possibly will go through minor adjustments.
New values were accepted in SDF_OVERVIEW (however some of the values still does show up in "diffs", that's because SDF_OVERVIEW cannot handle very small values).
Details
Earlier we had a hard time getting reasonable values for kappas, in Maddie's GDS code as well as recent Greg Mendell and Sudarshan's Matlab code using PCALMON data (see LHO alog 21102). One of the reasons why calculations wouldn't fully agree with actual parameters was that EPICS records had values calculated from ER7 model, which we know is not up to date anymore. But I belive that even after we got a fairly good approximated ER8 model there's probably something missing in EPICS actuation coefficient for x_{tst} calibration line. We're still investigating the source of this discrepancy.
The script that extracts values needed for calculation of kappas from DARM OLGTF model was committed to calibration SVN:
CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/CAL_EPICS/writeER8model_TDEP_EPICS.m
Output files (raw epics values and more verbose log) were committed to the same directory, detailed log is also attached to this alog.
The EPICS values correspond to the model file and the parameter file in calibration SVN at rev. 1374:
CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/DARMOLGTFs/H1DARMOLGTFmodel_ER8.m
CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/DARMOLGTFs/H1DARMparams_1124954397.m
This morning Elli and I spent some time on phaseing of AS WFS, and again this evening we picked it up again. More details to follow, but here are some highlights:
We can use a consistent phase on all 4 quadrants to get reasonable signals.
In PRMI we can use 40 degrees for all quadrants on ASA36 to get most of the static signal into the I phase, and 90 degrees on AS 36 B. In this situation the Q phase has a good signal for the BS, with the transfer function from BS pit drive (1000 cnts, 12 Hz) to each quadrant having all the right phase relationships.
In DRMI 90 degrees still seems like a good setting for keeping the static signals in the I phase for AS B, while AS A it seems like 20 degrees is a better phase, except for quadrant 4 where it seemed like the phase should be 70. I assume that this could be caused by a misalingment, and should not be a real since all the phases were consistent in PRMI.
We closed the loops on AS B in DRMI, which worked fine, and locked. The phases of the transfer function to each quadrant stayed consistent in DRMI, full lock, and at 10 Watts. I left the excitation on the BS as I powered up, starting around 7:08 UTC sept 10th. Durring this time all the quadrants on AS 36 A had an R phase of 20 degrees, all the R phases for AS B 36 were 90, and the yaw matrix for both wfs was normal (it included all quadrants.) After powering up to 10 Watts we saw the usual runaway where POP90 grows. We can perhaps look at this data to see if our sensing matrix was changing durring some thermal process.
After several minutes at 10 Watts, I tried to recover a good operating point by moving the offset in AS_C, which did not work and broke the lock.
Everything is reverted and we are going back to locking with our old arangement for AS WFS.
LVEA: Laser Hazard IFO: UnLocked Intent Bit: Commissioning All Times in UTC (PT) 23:30 (16:30) Take over from TJ 23:30 (16:30) Relocking the IFO 23:52 (16:52) Locked at NOMINAL_LOW_NOISE, 22.7W, 68Mpc 00:02 (17:02) Lockloss – commissioning activities 00:36 (17:36) Locked at NOMINAL_LOW_NOISE, 22.7W, 68Mpc 00:38 (17:38) Lockloss – commissioning activities 01:09 (18:09) Locked at NOMINAL_LOW_NOISE, 23W, 68Mpc 01:10 (18:10) Lockloss – commissioning activities 01:31 (18:31) Locked at NOMINAL_LOW_NOISE, 22.9W, 63Mpc 01:47 (18:47) Set intent bit to Observing, 22.9W, 69Mpc 02:25 (19:25) Set intent bit to Commissioning – Evan working on noise characterization 03:05 (20:05) Range bump (over 80Mpc) due to calibration work 05:13 (22:13) Lockloss – Commissioning activities 05:30 (22:30) Adjust ISS diffracted power from -2.02v to -2.00v – diffracted power now at 8.4% Shift Summary & Observations: - Attend all hands meeting until 23:20 (16:20) - Evan working on noise characterization - Over 80Mcp range bump due to calibration work – NOT REAL - At the 05:13 Lockloss, the OMC watchdog tripped 4 seconds after the Lockloss - The first part of the shift was given to the commissioners while LLO was down. After LLO came back up and was in Observing mode, LHO switched to Observing mode as well. We had a 4-hour lock stretch in the middle of the shift with good range, moderate wind, no significant seismic activity. After losing lock, commissioning work continued until the last hour of the shift. Balance of the shift was spent relocking and doing fine tuning commissioning.
While we were in commisioing mode, we had a strange non stationary spectrum, with large non stationary noise also appearing in MICH and PRCL. I quickly looked at some triple witness sensors, since we've had issues with these recently, but didn't see anything obvious.
The strange noise started a few minutes before 2:50:34 UTC Sept 10, seemed to get better for a few minutes and then got worse again around 2:57:21.
We don't think that this was related to any comissioning we were doing. Evan had temporarily lowered the DARM ugf and put a notch in, (which should cause anything like this). He undid both changes when we saw the unusual noise, but the noise was unaffected by his change.
We aren't going to investigate any further right now since the problem seems to have gone away.
This is happening again. (this time it seems worse) It started about a half hour ago, at 22:30 UTC on Sept 14th
We ran a bruco on 5 minutes of this excessively noisy period (2015-09-14 22:55:00). Results here.
From 10 Hz to 1 kHz, most of the LSC and ASC signals are coherent with DARM, so it's hard to say what's going on here.
Above 1 kHz, there is some coherence with the RFAM monitor on the EOM driver. A series of spectra are attached showing elevated RIN in this monitor, compared to a quiet period roughly 30 minutes later. What mechanism could cause this? Also included are spectra of the LSC magnetometer in the CER, since this also showed elevated coherence.
RF45 AM stabilization looks fishy.
Look at feedback signal (Ch3) VS the range. They're in sync today (left) as well as back on 10th of Sept. (right).
Last night, I moved the PRM coil driver switching to happen before the CARM offset reduction (21321 ) with the idea that this would make the locklosses it causes less painfull as they would happen earlier in the sequence. We tried it this way once and things seemed fine, but then Travis later had a hard time locking. Many of his locklosses were due to PRM coil driver switching. This morning TJ and I moved this back to the DRMI on POP state (after CARM ofset reduction) and we also had locklosses caused by it, although we were able to make it to full lock sometimes.
This afternoon TJ and I looked into this a bit. We looked at the amplitude of the glitches as seen by noisemon, and they are the same today as they were August 30th. We also looked at how the PRM dirves respond to this glitch. Two weeks ago, we were nearly saturating PRM M3 each time we switched, in the past few days we have been nearly saturating M1 as well. This is not suprising with the higher offloading gain intended to help us lock durring high ground motion (alog 21216)
I put an elliptic low pass at 10 Hz in PRM M1, this reduces the glitch in M1 by an order of magnitude. We've survived the switching 3 times since we did this, and it is in the DRMI guardian. In the two screenshots you can see an example of the glitch with switching this morning, and the second one shows the glitch with the new low pass engaged. We are still in danger of saturating M3, so it would still be a good idea to look at tuning the delay if we get a chance.
Log:
Times in UTC (PST)
Summery:
Commissioning or other activities for my entire shift. Not much to report from me.
Hugh and I have switched the SDFs of many of the SEI, SUS, the LSC, and ASC to compare current settings (OBSERVE.snap) to the NOMINAL LOW NOISE LOCK from around 2:30pm (local) today. It then broke lock so we have to wait for the next lock stretches to finish - likely to be done tomorrow, so bear with us while we get this up and running. Background: we are redirecting to use SDF as the main settings configuration control monitor for O1, so we are now working quickly to do some smallish reconfiguring. This involves setting the SDF such that it will monitor ~all settings of the IFO and will be all-GREEN when the IFO is in NOMINAL LOW NOISE. This means it wiill have lots of reds after lockloss and during lock acquisition. (We will work to reincorporate it's feature to be a troubleshooting tool soon.)
For the next day or so, while in-flux, it may mean that there are some red SDF alarms that will not let you hit the INTENT button. If this is the case, please feel free to call me and we'll work through the short diff list, or simply set any DIFFs to NOT MON and I will adjust them as needed tomorrow.
In the long run, the idea is that the SDF will monitor all settings, including guardian manipulated ones, except the few things which change between lock stretches (such as VIOLIN mode damping, HPI Setpoints, ISS REF CAV adjustments, and OPTICALIGN slider values).
One possible cause of the DAC overflows we've been seeing (the ETMY saturation warnings) is something at high frequency using up the range of the drive. We only record the last output to the ESD DAC at 2048 Hz, so we don't see anything happening above a kHz. It's possible some high frequency line or feature bumps up against 2^17 counts and that makes the DAC overflow, which then causes a much larger glitch - once there's an overflow the whole thing will go crazy. To try to investigate this, the simplest thing to do is to record a channel like SUS-ETMY_L3_MASTER_OUT_LL_DQ at the highest rate possible, or something equivalent that monitors the actual signal sent to the ESD DAC. Once an ETMY overflow happens, save the data for several seconds around the time somewhere that people offsite can access it, preferably in a common format like ASCII or frames. Some time without an overflow would be good for comparison. Then we can see if there's something using up the drive range, or if it's just something sudden from somewhere else in the DARM loop. A few instances of overflows would be nice. If this is not possible, there's another tack we can take but it's more involved, so I'd prefer this.
If it's any help, we do record H1:SUS-ETMY_L3_ISCINF_L_IN1_DQ at the full rate (16 kHz). In full lock, this is the only signal (other than the calibration line) that is sent to the ESD. With an adequate knowledge of the filters between this channel and the ESD master outs, it should (in theory) be possible to reconstruct the master out signals at the full rate. That was the original motivation for recording this channel at full rate, anyway.
Yes, using ISCINF was the other tack that I meant. I checked the MEDM screen when I knew the detector was locked. Here's what I see in the chain: ETMY_L3_ISCINF_L - seems to be empty ETMY_L3_LOCK_L - FM1,3,5,6,8,9,10; gain is 1.25 ETMY_L3_DRIVEALIGN - L2L is only thing used, just a gain of -30 ETMY_L3_EULER2ESD matrix - factor of 0.25 to each OSEM ESD linearization - bypassed, so I think we can ignore it ETMY_L3_ESDOUTF_LL - FM6,7; gain is 1. (I think the four coils get nearly the same drive, so any quadrant is fine). The needed filters are available from the web SVN in this text file. So there's only two filter banks to apply, and a few gains. I think this is doable, though it seems like it would just be easier to record SUS-ETMY_L3_MASTER_OUT_LL_DQ at 16384 Hz for a few hours of lock. Maybe one or two of the LSC Fellows could take on this more complicated method.
I managed to capture the full data for SUS-ETMY_L3_MASTER_OUT_LL around the time of the most recent glitch. I put it on my account on caltech at /home/jordan.palamos/detchar/ETMY_glitch.txt and also at https://lhocds.ligo-wa.caltech.edu/exports/jordan.palamos/.
The start time is 1125946072.8
Similar to what I did yesterday for HEPI re 21309 I made OBSERVE.snap files for all the ISIs.
However, I came upon an issue for the BSC ISIs where toggling the MONITOR SELECT choice button to ALL left the SDF with differences.
These come from the following: When an ISI or HEPI Isolates, it observes and averages the free hanging position for 5 seconds and puts this mean into the SETPOINT_NOW. This makes the residual error of the loop zero when it is turned on. After the loop is completely on, the Guardian puts the Reference Location (e.g.: H1:ISI-BS_ST1_CPS_X_TARGET) into the SETPOINT_NOW record causing the Isolation loop to servo to that same position from platform trip to platform trip. That is for some but not all platforms/dofs. All HEPIs and all HAM ISIs are resorting the TARGET. No BSC ISI is restoring the Target location.
So even though I just took a new OBSERVE.snap, many of the BSCs showed diffs out at e-10 to -15 in the SDF. I'm not sure why that was variable but it brought the realization forward that each time a BSC ISI tripped, the new SETPOINT_NOW would be different (as it would for all the HEPIs and HAM ISI,) and would not be replaced by the TARGET Location at the end of Isolation (as it would be for the HEPIs and HAM ISIs.)
So for the BSC ISIs, after getting the OBSERVE.snap loaded into SDF as done for the HEPIs and HAM ISIs, the NOT MONITORED channels were all set to monitor and then the SETPOINT_NOW channels were set back to NOT MONITORED. Given this, the BSC ISIs will not be pink and the MONITOR SELECT button will be set to MASK, rather than ALL like the HEPIs and HAM ISIs. It may be that I will do the same for all the HEPIs and HAM ISIs if that is best for consistency, TBD.
It seems like shifting the pole from 120Hz to 1.2Hz (alog 21300) reduced the beam jitter greatly only at 60 and 180 Hz. The broad jitter noise floor might have been somewhat reduced between 10 and 100Hz, but I see no change at 300Hz. In the attached, references are before, current traces are after.
Robert reported that the coupling to the chiller water flow dominates the periscope motion at around 300Hz (alog 21273).
Laser Status:
SysStat is good
Front End power is 33.02W (should be around 30 W)
Frontend Watch is GREEN
HPO Watch is RED
PMC:
It has been locked 8.0 days, 1.0 hr 19.0 minutes (should be days/weeks)
Reflected power is 2.097Watts and PowerSum = 26.62Watts.
FSS:
It has been locked for 0.0 days 1.0 h and 10.0 min (should be days/weeks)
TPD[V] = 1.498V (min 0.9V)
ISS:
The diffracted power is around 7.045% (should be 5-9%)
Last saturation event was 0.0 days 1.0 hours and 10.0 minutes ago (should be days/weeks)
In prep for turning ~all settings configuration control over to SDF, I have turned off the Set Point Monitoring (SPM) feature on the SUS guardians that were in false alarm (yellow). We will not be using the SPM feature as a requirement for O1, although any ones that go into alarm which are stll enabled may be used as a good troubleshooting tool. (Most that are enabled have been green and running for many months now, so can't hurt to stay on.)
J. Kissel, K. Izumi We haven't gotten these two measurements since the tail end of Calibration week, and we want to confirm that our model (that's under development) is still valid after 1.5 weeks, so we've taken new DARM OLG TFs and PCAL to DARM TFs. We'll analyze these in the next few days and confirm what we can determine from the calibration lines. Data lives here: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/ER8/H1/Measurements/ DARMOLGTFs/2015-09-08_H1_DARM_OLGTF_7to1200Hz.xml PCAL/2015-09-08_PCALY2DARMTF_7to1200Hz.xml For the record, this pair of measurements takes almost exactly 1 hour. However, to knock down the measurement time to that one hour, one needs to run the DARMOLGTF first, and start the PCAL TF once the DARM OLG TF hits the frequency points around 15 to 10 [Hz]. Note, calibration lines were turned OFF during these measurements. They're now back ON.
Paramter file:
aligocalibration/trunk/Runs/ER8/H1/Scripts/DARMOLGTFs/H1DARMparams_1125813052.m
And the analysis results in pdf format are attached. It seems that adjusting the optical gain and cavity pole such that match the Pcal measurement results in frequency-dependent descrepancy in the open loop model by 3% at 120 Hz. The residual in open loop seems similar to what Darkhan reported (alog 21318).
Collected OBSERVE.snap files for use in full observing configuration monitoring.
With the SDF happy (green) and Guardian Nominal (Robust Isolated) and green, I made an OBSERVE.snap file with SDF_SAVE screen
choosing EPICS DB TO FILE & SAVE AS == OBSERVE
This saves the current values of all switches and maintains the monitor/not monitor bit into the target area, e.g. /opt/rtcds/lho/h1/target/h1hpietmy/h1hpietmyepics/burt as OBSERVE.snap
This file is then copied to the svn: /opt/rtcds/userapps/release/hpi/h1/burtfiles as h1hpietmy_OBSERVE.snap and added/commited as needed.
The OBSERVE.snap in the target area is now deleted and a soft link is created for OBSERVE.snap to point to the svn copy:
ln -s /opt/rtcds/userapps/release/hpi/h1/burtfiles/h1hpietmy_OBSERVE.snap OBSERVE.snap
Finally, the SDF_RESTORE screen is used to select the OBSERVE.snap softlink and loaded with the LOAD TABLE button.
Now, for the HEPIs for example, the not monitored channels dealt with by the guardian will be a different value from the safe.snap but, the not monitored channels are still not monitored so the SDF remains green and happy. And if the HEPI platform trips, it will still be happy and green because, the not monitored channels are still not monitored.
What's the use of all this you say? Okay, I say, go to the SDF_TABLE screen and switch the MONITOR SELECT choice to ALL (vs MASK.) Now, the not monitored channel bit flag is ignored and all records are monitored and differences (ISO filters when the platform is tripped for example) will show in the DIFF list until guardian has the platform back to nominal.
Notice too that the SDF_OVERVIEW has the pink light indicating monitor ALL is set. This should stay this way unless Guardian is having trouble reisolating the platform and then the operator may want to reenable the bit mask to make more evident any switches that guardian isn't touching more apparent.
But rather than rely on selecting ALL in the SDF_MON_ALL selection, I would suggest you actual set the monitor bit to True for all channels in the OBSERVE.snap. That way we don't have to do a two-step select process to activate it, and we can indicate if there are special channels that we don't monitor, for whatever reason.
Yes Jameson. That is why I selectied the ALL button allowing all channels to be monitored.
Hugh, I think the OBSERVE snaps should have the montor bit set for all channels. In some sense that's the whole point of having separate OBSERVE files to be used in this way, that we use them to define setpoints against which every channel should be moniotred.
Evan, Sheila, Jeff B Ed
Earlier today we were knocked out of lock by a 5.2 in New Zealand, the kind of thing that we would like to be able to ride out. The lockloss plot is attached, we saturated M3 of the SRM before the lockloss and PRM M3 was also close to saturating.
While LLO was down, we spent a little time on the offloading, basically the changes described in alog 21084. This offloading scheme worked fine in full lock for PRM, however we ran into trouble using it durring the acquisiton sequence. Twice we lost lock on the transition to DRMI, and twice we lost lock when the PRM coil state swtiched in DRMI on POP. Hpwever, we can acquire lock with the new filter in the top stage of PRM and SRM, but the old low gain (-0.02). We've been able to turn the gain up by a factor of 2 in full lock twice, so I've left the guardian so that it will turn of the gain in M2 (before the intergrator) in the noise tunings step.
If anyone decides they need to undo this change overnight, they can comment out lines 344-347 and 2508-2514 of the ISC_LOCK guardian.
Before we started this, the PRM top mass damping was using 10000 cnts rms at frequencies above a few hundred Hz, because of problems in the OSEMS (alog 21060 ). Evan put some low passes at 200 Hz in RT and SD OSEMINF which reduces this to 2000 cnts rms. Jeff B accepted this change in SDF.
The second attached screenshot shows the PRM drives, the references are in the minutes before the earthquake dropped us out of lock. The red and blue curves show the current drives, with the high frequency reduction in M1 due to Evan's low pass, and the new offloading on. The last attached screenshot shows SRM drives with the new offloading.
I don't think the 5.2 EQ is the cause of the lockloss.
According to your plot, the lockloss happened on Sep 08 at 00:31:07 UTC. The 5.2 EQ happened on Sep 07 at 20:24:56.84 UTC and hit the site at 20:38:21 UTC according to Seismon (so 4 hours earlier). The BLRMS plot confirm that statement (see attachment).
Around loss time, the ground seems as quiet as usual.
I was mistaken in identifying the earthquake, but the ground motion did increase slightly, which seems to be what caused the lockloss.
As far as the overflows are concerned, A1 and A2 overflow in OMC is OK as the only channels in A1 and A2 used by OMC are the ones used for acquisition/transition.
If you look at H1OMC_GDS_TP screen, there are four potential source of overflow, A0, A1, A2 and D0 representing ADC card 0, 1, 2, and DAC card 0. In the attached, you can see that ADC2 is overflowing.
If you press "A2" button, you'll see that the channel overflowing is C_DIFF_PLL_CTRL_INMON (second attachment), and that this channel is the only one in ADC2 that is used by the OMC frontend.
This channel is only used before RF lock and always rails after full lock. It's safe to ignore this in observation mode as the channel is not used in full lock.
Likewise, if A1 overflows in observation mode, that's fine as the channels in ADC1 used by OMC (ASAIR_A_RF45_I and Q) are only used before DC lock.
If A0 or D0 was constantly overflowing, that would have been a concern.
To follow up, here's Kiwamu's email regarding the outstanding OMC filter file that was modified last night causing additional confusion and his clearing of it this morning:
"I loaded the coefficients. The CFC bit is now back to green.
$ diff filter_archive/h1omc/H1OMC_1125809279.txt H1OMC.txt
173a174,175
> # DESIGN LSC_OMC_DC 3 zpk([1],[2.5],1,"n")gain(0.4)
> # DESIGN LSC_OMC_DC 4 zpk([2.5],[5],1,"n")gain(0.5)
180a183,184
> LSC_OMC_DC 3 12 1 49152 0 1:2.5:ac 0.9997125807026548 -0.9990417213032660 0.0000000000000000 -0.9996165783185160 0.0000000000000000
> LSC_OMC_DC 4 12 1 49152 0 2.5:5:ac 0.9995213195808776 -0.9980843600250285 0.0000000000000000 -0.9990417213032660 0.0000000000000000
kiwamu"
Upon some closer inspection, I'm not totally convinced that the lockloss at 10:30:46 UTC was due to the earthquake.
Attached is a plot for a single seismometer (located at End-X), but all 5 STS-2s that are mounted on the floor look the same. The top row is x-axis data, the middle row is y-axis and the bottom row is z. The columns from left to right are in increasing order of BLRMS bands. The lockloss is at about 0 seconds (actually -0.9ish), and the plots are 10 minutes of data, centered around the lockloss. We don't see the earthquake until about 150 seconds after the lockloss. This is consistent with the anticipated P-phase arrival time from Terramon - 10:33:12 UTC, but much earlier than the S-phase time (10:38:25 UTC) or the R-phase time (10:43:56 UTC).
I don't know yet what did cause the lockloss, but I don't think it was the Alaskan earthquake.