The 3 SSD drives that failed yesterday for the Raid on h1tw0 had been replaced. Jim will work on mount process for minute trend.
Sheila, Evan
Since we have recently gotten rid of the excess oscillator AM noise in DARM, we wanted to look at the residual of the DCPD sum and null. This test has been done at LLO several times (most recently here).
Unlike the previous look at this noise, we decided to follow the LLO technique and notch the DARM loop by 20 dB between 100 Hz and 700 Hz. In order to do this, we had to reduce the DARM ugf to 20 Hz or so. Unfortunately, this rang up the quad bounce modes. Rather than recommission the bounce mode damping, we just turned it off and collected the sum/null data in a few 20-minute intervals, and then applied damping inbetween using the usual DARM loop shape. In total we collected a little over an hour's worth of data, with the usual 20 mA of current on the sum. The attached plot shows the data processed into 1-second ASDs.
The dc current of the sum was 20.0 mA, meaning we expect a shot noise of 8.00×10−8 mA/Hz1/2. On the other hand, the ratio of sum/shot and null/shot (attached) indicates that we might be too low by a few percent. It's a bit complicated, since it relies on a combination of the frontend calibration (Koji's preamp and whitening compensation filters) and a post facto calibration (Kiwamu's combination of the IOP inversion, the supernyquist preamp poles, the AA response, and the mystery 10.7 kHz pole).
Why is the coherence between A and B not one at the jitter peaks (300-400 Hz)?
You band-stopped the DARM loop in that region, so there should be no correlation induced by DARM. So I would expect that the coherence is one, if the peaks are coming through intensity noise on the laser beam (possibly converted from jitter by the OMC). Instead there is low coherence, which seems to me not compatible with the shot noise level: this is visible in Evan's plot too, since the uncorrelated noise is closer to the value of the sum than to the null...
Two hypothesis
I think the numbers hang together. Let SAA = S00 + SA'A' and SBB = S00 + SB'B', where S00 is the PSD of the correlated component of the noise between the DCPDs, and SA'A' and SB'B' are the PSDs of the uncorrelated noises on each PD.
From last night's data, at 315 Hz we have SAA = SBB = (1.14×10−7 mA/Hz1/2)2. Looking nearby at 540 Hz, where the DCPDs seem to be shot-noise dominated, we can estimate SA'A' = SB'B' = (0.577×10−7 mA/Hz1/2)2, so therefore S00 = (0.983×10−7 mA/Hz1/2)2.
For the CSD, we have SAB = S00.
Therefore, the coherence we expect is γ2 = |SAB|2/(SAA×SBB) = 0.55.
Or in terms of the PSDs of sum (S++) and null (S−−), the coherence should be γ2 = [(S++ − S−−)/(S++ + S−−)]2.
I agree with your computations. But why is your projection of the residual showing the peaks? it should line up with the shot noise if that's what is limiting the coherence.
New ion pump OK
As a periodic evaluation of condition, we'll be trending the Pump Station Pressures to watch for changes in the Pump Performance.
The attached plots show the pressure trends from the pump stations at the three buildings for the past 30 days.
There are no obvious steps indicating a pump stating cavitation--I don't really expect this out of the blue. But if the system is taken down (Pump spun down) for any reason and then brought up, that is when cavitation has occurred and the pump pressures change. This will be obvious for the corner station as the other pumps will compensate for the cavitating pump. Hmmm, if this happens at an end station though...I expect the VFD output will have to increase to make up for this if if can. I've only seen this at the corner so... Anyway, I include the VFD output for the ends to look for this.
Observations:
Corner Station: Bizarrely, the first pressure out of the Pumps all show almost perfectly flat tends--look at the scales; I can't explain this. All the other pressures show a somewhat regular glitch up; this may be dailyish but seem a bit irregular.
EndY: Nothing new here. Pressure and VFD output steady. The long documented daily and weekly glitches still present and the daily temperature swings evident on the VOUT.
EndX: Same story except there is no regular glitches and the building must insulate things from the temperature fluctuation as there is no daily signal in the VOUT.
LLO still down, and Kyle needs to unplug the aux cart down at EX.
While LLO is down, the cleaning crew needs to use the roll up door and head into the garb room breifly. Should be right back in as soon as they're done.
Back to Observing Mode.
After the earth settled back down, I had no trouble relocking the IFO. The OMC issue persists but our range has been stable. I also had to clear SDF again after relocking, but I imagine this will be resolved when the transition to the Observe.snap state is completed.
12:49 locked in Observing Mode.
8:08 locked Low Noise
8:36 set to Observing after clearing a bunch of SDF diffs (Betsy had told me to set them to Not Monitored since they are still in the process of creating the observe.snap). Sheila helped clear some that couldn't be Not Monitored due to the 'too many decimal places' issue.
9:30 I noticed that the OMC had a red CFC block on the CDS overview. The overflow was incrementing rapidly. I tried to reset it anyways, but to no avail. I then tried calling several people, none answered. Finally I called Corey to let him know about the situation. Eventually, I got a couple of calls back suggesting that I let it ride since we were already in Observing. Looking at a few trends, I saw that this began during the locking sequence. Perhaps the CDS overview had a red block in it, but I failed to notice (at least it didn't prevent me from going to Observing).
10:30 lockloss due to 5.9 EQ in Alaska.
During the ~2.5 hours lock stretch, our range has been steadily trending downward, perhaps related to the OMC issue (?).
As far as the overflows are concerned, A1 and A2 overflow in OMC is OK as the only channels in A1 and A2 used by OMC are the ones used for acquisition/transition.
If you look at H1OMC_GDS_TP screen, there are four potential source of overflow, A0, A1, A2 and D0 representing ADC card 0, 1, 2, and DAC card 0. In the attached, you can see that ADC2 is overflowing.
If you press "A2" button, you'll see that the channel overflowing is C_DIFF_PLL_CTRL_INMON (second attachment), and that this channel is the only one in ADC2 that is used by the OMC frontend.
This channel is only used before RF lock and always rails after full lock. It's safe to ignore this in observation mode as the channel is not used in full lock.
Likewise, if A1 overflows in observation mode, that's fine as the channels in ADC1 used by OMC (ASAIR_A_RF45_I and Q) are only used before DC lock.
If A0 or D0 was constantly overflowing, that would have been a concern.
To follow up, here's Kiwamu's email regarding the outstanding OMC filter file that was modified last night causing additional confusion and his clearing of it this morning:
"I loaded the coefficients. The CFC bit is now back to green.
kiwamu"
Upon some closer inspection, I'm not totally convinced that the lockloss at 10:30:46 UTC was due to the earthquake.
Attached is a plot for a single seismometer (located at End-X), but all 5 STS-2s that are mounted on the floor look the same. The top row is x-axis data, the middle row is y-axis and the bottom row is z. The columns from left to right are in increasing order of BLRMS bands. The lockloss is at about 0 seconds (actually -0.9ish), and the plots are 10 minutes of data, centered around the lockloss. We don't see the earthquake until about 150 seconds after the lockloss. This is consistent with the anticipated P-phase arrival time from Terramon - 10:33:12 UTC, but much earlier than the S-phase time (10:38:25 UTC) or the R-phase time (10:43:56 UTC).
I don't know yet what did cause the lockloss, but I don't think it was the Alaskan earthquake.
Summary
EPICS values used for calculation of DARM time-varying parameters (DCC T1500377) in GDS pipeline and with PCALMON data were updated on 10-Sep-2015, 01:03:38 PDT (10-Sep-2015, 08:03:38 UTC) using H1 DARM model for ER8/O1. We will analyze DARM time-varying parameters (sometimes generalized as "kappas") calculated in GDS pipeline in GDS frames from nominal low-noise lock stretches after this change.
Previously values written in these EPICS records were calculated based on ER7 model (see LHO alog 20452), now we used ER8/O1 model with parameter file "H1DARMparams_1124954397.m", the summary of this param file residuals were given in LHO alog 21318. There is an additional phase of 136.7 degrees that was fudged into H1:CAL-CS_TDEP_REF_INVA_CLGRATIO_TST_(REAL|IMAG) that should provide kappa_tst with a reasonable phase value (ideally the phase of the kappa_tst should be 0), this was done so that we can test overall performance of GDS pipeline when calculating kappas.
These are not the final values for ER8/O1, currently H1 DARM OLGTF model is under testing using different DARM OLGTF and PCAL2DARMTF measurements and possibly will go through minor adjustments.
New values were accepted in SDF_OVERVIEW (however some of the values still does show up in "diffs", that's because SDF_OVERVIEW cannot handle very small values).
Details
Earlier we had a hard time getting reasonable values for kappas, in Maddie's GDS code as well as recent Greg Mendell and Sudarshan's Matlab code using PCALMON data (see LHO alog 21102). One of the reasons why calculations wouldn't fully agree with actual parameters was that EPICS records had values calculated from ER7 model, which we know is not up to date anymore. But I belive that even after we got a fairly good approximated ER8 model there's probably something missing in EPICS actuation coefficient for x_{tst} calibration line. We're still investigating the source of this discrepancy.
The script that extracts values needed for calculation of kappas from DARM OLGTF model was committed to calibration SVN:
CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/CAL_EPICS/writeER8model_TDEP_EPICS.m
Output files (raw epics values and more verbose log) were committed to the same directory, detailed log is also attached to this alog.
The EPICS values correspond to the model file and the parameter file in calibration SVN at rev. 1374:
CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/DARMOLGTFs/H1DARMOLGTFmodel_ER8.m
CalSVN/aligocalibration/trunk/Runs/ER8/H1/Scripts/DARMOLGTFs/H1DARMparams_1124954397.m
This morning Elli and I spent some time on phaseing of AS WFS, and again this evening we picked it up again. More details to follow, but here are some highlights:
We can use a consistent phase on all 4 quadrants to get reasonable signals.
In PRMI we can use 40 degrees for all quadrants on ASA36 to get most of the static signal into the I phase, and 90 degrees on AS 36 B. In this situation the Q phase has a good signal for the BS, with the transfer function from BS pit drive (1000 cnts, 12 Hz) to each quadrant having all the right phase relationships.
In DRMI 90 degrees still seems like a good setting for keeping the static signals in the I phase for AS B, while AS A it seems like 20 degrees is a better phase, except for quadrant 4 where it seemed like the phase should be 70. I assume that this could be caused by a misalingment, and should not be a real since all the phases were consistent in PRMI.
We closed the loops on AS B in DRMI, which worked fine, and locked. The phases of the transfer function to each quadrant stayed consistent in DRMI, full lock, and at 10 Watts. I left the excitation on the BS as I powered up, starting around 7:08 UTC sept 10th. Durring this time all the quadrants on AS 36 A had an R phase of 20 degrees, all the R phases for AS B 36 were 90, and the yaw matrix for both wfs was normal (it included all quadrants.) After powering up to 10 Watts we saw the usual runaway where POP90 grows. We can perhaps look at this data to see if our sensing matrix was changing durring some thermal process.
After several minutes at 10 Watts, I tried to recover a good operating point by moving the offset in AS_C, which did not work and broke the lock.
Everything is reverted and we are going back to locking with our old arangement for AS WFS.
LVEA: Laser Hazard IFO: UnLocked Intent Bit: Commissioning All Times in UTC (PT) 23:30 (16:30) Take over from TJ 23:30 (16:30) Relocking the IFO 23:52 (16:52) Locked at NOMINAL_LOW_NOISE, 22.7W, 68Mpc 00:02 (17:02) Lockloss – commissioning activities 00:36 (17:36) Locked at NOMINAL_LOW_NOISE, 22.7W, 68Mpc 00:38 (17:38) Lockloss – commissioning activities 01:09 (18:09) Locked at NOMINAL_LOW_NOISE, 23W, 68Mpc 01:10 (18:10) Lockloss – commissioning activities 01:31 (18:31) Locked at NOMINAL_LOW_NOISE, 22.9W, 63Mpc 01:47 (18:47) Set intent bit to Observing, 22.9W, 69Mpc 02:25 (19:25) Set intent bit to Commissioning – Evan working on noise characterization 03:05 (20:05) Range bump (over 80Mpc) due to calibration work 05:13 (22:13) Lockloss – Commissioning activities 05:30 (22:30) Adjust ISS diffracted power from -2.02v to -2.00v – diffracted power now at 8.4% Shift Summary & Observations: - Attend all hands meeting until 23:20 (16:20) - Evan working on noise characterization - Over 80Mcp range bump due to calibration work – NOT REAL - At the 05:13 Lockloss, the OMC watchdog tripped 4 seconds after the Lockloss - The first part of the shift was given to the commissioners while LLO was down. After LLO came back up and was in Observing mode, LHO switched to Observing mode as well. We had a 4-hour lock stretch in the middle of the shift with good range, moderate wind, no significant seismic activity. After losing lock, commissioning work continued until the last hour of the shift. Balance of the shift was spent relocking and doing fine tuning commissioning.
While we were in commisioing mode, we had a strange non stationary spectrum, with large non stationary noise also appearing in MICH and PRCL. I quickly looked at some triple witness sensors, since we've had issues with these recently, but didn't see anything obvious.
The strange noise started a few minutes before 2:50:34 UTC Sept 10, seemed to get better for a few minutes and then got worse again around 2:57:21.
We don't think that this was related to any comissioning we were doing. Evan had temporarily lowered the DARM ugf and put a notch in, (which should cause anything like this). He undid both changes when we saw the unusual noise, but the noise was unaffected by his change.
We aren't going to investigate any further right now since the problem seems to have gone away.
This is happening again. (this time it seems worse) It started about a half hour ago, at 22:30 UTC on Sept 14th
We ran a bruco on 5 minutes of this excessively noisy period (2015-09-14 22:55:00). Results here.
From 10 Hz to 1 kHz, most of the LSC and ASC signals are coherent with DARM, so it's hard to say what's going on here.
Above 1 kHz, there is some coherence with the RFAM monitor on the EOM driver. A series of spectra are attached showing elevated RIN in this monitor, compared to a quiet period roughly 30 minutes later. What mechanism could cause this? Also included are spectra of the LSC magnetometer in the CER, since this also showed elevated coherence.
RF45 AM stabilization looks fishy.
Look at feedback signal (Ch3) VS the range. They're in sync today (left) as well as back on 10th of Sept. (right).
One possible cause of the DAC overflows we've been seeing (the ETMY saturation warnings) is something at high frequency using up the range of the drive. We only record the last output to the ESD DAC at 2048 Hz, so we don't see anything happening above a kHz. It's possible some high frequency line or feature bumps up against 2^17 counts and that makes the DAC overflow, which then causes a much larger glitch - once there's an overflow the whole thing will go crazy. To try to investigate this, the simplest thing to do is to record a channel like SUS-ETMY_L3_MASTER_OUT_LL_DQ at the highest rate possible, or something equivalent that monitors the actual signal sent to the ESD DAC. Once an ETMY overflow happens, save the data for several seconds around the time somewhere that people offsite can access it, preferably in a common format like ASCII or frames. Some time without an overflow would be good for comparison. Then we can see if there's something using up the drive range, or if it's just something sudden from somewhere else in the DARM loop. A few instances of overflows would be nice. If this is not possible, there's another tack we can take but it's more involved, so I'd prefer this.
If it's any help, we do record H1:SUS-ETMY_L3_ISCINF_L_IN1_DQ at the full rate (16 kHz). In full lock, this is the only signal (other than the calibration line) that is sent to the ESD. With an adequate knowledge of the filters between this channel and the ESD master outs, it should (in theory) be possible to reconstruct the master out signals at the full rate. That was the original motivation for recording this channel at full rate, anyway.
Yes, using ISCINF was the other tack that I meant. I checked the MEDM screen when I knew the detector was locked. Here's what I see in the chain: ETMY_L3_ISCINF_L - seems to be empty ETMY_L3_LOCK_L - FM1,3,5,6,8,9,10; gain is 1.25 ETMY_L3_DRIVEALIGN - L2L is only thing used, just a gain of -30 ETMY_L3_EULER2ESD matrix - factor of 0.25 to each OSEM ESD linearization - bypassed, so I think we can ignore it ETMY_L3_ESDOUTF_LL - FM6,7; gain is 1. (I think the four coils get nearly the same drive, so any quadrant is fine). The needed filters are available from the web SVN in this text file. So there's only two filter banks to apply, and a few gains. I think this is doable, though it seems like it would just be easier to record SUS-ETMY_L3_MASTER_OUT_LL_DQ at 16384 Hz for a few hours of lock. Maybe one or two of the LSC Fellows could take on this more complicated method.
I managed to capture the full data for SUS-ETMY_L3_MASTER_OUT_LL around the time of the most recent glitch. I put it on my account on caltech at /home/jordan.palamos/detchar/ETMY_glitch.txt and also at https://lhocds.ligo-wa.caltech.edu/exports/jordan.palamos/.
The start time is 1125946072.8
Evan, Sheila, Jeff B Ed
Earlier today we were knocked out of lock by a 5.2 in New Zealand, the kind of thing that we would like to be able to ride out. The lockloss plot is attached, we saturated M3 of the SRM before the lockloss and PRM M3 was also close to saturating.
While LLO was down, we spent a little time on the offloading, basically the changes described in alog 21084. This offloading scheme worked fine in full lock for PRM, however we ran into trouble using it durring the acquisiton sequence. Twice we lost lock on the transition to DRMI, and twice we lost lock when the PRM coil state swtiched in DRMI on POP. Hpwever, we can acquire lock with the new filter in the top stage of PRM and SRM, but the old low gain (-0.02). We've been able to turn the gain up by a factor of 2 in full lock twice, so I've left the guardian so that it will turn of the gain in M2 (before the intergrator) in the noise tunings step.
If anyone decides they need to undo this change overnight, they can comment out lines 344-347 and 2508-2514 of the ISC_LOCK guardian.
Before we started this, the PRM top mass damping was using 10000 cnts rms at frequencies above a few hundred Hz, because of problems in the OSEMS (alog 21060 ). Evan put some low passes at 200 Hz in RT and SD OSEMINF which reduces this to 2000 cnts rms. Jeff B accepted this change in SDF.
The second attached screenshot shows the PRM drives, the references are in the minutes before the earthquake dropped us out of lock. The red and blue curves show the current drives, with the high frequency reduction in M1 due to Evan's low pass, and the new offloading on. The last attached screenshot shows SRM drives with the new offloading.
I don't think the 5.2 EQ is the cause of the lockloss.
According to your plot, the lockloss happened on Sep 08 at 00:31:07 UTC. The 5.2 EQ happened on Sep 07 at 20:24:56.84 UTC and hit the site at 20:38:21 UTC according to Seismon (so 4 hours earlier). The BLRMS plot confirm that statement (see attachment).
Around loss time, the ground seems as quiet as usual.
I was mistaken in identifying the earthquake, but the ground motion did increase slightly, which seems to be what caused the lockloss.
