Attached is agallery of 5 "dust" glitches. Still clueless of what they are, but - ETMY saturation is a symptom, not a cause - it is not possible to produce such a white glitch from saturating a drive. - The DCPD spectrum shows a roll-off for all of them - But the roll-off frequency (i.e. glitch duration) varies significantly = from about 300Hz to 3kHz. Example 2: GPS: 1126294545 UTC: Sep 14 2015 19:35:28 UTC ETMY saturation: yes Example 3 GPS: 1126437892 UTC: Sep 16 2015 11:24:35 UTC ETMY saturation: yes Example 4 GPS: 1126434798 UTC: Sep 16 2015 10:33:01 UTC ETMY saturation: yes Example 5 GPS: 1126441165 UTC: Sep 16 2015 12:19:08 UTC ETMY saturation: yes Example 6 GPS: 1126442379 UTC: Sep 16 2015 12:39:22 UTC ETMY saturation: yes
WIth Hang's help, I managed to investigate these glitches with the new lockloss tool using SUS-ETMY_L3_MASTER_OUT_LL_DQ as a reference channel. The script couldn't find any other optics that glitch prior to the ETMY. And sometimes the glitches are seen by ETMX 30-40 miliseconds after.
I've attached the plot of the glitches at the time you've given. I've also attached the list of channel I told the script to look. Basically all the SUS MASTER OUT DQ channels. Please let me know if you have any suggestions on whereelse I should look at.
Attached are time traces of the DCPD_SUM for the 5 examples.
We just loaded two minor guardian changes that should save us some time and locklosses during acquisition. They don't change anything about the configuration of the IFO in observing mode.
The first was the fix the issue that Jenne and Patrick wrote about 21508. When the SOFT loops had not converged before the gain was increased, it could cause us to loose lock (which could be because the SRC1 loop is strongly contaminated by these signals, and we see POP90 running away in the usual way.) I've simply added another if statement to the ENGAGE_ASC part 3 state, which will wait for all the SOFT loop control signals to become less than 500:
The Initial Alignment Checklist and ops wiki page's "Initial alignment brief version" have been updated to reflect this change.
When you compare "H1 SNSW EFFECTIVE RANGE (MPC) (TSeries)" data in DMT SenseMonitor_CAL_H1 with its copy in EPICS (H1:CDS-SENSEMON_CAL_SNSW_EFFECTIVE_RANGE_MPC), you will find that the EPICS data is "delayed" from the DMT data by about 109 seconds (109.375 sec in this example, I don't know if it varies with time significantly).
In the attached, vertical lines are minute markers where GPS second is divisible by 60. Bottom is the DMT trend, top is its EPICS copy. In the second attachment you see that this results in the minute trend of this EPICS range data becoming a mixture of DMT trend from 1 minute and 2 minutes ago.
This is harmless most of the time, but if you want to see if e.g. a particular glitch caused the inspiral range to drop, you need to do either a mental math or a real math.
(Out of this 109 seconds, 60 should come from the fact that DMT takes 60 seconds of data to calculate one data point and puts the start time of this 1 min window as the time stamp. Note that this start time is always at the minute boundary where GPS second is divisible by 60. Remaining 49 seconds should be the sum of various latencies on DMT end as well as on the copying mechanism.)
The 109s delay is a little higher than expected, but not to strange. I'm not sure where DMT marks the time, as the start/mid/end of the minute it outputs.
Start Time | Max End Time | Stage |
0 | 60 | Data being calculated in the DMT. |
60 | 90 | The DMT to EPICS IOC queries the DMT every 30s. |
90 | 91 |
The EDCU should sample it at 16Hz and send to the frame writter. |
The 30s sample rate of the DMT to EPICS IOC is configurable, but was chosen as a good sample rate for a data source that produces data every 60 seconds.
It should also be noted that at least at LHO we do not make an effort to coordinate the sampling time (as far as which seconds in the minute) that happen with the DMT. So the actual delay time may change if the IOC gets restarted.
EDITED TO ADD:
Also, for this channel we record the GPS time that DMT asserts is associated with each sample. That way you should be able to get the offset.
The value is available in H1:CDS-SENSMON_CAL_SNSW_EFFECTIVE_RANGE_MPC_GPS
20:28 UTC Back to observing mode.
C. Cahillane I have generated the latest uncertainty results for LHO. I have characterized the uncertainty in the actuation function by quadratically summing the statistical and systematic errors. There are more elegant ways to combine systematics and statistical errors, but for now we will see if this is sufficient. Plot 1 and 2 are the Mag and Phase components of σ_h^2, and Plots 3 and 4 are the Mag and Phase of σ_h itself. We have greater than 10% and 10 degree uncertainty just below 100 Hz with our current analysis, but this is just an update with inflated flat uncertainties in the kappas and f_c. Note that the linear sum of our PUM and UIM stage uncertainties into our A_pu uncertainty is not correct, but only an estimate used for calculations in the meantime. I have performed the correct calculations in Mathematica and will implement them given time. The following are the sigma values I have used: σ_|A_tst| = A_coeff_sigma_mag_A_tst .* abs(A_tst); σ_|A_pu| = A_coeff_sigma_mag_A_pum .* abs(A_pum) + A_coeff_sigma_mag_A_uim .* abs(A_uim); σ_|C_r| = 1.0 .* abs(C_r); σ_|kappa_tst| = 5; σ_|kappa_pu| = 5; σ_φ_A_tst = A_coeff_sigma_phase_A_tst; σ_φ_A_pu = A_coeff_sigma_phase_A_pum + A_coeff_sigma_phase_A_uim; σ_φ_C_r = 5; σ_φ_kappa_tst = 5; σ_φ_kappa_pu = 5; σ_kappa_C = 5; σ_f_c = 23.4731; I have inflated the errors in the kappas and f_c to 5% and 5 degrees based on our systematics seen from the calibration group calculations. We hope that we can deflate these errors when the calibration group discovers the systematics responsible for our kappa deviations from 1. The next step is to propagate uncertainty for the sensing function, after which I will have finished analysing the uncertainty of measurements at LHO. Then I may move onto LLO. Steps: 1) Get sensing function uncertainties at LHO 2) Implement correct A_pu uncertainty propagation 3) Check sanity 4) Move on to LLO uncertainty measurements (?) Correct for kappas and f_c uncertainties as better calculations are made available. For now, apply flat 5% and 5 degrees.
15:55 David N. through gate Had trouble locking on DRMI. Tried locking on just PRMI. Had a few locks on PRMI, but would not stay. Eventually gave up and started an initial alignment at 17:47 UTC. Finished initial alignment with help from Sheila at 18:55 UTC. Had trouble with ALS during initial alignment. Sheila and Jenne now suspect we may be losing lock engaging the gain for the DSOFT loop. They are starting a test now.
[Sheila, Patrick, Jenne]
We discovered that the problem was we had some medium-large offsets on the input of the SOFT yaw loops, and when the gain was being increased in ENGAGE_ASC_PART3, the loops started to run away.
For this lock, we hand-ran the "main" part of ENGAGE_ASC_PART3, which turns the loops on with low gain. Once the loops had converged (~ 10 minutes for all of them to finish!), we let guardian run the full ENGAGE_ASC_PART3 state, and continue on. We have just arrived at NOMINAL_LOW_NOISE, and we don't have any SDF diffs, so there is no net change in the configuration of the IFO.
We would like to think about putting a "convergence checker" in this guardian state, so that it waits until the loops are happy before continuing. Sheila has implemented this with great success several weeks ago in the DRMI ASC parts of the guardian code, so it should be relatively simple to do the same thing for the arm loops.
TITLE: Sep 16 OWL Shift 07:00-15:00UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Was observing at ~70 Mpc for 4 hours after the 6.3M earthquake in Indonesia. We lost lock again due to a 6.0 earthquake in Papua New Guinea. Mother nature wasn't happy.
SUPPORT: Rick S., Darkhan T., Corey G.
SHIFT SUMMARY: After the 6.3M earthquake in Indonesia, things had been quiet until another 6.0M earthquake in Papua New Guinea. There were several small earthquakes (less than 3 mag) in California, Oregon, and Alaska but the interferometer hardly noticed them. Wind speed >=10 mph. 4 ETMY glitches that were loud enough to be noticable on range plot. Overall trend of the microseism is slowly coming down. LLO has been down since earthquake. Danny said there were a problem with ALS PDH servo board.
INCOMING OPERATOR: Patrick
Activity log:
07:00 Took over from Ed. The ifo was locking and observing. No more nasty 2Hz comb (yay!).
08:07 Rick called about LVEA light. I switched to commissioning to go into the LVEA to turn off the light.
08:12 Back to Observing. The light in the PSL area was still on. Called Rick again. We thought it was the light from the change room.
08:30 After LLO went down I switched the intent bit to commissioning again and went to find PSL change room light switch. It wasn't the change room light. It was the crain.
Came back few minutes after and found out we had a lock loss at 08:01 Due to an earthquake in Indonesia.
09:02 Began lock acquisition. The earthquake band amplitude was still 10^-1 um/s but ALS-TRX and TRY didn't glitch much so I gave it a try.
09:15 Craig and Darkhan left the control room.
09:18 Corey called about the crain light. I went out to LVEA to take care of it. I switched off the door access reader on my way out.
09:23 Back in the control room. I realized that the operating mode was still commissioning. It should have been in Environmental for ~30 minutes. Since I have started the lock acquisition I switched the operating mode to LOCK ACQUISITION.
09:47 Back to Observing. The ifo have reached NOMINAL_LOW_NOISE few minutes ago but I thought the 3001 Hz Pcal line wasn't there. I called Darkhan and apparently the spectrum needed a narrower BW for the line to show (I saw it at 0.01Hz BW).
12:00 Karen and Christina on site
13:00 Richard on site.
14:28 Lockloss. 6.0M earthquake in Papua New Guinea.
15:00 Handing off to Patrick
Note:
- EX dust monitor looks bogus. The alarm that went off probably wasn't real neither.
STATE Of H1: Earthquake OUTGOING OPERATOR: Nutsinee QUICK SUMMARY: Unlocked, possibly due to earthquake. Winds less than 10 mph. Peak to around 1 um/s in 0.03 - 0.1 Hz seismic. 0.1 - 0.3 Hz seismic around 10^-1 um/s. Lights off in LVEA. Lights appear off in PSL enclosure. Lights appear off at mid and end stations.
ER8 Day 30. h1ecaty1 all three PLCs restarted due to Beckhoff chassis work.
Tue 15 Sep 2015 11:34:25 h1ecaty1plc[1,2,3]
ER8 Day 29. No restarts reported.
ER8 Day 28. No restarts reported
Due to a 6.0M earthquake in Papua New Guinea. Requested ISC_LOCK to DOWN. Mother nature hasn't been very happy today....
Terramon showed this earthquake 4 minutes after the lockloss.
Due to a 6.3M earthquake in Indonesia. I didn't take anything down and hoped that we could survive. Nothing has tripped. I've requested ISC_LOCK to DOWN waiting for the earthquake to pass.
Earlier I set the bit to commissioning briefly and went in to turn the light off in the LVEA as Rick suggested. He thought it'd be better if the LVEA was to be in the same condition as couple nights before and I agreed. However I noticed the light was on near the PSL area and I couldn't find the light switch for it. I came back, set the intent bit to Observing and called Rick. He suggested the light switch would be in the change room by the PSL. Shortly after the earthquake took down LLO I set the intent bit to commissioning and went in again and realized that the light didn't come from the change room but came from the ceiling (part of the crain?). I didn't know where to turn it off so I left it as that. I've attached a photo. If you know where/how to kill the light please call and I will take care of it.
Just chatted with Nutsinee about the southbay crane. I believe this one has the standard controller pendant which hangs from the bridge. The hand controller has controls for moving the crane (i.e. north/south/east/west/up/down), and it also has a red emergency button on the top. To power off the entire crane (including the lights), one just pushes this red button in.
We do have a remote/wireless controller for the West Bay crane (can't remember if we have other cranes with remote/wireless controllers). The controller for this is kept in a "mailbox" attached on the north wall of the west bay (near the "parking spot" of the crane).
Back to Observing 9:47 UTC at 75 Mpc.
I turned off the crain light by pushing the emergency button as Corey suggested. I also turned off the door access reader.
TITLE: Sep 16 Owl Shift 07:00-15:00UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Observing at ~70 Mpc for the past 6.5 hours
OUTGOING OPERATOR: Ed M.
QUICK SUMMARY: 2Hz comb disappeared. Microseismic seems to have flatten between 0.1 and 0.2 um/s. Wind speed <=10 mph. Craig and Darkhan doing calibration work. LLO is also observing. LVEA light is on.
Hugh, Sheila, Jenne, filling in for Patrick
We are filling in for Patrick at the moment because he is helping Fil to use the beckhoff test stand in the H2 building. We are trying to lock now after Patrick and Jenne did a full initial alignment.
This morning, Fil, Vern, Elli, Nutsinee and I swapped out the End Station 2 Beckhoff Chassis, because all the modules on the right rail were in the state INIT NO_COMM, and we could not change their state by requesting different states (including safe) or clearing errors. We swapped the chassis with a spare. Hugh and Patrick burt restored plc1,2,3 to 10 pm last night.
When we first arrived we saw that the high voltage power supplies for the ESD were off, Fil turned them on and restored the settings. This may be because the vacuum gauges which are controlled by beckhoff would have tripped them off. The ESD driver was off, but we did not reset it believing that we could reset it remotely. Later Betsy reset it by driving back to EY after she could not reset it in the normal way remotely.
Elli quickly noticed that something was amiss with the ring heaters at end Y, which are controlled by the same beckhoff chassis. We checked that the settings were all the same, it seems like there is some difference between the 2 chassis.
Fil, with help from several people, has spent most of the afternoon with the chassis we removed. We had trouble getting a test set up going that we could use to diagnose the problem, so Hugh, Jenne, and I are relieving Patrick for a few hours so he and Fil can work together on the removed chassis. They think that they have found a problem with one of the modules, and are currently replacing that module.
We are about to reach low noise (with the spare chassis, and incorrect power on the ring heaters). We hope to see that the comb is gone, and if we see that we may break the lock to revert to the old chassis if possible.
Patrick's notes about his shift:
10 am PDT safety tour changed from X end to Y end(this may have broken our noisy lock this morning)
18:11 UTC placed beckhoff chassis at End Y
18:20 UTC Rick retrieves (?) PCAL out of optics lab
18:30 UTC Dick in optics lab out at 12:41 PDT
18:31 UTC Fil restarts beckhoff computer (Guardian to down)
a little while later h1ecaty1 burt restores to 22:10 Sep 14
18:53 UTC Jeff B, Jason retrieve part from optics lab (?)
12:38 PDT UPS truck at LSB
20:37 start of initial alignment finished at 21:22
20:54 Sheila to H2 electronics back 20:56, Fil is still there
20:58 Betsy to EY to restart ESD back 14:31
14:58 UTC Ryan B. patching and rebooting alog 15:09 UTC Gate phone rang. Bubba G. answered first. It was the fire department. Bubba let them in. They wanted to do fire hydrant tests. Bubba told them not to. 15:11 UTC Fire department leaving 15:12 UTC David N. through gate 15:59 UTC Turned away fire department from gate. They wanted to check fire extinguishers. 16:06 UTC ETMY saturation alarm, lock loss, end Y ALS in fault 16:16 UTC Christina to mechanical room to get supplies 17:51 UTC King Soft water through gate. Parking in main parking lot. Wheeling in equipment for RO system on hand truck. 17:56 UTC Dave B. running an svn update 18:20 UTC Rick S. retrieving part from optics lab ? UTC Rick S. back 18:30 UTC Dick G. in optics lab 18:53 UTC Jeff B, Jason O. retrieving part from optics lab 19:30 UTC Jeff B, Jason O. done 19:38 UTC UPS at LSB 19:41 UTC Dick G. out of optics lab 20:37 UTC Starting initial alignment 20:54 UTC Sheila D. to H2 electronics room 20:56 UTC Sheila D. back 20:58 UTC Betsy to end Y for ESD restart 21:22 UTC Finished initial alignment 21:31 UTC Betsy back
To clarify, I didn't go to the optics lab as noted in Patrick's log. When retrieving the PMC from the optics lab was approved, I let Jason know that there was an opportunity to go to the lab, which he did as noted by Patrick.
Things were quiet. Then Stefan arrived, and I realized I had set the observatory mode but forgot the Intent Bit, and which spiraled into a bunch SDF differences. The LSC guardian showed a difference on a t-ramp (annoying that this kind of difference would keep us from going into Observe) and ASC had two diffs that I guess Evan logged about last night but didn't accept. Then Stefan went and set a bunch of ODC masks (this also would have kept us from going into observe, so also annoying) as well as tuning the whitening the IM4 QPD (which was saturating, Stefan assures me is out of loop so shouldn't affect anything). Guardian recovered from the earlier lock loss in 20 minutes (not annoying!) back to 70 mpc and it has been otherwise quiet.
We just came out of Observe for a minute so Stefan could set more ODC bits. Service has resumed.
The ODC status should not affect OBSERVATION READY in any way. If it does, then ODC is misconfigured and needs to be fixed.
SDF is unfortunately looking at all channels - including ODC mask bits. When anyone updates the ODC masks, the SDF goes red, and you then havta accept the ODC mask channel value in SDF, or ignore it. Again, it's a one-time thing to update these ODC values, just late in the game and not all at one time.