Keita noticed that the ALS laser was still present in the arms. With his assistance, I have misaligned the PZT2_PIT for both the arms and the PZT1_PIT for the X arm. This brought our range back to where it was prior to the 'big dip'.
We will need to remember to relign these when we need the ALS system next.
Note for data analysts: This misalignment occurred at ~17:40, with the Intent Bit still set to Undisturbed.
The full report can be found on the detchar wiki
Highlights:
ER8 Day 3. No restarts reported.
ER8 Day 2. Maintenance Tuesday
model restarts logged for Tue 18/Aug/2015
2015_08_18 08:35 h1susetmx
2015_08_18 08:35 h1susetmy
2015_08_18 08:42 h1susitmx
2015_08_18 08:42 h1susitmy
2015_08_18 08:52 h1odcmaster
2015_08_18 09:07 h1oaf
2015_08_18 09:15 h1dc0
2015_08_18 09:16 h1dc0
2015_08_18 09:50 h1broadcast0
2015_08_18 09:50 h1dc0
2015_08_18 09:50 h1fw0
2015_08_18 09:50 h1fw1
2015_08_18 09:50 h1nds0
2015_08_18 09:50 h1nds1
2015_08_18 10:12 h1iopsush56
2015_08_18 10:14 h1susomc
2015_08_18 10:14 h1sussr3
2015_08_18 10:14 h1sussrm
2015_08_18 14:55 h1broadcast0
various model changes. Bad DAQ restart. sush56 restart for DAC calibration. GDS reconfiguration.
LVEA: Laser Hazard
IFO: Locked
Observation Bit: Commissioning
All Times in UTC
07:00 Take over from TJ.
07:00 IFO Locked at NOMINAL_LOW_NOISE
07:57 Set intent bit to Undisturbed – 70Mpc range, Good noise curve
10:30 Increasing noise – Decreasing range – Seismic activity
12:30 Noise starting to decline – Range increasing
13:50 After a couple of small glitches – Range has recovered to ~60Mpc
14:20 5.5 EQ in Papua New Guinea – Small blip decrease in range with similar increase in noise
Shift Summary & Observations:
At start of shift had good lock with low noise and a 70Mpc range.
There were a series of glitches between 08:54 & 10:30 – Saturation of ETM-Y.
10:30 – Started to see an increase in noise and decrease is range. Seismic activity on the 30m-100m StripTool plot. No wind. 4.5 EQ in the Northern Marianas and 5.7 EQ in Indonesia around this time. Also had building heavy traffic (for Rt 10) on Rt 10 during this time.
14:00 – Range back up to around 60Mpc. Took no action during the dip and recovery in range.
Using H1:SUS-ITMY_L2_DAMP_MODE2, FM1, FM2, FM4. gain=-2000 Monitor: H1:SUS-ITMY_L2_DAMP_MODE6_RMSLP_LOG10_OUTMON This was also added to the wiki: https://awiki.ligo-wa.caltech.edu/aLIGO/H1%20Violin%20Mode
Using H1:SUS-ETMY_L2_DAMP_MODE10, FM6, FM9, gain=-800 Monitor: H1:SUS-ETMY_L2_DAMP_MODE10_RMSLP_LOG10_OUTMON Also had to add a notch in H1:SUS-ETMY_L2_DAMP_MODE3 FM10, because it was ringing the mode up. All this was added to the wiki: https://awiki.ligo-wa.caltech.edu/aLIGO/H1%20Violin%20Mode PS: actually -5000cts worked too.
Not much to report from me. Commissioners wroked their magic pretty much the whole night. Sheila and I did an Initial alignment at 0:42 UTC. Locked in NOMINAL_LOW_NOISE right now, handing it off to Jeff B.
Stefan, Sudarshan, Evan
After recovering low-noise locking, we noticed something was wrong with the calibration: the region from 10 to 30 Hz was too low, and the region from 50 to 200 Hz was too high.
We looked at the calibration lines, we took a DARM OLTF, we compared SDF settings, we inspected foton coefficients, but we could not figure out what was wrong. [Actually, the EY L3 lock gain in the CAL CS model was 1.0 instead of 1.25, but that wasn't the root of the problem.]
In the end it turned out that FM1 of CAL-CS_DARM_FE_ETMY_L1_LOCK_L has a pole at 0 Hz, and the filter had charged up, causing the CAL CS version of the L1 control signal to show a fake, white-ish noise floor above 10 Hz.
We turned the filter off and then back on again, which fixed the problem. However, we can already see that the filter has started to charge up again. So we've replaced the 0 Hz pole with a 0.001 Hz pole.
Sudarshan also found that the CAL CS version of the whitened DARM control signal had a fake 1/f shape between 8 and 80 Hz, which was due to insufficient whitening. So now there is an extra pair of 1 Hz zeros and 100 Hz poles in CAL-CS_DARM_DELTAL_CTRL_SUM_WHITEN.
Plots of DELTAL_CTRL signal before and after this fix was implemented.
Just a note:
As for the pole at 0 Hz, this has been a long standing issue and we have mitigated it by not enagling this particular filter (see (B) part of alog 17528). The issue is now exposed again because of my automation scripts (see alog 20617). I was hoping to get around it by clearing the past history of the filter every time when a new lock starts.
Sheila, Jenne, Nergis, Daniel, Stefan Most of the things we tried today we tired today were goose chases, but here are the things that we left in: - We moved SR3 mack to where is was before we turned on the cage servo. - We (for now) switched back from the cage servo to the OPLEV servo. - We switched back to the old SRC1 yaw input matrix: AS_A_RF36_I_YAW to SRC1_Y : -3 AS_B_RF36_I_YAW to SRC1_Y : 1 That's the matrix for which we had done the careful SRCL FF tuning. - We scratched our heads for half a day about why the MICH_P loop goes unstable at high power. - We then threw our hands in the air, looked at what other signals see MICH_P (i.e. BS pitch motion) - turns out to be AS_A_RF36_I, with a gain that is roughly -3/2 of that one in AS_B_RF36_Q (at least at low power) - We punched in an input matrix that includes both error signals: AS_A_RF36_I_PIT to MICH_P : -0.666 (remember that AS_A was never phased for PIT, so don't wonder too much about the I...) AS_B_RF36_Q_PIT to MICH_P : 1.0 With those signals we were able to hold lock at 23W, although we still could improve the ASAIR_90 buildup by slightly offsetting the error point - This also significantly reduced the drop in recycling signals right after power-up. - After about 30min at high power, we stepped the MICH_P offsets by 100cts, and noticed that there was no signals left in AS_B_RF36_Q_PIT - no surprise that we lost lock on this signal before... - After 3h of locking I revisited the MICH_P offsets, and found that no offset, but completely switching to AS_A_RF36_I_PIT gave us the best performance: AS_A_RF36_I_PIT to MICH_P : -1.333 AS_B_RF36_Q_PIT to MICH_P : 0 After than we damped some violins (see above), and turned on whitening. All this stuff is not in the Guardian yet - to be done tomorrow.
As part of the recovery effort today we stumbled upon a way that seems to fairly reliable allow us to transition from PRMI to DRMI (we were 3 for 3 today - we'll try again tomorrow). The procedure is: - Let Guardian set up for DRMI locking. - Misalign SRM - turn off H1:LSC-SRCL OFFSET, INPUT and OUTPUT - Let PRMI lock - make sure H1:LSC-SRCL OFFSET, INPUT and OUTPUT are off (guardian turns on OFFSET), and clear history. - Pause ISC_LOCK and ISC_DRMI guardian - Align SRM, wait until it damps down. Today the PRMI always stayed locked in this configuration. - Turn on H1:LSC-SRCL OUTPUT - Turn on H1:LSC-SRCL INPUT. DRMI is now locked. - Turn on H1:LSC-SRCL OFFSET, if desired for mode-hopping suppression
Already done at Livingston a long time ago...
https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=11340
Except this scheme seems much faster than waiting for DRMI...
As recorded in r11378, I have fixed a bug in the ext_alert.py GRB monitor that prevented it from running.
Can this monitor be restarted at the earliest available opportunity?
S. Dwyer, D. Hoak, J. Driggers, J. Bartlett, J. Kissel, C. Cahillane
We had trouble with the transition to ETMY, which turned out to be the same problem that we had with ETMX ESD driver this morning (20652) -- the Binary IO configuration was toggled such that high-voltage driver was disconnected, but we were still driving through the high voltage driver. This was the source of several lock losses during this evening's maintenance recovery. We'd found it by comparing a conlog between now and during yesterday's DARM OLGTF measurements by the calibration group. Once we fixed it, we made sure to accept the configuration in the SDF (where we looked first, but it turns out that the wrong configuration had been stored there).
For the record, the attached screenshot shows the correct configuration for ETMY BIO. The good configuration is with have HI/LO Voltage OFF to run with the low voltage driver and the HI Voltage Disconnected (i..e the switch is OFF). In the same screenshot, we show what the DARM loop gain looks like with (yellow) locked on EX alone, (blue) a half-transition to EY when EY is in the bad configuration, (red) a half transition to EY with EY in the good configurtation.
The noise tonight: excess stuff between 30 and 200Hz (first plot). The lower frequency end is very nonstationary and coherent with LSC (second plot) and ASC (third plot) signals. Between 100 and 200Hz it's quite stable. The ISS second loop is not on.
MICH Feedforward appears to be off or mistuned.
The ISS should not be running far from the hard-coded 8% diffracted power. 13% (reported in 20663) is probably too close to the upper limit... I would not change the second loop target value, but rather set the inner loop offset so that the diffrected power runs near 8%.
The amount of excess noise was anomalously bad from this lock stretch. It's not so surprising that the MICH FF was not working.
The current 17+ hour lock has a more typical DARM noise, at least during the high-range stretches. Here the MICH FF seems to be mostly working, although we could get rid of some more coherence by retuning it.
Also note the coherence with PRCL in the region where we have the PSL PZT peaks...
While Travis was working on initial alignment earlier today, we found that we were once again having trouble turning on the ASC for the X green arm. I think the trouble was that the DC centering loops' error signals (from the green ITMX camera) were a little bit too large. This isn't something that happens too often, but it definitely caused the ASC to pull the arm away from resonance. Engaging the ASC without the DC centering was fine, so it's definitely something with that pair of loops.
By hand, I turned the gain on the DOF 3 loop for both pitch and yaw (which handles the DC centering) to half of the nominal value. Once the error signals were closer, I put the gain back up at the nominal value for both loops.
Since this is a particularly tricky problem to troubleshoot, I've tried to handle it in the ALS Arm guardians. The modifications were made in the "generator" states, so they are the same for each arm. The actual gain values for each arm are stored in the alsconst.py file, which is already loaded by the generator states.
I have loaded this new code, but it has not been fully tested, since we have not done initial alignment since I loaded it (and that's where it'll get used). I've tried to test the logic in the guardian shell and that all seems to be fine, but there's no test like doing it live. If it is giving errors that seem insurmountable, I did an svn checkin of the as-found code before I started modifying it (as well as another checkin after my edits), so we can go back one svn version.
This is a sign of ghosts past! I seem to remember that we solved this problem months ago with a delayed boost/gain which is engaged by the WFS FM triggers. Is it clear that going back to the original settings doesn't address it?
Here is the current SVN status of the front end filter file's svn status.
A reminder, in the /opt/rtcds/lho/h1/chans/ directory, all the filter files should be symbolic links to the appropriate userapps filterfiles area.
I found that the following chans directory files are not symbolic links:
H1CAL[CS,EX,EY].txt, H1SUSETM[X,Y]PI.txt
I found that all symbolic links used the absolute path of /opt/rtcds/userapps/release/ except for:
../../../userapps/release/sus/h1/filterfiles/H1SUSBS.txt
Here is the list of the svn status:
M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM1.txt
M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM6.txt
M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM2.txt
M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM3.txt
M /opt/rtcds/userapps/release/isi/h1/filterfiles/H1ISIHAM3.txt
M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM4.txt
M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM5.txt
M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIITMY.txt
M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIBS.txt
M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIITMX.txt
M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSPRM.txt
M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSPR3.txt
M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSSRM.txt
M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSSR3.txt
M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSOMC.txt
M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSITMY.txt
M ../../../userapps/release/sus/h1/filterfiles/H1SUSBS.txt
M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSITMX.txt
M /opt/rtcds/userapps/release/pem/h1/filterfiles/H1PEMCS.txt
M /opt/rtcds/userapps/release/isc/h1/filterfiles/H1OAF.txt
M /opt/rtcds/userapps/release/lsc/h1/filterfiles/H1LSC.txt
M /opt/rtcds/userapps/release/omc/h1/filterfiles/H1OMC.txt
M /opt/rtcds/userapps/release/lsc/h1/filterfiles/H1LSCAUX.txt
M /opt/rtcds/userapps/release/asc/h1/filterfiles/H1ASC.txt
M /opt/rtcds/userapps/release/asc/h1/filterfiles/H1ASCIMC.txt
M /opt/rtcds/userapps/release/pem/h1/filterfiles/H1PEMMX.txt
M /opt/rtcds/userapps/release/pem/h1/filterfiles/H1PEMMY.txt
M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSETMY.txt
M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIETMY.txt
M /opt/rtcds/userapps/release/isc/h1/filterfiles/H1ISCEY.txt
M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSETMX.txt
M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIETMX.txt
and here are the scripts which generated it (running in the chans directory):
for i in 'awk '{print $1}' ../cds/model_info.txt';do echo $i|tr [a-z] [A-Z]|awk '{print $i".txt"}'>>/tmp/filtfilelist.txt;done
for i in 'cat /tmp/filtfilelist.txt' ;do ls -al $i >> /tmp/filelonglisting.txt;done
for i in 'grep "->" /tmp/filelonglisting.txt |awk 'BEGIN{FS=" -> "}{print $2}'';do svn st $i;done
I've copied the H1CAL*.txt files over to the userapps repo, committed them, and created a soft link in the chans directory, lrwxrwxrwx 1 jeffrey.kissel controls 58 Aug 18 19:25 H1CALCS.txt -> /opt/rtcds/userapps/release/cal/h1/filterfiles/H1CALCS.txt lrwxrwxrwx 1 jeffrey.kissel controls 58 Aug 18 19:25 H1CALEX.txt -> /opt/rtcds/userapps/release/cal/h1/filterfiles/H1CALEX.txt lrwxrwxrwx 1 jeffrey.kissel controls 58 Aug 18 19:25 H1CALEY.txt -> /opt/rtcds/userapps/release/cal/h1/filterfiles/H1CALEY.txt I've also fixed the BS softlink, and committed the current filter file to the repo, lrwxrwxrwx 1 jeffrey.kissel controls 58 Aug 19 01:14 H1SUSBS.txt -> /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSBS.txt
All the HEPI and ISI files were commited. The mass HEPI mods were from the running of foton -c on the files. The HAM3 ISI change was related to weiner filters--ready to test these at opportunity.
What happened to our shutters?
Shutters were closed at ~18:45 UTC. Sheila has since returned the shutters and beam diverters to their appropriate states in Guardian. The ALS Guardians will need to be reloaded when they are needed again.