The full report can be found on the detchar wiki
Highlights:
ER8 Day 3. No restarts reported.
ER8 Day 2. Maintenance Tuesday
model restarts logged for Tue 18/Aug/2015
2015_08_18 08:35 h1susetmx
2015_08_18 08:35 h1susetmy
2015_08_18 08:42 h1susitmx
2015_08_18 08:42 h1susitmy
2015_08_18 08:52 h1odcmaster
2015_08_18 09:07 h1oaf
2015_08_18 09:15 h1dc0
2015_08_18 09:16 h1dc0
2015_08_18 09:50 h1broadcast0
2015_08_18 09:50 h1dc0
2015_08_18 09:50 h1fw0
2015_08_18 09:50 h1fw1
2015_08_18 09:50 h1nds0
2015_08_18 09:50 h1nds1
2015_08_18 10:12 h1iopsush56
2015_08_18 10:14 h1susomc
2015_08_18 10:14 h1sussr3
2015_08_18 10:14 h1sussrm
2015_08_18 14:55 h1broadcast0
various model changes. Bad DAQ restart. sush56 restart for DAC calibration. GDS reconfiguration.
LVEA: Laser Hazard
IFO: Locked
Observation Bit: Commissioning
All Times in UTC
07:00 Take over from TJ.
07:00 IFO Locked at NOMINAL_LOW_NOISE
07:57 Set intent bit to Undisturbed – 70Mpc range, Good noise curve
10:30 Increasing noise – Decreasing range – Seismic activity
12:30 Noise starting to decline – Range increasing
13:50 After a couple of small glitches – Range has recovered to ~60Mpc
14:20 5.5 EQ in Papua New Guinea – Small blip decrease in range with similar increase in noise
Shift Summary & Observations:
At start of shift had good lock with low noise and a 70Mpc range.
There were a series of glitches between 08:54 & 10:30 – Saturation of ETM-Y.
10:30 – Started to see an increase in noise and decrease is range. Seismic activity on the 30m-100m StripTool plot. No wind. 4.5 EQ in the Northern Marianas and 5.7 EQ in Indonesia around this time. Also had building heavy traffic (for Rt 10) on Rt 10 during this time.
14:00 – Range back up to around 60Mpc. Took no action during the dip and recovery in range.
Using H1:SUS-ITMY_L2_DAMP_MODE2, FM1, FM2, FM4. gain=-2000 Monitor: H1:SUS-ITMY_L2_DAMP_MODE6_RMSLP_LOG10_OUTMON This was also added to the wiki: https://awiki.ligo-wa.caltech.edu/aLIGO/H1%20Violin%20Mode
Using H1:SUS-ETMY_L2_DAMP_MODE10, FM6, FM9, gain=-800 Monitor: H1:SUS-ETMY_L2_DAMP_MODE10_RMSLP_LOG10_OUTMON Also had to add a notch in H1:SUS-ETMY_L2_DAMP_MODE3 FM10, because it was ringing the mode up. All this was added to the wiki: https://awiki.ligo-wa.caltech.edu/aLIGO/H1%20Violin%20Mode PS: actually -5000cts worked too.
Not much to report from me. Commissioners wroked their magic pretty much the whole night. Sheila and I did an Initial alignment at 0:42 UTC. Locked in NOMINAL_LOW_NOISE right now, handing it off to Jeff B.
Stefan, Sudarshan, Evan
After recovering low-noise locking, we noticed something was wrong with the calibration: the region from 10 to 30 Hz was too low, and the region from 50 to 200 Hz was too high.
We looked at the calibration lines, we took a DARM OLTF, we compared SDF settings, we inspected foton coefficients, but we could not figure out what was wrong. [Actually, the EY L3 lock gain in the CAL CS model was 1.0 instead of 1.25, but that wasn't the root of the problem.]
In the end it turned out that FM1 of CAL-CS_DARM_FE_ETMY_L1_LOCK_L has a pole at 0 Hz, and the filter had charged up, causing the CAL CS version of the L1 control signal to show a fake, white-ish noise floor above 10 Hz.
We turned the filter off and then back on again, which fixed the problem. However, we can already see that the filter has started to charge up again. So we've replaced the 0 Hz pole with a 0.001 Hz pole.
Sudarshan also found that the CAL CS version of the whitened DARM control signal had a fake 1/f shape between 8 and 80 Hz, which was due to insufficient whitening. So now there is an extra pair of 1 Hz zeros and 100 Hz poles in CAL-CS_DARM_DELTAL_CTRL_SUM_WHITEN.
Sheila, Jenne, Nergis, Daniel, Stefan Most of the things we tried today we tired today were goose chases, but here are the things that we left in: - We moved SR3 mack to where is was before we turned on the cage servo. - We (for now) switched back from the cage servo to the OPLEV servo. - We switched back to the old SRC1 yaw input matrix: AS_A_RF36_I_YAW to SRC1_Y : -3 AS_B_RF36_I_YAW to SRC1_Y : 1 That's the matrix for which we had done the careful SRCL FF tuning. - We scratched our heads for half a day about why the MICH_P loop goes unstable at high power. - We then threw our hands in the air, looked at what other signals see MICH_P (i.e. BS pitch motion) - turns out to be AS_A_RF36_I, with a gain that is roughly -3/2 of that one in AS_B_RF36_Q (at least at low power) - We punched in an input matrix that includes both error signals: AS_A_RF36_I_PIT to MICH_P : -0.666 (remember that AS_A was never phased for PIT, so don't wonder too much about the I...) AS_B_RF36_Q_PIT to MICH_P : 1.0 With those signals we were able to hold lock at 23W, although we still could improve the ASAIR_90 buildup by slightly offsetting the error point - This also significantly reduced the drop in recycling signals right after power-up. - After about 30min at high power, we stepped the MICH_P offsets by 100cts, and noticed that there was no signals left in AS_B_RF36_Q_PIT - no surprise that we lost lock on this signal before... - After 3h of locking I revisited the MICH_P offsets, and found that no offset, but completely switching to AS_A_RF36_I_PIT gave us the best performance: AS_A_RF36_I_PIT to MICH_P : -1.333 AS_B_RF36_Q_PIT to MICH_P : 0 After than we damped some violins (see above), and turned on whitening. All this stuff is not in the Guardian yet - to be done tomorrow.
As part of the recovery effort today we stumbled upon a way that seems to fairly reliable allow us to transition from PRMI to DRMI (we were 3 for 3 today - we'll try again tomorrow). The procedure is: - Let Guardian set up for DRMI locking. - Misalign SRM - turn off H1:LSC-SRCL OFFSET, INPUT and OUTPUT - Let PRMI lock - make sure H1:LSC-SRCL OFFSET, INPUT and OUTPUT are off (guardian turns on OFFSET), and clear history. - Pause ISC_LOCK and ISC_DRMI guardian - Align SRM, wait until it damps down. Today the PRMI always stayed locked in this configuration. - Turn on H1:LSC-SRCL OUTPUT - Turn on H1:LSC-SRCL INPUT. DRMI is now locked. - Turn on H1:LSC-SRCL OFFSET, if desired for mode-hopping suppression
Already done at Livingston a long time ago...
https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=11340
Except this scheme seems much faster than waiting for DRMI...
As recorded in r11378, I have fixed a bug in the ext_alert.py GRB monitor that prevented it from running.
Can this monitor be restarted at the earliest available opportunity?
16:00-17:00 Kiwamu confesses he has been tuning OMC without realizing the Intent Bit was set to Observing
17:05 Stefan doing some tuning
17:06 Lockloss
17:45 Fil to EY
18:10 Fil back from EY
Locking summary: IFO was locked when I arrived (5 hour lock stretch), although at poor range. Lockloss due to commissioning efforts. After the tuning was satisfactory, I did an initial alignment. After some Guardian issues were worked out with Kiwamu's help, we were able to bring the IFO back to Low Noise, albeit shortlived locks. Commissioning continued thereafter.
The lockloss analysis tool at "/opt/rtcds/userapps/trunk/isc/common/scripts/lockloss" is updated. Now it will be triggered only when a lock loss from nomimal low noise state happens; losses during lock acquisation cannnot trigger this tool.
Besides, a "readme.txt file" is added to help you use this tool.
I "destroyed" the TEST guardian node, so that we have only critical nodes active during the run.
I have taken the transfer function of the coil A, B, C, and D inputs to outputs on the Satellite Box (D0901284). The measurement was taken with an SR785 sourcing the input through a Coil Driver Chassis to turn the input into a differential, and then fed through the Satellite Box. We expected the Satellite Box to be have flat phase response and gain. We have confirmed that to a tenth of a percent. (0.1%) The Reference TF was taken of the Coil Driver Chassis by itself, and the residual calculation took our response from the Satellite Box Coil A over Reference TF. The following was calculated using Satellite_Box_TF_Plotter.m located in /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER8/H1/Measurements/Satellite_Box/2015-08-17/.
I remade the plots above since they didn't show up nicely.
While Travis was working on initial alignment earlier today, we found that we were once again having trouble turning on the ASC for the X green arm. I think the trouble was that the DC centering loops' error signals (from the green ITMX camera) were a little bit too large. This isn't something that happens too often, but it definitely caused the ASC to pull the arm away from resonance. Engaging the ASC without the DC centering was fine, so it's definitely something with that pair of loops.
By hand, I turned the gain on the DOF 3 loop for both pitch and yaw (which handles the DC centering) to half of the nominal value. Once the error signals were closer, I put the gain back up at the nominal value for both loops.
Since this is a particularly tricky problem to troubleshoot, I've tried to handle it in the ALS Arm guardians. The modifications were made in the "generator" states, so they are the same for each arm. The actual gain values for each arm are stored in the alsconst.py file, which is already loaded by the generator states.
I have loaded this new code, but it has not been fully tested, since we have not done initial alignment since I loaded it (and that's where it'll get used). I've tried to test the logic in the guardian shell and that all seems to be fine, but there's no test like doing it live. If it is giving errors that seem insurmountable, I did an svn checkin of the as-found code before I started modifying it (as well as another checkin after my edits), so we can go back one svn version.
This is a sign of ghosts past! I seem to remember that we solved this problem months ago with a delayed boost/gain which is engaged by the WFS FM triggers. Is it clear that going back to the original settings doesn't address it?
Here is the current SVN status of the front end filter file's svn status. A reminder, in the /opt/rtcds/lho/h1/chans/ directory, all the filter files should be symbolic links to the appropriate userapps filterfiles area. I found that the following chans directory files are not symbolic links: H1CAL[CS,EX,EY].txt, H1SUSETM[X,Y]PI.txt I found that all symbolic links used the absolute path of /opt/rtcds/userapps/release/ except for: ../../../userapps/release/sus/h1/filterfiles/H1SUSBS.txt Here is the list of the svn status: M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM1.txt M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM6.txt M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM2.txt M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM3.txt M /opt/rtcds/userapps/release/isi/h1/filterfiles/H1ISIHAM3.txt M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM4.txt M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIHAM5.txt M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIITMY.txt M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIBS.txt M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIITMX.txt M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSPRM.txt M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSPR3.txt M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSSRM.txt M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSSR3.txt M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSOMC.txt M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSITMY.txt M ../../../userapps/release/sus/h1/filterfiles/H1SUSBS.txt M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSITMX.txt M /opt/rtcds/userapps/release/pem/h1/filterfiles/H1PEMCS.txt M /opt/rtcds/userapps/release/isc/h1/filterfiles/H1OAF.txt M /opt/rtcds/userapps/release/lsc/h1/filterfiles/H1LSC.txt M /opt/rtcds/userapps/release/omc/h1/filterfiles/H1OMC.txt M /opt/rtcds/userapps/release/lsc/h1/filterfiles/H1LSCAUX.txt M /opt/rtcds/userapps/release/asc/h1/filterfiles/H1ASC.txt M /opt/rtcds/userapps/release/asc/h1/filterfiles/H1ASCIMC.txt M /opt/rtcds/userapps/release/pem/h1/filterfiles/H1PEMMX.txt M /opt/rtcds/userapps/release/pem/h1/filterfiles/H1PEMMY.txt M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSETMY.txt M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIETMY.txt M /opt/rtcds/userapps/release/isc/h1/filterfiles/H1ISCEY.txt M /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSETMX.txt M /opt/rtcds/userapps/release/hpi/h1/filterfiles/H1HPIETMX.txt and here are the scripts which generated it (running in the chans directory): for i in 'awk '{print $1}' ../cds/model_info.txt';do echo $i|tr [a-z] [A-Z]|awk '{print $i".txt"}'>>/tmp/filtfilelist.txt;done for i in 'cat /tmp/filtfilelist.txt' ;do ls -al $i >> /tmp/filelonglisting.txt;done for i in 'grep "->" /tmp/filelonglisting.txt |awk 'BEGIN{FS=" -> "}{print $2}'';do svn st $i;done
I've copied the H1CAL*.txt files over to the userapps repo, committed them, and created a soft link in the chans directory, lrwxrwxrwx 1 jeffrey.kissel controls 58 Aug 18 19:25 H1CALCS.txt -> /opt/rtcds/userapps/release/cal/h1/filterfiles/H1CALCS.txt lrwxrwxrwx 1 jeffrey.kissel controls 58 Aug 18 19:25 H1CALEX.txt -> /opt/rtcds/userapps/release/cal/h1/filterfiles/H1CALEX.txt lrwxrwxrwx 1 jeffrey.kissel controls 58 Aug 18 19:25 H1CALEY.txt -> /opt/rtcds/userapps/release/cal/h1/filterfiles/H1CALEY.txt I've also fixed the BS softlink, and committed the current filter file to the repo, lrwxrwxrwx 1 jeffrey.kissel controls 58 Aug 19 01:14 H1SUSBS.txt -> /opt/rtcds/userapps/release/sus/h1/filterfiles/H1SUSBS.txt
All the HEPI and ISI files were commited. The mass HEPI mods were from the running of foton -c on the files. The HAM3 ISI change was related to weiner filters--ready to test these at opportunity.
Plots of DELTAL_CTRL signal before and after this fix was implemented.
Just a note:
As for the pole at 0 Hz, this has been a long standing issue and we have mitigated it by not enagling this particular filter (see (B) part of alog 17528). The issue is now exposed again because of my automation scripts (see alog 20617). I was hoping to get around it by clearing the past history of the filter every time when a new lock starts.