J. Kissel, N. Kijbunchoo, K. Izumi While I was peacefully explaining tilt-horiztonal coupling to Nutsinee waiting for the DRMI to lock, it acquired, but the ISC_DRMI guardian node got stuck in the DRMI_1F_LOCKED_ASC state complaining in the SPF DIFFs that the channel H1:ASC-INMATRIX_P_1_9 (the REFLA RF9I to element INP1_P) is dead. Kiwamu pointed us to Jamie's solution the last time this had occurred (see LHO aLOG 17545 for problem, and LHO aLOG 17548 for fix), but this time we're 100% confident that no one has made any change to guardian code. I've tried reloading the guardian code, but that's all I'm willing to do. We've been working so hard to get DRMI up since we lost lock from violin mode problems. I note that Dave has been reporting that the guardian machine is grossly overloaded today (LHO aLOG 18152), but at this point I can only claim these two things are connected anecdotally. I've left a message at the Guardian Help Desk.
D. Barker, J. Kissel, and then J. Rollins Pre call from Jamie: Dave and I tried chasing a solution to the above problem a little further by not just reloading the guardian code, but restarting the node as Sheila had done in LHO aLOG 17545. Unlike that previous situation, though, the problem cleared with the restart. Regrettably, the node comes up in "INIT" and when I tried requesting the same state it had frozen in, "DRMI_1F_LOCKED_ASC," it dropped the DRMI lock. It didn't kill the ALS COMM over DIFF lock though, which is nice. Then with Jamie on the phone: By then DRMI had recovered to LOCK_DRMI_1F, but remained stationary because when you restart a subordinate node, his manager -- in this case ISC_LOCK -- loses management possession and doesn't know what to do. The trouble is that the only way for ISC_LOCK to regain possession of all of its subordinates is to go to the INIT state, which I didn't want to do because we already had an ALS COMM, ALD DIFF, and DRMI locked up. Jamie recommended that I try force regaining possession of: - Put the ISC_LOCK in MANUAL mode (via the "all" states subscreen) - Jump to INIT (this *worked* and repossessed the ISC_DRMI node) - Jump back to the state it *was* in before it had been put into manual, and switch back to EXEC. However, upon switching back to EXEC, the IFO lost all locks. Jamie thinks it's because I should have requested the state *after* the one where it was stuck, I think (now while writing this log) that I just went to the wrong state period (i.e. not what it was before I went to manual). *sigh* Oh well. We're on our way back up...
I have created a 10 minute injection file simulating a stochastic source at omega_GW=1 at 100Hz, and placed it on the cds system in:
/ligo/home/edward.daw/research/hardware_injections/2015_05_01/inj10mins.txt
The file was created as follows:
on the h1hwinj1 machine,
cd /ligo/home/edward.dad/research/hardware_injections/dependencies/sources/virgo/NAPNEW/SCRIPTS/IsotropicSbGenerator
python IsotropicSbGenerator.py --init IsotropicSbGenerator2.ini
This code generates a single 600 second frame which I subsequently moved to /ligo/home/edward.daw/research/hardware_injections/2015_05_01/SB_HI_L1-1114555770-600.gwf.
To convert the frame to an ascii file, I tried running a local matlab, but I couldn't get a license. I therefore shipped the frame to my laptop, and used matlab interactively:
>> [data,tsamp]=frgetvect('SB_HI_L1-1114555770-600.gwf','H1:strain',1114555770,600);
>> outfile=fopen('inj10mins.txt','w');
>> fprintf(outfile,'%g',data);
>> fclose(outfile);
...and finally I used gsisftp to move the resulting text file back to the cds machine at the above location.
The above matlab code could easily be used to scale the data by a factor, as it seems you have done with previous injections, if the existing scale proves inappropriate for the injection. Please inject this 10 minute duration signal once the machine is stable and you are ready for more injection tests.
Thanks.
Ed
Nice work Ed, Jeff, and Giancarlo getting this ready in time for the mini run. I have a similar question to the one posed by Jeff. Is the output of the file in units of strain or is in units of Initial LIGO counts? The reason I ask is because, during Initial LIGO, we used this code, or code like it, to create injection files with a frequency-dependent transfer function applied. For aLIGO, we don't want to apply this transfer function. Would it be possible to make a plot of the amplitude spectral density of the injection file? It should have a power-law shape with index -3/2.
J. Kissel, N. Kijbunchoo, K. Izumi After deciding that we've damped the 504.8 [Hz] violin mode enough to advance to DC readout, we got a few minutes of DC readout on ETMX, and then we noticed that wind began to pick up speed a few hours ago. We lost lock, and during recovery DRMI was taking particularly long to lock back up. From Izumisan we learned that - If it takes particularly long for DRMI to lock up, e.g. greater than ~15 minutes, - If one hasn't run initial alignment for quite some time, and - The AS port shows flashes that are not strictly LG00 modes, but LG10/01 or any higher order modes it's likely that one needs to redo initial alignment. We began embarking on initially alignment, and found the arms difficult to lock on green. I suggested we move the blend frequency up on the beam line directions of ETMX and ETMY, as is common practice when winds are > 20 [mph] as they are now. Just after we increased the blend frequency, the PSL laser tripped at 16:50 PDT / 23:50 UTC. Quick (and at this point only superficial) investigations do not reveal a reason. Regrettably, the global PSL team is rather busy with LLO's laser at the moment (see LLO aLOG 17959, and subsequent entries). Kiwamu and Nutsinee are in the PSL diode room now recovering (and have done so as I finish this log). The usual perfect storm on the first day of an attempt to leave the IFO alone and collect data!
M. Landry, J. Kissel
We injected the same CBC waveform on fourteen occasions with a variety of gains, into the DC locked machine. The strain waveform is a 1.4-1.4 solar mass BNS coalescence, optimally-oriented, at 45Mpc (the same waveform injected into L1 during ER6). Below we summarize these injections:
Channel = H1:CAL-INJ_TRANSIENT_EXC
Waveform: /ligo/home/jeffrey.kissel/2015-04-30/injection_strain.txt
Logfiles: /ligo/home/jeffrey.kissel/2015-04-30/*.log
|
gps time |
utc time | scalar gain |
| 1114538072 | May 01 2015 17:54:16 UTC | 1.0 |
| 1114538434 | May 01 2015 18:00:18 UTC | 2.0 |
| 1114538619 | May 01 2015 18:03:23 UTC | 10.0 |
| 1114539102 | May 01 2015 18:11:26 UTC | 10.0 |
| 1114539462 | May 01 2015 18:17:26 UTC | 10.0 |
| 1114540301 | May 01 2015 18:31:25 UTC | 50.0 |
| 1114540790 | May 01 2015 18:39:34 UTC | 50.0 |
| 1114541450 | May 01 2015 18:50:34 UTC | 10.0 |
| 1114543037 | May 01 2015 19:17:01 UTC | 10.0 |
| 1114543403 | May 01 2015 19:23:07 UTC | 2.0 |
| 1114543833 | May 01 2015 19:30:17 UTC | 1.0 |
| 1114544201 | May 01 2015 19:36:25 UTC | 0.1 |
| 1114544554 | May 01 2015 19:42:18 UTC | 0.01 |
Daytime seismic noise from Hanford truck traffic has decreased during the last several weeks and swing shift hauling ceased completely on 4/29. Some new work at the 300 area could bring the daytime haul numbers back up in a month, but no resumption of swing shift activity is planned for the remainder of 2015.
Scott L. Ed P. Chris S. 4/29/15 the crew clean 61.7 meters ending 7.6 meters north of HNW-4-021. Test results posted here. Removed lights and relocated to next tube section. Ed P. Chris S. Joe D. Cris M. 4/30/15 the crew cleaned 49.3 meters ending 10 meters north of HNW-4-024. Ed P. Chris S. 5/1/15 - 49 meters cleaned ending at HNW-4-027. Beam tube pressures are continually monitored during cleaning operations by control room operator.
Quiet day of locking for the mini-run.
J. Kissel, B. Weaver, J. Warner, C. Vorvick We lost lock about 2 hours ago, and the cause was excessively rung up violin modes, specifically the 504.8 [Hz] ITMY FL mode (ref LHO aLOG 17610), which are damped by the MODE2 and MODE4 banks of ITMY_L2_DAMP block. I attach a screenshot of the ~hours of adventure. To solve the problem, there's no magic bullet unfortunately. While DARM is still locked on RF, we turn on the damping loops one at a time. The "nominal" gains of -100 and 300 for these banks are too large for the current amplitude of the lines, so we have to start with a small gain and work our way up. The attached screenshot shows the progress. I'm hoping to have them reduce enough (roughly equivalent to the current Bounce and Roll mode amplitudes) to move on to low-noise in an hour or so.
J. Kissel Again, more analysis details later, but I attach the DARM OLG TF taken this morning. The measurement has been committed to the CalSVN repo here: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Measurements/DARMOLGTFs/2015-05-01_H1_DARM_OLGTF_LHOaLOG18144.xml
I have noticed that there is a mismatch between Stage-2 BSC-ISI CART2ACT basis change matrix. This is true for all the BSC-ISI chambers. The first attachment shows the designed value (https://dcc.ligo.org/DocDB/0013/T1000388/017/T1000388%20SEI%20Sensors%20Location.pdf) of CART2ACT matrix while the second and third attachments show the current matrices in ITMX (or ETMY) and ITMY (or BS/ETMX) respectively. I have cross checked that the ST2-CART2ACT matrix stored here- /ligo/svncommon/SeiSVN/seismic/BSC-ISI/Common/Basis_Change_BSC_ISI/aLIGO_BSC_ISI_BS_ITMY_ETMX.mat is in agreement with the technical document.
This was an medm error on the ST2_CART2ACT matrix. I have fixed this and commited it to the svn.
We have noticed that h1guardian0 is running slowly. It has a load average of 45, which is oversubscribed for a 12 core machine. Jamie has been contacted.
What: L2 Damp Mode Filters rung up and tripped the analog watchdog.
Where: mode 2 and mode 4, UR coil, 504.9HZ
We see some bursts of noise broadly around 200 Hz (first plot - you may need to squint a bit). I guessed that this was due to non-stationary coupling with SRCL, a la Gabriele. The second plot is a coherence spectrogram for the same five minutes. The coherence is indeed changing in bursts. The third plot is the coherence with MICH, PRC, and SRC over the same time. The coherence with MICH especially is extremely high; I think I saw that the feedforward is disabled?
Yes, we have not yet finished commissioning the MICH and SRCL feed-forward with the better recycling gain. For the sake of robustness, we've decided not to advance beyond "LOWNOISE_ESD_ETMY" to the "LSC_FF" state for the mini run.
We noticed that the violin modes were very rung up in the DARM spectrum on the wall causing a huge comb of peaks. Kissel turned off the DARM ERR DAMP filters (violin mode filters) on all 4 TMs to see if that will help. We did not toggle the intent bit - too busy trying to save the lock.
J. Kissel, M. Landry More detailed aLOGs to come, but for those watching we've now completed all the tests we plan to do today with hardware injections and calibration measurements as of ~19:45 UTC / ~12:46 PDT. The observation intent bit is on. For the record, the "nominal" guardian state is still set to be "LSC_FF," (STATE_N = 520) but we will *not* be going to that state for not, as we still believe the MICH and SRCL subtraction are unstable and/or cause non-gaussianity. Also the cut-off filters for the CHARD and DHARD ASC loops are not engaged (because those also have not been tuned for the new higher recycling gain). We are leaving it in this state (called LOWNOISE_ESD_ETMY, STATE_N = 515) for this lock stretch, even though guardian does not think it's "nominal." Thanks for your patience! For the record, the first part of this lock stretch was glitchy (as seen on the spectrum in the wall and in the time series of DARM control). We're not sure why, but we'd convinced ourselves that it did not directly correlate with our hardware injection tests. It would be interesting to know what the problem was,
Could the nominal state be temporarily changed to Low noise ESD ETMY (or whichever state the IFO is most likely to be in for the mini run)?
If the Guardian isn't reporting that it's in the nominal state, the science mode bit will remain off and downstream tools that we want to test won't automatically run.
"As you wish..." (said as Welsey to Princess Buttercup) The Guardian Nominal state for the ISC_LOCK has been temporarily changed to "LOWNOISE_ESD_ETMY" and the guardian code has been reloaded.
Since DC lock was achieved at 11:09 (after starting to try locking at 9:00 this morning), I've set the intent bit to Undisturbed. Injections are ongoing, the range isn't great and there have been earlier lock losses, but for now I think we can call an official start to the mini-run at LHO.
We see a swept sine in the data, but the intent bit seems to still be in undisturbed. I think the agreement on the detchar call is that the intent bit would be treated like science mode, with no excitations or other interventions. We also see some hardware injections, which are fine, since they will be flagged.
Andy, all: yep we had one calibration sweep in place during the first lock, apologies. From now on, we'll only allow for hardware injections when the intent bit is set.
Let us know if there is an interval you require (e.g. 2048s?) betwen injections.
Mike: I think that five minutes should be fine between CBC injections. But if there was a specific request from the CBC group, I don't want to override that.
We lost lock ~20 minutes ago. I've reset the intent bit. Looks like a rung up violin mode on ITMY. Jeff is working on damping it.