model restarts logged for Sat 17/Jan/2015
2015_01_17 07:40 h1fw0
2015_01_17 08:06 h1fw0
2015_01_17 10:12 h1fw0
2015_01_17 10:30 h1fw1
2015_01_17 11:34 h1fw1
2015_01_17 15:07 h1fw1
2015_01_17 15:33 h1fw1
2015_01_17 22:17 h1fw0

model restarts logged for Sun 18/Jan/2015
2015_01_18 03:02 h1fw0
2015_01_18 20:26 h1fw1

all unexpected restarts. We will try power cycling the solaris QFS writers Tuesday to see if this improves the situation.

Conlog frequently changing channels log files attached.

Alexa, Evan, Sheila

We have a DRMI ASC cnfiguration in the guardian that seems good enough for now.  We have PRC2 (REFLB 9I to PR3), SRC1 (combination of AS 36 WFS to SR3),  SRC2 (combination of AS_C to SRM and SR2) with low bandwidth, and MICH (AS A 36 combination) to BS, oplev damping off. 

We have also edited the offloading in the guardian, so that we have a 20 second ramp time and we leave the loops running.  This works now.  

We have had several incidents where MC1+MC3 trip, and sometimes also cause HAM2 ISI to trip.  Kiwamu found that there were large signals coming through DOF4 (16128), although there was some other underlying problem which was causing MC2 to get large length signals. 

We Had DOF 4 off for most of the day saturday, once we found and fixed the PSL noise eater oscillation we turned it back on.  This morning we dropped the mode cleaner lock when we were attempting to lock the X arm in IR, which also tripped MC1, MC3, and HAM2 ISI (this lockloss is the kind of thing that is expected to happen once in a while, and should not cause a cascade of trips).  We have again held the offsets on DOF 4,  because somehow this loop makes the sysem more fragile.  Since holding the outputs we have had several mode cleaner lock losses without any trips.  

Alexa, Sheila, Dan, Evan

With the COMM PLL locking robustly again, we have been able to proceed with little trouble to DRMI3f with arms held off resonance. We then tried to proceed further by letting the ISC_LOCK guardian transition the CARM sensor from ALS COMM to sqrt(TRX+TRY), but this blew the lock. So we will have to diagnose the transition sequence more carefully.

DRMI ASC with arms off resonance continues to be fragile. We already know that the PRCL loops require careful tuning of the pointing onto the POP QPDs in order to function. But tonight, even the MICH and SRCL loops would blow the lock when the guardian turned them on.

Alexa and I have left a measurement running on the IMC overnight.

Koji and I will be in tomorrow to work on the OMC, I will clean up after the measurement in the morning.

Alexa, Evan, Dan, Sheila

We have been having intermittent problems for the last two or three days.  This evening we traced the problems we've been having with ALS COMM (alog 16129 ) to an oscillation of the PSL noise eater.  The tell tale symptom was amplitude noise at around 900 kHz on the PSL light.  We don't know of a good indicator of this problem from the control room.

We do not know if this was the cause of our mode cleaner lock losses over the last few days (alog 16128 ),  or to tripping of MC1+MC3 suspensions and HAM2 ISI.  

After I toggled the noise eater switch the ISS first loop was unable to lock.  For now we have turned it off. 

We had the outputs held on the IMC WFS DOF4 from late last night until 8 pm today. We didn't see any trips of MC1+MC3 today.  Now we have turned DOF4 back on.

model restarts logged for Fri 16/Jan/2015
2015_01_16 09:43 h1fw0
2015_01_16 19:03 h1fw0
2015_01_16 20:27 h1fw1

all unexpected restarts. Conlog frequently changing channels list attached.

Alexa, Sheila, Kiwamu, Evan

Tonight we hoped to proceed further with the full locking procedure.

Unforunately, we have been stymied by repeated MC1/MC3 trips described in LHO#16128, and frequent unlocking of the modecleaner.

Eventually we noticed that some of these unlocking events were correlated with our attempts to lock ALS COMM. Sheila went out to check the IMC PDH loop, but found it is healthy, with a 28 kHz UGF with 40 degrees of phase.

Next, we noticed that even when the IMC stayed locked, ALS COMM would not lock. Sheila went out into the LVEA to check the health of COMM, and found that the PLL was not even locking (despite the digital system reporting that it was locked). Eventually, we shifted over the COMM signals from the COMM PFD to the DIFF PFD, and we were able to get COMM locked. We thought this meant that the COMM PFD was broken, but when we plugged the COMM signals back into the COMM PFD, we could get COMM to lock as well. Loose connection? Badly crimped cable? Unclear.

The following UTC times give some MC1/MC3 trips:

• 2015-01-16 22:24:00
• 2015-01-17 04:33:00
• 2015-01-17 04:46:00

Sheila told me that there were two kinds of mysterious events in IMC

1. IMC unlocked during ALS diff
2. MC1 and MC3 suspensions tripped for some reason

So I looked into these two kinds of incidents. Here is a summary of my understanding of what were happening, although they are not 100% understood yet.

(IMC unlocking events)

This was due to saturation in the longitudinal drive at the M3 stage of the MC2 suspension. I picked up two unlocking events from this after noon at  around 22:24 and 22:53 UTC respectively. Both unlocking events were caused by saturation in the M3 stage of the MC2 suspension. I attach a screenshot of the lockloss analyzer for the 2nd event:

In the upper right panel of the attached lock loss, it is clearly seen that the M3 longitudinal drive hit the limiter (which had been set to be close to the DAC saturation counts). According to T1300079-v2, the range of the M3 stage is about 0.5 um in peak. Therefore, roughly speaking, the IMC cavity must have moved more than 0.5 um in order to saturate the DACs.

On the other hand, we have no idea of why the IMC length motion (or perhaps PSL frequency) moved by such amount on a time scal of 200 msec or so.

We switched the coil driver setting to "Acq on, LP off" to see if it helps, but we then had another incident which was also caused by the same issue. We then turn them back into the low noise mode i.e. state request=1.

(MC1 and 3 tripping)

At the beggining, I was imagining that the MC1 and MC3 suspensions were kicked by the ASC loops right after the IMC unlocked. However, carefully looking into them, I found that they tripped some times after a lock loss. In a case, they tripped ~40 seconds after the lock loss event at around 22:53 UTC. According to various IMC ASC data, it seems that DOF4 gave a huge impulse (~10⁷ cnts ! ) to the MC1 and MC3 suspensions. This happened when the IMC LOCK guardian catched a fringe and hence a high build up. See the attached below:

It looks as if the DOF4 maintained a large number during they were off and then released it when the ASC was triggered. A strange point is that any of the IMC ASC loops, including this DOF4, should not maintain the past values as all the intergrators are cleared by the filter trigger in every lock loss. We double-checked the history handling of the integrators in foton, but they were set right -- no history maintained. Also, we reloaded the latest filters in order to be 100% sure that the filters in the front end are actually the ones in the latest foton file.

P.S.

We hoped that the issue would be gone by loading the latest foton file, but it seems to still persist. We had another incident where the MC1 and MC3 suspensions were tripped again when the IMC lost the lock. The lock loss was initated by saturation in the M3 stage of the MC2 suspension. Almost at the same time as the lock loss, the DOF4 kicked the MC1 and MC3 suspensions again. Hmmmm.... We need to further investigate this issue.

I updated the generate_QUAD_Model_Production.m function so that you can now specify a foton filter for the damping loops. Oplev damping is not yet supported.

The generate script is in

.../SusSVN/sus/trunk/QUAD/Common/MatlabTools/QuadModel_Production

You will also need to svn up

.../SusSVN/sus/trunk/Common/MatlabTools

since this is where the foton file reading functions are located (imported from the SeiSVN)

You can still specify the usual .mat struct file as before. The generate script looks for the .txt extension to determine if you are sending it a foton file.

Here is an example of how to make a model with damping filters read from foton:

quadmodel = generate_QUAD_Model_Production(frequency_vector_for_plots, 'fiber' , [] , 0 , 1 ,'/opt/rtcds/lho/h1/chans/H1SUSETMX.txt');

The function has more instructions commented into its header.

While answering questions that Jim Warner had about the front end monitoring in the 2.9 rcg, we noted that he wanted a better way to set the monitored bit than a text editor.

So I updated the sdf_set_monitor script to optionally limit the list of channels it operates on. Now you can build a list of channels that you want to (un)monitor, save it as a file and quickly change the monitoring flag in the safe.snap file.

Examples:

Set all channels to be monitored:

sdf_set_monitor 1 safe.snap

Set all channels to not be monitored:

sdf_set_monitor 0 safe.snap

Set some channels to be monitored (leaving the rest as is):

sdf_set_monitor -c ~/monitor_list.txt 1 safe.snap

Set some channels to not be monitored (leaving the rest as is):

sdf_set_monitor -c ~/dont_monitor_list.txt 0 safe.snap

You need to create a file with the list of channels.  It reads one channel per line, and only reads until the first space, tab, ... so you can actually hand it lines from a snap file and it will extract the channels you give it.

This change is LHO only right now as I lack commit rights to the proper svn repository.

See Jamie's log at: https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=15907.

The following microphones were powered and connected to the PEM AA chassis in the CER. 
HAM2 MIC
BEER GARDEN MIC
CER MIC

The electrical grounding situation in the HEPI Pump Station controller/servo is not robust.  See the attached where while fussing about in the area in early December I somehow disrupted the ground.  The pumps went down at this transition (not necessarily causal) but when the platfoms comeback under control, the local sensors will usually have some offset.  What I done in the grace window is repeat the channels in 2 & 3 and 4 & 5 and zoomed into them to a similar level to see if the position sensor is noisier after the increase in noise on the pressure channel (trace1.)  I'd say the answer is maybe..maybe not.  Maybe JeffK will suggest a better way to study this.  I can alwasy just switch the servo into manual mode and that way the drive to the pump station will be unchanging and not effected by the controllers response to the pressure channel noise.

Meanwhile, we should continue to get back to the quiet signal time however that may be achieved.

Nergis, Evan

The ETMX oplev was badly miscentered in yaw. We have recentered it.

We did not make any changes to the whitening board settings. A picture is attached.

Alexa, Kiwamu, Sheila, Koji, Evan

DRMI ASC

Finally we were able to lock DRMI with the high-bandwidth ASC loops.

The key here was to move IM4 so as to center the forward-transmitted beam on POP B. In addition to reducing the amount of offset for the INP error signals, we believe (based on camera images) that this reduced the amount of light scattered on the PR2 baffle.

After moving IM4, we then adjusted PRM and PR2 so that PRX would lock again. We then proceeded with the usual initial alignment of the corner optics.

Once DRMI had locked, we engaged the MICH, SRC1, and SRC2 loops without issue, and then transitioned them to high bandwidth (by turning off the -20 dB filters and ramping down the BS oplev damping).

Then we were able to engage the PRC1_P and PRC2_P loops without issue, and transition them to high bandwidth (by turning off the -20 dB filters, and turning on the PRM M1 and PR3 M1 locking filters).

Initially we had difficulty turning on PRC1_Y and PRC2_Y. However, we found that we could get them to work by engaging them in close succession. Kiwamu conjectures that there may be some gain heirarchy at work here.

Then we were able to engage INP1_P. Initially we put in an offset at the error point so that the loop would not immediately try to integrate away the error signal dc value. However, we were able to turn the offset off without issue.

The only tricky business here was INP1_Y. At one point (before working on the PRC loops), we turned it on (with an offset) and found that we had to flip the sign of the gain (from 300 ct/ct to -300 ct/ct) to keep the POP buildup stable. However, once we engaged it last (after all the other loops), we found that the original gain works fine. It's still unclear what's going on here.

The new slider values for IM4 are outside the "safe" range found by Keita and Alexa (LHO#). But since the IMC pointing has been changed since then, it's not clear that these safe values are still valid.

We started a (hopefully) long DRMI1f+ASC lock at 2015-01-10 05:21:00 UTC.

DRMI lock acquisition

When DRMI locking becomes sluggish, we found it is helpful to misalign the SRM, then wait for PRMI to lock, then adjust PRM and BS to maximize POPAIR_B_RF18. Then upon breaking the lock an realigning SRM, DRMI appears to lock more quickly.