Most cpu overruns associated with the faster SUS computers (which were installed in the end stations this Tuesday) only glitch the SUS-IOP and SUS-ETMX,Y models. Once in a while the glitch extends to the SEI IOP/ISI model and the ISC/ALS models and also glitches the SUS-TMS and SUS-PI models.

To keep the overview GREEN and help with trending, I have extended my end_station_sus_diag_resets.bsh script to clear these additional models. This script is ran roughly once a minute on opsws16.

WHAM6 ISI is lifting itself almost 40um from its free hanging position during isolation.  The drive to the Z actuators is running nearly 2000 cts on average.

What's the issue?  Excess drive ==> excess noise?  Unnecessary heating?  Trip vulnerablility?

The attached plot shows 15 minutes when I deisolated the ISI.  Channel 9 shows the free hang sag position.  This was not so low until the HAM6 work back in April/May when it decidedly dropped.  While I balanced and leveled the platform, maybe we need to bias that work high to deal with thermal sagging.  I don't see it obviously sagging lower over time.  See Channels 14 thru 16 for the vertical drives.

Second attachment is 120 days of the outside temp and the vertical drives.  I'd say there is nice inverse correlation with phase thrown in related to our typical step wise heating response to seasonal temperature changes.

My recommendation is remove some payload to raise the free hang position.  Maybe not to big a deal but the free hang positions of the Vertical CPS are on the low side (Channels 4 thru 6.)  This requires a vent of course so maybe not.

Otherwise, we should accept & show that the OMC position does not care about the 90um shift and reset.

Evan and I left the interferometer in the observing mode at around 20:00 pm local last night. The ISS 2nd loop now keeps correcting the diffraction power all the time via a feedack loop and maintains it at 8%.

This is yet another followup of Kiwamu's cage servo followup: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=25416

In the attached left panel is a 12-days trend of SR3 signals. Left column is the EPICS version of M3 witness sensors, and PMON is used for the cage servo. Middle column is a DQ-version of the same signal.

At first YMON and PMON were going happy together with DQ version, but at around Feb 02/03 midnight UTC, PMON and YMON got huge at the same time the cage servo went crazy, but WIT_Y_DQ didn't feel it. Later when Kiwamu turned down the cage servo gain, WIT_P_DQ (middle bottom) went back to normal-ish but PMON and YMON didn't.

It turns out that when you look at non-DQ version of these signals (attached middle panel), there are huge lines in all four M3 OSEMs of SR3 at 1.7kHz and its harmonics, totally dominating the RMS (top). Bottom plot shows the comparison between SR3 (green) and PR3 (red), and PR3 doesn't show any of these huge lines.

DQ version is decimated by the frontend, but it seems like PMON (as well as YMON and LMON) are just sparsely sampled version of the fast signals, no filter modules therefore no decimation, thus the high frequency junk directly goes into low frequency (right panel).

In the past, the solution for oscillating satellite amps was to power-cycle them repeatedly until the oscillation stops.

1515 - 1600 hrs. local -> To and from Y-mid 

Next over fill to be Friday, Feb. 12th before 4:00 pm

Today I cut off the excess length of HV cable at the controller end which included the suspect portion of cable previously noted but the short did not change.  I inspected the length within the raceway in the VEA but nothing looks suspicious.  Thus, all of the length which is accessible looks unremarkable.  There is still 10'-15' that is within EMT conduit in the VEA that could be exposed if pulled through the conduit body at the exterior wall penetration but other than that there isn't much else to do barring pulling another 900' run of "not free" 10,000V rated cable.  

Recall that the ion pump controller was found tripped off several days ago and could not restart the ion pump do to its power limiting feature and can only achieve 500VDC @ 0.5 amps implying 1000 ohms between HV conductors.  This "shorted" resistance value is duplicated with the "Megger" unit, as well as, with the low voltage VOM which measures 1080 ohms.

C. Cahillane

There was some concerned voiced by Alan and Peter about the LLO uncertainty budget combined with the systematic error corrections at around 100 Hz at gpstime = 1126259461.
This post is merely to mirror the study I did at LLO with a similar one at LHO explaining why we do not see the LLO 'hump' at LHO.  See LLO aLOG 24796
I have included the LHO C01 vs C03 response functions systematic error and statistical uncertainty plot. (See PDF 1) 

I began with the "perfect" C03 version of calibration response which includes all systematic corrections.  Then, one by one I removed each systematic correction and compared it to the original C03 response: (See PDF 2)
Sensing:    C_r (Plot 1)
Actuation:  A_tst, A_pum, and A_uim (Plot 2, 3, and 4)
Kappas:     kappa_tst, kappa_pu, kappa_C, and f_c (cavity pole) (Plot 5, 6, 7, and 8)

So you don't have to go through and view every plot yourself, I have compiled all of the systematic error values at 100 Hz for each of the parameters:
C_r = 0.98932
A_{tst} = 1.0018
A_{pum} = 0.99381
A_{uim} = 1
kappa_{tst} = 1.0251
kappa_{pu} = 0.99465
kappa_C = 1.002
f_C = 1

Total Syst Error = +0.0068

So the total systematic error at 100 Hz is < 1% because it is countered by the C_r, A_pum, and kappa_pu systematic errors.

model restarts logged for Tue 09/Feb/2016
2016_02_09 10:05 h1psliss
2016_02_09 10:23 h1broadcast0
2016_02_09 10:23 h1dc0
2016_02_09 10:23 h1nds0
2016_02_09 10:23 h1nds1
2016_02_09 10:23 h1tw1
2016_02_09 10:42 h1sysecaty1plc3sdf
2016_02_09 10:43 h1sysecaty1plc3sdf
2016_02_09 10:45 h1sysecaty1plc2sdf
2016_02_09 10:49 h1sysecaty1plc1sdf
2016_02_09 10:50 h1sysecatx1plc3sdf
2016_02_09 10:52 h1sysecatx1plc2sdf
2016_02_09 10:53 h1sysecatx1plc1sdf
2016_02_09 10:54 h1sysecatc1plc3sdf
2016_02_09 10:56 h1sysecatc1plc2sdf
2016_02_09 10:58 h1sysecatc1plc2sdf
2016_02_09 10:59 h1sysecatc1plc1sdf
2016_02_09 11:50 h1iopsusey
2016_02_09 11:52 h1susetmy
2016_02_09 12:02 h1iopsusey
2016_02_09 12:02 h1susetmy
2016_02_09 12:03 h1iopsusey
2016_02_09 12:03 h1susetmy
2016_02_09 12:04 h1susetmypi
2016_02_09 12:04 h1sustmsy
2016_02_09 12:28 h1iopsusey
2016_02_09 12:28 h1susetmy
2016_02_09 12:28 h1sustmsy
2016_02_09 12:29 h1iopsusey
2016_02_09 12:29 h1susetmy
2016_02_09 12:29 h1sustmsy
2016_02_09 12:30 h1susetmypi
2016_02_09 12:35 h1hpietmy
2016_02_09 12:35 h1iopseiey
2016_02_09 12:35 h1isietmy
2016_02_09 12:37 h1alsey
2016_02_09 12:37 h1caley
2016_02_09 12:37 h1iopiscey
2016_02_09 12:37 h1iscey
2016_02_09 12:37 h1pemey
2016_02_09 13:27 h1iopsusex
2016_02_09 13:27 h1susetmx
2016_02_09 13:27 h1susetmxpi
2016_02_09 13:27 h1sustmsx
2016_02_09 13:28 h1hpietmx
2016_02_09 13:28 h1iopseiex
2016_02_09 13:28 h1isietmx
2016_02_09 13:42 h1susetmx
2016_02_09 13:46 h1iopiscex
2016_02_09 13:46 h1pemex
2016_02_09 13:48 h1alsex
2016_02_09 13:48 h1calex
2016_02_09 13:48 h1iscex

Maintenance day. New PSLISS code, new Beckhoff C1PLC2 code and associated DAQ restart.

New Beckhoff SDF system code install.

Install of faster SUS computers at both end stations, required restart of all Dolphin attached computers for SEI and ISC.

model restarts logged for Mon 08/Feb/2016 No restarts reported

model restarts logged for Sun 07/Feb/2016 No restarts reported

model restarts logged for Sat 06/Feb/2016 No restarts reported

model restarts logged for Fri 05/Feb/2016 No restarts reported

IM2's alignment changes after a shaking event (ISI trip) such that the alignment drive is unchanged, but the pointing of the optics (as measured on OSEMs) is different.

These jumps in alignment are anywhere from 5urad to 42urad in pitch.

IM2 is the most effected, but IM1 and IM3 also show this behavior.

I've looked at six shaking / alignment change events, and recorded the amount of motion (shaking) the optic, ISI, and HEPI experienced.

The results shows that the amount of HEPI-ISI shaking can vary during an event that results in a change in IM2 alignment.

What is consistant is the average amount the optic shakes (pitch plus yaw divided by two) to produce a jump in alignment are between 2.1 and 6.1 urad of optic alignment change per urad of average optic shaking.

This suggests the conclusion that optic shaking is the cause of the alignment change, however, in looking at a few events where the optic tripped but HEPI and ISI didn't, I have not seen a significant alignment change, so this part of the IM alignment jump investigation is ongoing.

Based on my curent data, the mechanism that creats and alignment jump on IM2 is a combination of optic shaking and HEPI-ISI shaking.

Feb. 10 2016 ~ 21:51 UTC Stopped Conlog process on conlog-test-master. Had connected to channels on Feb. 3 2016 16:38 UTC (alog 25344).

Before O1, LLO installed some high UGF loops and blend filters on their HAM1 HEPI, in an attempt to better isolate some of the steering mirrors in HAM1. When I tried here, I found that our HAM1 piers weren't as stiff as LLO's in some DOF's because the piers had never been grouted. The grouting was done during O1, and Hugh got new tf's over a couple of down days. I just finished working through the commissioning scripts and have some higher UGF loops to install. These higher UGF loops are needed in order to get sufficient gain at a few hertz. Attached is a pdf of the loops. I'll wait for a convenient maintenance day to try installing.

After removing all partial filter loads yesterday during maintenance, I noticed today that SUS BS_M2_LOCK_L had been loaded. On further investigation I found that the H1SUSBS.txt filter file has not been changed since 20th January. So the front end is reporting a load, but there is no archived filter file since no actual filter change was made.

Conlog says that BS_M2_LOCK_L was loaded 18:50:13 PST last night (writing 1 to RSET performs a load) and then had its history cleared fifteen seconds later at 18:50:28  (writing a 2 to RSET clears history).

This looks like it could be a scripting or guardian bug, We should not be loading filters automatically.

I have performed a full filter load on SUS-BS to see if this comes back again.

Following Hugh's clean up of the VAC screen from yesterday, I went through the exercise of editing the Alarm Handler file for dust.alhConfig (located at /optrtcds/userapps/release/cds/h1/alarmfiles).  It actually wasn't that painful.

So now we have NO open (i.e. constantly ON) alarms for the Alarm Handlers (yay!), and should address any audio alarms and new & real.

For the DUST alarm handler, I commented out the following:

• H1:PEM-CS_DUST_LAB3*
• H1:PEM-CS_DUST_LVEA14*
• H1:PEM-EX_DUST_VEA1*
• H1:PEM-EY_DUST_VEA1*

We can add any new ones fairly easily.  For instructions on alarm handler editting, go here.

Gerardo performed a regen on the YEND non-evaporable getter(NEG) pump yesterday. The plot attached shows a 25% reduction in pressure in the end station pressure.

The last regen was performed in July prior to O1. Impressive performance for such a small package.

As I was doing my usual morning rounds, I discovered one of our main fire pumps running, apparently ran overnight pumping a large amount of water into the desert. I shut the pump down.
Richard had the HFD onsite yesterday so they were probably testing something that initiated the fire pump start. Someone should be informed when HFD completes their work and is ready to leave the site. This is not the first time this has happened.

I also found the door left open on the grey storage container located out by the water tank, again, apparently overnight. I checked for critters and did not see any large ones but could be some small ones trying to find a new home. I closed the door. If you remove things from the containers, PLEASE close the door when you are finished. If you are not able to close the door, contact me and I will be happy to close them for you. 

I'm not sure if the ESD changes today (alog 25468) are at fault, or if something else has changed today, but I cannot lock ALS Diff using the nominal loop. 

In order to lock ALS Diff, I have included a gain of 0.001 in the DARM2 filter bank (usually just has gain of 1, no filters yet).  Also, I have commented out the line in the Diff guardian that turns on DARM1 filters 2 and 3 (an integrator and resG respectively).  I can turn the DARM2 gain up to 0.002 or 0.004 most of the time, but at 0.006 the DARM loop starts to ring up over 2-3 seconds until we saturate the ETMX L3 coils.  Even at the very lowest teensy-tiny gain, I can't turn on either the integrator or the resG. 

There is so very little gain here that I can't measure the DARM loop in the 10-100 Hz band. 

Unsurprisingly, this means that Diff is too noisy to hold the Yarm anywhere in particular, so I can't move on with the locking sequence (with the thought that it would all be okay if I could get far enough to transfer to RF DARM). 

I am reverting my changes (uncommenting line 228 in ALS diff guardian and gain to 1 for DARM2 bank).

I'm calling it a night since I'm running low on ideas.

Also, Kiwamu straightened me out about the DARM filter module screen situation - the auto-generated screens live in the OMC folder.  He'll fix the overview screen tomorrow.

Summary

Varying the DARM offset by a factor of 2 has no effect on the unexplained noise above 70 Hz.

Details

Nominally we run with 20 mA of DCPD sum current, which (with an optical gain of 3.2 mA/pm) corresponds to a DARM offset of 13 pm. I took quiet data (15 minutes at a time) with sum currents varying from 10 to 40 mA, which corresponds to DARM offsets from 9 to 18 pm.

Thanks to the automatic gain scaling that Dan and Stefan installed before O1, no manual gain adjustment is needed either for the DARM loop or the calibrated freerunning channels. At each offset, I remeasured the DARM OLTF just to be sure.

I did not adjust the SRCL feedforward during this test, and indeed the coherence below 70 Hz between DARM and the SRCL control signal is increased at DARM offsets other than 13 pm. At 40 mA sum current, the coherence is >0.3 from 25 to 40 Hz. Therefore, the excess noise below 70 Hz during this test is due almost certainly to the SRCL feedforward becoming mistuned.

On Wednesday TJ and I tripped the ISI several times executing the HAM3 ISI GS13 gain switch in various ways.  I have several examples to look at but here is a comparison of one trip to a successfull switch of the gains using the SEI COMMAND perl script.

The first attachment has 20 seconds of the trip example next to the switch that does not trip.  It looks like there isn't anything different going on before the two cases: There are larger swings in X & Y in both and the other DOFs all look fairly similar too.  There is a larger glitch 5+ seconds before the trip in Z but that seems to be settled by the switch/trip time; these would be related to Reference Position ISO servoing.

In the second attachment, again side by side are the two cases zoomed in a bit closer (10 sec.)  The two stage switch by the perl script is evident with the gain switch doing more glitching and the whitening stage doing nothing.  There is a delay of the sensors wrt to the switching by the guardian which seems weird but I'm not looking at outputs from the filter bank as we have no DQ channels here...  These channels are the inputs to the blend bank after transforming to cartesian basis.   Again, nothing stands out as to why there is a problem with the guardian script.  We (TJ & I) did adjust the guardian code to separate the FMs switching like the perl script does but it still tripped the ISI.

The third attachment is a close zoom in on the ISI trip switch.  It is what it is...  Given that the Y and RZ are the dofs that ultimately go big, I'll suspect the horizontals but I could be fooled.