Something happened at about 10:30PM (11-12-2014 6:30 UTC) and the 363Hz oscillation of the end Y PZT mirror(s?) went totally unstable.

Daniel and Alexa are heading to EY to swap the controller.

In the attached right, references are from 24 hours ago, current traces are as of now. The trend shows the green QPD and the VEA temperature for 3 days.

As I reported in 14965, I suspected WHAM3 ISI OpLev Pitch and Yaw were reveresed as I believe is HAM2's.  See the attach where I yaw'd WHAM3's HEPI.  The HEPI Pitch moves as well but a couple of orders of magnitude less.  Likewise on the OpLev, the Pitch responds much more than the Yaw, so yes,

HAM3 ISI OpLev Pitch & Yaw are Reversed

The chilled water temperature at MIDY alarmed HIGH this morning. There is no demand for cooling so the water is bypassing the cooling coils completely and therefore the water pump is only heating the water.

I have turned both mid station pumps off.

This cold front has sent our temperature control loops out of whack - Last night I turned on heaters in both end stations. Today we will monitor and perhaps turn on more heat in both the LVEA and the end stations.

Alexa, Evan, Nic, Sheila

We seem to have lost the -24 V rail and the -12 V on several racks in the electronics room.  The PSL/IO/TCS beckhoff chassis power switch was in the off position when we went out there, I tried flipping it to on and the rocker switched off again immediately.  On the cds overview we have lots of red on the PSL frontends, this chassis is in the same rack as the PSL/IO/TCS beckhoff chassis, and they are on the same 24 V supply.  Nic restarted the front end models several times, then restarted the computer, then restarted the computer and restarted the PCIE expansion box.  (before we realized that we were missing power). 

(Alexa, Sheila, Evan)

As we were trying to lock ALS DIFF, we noticed that both the COMM and DIFF beatnote were too weak. We trended these values and found that the beatnote first disappeared when the PSL went down and then again at around 9am PST. I then tried locking the IR to the x-arm and found that I had to move PR3 by 4 urad in pitch to lock. The input wfs then moved IM4 and PR2 to increase the buildup.  We suspect things got moved around again with the work and trips on HAM2, HAM3. We didn't spend too much time investigating the exact cause, and we did not want to go through the entire alignment to lock DRMI and check the POP18 counts since we want to focus on DIFF right now. Hopefully we didn't introduce a new clipping in the PR2 baffle by the amount we moved. 

IMPORTANT: I have not saved these new alignment values, so if the guardians get realigned they will restore the old alignment values.

Also, the beam in the AS AIR camera looks too far to the right.

After yesterday and today recentering and tweaking, the ISS second loop is back to good working condition. The first attached plot compares some signal with the second loop open and closed. Dashed line, loop open, solid lines, loop closed.

The ISS second loop signals (SUM14 and SUM58) are calibrated in volts and the whitening filter is compensated. Using the slow DC channels, I could compute the total signal for each combination, which is about 11-12 V. I used those values to calibrate the signals in terms of RIN as in the figure. This should roughly correspond to about 20 mW total power hitting the array.

The out-of-loop signal is still showing some excess noise with respect to a flat level. Assuming that there are 10 mW impinging on the four out-of-loop diodes in total, the shot-noise-limited dP/P should be sqrt(2*nu*hplanck/P) = 6e-9 /rHz. Counting both the in-loop and out-of-loop shot noise, we get an additional sqrt(2). Therefore we expect the out-of-loop signal to be at the level of 8.6e-9 /rHz, not too far from the 1e-8 /rHz that we see at about 100 Hz.

The structures between 10 and 50 Hz are variable, but I was not able to change them significantly moving the IMC alignment or the picomotors in front of the ISS array. Notably, for most of the day there was an additional bump between 10 and 20 Hz, which disappeared some time at about 13.05 local time, without any conscious intervention on my side. See second plot. No idea what the origin of this thing is, but it was there also yesterday.

One more peculiar thing. We can use IM4_TRA_SUM to measure the RIN in transmission of the IMC, as an additional out-of-loop sensor. This is included in the attached plots. When the second loop is open, the RIN measured by IM4_TRANS agrees quite well with what is measured by the ISS second loop diodes. However, when the second loop is closed, IM4_TRA measures a significantly larger noise. This might be due to the sensor electronic or dark noise (I haven't checked it yet). However, at least at 10-30 Hz it measures the same structures visible in the ISS diodes, but a factor 4 larger. The shape is also slightly different. Surprinsingly, there is no coherence between IM4_TRANS and  SUM58. This might tell us something about the origin of those peaks, but what, it's not clear.

The peaks between 15 and 25 Hz are visible, with different amplitudes and shapes, in all the eight  photodiodes, but there is little coherence among them. See third plot. These peaks are not visible in the ISS QPD signals, even though there is a wider bump at similar frequencies. They may be due to scattered light. The most prominent structure is a 18 Hz peak.

Beckhoff. Daniel and Dave

Following Daniel's changes to the Beckhoff systems, I applied the latest INI files to the DAQ and restarted the DAQ. Unfortunately h1ecatx1 needed a restart shortly after this and we have some channels disconnected. I also copied the latest Beckhoff REQ files into the appropriate target areas and Patrick reconfigured the conlog using them.

Conlog: Patrick and Dave

I modified the script which creates the conlog channel list to use the generic include.txt file as well as the generated fe_include.txt and grd_include.txt. I changed the script which generates the Guardian autoBurt.req file (which is used to create grd_include.txt) to include the USERMSG byte encode strings for conlog logging. I created a script to convert the output of USERMSG to a string. Patrick reconfigured conlog's channel list.

Guardian: Dave

I checked that the running guardian nodes and the GUARD_OVERVIEW medm screen are in sync with each other (they are). I updated the guardian autoBurt.req file (using guardctrl list to get the node listing).

Filtermodules Load Status: Dave

Many H1 models are running with partial loaded filter files. Plus two models (SUSPR3 and SUSMC2) had modified filter files. I pressed the "LOAD COEFFICIENTS" button on the GDS_TP screen for the following models to resync them with their filter files:

ISIHAM[3,4,5,6], ISIETMX, SUS[PRM,PR2,SR2,SRM,BS,ETMY,ETMX,PR3,MC2], TCSCS, LSC, LSCAUX, OMC, ASC, ASCIMC, ISC[EX,EY]. No problems were encountered.

inicheck of all DAQ INI files: Dave

I ran inicheck on all the DAQ INI files, we have channels with mixed case on the systems: ISI[BS,ITMX,ITMY,ETMX,ETMY] and LSC.

foton check on filter files: Jim and Dave

We ran foton -c on a copy of all the running filterfiles. Most of the errors we saw relate to obsolete filters left in the body of the file but no longer in the MODULES header. At some point we should make the effort to clean up the filter files of obsolete material (quak generated files exempted from a foton cleanup).

I've restarted the CDS Matlab license server to install a new license file (that is not due to expire...).  Existing sessions fail over to the Caltech servers automatically.  I'll have to update the NLM_LICENSE_FILE environment variable later to remove the old Caltech triad license servers and add the new single server configuration for failover purposes.

08:00 PSL is DOWN - sent e-mail to D Cook, P King and R Savage

08:00 3IFO team is in the LVEA

08:01 the LVEA is in the bifurcated LASER hazard state

08:15Fil, Aaron and Manny out to EX

08:30 Rick called me and walked me through restarting the PSL frontend laser.

09:00 Jody into the LVEA 

09:12 Jody ou of the LVEA

09:20 Mitchell out of th LVEA

10:10 Karen into EY

10:30 Cris into EX

10:35 Patrick and Richard working on Beckhoff system for Mids and Ends

10:36 D Barker informed me of a DAQ restart at 12:00P

10:41 Karen heading back from EY

10:53 Karen heading out to MY

11:02 Cris at MX

11:07 Patrick and Richard are done working on Beckhoff

11:35 Jeff and Andres out of LVEA - LVEA returned to full LASER hazard

11:36 Gerardo and Kyle out to LVEA

11:40 Karen leaving MY

11:50 Kris back from MX

12:04 Gerardo and Kyle out of the LVEA

12:13 DAQ restart delayed- waiting for EY update

12:21 DAQ restarted at 12:25

12:35 Fil, Aaron and Manny back from EX

13:13 HAM2,4,5 ISI WDs tripped

13:20 Travis in LVEA w/comm appr

13:22 WDs in HAM 2,4,5 reset after ISIs settled down

13:25 Dan into LVEA w/comm appr

13:53 HEPI WD accumulators cleared - HAM2 (2) and HAM5 (348) none were incrementing at the time.

13:55 CP-7 on H0VE_SITE_OVERVIEW.adl screen was alarming earlier in the day and now has remained red since right before lunch.

13:57 Travis out of LVEA

14:12 Dan into LVEA to execute LHO WP #4942

14:30 Joe into the LVEA w/comm appr - checking batteries in foklift

14:39 Sheila and Evan to End Y to investigat a problem wih the ALS

14:45 Joe out of the LVEA.

Suspension MC3 had a tripped watchdog that was brought to my attention that wasn't being reflected on the Guardian Ops Overview screen. (didn't turn red).

Summary:

Following the PR2 scan from last week (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=14845), I used a single bounce beam from IX and scanned the BS while using SR2 to center the beam on AS_C. I also scanned SR3 while using the same ASC feedback on SR2.

There's at least 0.7% or so in the SRC-AS path donwstream of SR3. Tomorrow I'll roughly center SR2 by looking at the SR2 baffle and scanning SR3, then scan SR2 to maximize AS_C while keeping AS_C centered using pico.

The beam might become off-centered on SRM depending on how well the Faraday is centered on SRM, but that should be fine.

Details:

In the first attachment, the plot on the left  is AS_C_SUM normalized VS the BS Y slider value (while the ASC loop was acting on SR2 to center AS_C).  The scan started from Y=-296.5.

As the BS Y was increased, at some point PIT feedback on SR2 became large and eventually railed the SR2 BOSEMs, and that's probably why the plot shows a kink at around BS Y=0. I had to stop the scan at BS Y=83.5. During this scan BS BOSEMs never railed. Though it's not shown here, I also scanned SR3 while using the same SR2 feedback to center the AS_C, and the data also showed similar kink.

The plot on the right is a similar plot for SR3 scan. Note that these data were taken on a different days, so the laser power could be different at sub-% level, and the IFO alignment could be substantially different, but you can see that the two plots show the same kink.

Based on the above, I'd guess that the reason why SR2 had to be moved in PIT despite YAW scan is because the centering on SRM or SR2 was poor in PIT. (Remember, ROC of SR2 is -6.4m and that of SRM is -5.7m.)

In the second attachement, I just plotted the beam displacement for the BS scan at SR3, SR2, SRM and OFI input normalized by either the beam radius (left) or optic radius (right), taking into account the fact that ASC is keeping AS_C centered using SR2. This doesn't prove anything, but suggest that SR2 and OFI are kind of dangerous as far as the clipping by the beam motion is concerned in this measurement, and we don't have to worry much about SR3.

(Note: The reason why AS_C should be centered is because the centering affects SUM number at sub % level.)

Ed reported that HAMS 2,4 and 5 all tripped at about 13:15 today. All tripped on actuators, but the spikes are small, and none of the in chamber instruments show much prior. HAM4 shows a trip that looks like something maybe hitting the chamber, so more understandable, but that trip was much later. HAM2's and 5's trips happened within 2 minutes  of each other, and look very similar. HAM2 first, then HAM5, I don't know when HAM4 happened as therehas been a subsequent trip. I've attached the HAM5 watchdog plots.

similar to last week's restart of the end stations beckhoff computers, when h1ecatx1 was restarted after lunch today the DAQ EDCU did not completely reconnect to all the channels. This is a strange reconnection event, with the EDCU code giving conflicting reports on the connection status:

daqd>
23363 channels
23335 connected
28 disconnected
74179 connection events processed
126179362 value change events processed
Total 0 disconnected.
daqd>
 

(normally the list of disconnected channel names would be given)

The number of connected channels as reported in the DAQ EPICS PVs is the total number minus 28.

Cyrus suggested we try restarting the EPICS gateway between the H1 front end LAN and the Slow Controls LAN. I did this at 14:15 but we still have 28 missing connections. I have asked the slow controls group for a copy of the Beckhoff h1ecax1 databases to see if the number 28 relates to particular epics records.

At the moment the only way to cleanly reconnect all Beckhoff channels to the DAQ is to restart the DAQ.

There's a huge 364Hz line in ALS TRY.

I'm not sure if this was already there for a while. Though we probably changed the loop gain of the ALSY PDH loop when we realigned the green path on ISCTEX today.

Gerardo & Hugh

We recentered these oplevs to make better use of their tracking.  And especially HAM2 as it seemed to be way off on one quadrant.  Luckily we didn't lose the beams.

First attached is a three day trend showing yesterdays fun with HAM2 and the recentering this morning.

Next is the last eighty minutes, maybe given the wandering, the X-Y plots on the medm should have their window zoomed out some.  This seems to be an issue mostly on HAM3 Yaw.

The third plot shows the five hour detail around yesterday morning on HAM2 with the HEPI Cartesian RZ (yaw) RY (pitch) and HP, the horizontal pringle mode.  This further shows even though the HEPI RZ has servo'd back to its reference, the OpLev Yaw would disagree.  Likewise, even though the HEPI pitch shifted way off for most of the time and then came almost all the way back, the OpLev Pitch doesn't follow that in any reasonable way.  Even though these OpLevs have no caibration as I understand, this says alot about system distortion.  I've included a feducial from the large shift in the HP calculation but it doesn't appear to correlate to anything in the OpLev response.