Following the Beckhoff chassis work, we brought the h1psl0 system back online by power cycling the CPU and IO Chassis. We are seeing a regular excursion of the IRIGB timing signal on the IOP model (every restart in the past two days has seen this). It takes about 30mins to ramp back to its nominal value. In the plot you can see the current ramp and the one associated with last night's reboot.
Removed chassis and found similar failure as yesterday. This time it was the TCS X rotation stage power board that failed. Removed failed board from chassis and re-installed PSL / TCS chassis back in rack so PSL could be brought back up. For now we are not connecting the TCS Y and X rotation stage field cables to the EtherCat chassis. Further investigation needs to be done to see why the same part (diode) has failed on two separate boards.
model restarts logged for Wed 12/Nov/2014
2014_11_12 09:21 h1ioppsl0
2014_11_12 09:21 h1psldbb
2014_11_12 09:21 h1pslfss
2014_11_12 09:21 h1psliss
2014_11_12 09:21 h1pslpmc
2014_11_12 22:27 h1fw1
2014_11_12 23:20 h1ioppsl0
2014_11_12 23:20 h1psldbb
2014_11_12 23:20 h1pslfss
2014_11_12 23:20 h1psliss
2014_11_12 23:20 h1pslpmc
2014_11_12 23:28 h1ioppsl0
2014_11_12 23:28 h1psldbb
2014_11_12 23:28 h1pslfss
2014_11_12 23:28 h1psliss
2014_11_12 23:28 h1pslpmc
h1fw1 unexpected restart. PSL restarts due to IO Chassis +24V DC power glitching related to Beckhoff chassis issues.
(Alexa, Evan, Dan, Nic, Sheila)
We wanted to try DRMI with the arms off resonance again, but were running into some troubles. We haven't really paid any attention to input alignment since maintence day happened... First, when we lock PRX and use the wfs to align PRM the ASAIR build-up is too low by a factor of 2. It's possible that our alignment onto ASAIR has degraded since Keita adjust SR2, SR3, especially given that the image position on the camera has moved so much. More concerning is the fact that when we try to lock PRMI on the SB the POPAIR18 build-up is also very low; the maximum we got was 200ish; when we expect more like 360. We did not have a chance to try DRMI, or investigate too deeply into the PRC gain because of the beckhoff issues.
It might be that we've been following the temperature drift in the LVEA. There has been a big downward temperature spike, then the FMCS responded, and totally totally overshot.
In the attached, colored vertical lines indicate Suresh's laser change, Hugh's HAM2 change and commissioners' changing input alignment. From the HAM2 oplev, Hugh's work looks like a minimal impact.
But the time the commissioners responded to whatever change by turning the mirrors corresponds to the temperature change.
Appears to be the same failure mode as yesterday.
We have left the Beckhoff chassis in the blown-fuse state. We have successfully restarted the frontend models (see below), but the Beckhoff will not run properly without this chassis, so we cannot continue tonight.
To bring the frontend models back: Nic shut off the PSL frontend and Sheila power cycled the IO chassis. Then Nic started the PSL frontend. This didn't work the first time, so Nic shut down the PSL frontend again, unplugged and plugged in the power cable, then turned it back on.
according to Daniel, we could have recovered partially from this last night by going into the system manager, right clicking on the bad chassis and selecting disable. Then a red X would appear over that chassis. If we redo the generate mappings, check configuration, activate confguration, the system should run again.
Attached is a screenshot of the TwinCAT System Manager. The context menu will bring up the option to disable an entire EK1100 chain. Once disabled, a red X should be visible in the overview. After disabling the offending chassis or module, one has to generate the mapping, check the configuration and activate the configuration (see second screenshot). Make sure you are in run-time mode.
Caveat: The chassis are daisy chained. Turning off a chassis will no only disable the chassis itself, but all units which are hanging off this chassis. Using an Ethernet coupler it is possible to bypass a single chassis.
(Alexa, Evan, Sheila, Nic, Kiwamu)
Tonight we locked ALS DIFF with a UGF of 8 Hz, phase margin of 60 deg (2014-11-12-ALS_OLTF.pdf). The in loop noise is 1e-3 um/rtHz RMS down to 0.1 Hz (ALS_Diff_spectra_11122014.pdf). This is comparable to our old ALS_DIFF spectra, which was comparable to HIFOXY days.
We are running under a similar configuration to LLO. We are using an offloaded distribution for the Quads. Our LSC-DARM and L1_LOCK_L filters are similar to LLO's as described in alog 14961. We have added a resG filter in LSC-DARM to supress noise at 1 Hz and 0.4 Hz. We also added an additional boost in the L1 LOCK L stage to improve our RMS at low frequency. The ESD LOCK L filter was initially empy; however, we added an elliptic filter in L3_LOCK_L to supress noise we saw at around 100 Hz in the ESD coil output, and to reduce the range we use on the ESD coils. The current LSC-DARM filter is shown in LSCDARM_filters.pdf. Meanwhile the L1_LOCK_L filter with the addition of the ESD ellpitcal filter is shown in UIM_filters.pdf.
We have about a factor of 7 headroom in the ESD before we would saturate, and about a factor of 30 head room in the ESD. Tonight the 1-3 Hz seismic is fairly high (0.1 um/sec) but the low frequency is low (0.2 um/sec at 0.1-0.3 Hz). So it seems like we have some room for the seismic to get worse without needing to add the top mass.
Our UIM/ESD crossover is at 0.9 Hz with 50 deg phase margin. The L1_LOCK_L_GAIN = 0.28, L3_LOCK_L_GAIN = 1. We collected data from L1_LOCK_L IN1/IN2 to measure the crossover (UIM_measure_EXonly.pdf). The data is also depicted in DARM_crossovers.pdf; this image also shows the same TF produced by the model along with the UIM/ESD. The model seems to agree with the data well; albeit with a gain of 5 fudge factor. I have also attached the OLTF produced by the model (DARM_TFs.pdf).
We found we could stabily and repeatbly lock ALS DIFF with a LSC-DARM gain of 400. We were ONLY feeding back to EX. We are suspicious of the EY ESD; when we feed back to EY the y-arm eventually drops lock even with a low DARM gain. We did little investigation into this issue.
ALS_DIFF guardian has been updated, and can lock DIFF in this configuration.
Nice to see this working again. I'm confused by your sentence: We have about a factor of 7 headroom in the ESD before we would saturate, and about a factor of 30 head room in the ESD. Do you mean the UIM stage for one of these? Also, in the future 'ESD coil' should be 'ESD electrode'.
Sorry Peter, I meant to say: We have about a factor of 7 headroom in the ESD before we would saturate, and about a factor of 30 head room in the UIM. I have also attached the coil spectrum now.
J. Kissel Sheila has tasked me into figuring out what's wrong with ETMY. Here're some additional notes that I've gathered from talking to Alexa / Sheila / Evan/ Nic that were not included in the (already amazingly and delightfully detailed) entry above. - I found EX & EY Yaw damping loops *without* the +12 [dB] gain-only filter on. Sheila / Alexa didn't know when / why the filter was turned OFF; a dataviewer trend reveals that it was just never turned ON after install on Nov 11th. Since this bug is common to both ETMs, I don't expect it was the problem. I've now turned them ON so we have as little difference between the damping loop design as possible, so we have less things to blame. - EX (ONLY) has a new "NicLP" low-pass filter engaged in FM2 of the L3_LOCK_L control. From what I gather, this elliptic low-pass filter was installed *after* they'd stopped using EY, so I also don't suspect this difference. - EY (ONLY) has optical lever damping in PITCH engaged. This also was turned on right as the switch to only using EX only, when it was determined that there was a problem with EY. It's also the only QUAD damping loop that has a sufficient amount of documentation associated with it (see LHO aLOG 14878) that we trust it to be functional (thanks Evan)! I'll now begin measuring stuff, in hopes to find problems...
This morning, after a model restart, the ISS second loop configuration was no more correct. We should save a reference snapshot at some point. For the moment, here is a picture of the "standard working configuration" at the moment.
This configuration ensures that the switches and gain sliders of the second loop are working properly. For some reason, the model sums the analog SUM signal to the digital SUM signal. To avoid this, the filter banks PD_14_SUM and PD_58_SUM shuold have the output turned off.
Since only the SUMXX_REL_OUT is saved to disk, for the moment being I hacked the configuration of SECONDLOOP_SUMXX_DC to provide a constant offset equal to the total DC in the PD sum, in volts. Therefore the _REL signals are now calibrated in units of dP/P and saved to disk.
Here is a collection of suspicious CPS trips on ITMX. This is a problem only on this chamber, which has just cropped up in the last few weeks. It needs to be investigated.
another example
Still there, still a problem....
I made a few plots today. The spectrum of the CPS right before and right after the trip, and the zoom-in time series plot around the time of the trip. I notice that the time that the CPS signal goes down does not match the GPS time indicated the trip exactly (off by a few hundred milliseconds). While making the spectrum I also notice the peak near 4Hz in the "beforetrips" spectrum plot. So I went and look at the LHO summary page and found that there's a small gaussian-looking bump around 4Hz that seems to appear everyday. Probably not very important but just wanted to point that out. Also, please note that V2, H2, and H1 signals are almost perfectly overlapped (that's why you only see two colors). I'll start looking into other related channels that *should* have seen the signals. Although I believe this might be an electronics issue.
Morning meeting:
Shift summary:
7:40 Karen to LVEA cleaning
8:24 Jim B powered off PSL computers for power cycle
8:47 Bubba with Freestone Eng. LVEA tour
9:13 Fil and Aaron to EX PEM cabling
9:19 Richard replaced PSL Beckhoff chassis and powered on IO chassis
10:13 Jim B and Dave to CER
11:08 Betsy to LVEA SUS test stand
11:30 Fire Dept. on sit
11:45 Fire Dept. done
12:20 Fil and Aaron back from EX
12:36 Dave to CER checking drawing
13:44 Dan H to HAM6
15:00 King Water on site
15:15 Freestone Eng. on site picking up container on both arms with Bubba
15:22 Suresh to HAM2, OpLev work
Looking at the SR2 peanuts-shaped baffle, I set the new SR3 alignment offset.
SR3 Old: [430.3, 142.6]
SR3 New: [654.3, -38.8]
The original position was not that bad, it was a bit too the left and low.
After this, I scanned SR2 while centering the AS_C QPD using picos.
SR2 Old: [2963.7, 2728.0]
SR2 New: [1339.7, 1575]
The new AS beam position on the camera is to the left and high.
SR3 details:
Peanuts-shaped SR3 baffle was used as a reference.
The IFO beam is supposed to be in the right hole, the angle between SR3-SR2 beam and SR2-SRM is 29 mrad, the distance from the baffle to SR2 is about 460mm (just by eyeballing the drawing), so the beam separation on the baffle is about 17.5mm.
This means that the SR3-SR2 beam should be about 17.5/2 mm to the left from the center of the right baffle hole.
PR3 YAW offset when the beam just hits the right edge of the baffle was -1394.4, while the offset when the beam was on the center line of the right baffle hole was -194.4.
When looked from HAM5, the right baffle hole looks like an ellipse with the major axis vertical, and the minor radius is 67.5mm.
Therefore the PR3 YAW offset should be
SR3 Y = -194.4 + (-194.4+1394.4) * 17.5mm/2 / 67.5mm = -38.8
PR3 P offset when the beam hits the top/bottom edge of the tallest part of the hole was 2350.3/-1041.7, and the average is 654.3.
This also means that the SR3 YAW slider calibration is off.
For YAW, 1200 urad of the slider offset produced about 67.5mm.
2*angle*16m = 67.5mm -> angle = 2.1mrad.
Therefore, in reality, the SR3 slider calibration for Y is 2.1mrad/1.2mrad = 1.76. (But I'd claim that the measurement error is probably as large as 30% or so).
For SR3 PIT, it's good.
The beam moved from the bottom to the top with -3392 urad slider step. The hole height is 114mm.
2*angle*16m=114mm -> angle = 3.6mrad.
The SR3 slider calibration for P is 3600/3392 = 1.06
SR2 details:
Before SR3 bias was set, AS_C was centered using pico.
After SR3 was done, AS_C was centered using SR2 bias. After this, SR2 bias was [1399.7, 1485.0]. This is our "initial position".
Then SR2 bias was scanned first in Y, then in P, and then Y again.
Attached left is the Y scan and the right is the P scan. Blue vertical lines indicate where I started.
As I was sort of suspecting, PIT was more off than YAW, and anyway I settled on [P,Y]=[1339.7, 1575].
One caveat is that the final position corresponds to one data point where I see a jump in the AS_C SUM (look at the green circle that goes to 1.04). This is repeatable, and I think this is where some stray beam or maybe the reflection from the AS_C goes into the AS_C. I don't think this corresponds to smaller loss. This position was chosen as the final position by eyeballing the peak of the green plot excluding that abnormal data point.
K. Venkateswara, J. Warner
The BRS damper (see 14388) has been malfunctioning since yesterday afternoon. It's controller is an open loop system that assumes it is in the centered position when it is first started. Occasionally it can get restarted accidentally when the insulation box is bumped (probably due to a short somewhere, but I'm not sure where). It looks like this happened yesterday around noon.
This requires a manual re-centering of the turn-table for the feedback system to work as designed. I've attached the directions to do this. Jim will do this later today or tomorrow.
I checked on the BRS this morning, and found the code frozen. I followed Krishna's fairly straightforward document up to the point of opening the box and found the masses 90 degrees out of position (NW/SE instead of SW/NE), I was unsure of what to do, so I closed up, restarted the code and called Krishna to consult.
Afterwards, I went back to EX this afternoon, and re-aligned the masses on the damper. I did this by first modifying the code as described in Krishna's document, then opening the large insulating panel on the west side of the BRS enclosure, and just turned the turn table until the masses were approximately SW/NE. The turn table moved quite easily, I was expecting more resistance.It was dark down there, so it was hard to get a good picture, but I've attached an image taken from the west side of the BRS enclosure, showing how I left the masses. I then went back to the rack, re-commenting the appropriate code and restarting. Haven't had a chance to check on the BRS since then, but it looked okay from what I could see on the laptop readouts.
Looking back at the data, it seems that the changes made in the morning, resulted in a positive feedabck loop leading to large amplitude oscillations in the BRS. The changes made in the afternoon reversed it but it took ~5-6 hours to damp. However, the software appears to have crashed again later. Tomorrow, I'll investigate for intermittent shorts or other grounding issues.