I have started to tar and compress older autoburt snapshot files to free up disk space - the snapshots saved so far consume over 1TB on disk, which is about 25% of the capacity of the /ligo filesystem. Currently the plan is to retain roughly the last three months of data in it's uncompressed form, with data older than that retained on disk in a compressed tar archive only. Files will be archived by month (so all of the files from a given month will be in a single tarball). Since the snapshot files are all ASCII text, they should compress down well. This should also speed up the rsync backups, since there will no longer be 2.8 million files to traverse looking for changes. If you need assistance accessing older snapshot files stored in the tar archives, please see one of the CDS admins.
Something happened at about 10:30PM (11-12-2014 6:30 UTC) and the 363Hz oscillation of the end Y PZT mirror(s?) went totally unstable.
Daniel and Alexa are heading to EY to swap the controller.
In the attached right, references are from 24 hours ago, current traces are as of now. The trend shows the green QPD and the VEA temperature for 3 days.
Turning the controller off and on brought the phase margin back to ~1º (from maybe ~0.01°). Needs a more permanent fix.
After ~2 hours, it started oscillating again. Alexa and I went back and jammed a piece of damping material between the lower and upper stages of the PZT flex mount. This seemed to have damped the resonance to a level similar to the one observed in EX.
Here is a picture of the foam installed under PZT2 (2 inch). It was difficult to get the foam all the way into the gap; so we left it installed on one side of the gab -- this might drift over time. I have also attached the new spectra of the EY and EX QPDs as of now.
The attached plots show the oscillating state (page 1), the "stable" state (page 2) from before the damper, and the damped state. The damped EY version most closly resembles the undamped EX version.
We also made sure that the low bandwidth jumpers were at the correct position in the controller. Looking at the controller (see attached) we found the two capacitors which are likely to be used by the integrator. The bandwidth could possibly be further decreased by increasing C5/C6. The current values are 2 µF each.
As I reported in 14965, I suspected WHAM3 ISI OpLev Pitch and Yaw were reveresed as I believe is HAM2's. See the attach where I yaw'd WHAM3's HEPI. The HEPI Pitch moves as well but a couple of orders of magnitude less. Likewise on the OpLev, the Pitch responds much more than the Yaw, so yes,
JeffK suggested I might see coherence between the SEI inertial sensors and the OpLev channels. Not sure how good this is or maybe I need to taylor the measurement.
Anyway, the upper traces are YAW (RZ) channels of the SEI(HEPI IPS & L4Cs, ISI CPS & GS13) TF with the Pitch & Yaw of the OpLev. The strongest TF, only 0.7, is between CPS YAW and the OpLev Pitch. The lower traces are the Pitch (RY) of the SEI TF with the OpLev showing the strongest response of 1.0 is between CPS Pitch and the OpLev Yaw. There is some 'correct response too but it is lower.
There is no drive here just the input ground motion. This may be more clear with an active excitation.
The chilled water temperature at MIDY alarmed HIGH this morning. There is no demand for cooling so the water is bypassing the cooling coils completely and therefore the water pump is only heating the water.
I have turned both mid station pumps off.
This cold front has sent our temperature control loops out of whack - Last night I turned on heaters in both end stations. Today we will monitor and perhaps turn on more heat in both the LVEA and the end stations.
Alexa, Evan, Nic, Sheila
We seem to have lost the -24 V rail and the -12 V on several racks in the electronics room. The PSL/IO/TCS beckhoff chassis power switch was in the off position when we went out there, I tried flipping it to on and the rocker switched off again immediately. On the cds overview we have lots of red on the PSL frontends, this chassis is in the same rack as the PSL/IO/TCS beckhoff chassis, and they are on the same 24 V supply. Nic restarted the front end models several times, then restarted the computer, then restarted the computer and restarted the PCIE expansion box. (before we realized that we were missing power).
There is not problem with the Power in these racks. The Beckhoff and IO expansion chassis only need +24 so we do not feed them -24Vdc and the 12V in the rack next to the PSL is the timing system which also only needs +12V so no - 12 V available. The Beckhoff chassis must have an internal problem which took down the PSL. I will open it up this AM to verify and hopefully fix the problem.
The TCS Y rotation stage power board had a problem (investigating now) that caused the 24V supply breaker to trip. We have replaced this board and the system is back up and running. Assuming this fault in the Beckhoff chassis caused a drop on the 24Vdc supply to the Front End IO chassis this caused the PSL front end to have problems. I have power cycled the IO chassis and restarted the FE computer. The system should be back and ready to lock.
(Alexa, Sheila, Evan)
As we were trying to lock ALS DIFF, we noticed that both the COMM and DIFF beatnote were too weak. We trended these values and found that the beatnote first disappeared when the PSL went down and then again at around 9am PST. I then tried locking the IR to the x-arm and found that I had to move PR3 by 4 urad in pitch to lock. The input wfs then moved IM4 and PR2 to increase the buildup. We suspect things got moved around again with the work and trips on HAM2, HAM3. We didn't spend too much time investigating the exact cause, and we did not want to go through the entire alignment to lock DRMI and check the POP18 counts since we want to focus on DIFF right now. Hopefully we didn't introduce a new clipping in the PR2 baffle by the amount we moved.
OLD | NEW | |
PR3 (P, Y) | (-246, 88) | (-242,88) |
IM4 (P, Y) | (11414.2, -5002.3) | (14326.4, -5102.9) |
PR2 (P, Y) | (1035.2, 2981.28) | (1006.9, 2973.95) |
IMPORTANT: I have not saved these new alignment values, so if the guardians get realigned they will restore the old alignment values.
Also, the beam in the AS AIR camera looks too far to the right.
I moved PR3 back to the old values, whic today gave us a better COMM beatnote.
After yesterday and today recentering and tweaking, the ISS second loop is back to good working condition. The first attached plot compares some signal with the second loop open and closed. Dashed line, loop open, solid lines, loop closed.
The ISS second loop signals (SUM14 and SUM58) are calibrated in volts and the whitening filter is compensated. Using the slow DC channels, I could compute the total signal for each combination, which is about 11-12 V. I used those values to calibrate the signals in terms of RIN as in the figure. This should roughly correspond to about 20 mW total power hitting the array.
The out-of-loop signal is still showing some excess noise with respect to a flat level. Assuming that there are 10 mW impinging on the four out-of-loop diodes in total, the shot-noise-limited dP/P should be sqrt(2*nu*hplanck/P) = 6e-9 /rHz. Counting both the in-loop and out-of-loop shot noise, we get an additional sqrt(2). Therefore we expect the out-of-loop signal to be at the level of 8.6e-9 /rHz, not too far from the 1e-8 /rHz that we see at about 100 Hz.
The structures between 10 and 50 Hz are variable, but I was not able to change them significantly moving the IMC alignment or the picomotors in front of the ISS array. Notably, for most of the day there was an additional bump between 10 and 20 Hz, which disappeared some time at about 13.05 local time, without any conscious intervention on my side. See second plot. No idea what the origin of this thing is, but it was there also yesterday.
One more peculiar thing. We can use IM4_TRA_SUM to measure the RIN in transmission of the IMC, as an additional out-of-loop sensor. This is included in the attached plots. When the second loop is open, the RIN measured by IM4_TRANS agrees quite well with what is measured by the ISS second loop diodes. However, when the second loop is closed, IM4_TRA measures a significantly larger noise. This might be due to the sensor electronic or dark noise (I haven't checked it yet). However, at least at 10-30 Hz it measures the same structures visible in the ISS diodes, but a factor 4 larger. The shape is also slightly different. Surprinsingly, there is no coherence between IM4_TRANS and SUM58. This might tell us something about the origin of those peaks, but what, it's not clear.
The peaks between 15 and 25 Hz are visible, with different amplitudes and shapes, in all the eight photodiodes, but there is little coherence among them. See third plot. These peaks are not visible in the ISS QPD signals, even though there is a wider bump at similar frequencies. They may be due to scattered light. The most prominent structure is a 18 Hz peak.
Beckhoff. Daniel and Dave
Following Daniel's changes to the Beckhoff systems, I applied the latest INI files to the DAQ and restarted the DAQ. Unfortunately h1ecatx1 needed a restart shortly after this and we have some channels disconnected. I also copied the latest Beckhoff REQ files into the appropriate target areas and Patrick reconfigured the conlog using them.
Conlog: Patrick and Dave
I modified the script which creates the conlog channel list to use the generic include.txt file as well as the generated fe_include.txt and grd_include.txt. I changed the script which generates the Guardian autoBurt.req file (which is used to create grd_include.txt) to include the USERMSG byte encode strings for conlog logging. I created a script to convert the output of USERMSG to a string. Patrick reconfigured conlog's channel list.
Guardian: Dave
I checked that the running guardian nodes and the GUARD_OVERVIEW medm screen are in sync with each other (they are). I updated the guardian autoBurt.req file (using guardctrl list to get the node listing).
Filtermodules Load Status: Dave
Many H1 models are running with partial loaded filter files. Plus two models (SUSPR3 and SUSMC2) had modified filter files. I pressed the "LOAD COEFFICIENTS" button on the GDS_TP screen for the following models to resync them with their filter files:
ISIHAM[3,4,5,6], ISIETMX, SUS[PRM,PR2,SR2,SRM,BS,ETMY,ETMX,PR3,MC2], TCSCS, LSC, LSCAUX, OMC, ASC, ASCIMC, ISC[EX,EY]. No problems were encountered.
inicheck of all DAQ INI files: Dave
I ran inicheck on all the DAQ INI files, we have channels with mixed case on the systems: ISI[BS,ITMX,ITMY,ETMX,ETMY] and LSC.
foton check on filter files: Jim and Dave
We ran foton -c on a copy of all the running filterfiles. Most of the errors we saw relate to obsolete filters left in the body of the file but no longer in the MODULES header. At some point we should make the effort to clean up the filter files of obsolete material (quak generated files exempted from a foton cleanup).
I've restarted the CDS Matlab license server to install a new license file (that is not due to expire...). Existing sessions fail over to the Caltech servers automatically. I'll have to update the NLM_LICENSE_FILE environment variable later to remove the old Caltech triad license servers and add the new single server configuration for failover purposes.
08:00 PSL is DOWN - sent e-mail to D Cook, P King and R Savage
08:00 3IFO team is in the LVEA
08:01 the LVEA is in the bifurcated LASER hazard state
08:15Fil, Aaron and Manny out to EX
08:30 Rick called me and walked me through restarting the PSL frontend laser.
09:00 Jody into the LVEA
09:12 Jody ou of the LVEA
09:20 Mitchell out of th LVEA
10:10 Karen into EY
10:30 Cris into EX
10:35 Patrick and Richard working on Beckhoff system for Mids and Ends
10:36 D Barker informed me of a DAQ restart at 12:00P
10:41 Karen heading back from EY
10:53 Karen heading out to MY
11:02 Cris at MX
11:07 Patrick and Richard are done working on Beckhoff
11:35 Jeff and Andres out of LVEA - LVEA returned to full LASER hazard
11:36 Gerardo and Kyle out to LVEA
11:40 Karen leaving MY
11:50 Kris back from MX
12:04 Gerardo and Kyle out of the LVEA
12:13 DAQ restart delayed- waiting for EY update
12:21 DAQ restarted at 12:25
12:35 Fil, Aaron and Manny back from EX
13:13 HAM2,4,5 ISI WDs tripped
13:20 Travis in LVEA w/comm appr
13:22 WDs in HAM 2,4,5 reset after ISIs settled down
13:25 Dan into LVEA w/comm appr
13:53 HEPI WD accumulators cleared - HAM2 (2) and HAM5 (348) none were incrementing at the time.
13:55 CP-7 on H0VE_SITE_OVERVIEW.adl screen was alarming earlier in the day and now has remained red since right before lunch.
13:57 Travis out of LVEA
14:12 Dan into LVEA to execute LHO WP #4942
14:30 Joe into the LVEA w/comm appr - checking batteries in foklift
14:39 Sheila and Evan to End Y to investigat a problem wih the ALS
14:45 Joe out of the LVEA.
Suspension MC3 had a tripped watchdog that was brought to my attention that wasn't being reflected on the Guardian Ops Overview screen. (didn't turn red).
Summary:
Following the PR2 scan from last week (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=14845), I used a single bounce beam from IX and scanned the BS while using SR2 to center the beam on AS_C. I also scanned SR3 while using the same ASC feedback on SR2.
There's at least 0.7% or so in the SRC-AS path donwstream of SR3. Tomorrow I'll roughly center SR2 by looking at the SR2 baffle and scanning SR3, then scan SR2 to maximize AS_C while keeping AS_C centered using pico.
The beam might become off-centered on SRM depending on how well the Faraday is centered on SRM, but that should be fine.
Details:
In the first attachment, the plot on the left is AS_C_SUM normalized VS the BS Y slider value (while the ASC loop was acting on SR2 to center AS_C). The scan started from Y=-296.5.
As the BS Y was increased, at some point PIT feedback on SR2 became large and eventually railed the SR2 BOSEMs, and that's probably why the plot shows a kink at around BS Y=0. I had to stop the scan at BS Y=83.5. During this scan BS BOSEMs never railed. Though it's not shown here, I also scanned SR3 while using the same SR2 feedback to center the AS_C, and the data also showed similar kink.
The plot on the right is a similar plot for SR3 scan. Note that these data were taken on a different days, so the laser power could be different at sub-% level, and the IFO alignment could be substantially different, but you can see that the two plots show the same kink.
Based on the above, I'd guess that the reason why SR2 had to be moved in PIT despite YAW scan is because the centering on SRM or SR2 was poor in PIT. (Remember, ROC of SR2 is -6.4m and that of SRM is -5.7m.)
In the second attachement, I just plotted the beam displacement for the BS scan at SR3, SR2, SRM and OFI input normalized by either the beam radius (left) or optic radius (right), taking into account the fact that ASC is keeping AS_C centered using SR2. This doesn't prove anything, but suggest that SR2 and OFI are kind of dangerous as far as the clipping by the beam motion is concerned in this measurement, and we don't have to worry much about SR3.
(Note: The reason why AS_C should be centered is because the centering affects SUM number at sub % level.)
There's a huge 364Hz line in ALS TRY.
I'm not sure if this was already there for a while. Though we probably changed the loop gain of the ALSY PDH loop when we realigned the green path on ISCTEX today.
alexa, nicolas
In order to determine whether the line we were seeing on ALS TRY was a result of aliasing of something higher frequency in the ADC, we checked the analog spectrum of the ALS TRY photodiode.
Attached photo shows the line at 364 Hz in analog as in digital, so no evidence for aliasing.
Sheila, Alexa, Nic, Evan
Sheila and I went down to EY to see if we could track this down.
This 365 Hz line appears to be caused by oscillation in the 2" PZT mirror on ISCTEY. If the green refl beam clips on the RFPD, this produces amplitude modulation that leaks into the PDH error signal.
Line is obvious on ALSY QPDs on the transmon.
It seems the oscillation is mostly yaw (364Hz), though there is a pitch line about 10x smaller (420Hz), as well as lines in ALSX QPDs (pit: 389Hz yaw: 363Hz).
Gerardo & Hugh
We recentered these oplevs to make better use of their tracking. And especially HAM2 as it seemed to be way off on one quadrant. Luckily we didn't lose the beams.
First attached is a three day trend showing yesterdays fun with HAM2 and the recentering this morning.
Next is the last eighty minutes, maybe given the wandering, the X-Y plots on the medm should have their window zoomed out some. This seems to be an issue mostly on HAM3 Yaw.
The third plot shows the five hour detail around yesterday morning on HAM2 with the HEPI Cartesian RZ (yaw) RY (pitch) and HP, the horizontal pringle mode. This further shows even though the HEPI RZ has servo'd back to its reference, the OpLev Yaw would disagree. Likewise, even though the HEPI pitch shifted way off for most of the time and then came almost all the way back, the OpLev Pitch doesn't follow that in any reasonable way. Even though these OpLevs have no caibration as I understand, this says alot about system distortion. I've included a feducial from the large shift in the HP calculation but it doesn't appear to correlate to anything in the OpLev response.
The last paragraph about the last plot is confused by the reversed Pitch and Yaw of this OpLev, see 14965.