P. King, J. Oberling, J. Bartlett, E. Merilh
We realigned the FSS RefCav this morning. It was down around 0.7V on the FSS TPD. We aligned both mirrors of the input periscope in both pitch and yaw. Yaw alignment did not increase the RefCav TPD signal by much; most of the adjustmen was to pitch on both mirrors. We left the FSS TPD reading 1.56V and locked the adjustment screws on both periscope mirrors. We also found that the input periscope mirror mounts (they both mount to a ~1.5 inch diameter post that in turn is mounted to the PSL table itself) were a little loose. Peter was able to tighten both mounts by ~1/8 to ~1/4 of a turn on the big screws that mount the mirrors to the post. Will continue to monitor the FSS TPD to see if it continues to drift down.
We measured the laser power in 2 places after the alignment:
We also measured the voltage on the FSS REFL PD when the RefCav was locked and unlocked:
model restarts logged for Mon 23/Feb/2015
2015_02_23 21:27 h1fw1
one unexpected restart. Conlog frequently changing channels report attached.
Evan, Alexa, Dan, Sheila
Tonight we made some more measurements of the plants for DHARD WFS, this time with no watchdogs tripped. The screens shots attached show both the open loop gains and the plant measurements for pitch and yaw. We plan to fit these and make a better loop for on resonance during maintence day today.
All of these measurements were made with the oplev damping on the ETMs off. The plant inversion used for the measurements at 50 pm was a little different than what we used on resonance.
In full lock, we had the following loops closed:
After engaging ASB36Q, we ramped down the yaw oplev damping on the BS. After a few seconds, ASB36Q began ringing at 1 Hz or so and the lock broke. Unclear whether this was a WFS loop instability or unsuppressed motion of the BS.
I altered the plant inversion for the diff ETM loops so that we are not applying so much drive above 10 Hz. Instead of five poles at 100 Hz and an ELP at 30 Hz, there are now three poles at 20 Hz, two poles at 100 Hz, and an ELP at 10 Hz. We lose some phase though, so we might not be able to get 1 Hz bandwidth out of these loops.
Today I followed up on last night's commissioning of the ETMY violin mode damping. I tested the ETMY L2 darm damping filter that was set up for the 508.1Hz violin mode (MODE2), feeding back to pitch. Last night I found that a positive gain of 500k had an effect on the mode, but this was while the L2 current watchdog was tripped. Tonight a gain of -30k worked well, so the current watchdog apparently reduces the drive AND flips the sign at high frequencies.
The attached plot shows progress of the mode with different gain settings. As of this writing the mode is as low as it's ever been. (To get this mode lower, we'll need to compete with the mode at 508.2Hz. Tricky.)
SUS_PRM guardian became unresponsive, but has how been restored.
The control room reported that the SUS_PRM guardian had become completely unresponsive. The log was showing the following epicsMutex error:
2015-02-24T03:45:51.412Z SUS_PRM [ALIGNED.run] USERMSG: Alignment not enabled or offsets have changed from those saved.
epicsMutex pthread_mutex_unlock failed: error Invalid argument
epicsMutexOsdUnlockThread _main_ (0x2999f20) can't proceed, suspending.
The first line is the last legit guardian log message before the hang. The node was not responding to guardctrl stop
or restart
commands. I then tried killing the node using the guardctrl interface to the underlying runit supervision interface:
controls@h1guardian0:~ 0$ guardctrl sv kill SUS_PRM controls@h1guardian0:~ 0$
This did kill the main process, but it unfortunately left the worker subprocess orphaned, which I then had to kill manually:
controls@h1guardian0:~ 0$ ps -eFH | grep SUS_PRM ... controls 18783 1 4 130390 37256 7 Feb19 ? 05:13:18 guardian SUS_PRM (worker) controls@h1guardian0:~ 0$ kill 18783 controls@h1guardian0:~ 0$
After everything was cleared out, I was able to restart the node normally:
controls@h1guardian0:~ 0$ guardctrl restart SUS_PRM stopping node SUS_PRM... ok: down: SUS_PRM: 49s, normally up starting node SUS_PRM... controls@h1guardian0:~ 0$
At this point SUS_PRM appears to be back to funcitoning normally.
However, I have no idea why this happened or what it means. This is the first time I've seen this issue. The setpoint monitoring in the new guardian version installed last week means that nodes with the monitoring enabled (such as SUS_PRM) are doing many more EPICS reads per cycle than they were previously. As the channels being monitored aren't changing very much, these additional reads shouldn't incur much of a perfomance hit. I will investigate and continue monitoring.
Sheila, Alexa, Elli, Evan, Gabriele
Because the DHARD WFS plant for pitch changes significantly with the CARM offset reduction, which we think could be caused by miscentering on the optics combined with radiation pressure. We have attempted to center the green beams on the ETMs this morning. We used the PCAL cameras and the code that Elli had been using, but did not use the fitting function, instead just looking by eye at the corsshairs that mark the center of the optic relative to the position of our beams.
For the Y arm we moved the green QPD A yaw from 0.5 (photo 100) to -0.5 (photo 105). For the X arm we ended up not moving, there are no QPD offsets on the X arm.
We found that we were not well aligned after this work, which might have been unrelated to what we did. Now we have reverted the offsets, but it woul be worth trying to put them back in at some time.
We had some suspicious alignment issues today. To align the two arms, we ran the green wfs which feeds back to the ITMs, ETMs, and TMSs, as we have been doing for several weeks now. Then we followed our input pointing, PRM aligning, MICH aligning, and SRM aligning. We could then lock IR with ALS COMM and produce a good build up in the x-arm; however, when we locked ALS DIFF, we were about 5% below the nominal IR build up in the y-arm. We kept seeing an oscillation in AS 45Q before transitioning to RF DARM, and we consistently were losing lock. DRMI was also taking longer to lock (about 12min). We were frusturated and decided to start aligning from scratch. We cleared all the WFS history, used the nominal QPD offsets as described above, and then used the baffle PDs to align the TMS. We then adjusted ITMs and ETMs by hand to get the green build up high in both the arms. This restored the IR build up in the y-arm when locked on ALS DIFF, and allowed us to reach full lock with DRMI locking on the order of minutes. For sanity check, I looked at the PCAL image in green of the Y-arm and it looked the same as above with the corresponding QPD offset (image LHOX 106). I am not sure why the green wfs took us to a "bad" alignment today; these have been reliable in the past.
[Stuart A, Jeff K] To accommodate QUAD top stage bounce and roll mode damping using AS WFS as sensors (see LHO aLOG entry 16868) it has been necessary to prepare h1sus QUAD models ready for tomorrows planned roll-out. All h1sus QUAD models links to the common library (QUAD_MASTER.mdl) were previously broken to provide DARM ERR damping at M0 (see LHO aLOG entry 16655). Therefore, we proceeded to work with the locally modified QUAD models, until such a time that the 'best' damping approach can be incorporated into the common library part for use at both sites. The following list of changes have been made: (1) ETMs, added RFM receiver for H1:ASC-ETMX_AS_B_RF45_Q_P & H1:ASC-ETMY_AS_B_RF45_Q_P at the top-level (e.g. see h1susetmx_top-level_new.png below). (2) ITMs, added IPC receiver for H1:ASC-ITM_AS_B_RF45_Q_P. (3) All QUADs, added input matrix within QUAD/M0/DARM_DAMP block for DARM_ERR + AS_B_RF45 (e.g. see h1susetmx_DARM_DAMP.png below). (4) All QUADs, routed AS WFS from top-level through to DARM_DAMP and into summation block (e.g. see h1susetmx_M0.png below). These models were test built which produced the following errors (IPC related): - H1:ASC-ETMX_AS_A_RF45_Q_P & H1:ASC-ETMX_AS_A_RF45_Q_P - H1:LSC-ETMX_L_SUSETMX & H1:LSC-ETMY_L_SUSETMY - H1:LSC-OAF_DARM_ERR The first is expected due to changes needing to be made to h1asc model. The remaining issues we need to coordinate with Kiwamu who is updating/splitting the h1lsc model. It is planned to build, install and restart models tomorrow.
The cdsutils installation has been updated to r440, which includes a new CDSMatrix object that, once initialized, can be used to reference CDS front end EPICS matrix elements by name.
NOTE: this object uses the standard (row, column) ordering, for consistency with the most matrix element reference standards.
Example usage:
jameson.rollins@operator1:~ 0$ cdsutils --version cdsutil 440 jameson.rollins@operator1:~ 0$ guardian -i -------------------- aLIGO Guardian Shell -------------------- ezca prefix: H1: In [1]: from cdsutils import CDSMatrix In [2]: m = CDSMatrix('LSC-PD_DOF_MTRX', rows={'DARM': 1, 'MICH': 2,}, cols={'OMC': 1, 'POP_A_RF45_Q': 5}) In [3]: m Out[3]:
In [4]: m('MICH', 'POP_A_RF45_Q') Out[4]: 'LSC-PD_DOF_MTRX_2_5' In [6]: m['MICH', 'POP_A_RF45_Q'] Out[6]: 0.0 In [7]: m['DARM', 'POP_A_RF45_Q'] = 0
Jeff, Dan, others...
Sheila and I noticed last night that the ETMY L2 stage RMS current watchdog had tripped - the circled indicator lights in the first figure were red. Today with Jeff we tested whether this has any effect on the L2 actuation and reset the watchdog.
Turns out that with the watchdog tripped you can still actuate on the L2 stage, but with about 3x less force at DC than with the watchdog untripped. So, maybe this explains our crummy alignment stability last night.
The second plot shows the effect of applying a large offset in pitch to ETMY L2 before and after the watchdog was reset, as observed by the optical lever.
I've added the SUS-ETM{X,Y}_BIO_L2_MON channels as conditions to the ETM warning lights on the OPS overview screen.
I ran my (now 3x faster) BruCo script using a lock stretch that Dan pointed to me, from last Friday (GPS time 1107840506 + 600 s). The report can be found at the following address:
https://ldas-jobs.ligo.caltech.edu/~gabriele.vajente/bruco_1107840506/
Here is an excerpt, but yuo should definitely read the whole book:
The second bullet is indeed very strange since it seems that SRCL/PRCL contributes pretty much uniformly to the DARM background. It seems rather strange to me that sensing noie of SRCL/PRCL can couple up to those frequencies. One simple way to test it would be to add a band-stop filter at 1-2 kHz in PRCL and SRCL loops, and see if the coherence with DARM gets reduced. If not, it means that the same souce of noise contributes in a similar way to both RF signals (used for PRCL and SRCL) and DC signals at the AS port. I can imagine things like: high order modes and RF modulation amplitude noise...
J. Kissel, S. Aston As per ECR E1500090, Integration Issue 1015, and Work Permit 5069 we are preparing to copy LLO's infrastructure implementation of using AS_A_RF45_Q_PIT (i.e. the pitch signal calculated from the Q phase of the 45 [MHz] demodulated signal from the AS A WFS in HAM6, see D1200666 and T1100472) to damping the four QUAD's roll mode. However, we found ourselves asking many questions of the implementation, namely if blindly copying is the right thing to do, mostly because of it requiring Reflected Memory (RFM) IPC connections between the ASC model and QUAD models. We think the answer is the blindly copying is NOT the right thing to do. We think we have a better sensor that is more suitable to how the roll modes appear in the H1 ASC sensors, and we think we don't have enough IPC head room. See discussion below. Please tell us your thoughts on this. If we don't hear any push back by ~9a PT tomorrow morning, we're going forward with the changes to the scheme as decided below. (1) Recently, LHO has pushed forward many different ways to *reduce* the number of ISC to ETM reflectec memory (RFM) communications: (a) We have split the ISC end-station models into two parts, an ALS and ISC model (LHO aLOG 16443) (b) We've moved the RFM connection for differential tidal control from being sent directly to the ETMs to being sent to the end-station ISC models first, and then sent to the SUS models via Dolphin PCIE. (T1400733) (c) Removed the SEI EX / EY computers from the RFM Loop (LHO aLOG 16775) (d) We will be (also tomorrow) re-distributing our corner-station LSC front end models to move the DARM path from the h1lsc.mdl to the h1omc.mdl (E1500041) Indeed, even LLO -- though they've only found to have *intermittent* IPC errors -- is testing out a faster CPU for their LSC computer to decrease the clock cycle turn around time. ... so, for the two new ASC to ETM RFM connections that LLO has added, should we do the same thing at (2) and feed the RFM connections to the end-station ISC models, and then to the SUS via Dolphin PCIE? Here is the current status of relevant IPC errors / clock-cycle turnaround time: Avg Clock Cycle IPC Receiving Channels Error Rate (CPU Max, [usec]) Errors? [n errors/sec] LLO LSC 35 No ASC 161 No ISCEX 27 No ISCEY 25 No [ALSEX] [does not exist] [ALSEY] [does not exist] SUSETMX 50 Intermittent L1:LSC-ETMX_[DARM_ERR,L_SUSETMX] less than 1 SUSETMY 49 Intermittent L1:LSC-ETMY_[DARM_ERR,L_SUSETMX] less than 1 LHO LSC 36 Yes H1:ALS-[X,Y]_ARM_RFM X: 100 to 1000, Y: 1-10 ASC 160 Yes H1:ALS-[X,Y]_REFL_SLOW_RFM 2048 ISCEX 5 No ISCEY 5 No ALSEX 32 No ALSEY 31 No SUSETMX 48 Yes H1:LSC-ETMX_[DARM_ERR,L_SUSETMX] 1 to 25, 5 to 40 SUSETMY 49 Yes H1:LSC-ETMY_[DARM_ERR,L_SUSETMX] 10 to 120, 5 to 60 Reminders: - all of the above models are running at a sampling rate of 16384 [Hz], so they have 60 [usec] to complete their clock cycle - all IPCs have a delay of 1 clock-cycle built in, but if all the computations do not complete, or just barely have enough time to complete, then the IPC misses the even the 1 clock-cycle and throws an error - The RFM loops (there are separate loops for each end station) is now LSC -> SUSETM[X,Y] -> ISCE[X,Y] -> OAF -> ASC -> LSC where the delay between the LSC and SUSETM[X,Y], as well as the ISCE[X,Y] and OAF, is the one-way four [km] light travel time through a glass fiber, n * L / c = 1.5 * 3995 [m] / 3e8 [m/s] = 20 [usec] and the delay between nodes in the same building is believed to be dt = 0.7 [usec]. This means from the LSC back to LSC delay is 2 * n * L / c + 3 * dt = 2 * 1.5 * 3995 [m] / 3e8 [m/s] + 3 * 0.7e-6 [s] = 42 [usec]. (2) The roll coupling to specifically AS_RF45_Q_PIT is the *only* feedback sensor that's piped to the SUS. But the coupling is a dirt effect do to imperfections in the real system. What guarantee do we have that we'll have equal success with this and only this sensor? To help answer the question, I took a look at 5 of the more recent times when the IFO was fully locked on DC readout, (a) 2015-02-12 07:35:56 UTC (b) 2015-02-13 05:23:45 UTC (c) 2015-02-21 01:53:30 UTC (d) 2015-02-21 03:08:20 UTC, and there doesn't appear to be a pattern on where they appear. For example, (e) 2015-02-23 09:14:31 UTC in which the there are one of two roll modes visible in DARM: 13.18 [Hz] in (a) and (b), 13.81 [Hz] in (b), (c), (d), and (e). At these times, I've taken a 0.05 [Hz] resolution amplitude spectral density of DARM and *every* ASC channel, i.e. AS A -- 45 and 36, I and Q, PIT and YAW AS B -- 45 and 36, I and Q, PIT and YAW REFL A -- 45 and 9, I and Q, PIT and YAW REFL B -- 45 and 9, I and Q, PIT and YAW TRX -- A and B, PIT and YAW TRY -- A and B, PIT and YAW POP -- A and B, PIT and YAW, or 52 channels plus DARM, and compared the amplitude and coherence to DARM in these channels for the 5 times. For each lock stretch, when either the 13.18 [Hz] mode or the 13.81 [Hz] mode is visible, it has an amplitude of ~2e-8 [ct] in DARM. After staring at the plots for an hour or three, I've convinced myself (and Stuart) that in 4 out of the 5 lock stretches, either AS B RF45 Q PIT or YAW is the sensor channel that sees both modes reliably *not* AS A RF 45 Q PIT. Further, observability does not preclude controllability, so it's not immediately clear which of the sensor signals will actually serve well for feedback. Finally, in the 5 lock stretches, only two of the four possible QUAD's highest roll modes have appeared. BUT I tested the hypothesis "does the lower frequency mode appear only in the TRs, and the higher in POP?" on the hunch that those two signals would be able to distinguish between an ETM roll mode or an ITM roll mode: again, I've convinced myself that the lower frequency mode (13.18 [Hz]) is only ever visible in POP -- implying its an ITM -- and the higher frequency mode is visible in the TRs -- implying it's an ETM. Maybe. Here's my point: if we want to be able to damp both of these modes, then just one ASC sensor isn't the right answer, and which sensor sees the mode the best is IFO dependent.
Comparisng two stretches of lock that Dan pointed to me (one from Friday, good stable sensitivity, and one from last night, worse sensitivity), I find a clear non-stattonary behavior of the sensitivity, see the first two plots, which are spectrograms of ten minutes of data during each lock.
Analyzing last night's lock, I computed the band-limited RMS between 100 and 300 Hz and it indeed shows an almost regular oscillation at about 120-140 mHz. I compared this oscillation with few auxiliary channels, and found out good correlation with the SRM angular motion (third plot), and expecially with AS_RF36 WFS signals (fourth plot).
The last two plots show my usual least square fit of the BLRMS using a set of auxiliary channels:
In summary, I think we need to commission an angular control for the SRM, and it seems we have good signals from AS_RF36. The main motion is in pitch.
= Ops Summary =
== Morning Meeting ==
Visitors: Stuart, Gabriele, Ken, Myron. Welcome to Tumbleweed Land.
SEI (Hugh) Things seems okay. Platform survived the weekend.
SUS (Betsy) I can't hear you. Sorry :-(
ISC *silence*
CDS (Richard) Things are somewhat stable. Loading/unloading ldas racks.
Vacuum ...
== Activities ==
9:58 Elli and Dave to IOT2R
*Difficulty requesting MICH_DARK_LOCKED. The beam was too low and the PD wasn't picking up all the signal. Alexa adjusted SR2 - All is well (for now)*
10:49 Mitchell to Mid X
11:54 Mitchell back
12:36 Got an automated call from Hanford Emergency System saying the alarm will go off at some point...
12:40 Elli and Dave back
13:38 Myron and Ken to Mid X
15:04 Myron and Ken back
Jodi - The big truck that supposed to come move the racks is not coming today.
Laser Status: SysStat is good Output power is 28.7 W (should be around 30 W) FRONTEND WATCH is RED HPO WATCH is RED PMC: It has been locked 1 day, 2 hr 34 minutes (should be days/weeks) Reflected power is 2 Watts and PowerSum = 24.4 Watts. (Reflected Power should be <= 10% of PowerSum) FSS: It has been locked for 0 h and 44 min (should be days/weeks) Threshold on transmitted photo-detector PD = 0.70 V (should be 0.9V) ISS: The diffracted power is around 13.8% (should be 5-15%) Last saturation event was 1 d 2 h and 4 minutes ago (should be days/weeks)
Added 250mL of water.
As I logged Friday (16836,) I set the minor alarm point to +- 2psi but that is just too tight for the EndY where the pressure sensor channels are the most noisy (by 4 to 10x.) I opened the non-alarm pressure to +-3psi. Please note if these alarms continue. The trends suggest that still might be just a tad too close.
Attached is 5 seconds of full data where the ISC sent a whole bunch (5e5nm) of LONG output to the HEPI all at once. The fast channels (shown on HPI ISO X) triggered the HEPI watchdog and the slow channels never see a thing. Why would ISC do such a thing?
Patrick T., Cyrus R. The work described under permit 5066 is complete. No issues came up.