corey.gray@LIGO.ORG - posted 09:58, Wednesday 15 May 2019 (49264)
PSL Chiller Water Level Top-Off (FAMIS #10509)
Topped off Crystal Chiller with 125mL. Diode Chiller OK. Filters looked normal.
Topped off Crystal Chiller with 125mL. Diode Chiller OK. Filters looked normal.
Alarm Handlers have a LOW temp alarm for MX. Looks like they took a step down in the last 24hrs (attached is a 3-week trend for MX VEA).
1) Initial Alignment
ALSy did not look normal. Started out looking nice with unlocked flashing just under 1, but after going to the INITIAL_ALIGNMENT state, the video looked much less active and much less flashing (and obviously no locking). Went to UNLOCKED and the flashes now looked much more noisy on ndscope.
While in UNLOCKED, decided to still touch up alignment by adjusting the ETMy. Managed to get flashes above 1. Went to INITIAL ALIGNMENT state and Y-arm immediately locked up! (This is different with Y-arm being very sensitive to alignment needed to lock.) Moved on to rest of alignment after this.
No other issues with all other INITIAL ALIGNMENT steps.
2) GWIstat: "Calib issue"
When I brought this tool up, noticed this was showing a newish status: "Calib issue". Apologies if this is a known issue, this was just new to me.
After an Initial Alignment, H1 made it to 30W fairly straightforwardly. Keita & commissioners discussed plan for H1 & they are going to log some COMMISSIONING time for higher power work this morning (Notified LLO/Virgo over TeamSpeak).
TITLE: 05/15 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Corey
SHIFT SUMMARY: started with H1 locked, then LockLoss, Relocking
LOG:
The LockLoss at 12:51:54 UTC was preceded by 4 seconds, by a Verbal Alarm for MC3, at 12:51:54 UTC.
Attached is a time series of the MC3 M1 OSEMS, and the H1:SUS-MC3_M1_OSEMINF_T2_OUT_DQ starts to glitch at 125.84 seconds into the plot, while the other MC3 M1 OSEMS, and Cal Delta and OMC DCPD A, are unaffected.
Other MC3 M1 OSEMS glitch next, then LockLoss.
Attachments:
While Relocking, DRMI lost lock and that also gave a Verbal Alarm for MC3.
Very interesting, good find Cheryl.
It's also in the drives, but only on MC3. So, it looks like it's something with the MC3 suspension, since if it was a glitch coming from the WFS it should show up on all of the IMC suspensions.
This happened to Sheila and me again this morning, as we were trying to finish up going to low noise at 40W, but this time it was MC1's T2 and T3. The WFS don't see anything until after the glitch, so this confirms my earlier suspicion that it is not coming from the WFS.
If MC1 and MC3 are served by the same coil driver box, it sounds a lot like it might be glitching on us.
The following units were power cycled during a lockloss:
SUS-C4 U41 S1001089 Coil Driver (MC1)
SUS-C4 U40 S1000242 Coil Driver (MC1/MC3)
SUS-C4 U39 S1001084 Coil Driver (MC3)
SUS-C4 U30 S1108081 AI
Attached are screenshots from both of these locklosses. The first screenshot shows all the top mass osem readbacks, in most of them the glitch show up slowly because it is going through the damping loops and the mechanical response of the suspension, but the glitch in T2 is much faster which indicates it is a problem with that sensor in both locklosses (MC3 T2 in Cheryl's low noise locklosses, MC1 T2 in th recent lockloss at 40W input power).
The AA (S1105215) chassis was power cycled yesterday afternoon after a lockloss.
TITLE: 05/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 0Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
Wind: 5mph Gusts, 4mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.13 μm/s
QUICK SUMMARY:
H1 dropped out during OFFLOAD_DRMI_ASC, and since this was 3rd attempt for locking, I'm going to start an Initial Alignment.
Cheryl mentioned seeing something with MC3 (she'll post an alog).
locked at 113Mpc, low winds, useiem continues to drift downward
TITLE: 05/15 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 111Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
Wind: 3mph Gusts, 2mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.15 μm/s
QUICK SUMMARY: locked for 1.5 hours
Sheila, Hugh (remotely), Niko, Georgia
After an epic battle to reacquire lock (see Niko, Sheila, and Jenne's posts about alignment references and DC centering loop notches), we found a couple of strange SDF differences.
The GS13's for HAM3 and HAM4 were in a non nominal state, with filter differences for H1:ISI-HAM[3,4]_GS13INF_[dof] for each degree of freedom (H1 H2 H3 V1 V2 V3). The Gain and DWH (de-whitening) filters were on when we reached nln, where nominally they are off. Hugh told us how to fix the problem (sitemap -> ISI -> HAM3 -> Commands -> GS13 !HI). Doing this did not break the lock.
We're not sure how we ended up in this state, maybe after the large earthquake today things were not quite returned to normal?
The SDF reported differences with PRM M3 (see attachment), but we found the filters were correct given the state of the coil drivers. Time machining to the last lock, we don't think there was really a difference.
Pretty sure I am to blame for HAM3 & HAM4 SEIs. :-/
Here's what I recall for SEI Land:
After seeing Georgia's alog about HAM3/4, immediately figured this probably due to me. Talked with TJ since he was in the area, and he (& also) Hugh mentioned this is probably due to my not hitting the Recover EQ button (which addresses GS13s). TJ said I could have perhaps taken HAM3/4 to a correct state "by hand", but I did not know how to do it.
Anyway, it's another learning experience.
The recovery script includes a step to put all of the GS13s in low gain which is unnecessary. This means that because HAM3&4 were already isolated when Corey pushed the recovery button, the other chamber guardians handled the sensor gains properly, but HAM3&4 were switched to low gain, and never set correctly by the guardians. I've taken those lines out of the script so this shouldn't happen again.
TITLE: 05/14 Eve Shift 23:00 – 07:00 (16:00-00:00), all times posted in UTC
STATE of H1: Locking
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Lost lock several times around ENGAGE_DC_VIOLINS, eventually did initial alignment. After initial we had issues getting past CARM_TO_ANALOG, so eventually we reverted the changes made earlier today and began initial alignment again. After that we reached NLN fairly quickly, sorted through sdf changes, and went into Observing.
LOG:
23:00 (16:00) Start of shift
01:35 (18:35) Dropped out during commissioning work, starting initial alignment
02:36 (19:36) Done with initial alignment, beginning to lock
03:56 (20:56) Reverting earlier commissioning changes, beginning initial alignment again
04:36 (21:36) Finished with initial alignment, starting to relock
05:24 (22:24) At NLN, sorting through sdf changes with Georgia and Sheila before Observing
05:52 (22:52) Going into Observing
We also had a problem where the CO2Y laser losing lock kicked us out of observing a couple of times shortly after reaching NLN.
I took some time during today's maintenance to attempt to measure the bounce/roll modes of the HSTS (with Jenne's help) in order to better identify some lines which are present in DARM spectra.
The obtained data is located in the SUS svn:
To do these measurements I used the already existent DTT templates located in the each SUS svn directory (after turning off the damping on each of the optics).
I have been unable to measure any bounce or roll mode: below 10 Hz, the data looks similar to the reference data, but above 10 Hz there seems to be some kind of noise which contaminates the measurements (although the coherence looks fine). This happens with the four optics, and driving with L_EXC, P_EXC or Y_EXC. The attachments show a couple of examples which show this behaviour. Sheila suggested that this noise might be coming from some cross-coupling of the cables driving/sensing the signals.
We haven't reset our initial alignment setpoints in a long time, and it likely needs redoing.
In prep for resetting the initial alignment setpoints, we need to have the IFO aligned and converged, but held at 2W. We also need the green shutters open and the green lasers locked on the TEM00 mode. With our current IR alignment and green QPD setpoints, the Xarm green doesn't want to lock on the 00 mode. Last lock I had opened the green shutters before the ADS loops came on for the arms, and the Xarm was locked on 00 for green, so I started running the green QPD setpoint script so that the green beam would follow along as the soft ADS loops converged. This seemed to be working, and the QPD offsets that had the Xarm green locked were:
ALS-X_QPD_A_PIT = -0.268
ALS-X_QPD_A_YAW = 0.708
ALS-X_QPD_B_PIT = 0.955 (this is extremely close to the edge, but it seemed okay hanging out here. The current value is 0.841)
ALS-X_QPD_B_YAW = -0.177
So, I think we should be able to slowly walk the offsets to this place during a future lock, or to use them for initial alignment if we also change the camera setpoint to be 251.2 for pitch, and leave yaw at the current value of 332.9 (the yaw CAM_ITM_YAW_ERR was basically zero, but the pitch was about 0.19). These setpoints wouldn't be our final answer for initial alignment setpoints, but would make it such that the Xgreen would lock on the 00 mode when the IFO is aligned with IR.
A somewhat confusing side note is that changing the QPD setpoints while in lock seems to have broken the lock twice when taking steps of size 0.1. Not totally sure why, but we're going to try this time to acquire without touching the offsets.
[Sheila, Jenne]
We have now reset the X and Y green initial alignment setpoints. Both arms' B pit offsets are extremely large - 0.96 - so we will need to pico these at some point, unless with the higher input power we decide that we need to move our spots (hopefully in a way that bring these closer to zero).
Attached is a screenshot of the QPD and ITM camera offsets after we reset them. We have now reverted this because we weren't able to make it through ANALOG CARM and DRMI on POP with the new alignment references.
We also found that the auto exposure was on for the ITMY green camera, it got turned on May 7th. This was causing glitches in the Y camera loop, we have turned it off.
We lost lock while powering up after this, so Niko ran an initial alignment.
Since we had so many locklosses this evening, we noticed a few more things that we could improve in the locking sequence.
We had a 0.5Hz pitch instability when reducing the RF45 modulation depth twice today, the second time I turned back on the 0.5 Hz notches in the pitch refl centering loops, which we had turned off last thursday (49160) (turning them off hadn't caused any problems until tonight).
It looks like the PR3 sliders weren't restored after the Dolphin crash last night, so our IFO alignment has been a bit wonky all morning (one lock we saw some beam on AS AIR even with the beam diverter closed, which indicates something is definitely funky).
I restored the PR3 sliders, and now JeffB is doing an initial alignment starting with INPUT_ALIGN (the green arm alignment should still be fine from Corey's alignment last night). Hopefully this will help things out a bit.
Oh wow! And the PRC_ALIGN section of the INITIAL_ALIGNMENT I ran, ran completely on its own, so I didn't worry about it so I could focus on the other problems (hence, I didn't mention PRC ALIGN in my Alog....it and the Green Arms did not give me issues when I tried them).
Sorry about that! With SUS, I only looked at sliders for mirrors which were giving me grief, and since PRC was taken care of by Guardian I didn't even look at them.
Good Luck!!!
P.S. Not sure of best check for SUS after dolphin crash. Sounds like we should go through and restore ALL of them to a good time (instead of do them willy nilly like I did). What's the best way to do this?
I caught this by looking at a time machine of IFO_ALIGN_COMPACT for 24 hours ago. So, I usually look at all of the slider values. Ndscope is also fine, but I find this to be quicker.
Also, I have re-monitored all of the slider values for the optics in the safe.snap files that are loaded when the models reboot. We don't accept values in the safes all that often, but we do sometimes. For example, I cleared all of the safe.snaps a week or so ago, but didn't think to look at the not-monitored channels. By re-monitoring them, we'll hopefully get a more recent value for all of the sliders when we have an unexpected reboot.
A consequence of this is that, for the suspensions whose observe.snaps are just links to their safe.snaps, we will have to accept any OPTICALIGN SDF diffs when we go to Observing. However, the suspensions that have this situation are ones like IMs 1-3, the RMs and the OMs that really shouldn't be changing ever. So, operators in general shouldn't see these as diffs every lock. If you do see these as diffs when you are trying to get to Observe, let's continue to screenshot them so that we can understand why they are changing.
Several other suspensions also have the safe.snap linked to the OBSERVE.snap, so each time we run inital alingment now we are having to accept sliders for many optics. I think it makes sense to un-monitor these, and find a different way to keep track of the slider values.