I am out at the end station working on the table (on the side closer to the chamber sometimes), and the ISI tripped. Stage 1 said CPSs were the first trigger, GS13s were the first trigger on stage2.
I don't think the plotting scripts work on the CDS Imacs, but these trips were probably due to me walking around.
On the Guardian Screen, I noticed a RED HAM4 ISI. Looking at the LOG, it said it had an ERROR from last Wed (4/9) after a WD Trip. Here are the last few lines from Guardian's Log regarding the issue:
20140409_18:58:49.069 ISI_HAM4 W: EzcaError: Could not connect to channel (timeout=2s): H1:ISI-HAM4_WD_MON_STATE_INMON
20140409_18:58:49.090 ISI_HAM4 ERROR in state WATCHDOG_TRIPPED_FULL_SHUTDOWN. See log for more info. MODE=>LOAD to reset
I went to MODE, hit LOAD (HAM4 Guardian went white for 10s of seconds & then came back YELLOW), and then hit EXEC (now it's GREEN).
Started morning with several systems tripped. Arnaud posted about an EQ around 1pm on Sat. So trips I'm posting about here are for trips on Sat around 5pm and on Sun around 5pm
So by about 9am, I got everybody back to untripped (GREEN). This was pretty easy:
NOTE: For the ISIs, I wasn't aware Guardian DOES NOT take care of the Blend Filters for us. So Operators (as long as it's OK with commissioners) should make sure to ensure the Blend Filters are enabled. Jim tells me there has been alogs stating what type of Blend Filters we are to engage per ISI.
Here is the list of commissioning task for the next 7-14 days:
Green team:
Red team:
Blue team (ALS WFS):
Blue team (ISCTEY):
SEI/SUS team:
model restarts logged for Fri 11/Apr/2014
model restarts logged for Sat 12/Apr/2014
model restarts logged for Sun 13/Apr/2014
No restarts reported for all three days.
Certainly because of an earthquake, 3 ISIs tripped two hours ago. ETMX and ITMX tripped on ST1 actuators, whereas ITMY tripped on L4C (a measurement was still running). ETMY survived though !
Chris, Alexa, Pablo, Keita, Sheila
This is a late alog of work done thursday afternoon and friday morning. The short summary is that we think the beam quality problem we are having on WFS B is caused by the Faraday used to reject the reflected beam, we don't think we have clipping between our on table PZTs and the TMS QPDs, but might have some in the path from the PZTs to the WFS.
First we used the technique Keita described in alog 11280 to look for evidence of clipping. When we dither the PZTs we see 30-40dB more resoponse in the QPD PIT or YAW than in NSUM, but for the WFS we the difference is more like 20-30dB, which suggested to us that we could have clipping in the WFS path itself (we had already checked on the table all the optics in this path, and didn't see anything that looked bad, so we suspected the Faraday.)
On friday morning Pablo and I went out to the end station and set up the beam profiler again. (images attached) We started by measuing the beam in the path to WFS B (this has been the troublesome one for beam quality) at a point upstream of where we have been, and see that the beam quality is also bad there. We moved the nanoscan to approximately the position of WFS B, and tried moving the first optic after the Faraday to see if this has the same impact on the beam quality that we have seen from moving TMS QPD servo offsets (see alog 11072), shown in the second page of the attachment. It didn't so we concluded that the problem must be upstream of that optic. We then tried moving the Faraday using the new 4 axis mount, and that did have a similar impact as moving the TMS QPD offsets. This may mean that the problem is in the Faraday. We tried several iterations of moving the Faraday in pitch, but didn't really suceed in improving the beam quality much and actually made it worse in the horizontal direction, and made the beam quality going into the chamber worse. We set the Faraday angle to restore a nice guassian profile for he beam going into the chamber.
We then tried putting an iris into the WFS path just after the slipt from the path for the LSC diode (photos will be attached soon.) This dramatically improved the beam quality on WFS B, but didn't change th situation on WFS A. We set the iris to be slightly more open than the position that gave us the best guassian fit to our beam profile, to allow for some alingment drifts. We saw that the iris really dosen't impact the beam quality on WFS A.
So as we left things, we had good beam quality on WFS B and the beam going into the chamber, and the same so so quality that we have had on WFS A since we reworked the path (alog 11074).
Thanks for letting this machine do this. Likely busy until late Sunday afternoon.
Phase 3b spectra (in chamber, in vacuum) was taken last week and has been processed today. During the measurement, HEPI was floating, and ISI was isolated with level 3 and blends at "start".
Also, I plotted a comparison of ETMX and ETMY main chain and reaction chain top mass transfer functions for the 6 DOF, from last week's measurement.
SPECTRA
1) ETMy / ETMx spectra comparison SUS damping off
2) ETMy / ETMx spectra comparison SUS damping on
3) ETMy main chain damping on / damping off comparison
4) ETMy reaction chain damping on / damping off comparison
Spectra look good. Only F2 osem of ETMY main chain has an unexpected sharp peak at ~30Hz (can be seen on 4 pdfs). ETMy amplitude is higher than ETMx due to the ISI state being different. Reaction chain Vertical and Yaw damping could be improved looking at the comparison damping on/off
TRANSFER FUNCTIONS
4) ETMx / ETMy M0-M0 and R0-R0 TF comparison damping OFF
5) ETMx / ETMy M0-M0 and R0-R0 TF comparison damping ON
Again, TFs look good and results of ETMy are matching the ones from ETMx, except for Pitch which has its second mode (1.33Hz) shifted down for ETMx
ETMY isn't working as well as ETMX.
ETMY is running TCrappy blends
ETMX is running TBetter with sensor correction (lower blend frequencies in rX and rY)
There is what looks like a loop instablity around 10Hz (stage 1 maybe?)
To much motion in ry & rX around 1/2 (probably the difference in the blend frequencies)
The references in the ETMX-GS13 and ETMX OPLev plots are with Sensor correction off, which is giving more peaking at the microsiesem then we want so I'm turning it off
The low frequency excess motion is likely due to the fact that we haven't finished doing the stage 1 tilt decoupling, Jim needs a few hours to finish that up
We opened the YEND to the beam tube today for ~9 hours.
The attached plot shows the pressures during the day at the BSC10 dome(pt410) and at the 80k pump(pt 424).
The pressure at the trap reached 3e-8 torr.
At ~7:00pm local time I closed GV18 , opened the valve to the main turbo, and opened the small turbo pump to the BSC6 annulus.
Alexa, Sheila, Keita, Daniel, Stefan, Chris ==== We did the y-arm sneak-peak initial alignment today. - A 200urad, 0.1Hz yaw drive on TMSY was big enough to get us the beam on the ITMY SPOOL camera. - We centred the single shot beam on ITMY using just the camera view. - Next we hit the ETMY baffle PD's using ITMX. We got about 3V with 20dB whitening gain on every PD. - Next we aligned ETMY until we got fringes on the camera. - Next we moved the BS until we saw the y-beam on ISCT1. (All other mirrors are common between x-arm and y-arm green path until we hit the ISCT1 prism.) Looking at POP baffle hole helped. - Next we measured the single shot power: y-arm: 10.1uWatt, x-arm 16.0uWatt. - Next we roughly aligned the ISCT1 table path for the y-arm beam. Note that so far we did NOT align the BS with the IR light. Instead we just separated, so the alignment will have to be redone. - We ensured the reflected beam was on the refl RF PD on iSCTEY. For calibration, we measured 4.3mW with ~-4V at the diode with 30dB gain. - We were able to see an RF signal out of the demodulator. This signal was very weak. IMPORTANT note; the RF MON on the demod board does not work. - We set the delay line to 11ns and were able to see a decent (but not great) PDH error signal out of the IMON of the demod. ==== We locked the Y-arm to the 00 mode (only a second at a time). - We aligned the ETMY and ITMY using transmission video image on ISCT1, green transmission level on ISCT1 (H1:ALS-C_DIFF_A_LF_OUT), PDH DC level, PDH demod, ITMY video and ETMY video. - At this point we were able to lock to some crappy mode for 0.5sec at a time when the masses were slow. - The fringe velocity was very high, probably more than 5um/sec. We turned off the ISI isolation. - At this point we were able to lock mostly to 00 mode. It would last for 1 sec or shorter (but we saw 2sec stretch too!), then would go to 01 or some higher order mode, and come back to 00. The motion is mainly PIT. - Seemed like the lock is lost when VCO output goes to some big number. ====== I have attached a picture showing the 00 modes for BOTH the x and y-arm locked at the same time!!!! (y-arm was locked for only a few seconds though).
This process is on opsws2.
A guardian process was created for ETMY (ST1, ST2 and Manager), and helped bringing back the ISI to "isolated" with blends on Tbetter on both stages.
ETMY tripped while switching blends on ST2. Attached ST2 plots
This is Arnaud
(Joseph D., John W., Justin B., Sheila D., and Gerardo M.)
Shutter installation finished today.
The viewport will need to be pulled sometime later for a close inspection and or possible replacement.
I opened GV7 at 2 pm.
(Rich M, Sheila, Alexa)
HEPI and ISI ETMX tripped because we were playing with the suspension watchdogs.
We had a hard time recovering from this trip which caused several more trips, the wd plotting script is not working this afternoon, possibly beause of our network problems so we don't have any plots. Once HEPI was isolated, we tried letting guardian isolate the ISI. It tripped immediately on CPSs (stage1) after we cleared this and tried again it tripped on GS13 (stage2).
We then paused the guardian, moved all the blends to start, and gave the gaurdian another try. (note, the ground motion is fairly quiet today). This tripped again, GS13s.
We then puased the guardian, and tried the command script, stage 1 level 3, tripped on the actuator limit. I then stopped taking notes.
The rough procedure that Rich used to bring it back was damping both stages, turning on 1 DOF at a time RX, RY, Z, RZ, X, Y (stage 1 first, then stage 2). Then Alexa moved the blends over to Tbetter without anything tripping.
We don't know why the things that used to work don't anymore, but now ETMX ISI is harder to untrip and neither the guardian nor the command script seem to be able to do it anymore.
Rich may not have mentioned in the alog, but after our trip the other day while trying to switch blends, he changed the filters from zero crossing to ramp, which should make it easier to switch blends without tripping the ISIs.
One thing about guardian, its not clear to me how we should unpause if we need to pause it for some reason. TO pause it we paused all three, stage 1, stage2 and the manager. Wen unpausing the subpodinates they don't come back in managed mode. I was able to get them back to managed mode by going to init on all three, but in the meantime they were doing their own thing, but happily didn't trip the ISI. What is the best way to do this? Also, maybe the right way to pause and unpause a manager with subordinates is something that needs to be made more obvious to a user.
Alexa also tried pausing, she only paused the manager, bringing that back was not a problem but it doesn't pause the subordinates.
quoth Sheila:
We don't know why the things that used to work don't anymore, but now ETMX ISI is harder to untrip and neither the guardian nor the command script seem to be able to do it anymore.
But you did mention one likely important thing that changed: new blend filters (Tbetter?). What happens if we go back to Tcrappy? Do we have the same problems?
also quoth Sheila:
One thing about guardian, its not clear to me how we should unpause if we need to pause it for some reason. TO pause it we paused all three, stage 1, stage2 and the manager. Wen unpausing the subpodinates they don't come back in managed mode. I was able to get them back to managed mode by going to init on all three, but in the meantime they were doing their own thing, but happily didn't trip the ISI. What is the best way to do this? Also, maybe the right way to pause and unpause a manager with subordinates is something that needs to be made more obvious to a user.
Alexa also tried pausing, she only paused the manager, bringing that back was not a problem but it doesn't pause the subordinates.
This is definitely sounds like something that could be improved. It is true that all nodes, manager and subordinates, need to independenty be paused and unpaused. I'll try to think about how to make that smoother. In the mean time, here's the procedure that should be used:
The manager INIT state resets all the works, restarts them, and puts them into managed mode. That should be the most straightforward way to do things for the moment.
Thanks Jamie.
We did try both the command script and guardian with all the blends on start, which used to work. I don't think we tried turning sensor correction off, which is another change.
I heard that rumors were spread that this alog was written during an earthquake that we hadn't noticed, so there is no reason to worry about our inability to use guardian or the command scripts to reisolate ETMX. That is false, there was no earthquake. PS, this is sheila, still loged in as Stefan.
No, sorry, that was me... too many of the same screens open.