WP 6087 The GDS software has been updated to branch 2.17.4 to address Bugzilla 897, 936, and 952. Release notes (T1600007) document has been updated. This affects Ubuntu 12 and Ubuntu 14 workstations.
Tues. maintenance task
(Corey, Fil, Richard M, Sheila)
Per Fil's Work Permit #6071 (Fil took care of running/connecting the camera's shielded network cable to the table's Patch Panel earlier):
Work Permit #6071 can now be CLOSED.
Title: 08/15/2016, Evening Shift 23:00 – 07:00 (16:00 – 00:00) All times in UTC (PT) State of H1: IFO is locking Commissioning: Working on IFO locking and stability. Outgoing Operator: TJ Activity Log: All Times in UTC (PT) 23:00 (16:00) Start of shift 23:14 (16:14) Kyle – Going to Vertex to adjust RGA 23:20 (16:20) Sheila & Evan – Going to PSL rack 23:27 (16:27) Kyle – Out of LVEA 23:39 (16:39) Gerardo – Going to Mid-Y to check/fill CP3 23:50 (16:50) Gerardo – Back from Mid-Y 01:08 (18:08) Both End HEPI & ISI WD tripped Title: 08/15/2016, Evening Shift 23:00 – 07:00 (16:00 – 07:00) All times in UTC (PT) Support: Sheila, Evan, Jenne, Hugh, Terra Incoming Operator: N/A Shift Detail Summary: Both end station HEPI and ISI WD tripped. Found HEPI Pump Controller had stopped. Both pump controllers VFDs had an OV3 error. Reset VFD and brought the pump stations back on line. Redid initial alignment, and commissioners are back at work. Commissioning work has wrapped up for the night. Although the IFO was locked, in an abundance of caution, put it into a DOWN state.
I've put together a diagram of all main aLIGO (red beam) optical sensors: DCC G1601619
Graphic is available in both .pdf (attached) and .svg format. It's built as a layer on the basics of Jeff's seismic isolation and suspension graphic, G1200071. Suggestions for clarity more than welcome.
Jenne, Evan, Terra, Sheila
Now that we are locking at least on RF, here is a summary of some of the problems we had and things we changed. It seems clear that the MICH ASC loop was not working well with the old TCS settings, and that changing TCS helped (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=29097). The other significant changes we made in the last few hours that may havbe helped with locking were reverting the MICH ugf for 3F to the old value (unclear), and commenting out the CARM gain redsitrbution.
CARM gain redistribution
We commented out the CARM gain redistribution from the ANALOG CARM state today. This gain redistribution was intended to keep the sliders all at positive gains for noise performance, but we saw that after the gain was resditributed we had large glitches in the MC2 drive. These large glitches were not present before the vent, and are gone now that we have commented this out.'
The first attached screenshot shows that on August 1st, the drive to MC2 did not change much after we went through analog carm, but that over the weekend it was much noisier after we went through Analog CARM. The next screenshot shows the glitches showing up as we do the gain redistribution, and in the last one the analog carm state has completed without large glitches once the gain redistribution is commented out.
It might be good to swap out the common mode board, in case the problem is that some gain slider which used to work no longer works.
DHARD gain peaking
Once we locked, we saw a a verry narrow line at 9.45 Hx once the ASC came on, which was due to gain peaking in DHARD. The only change we know of to this loop was correcting the mistake in the notches for ETMX However, this bounce/roll notch is the same as the one that has been in use for all the other test masses. For now we have commented out the gain increase in DHARD in the REFL POP WFS state, but we know that this will not be stable for powering up, so we will need to fix this.
We were able to engage the rest of the ASC except the soft loops, which have large offsets and cause the interferometer to move too fast for the SRM dither loop to keep up. This is probably because our green reference changed.
To summarize changes that we made over the last several days of trying to lock:
Changes that might matter:
Changes that shouldn't have mattered:
Things that we still need to fix:
Jenne and I reoptimized the offsets for the QPD pointing loops for the OMC suspension, so that the OMC guardian could find the 00 mode during the scan.
After doing this, the OMC guardian runs through the usual preparation procedure just fine. We made it to dc readout at 2 W with 15 mA of photocurrent.
Since the CO2 Y laser is busted, we have been trying to lock for the past few days with 0.5 W of central heating on the X CP, and no heating on the Y CP. This has resulted in a noticeable 02/20 contrast defect in the AS port during a simple Michelson lock.
We reduced the X CP heating to 0.25 W, which (in combination with 0 W on the Y CP) basically puts us at the O1 configuration. After making this change, we are able to keep the interferometer stable and fully resonant (at 2 W of PSL power, anyway).
Hugh, Evan, Terra, Jeff B.
At 01:08 (18:10) both end station HEPI and IS WD tripped. Both Pump Controllers were down. Called Hugh. I took End-X and Evan and Terra End-Y.
Found OV3 error on VFD. After ramping down the servo, reset the VFD. Ramped the servo slowly up until differential pressure was over 62 and set the servo control back to Auto. System came back up OK. Fluid level were unchanged from last logged level.
Posted is a plot of the Diff pressure and control voltage.
Nice Work Jeff, Thanks commissioners.
Attached are 2 hours of second trends and 3 seconds of full data with the site mainmons along with the Pump Station servo'd differential pressures.
On the minute trends plot I've circled more solid glitches on the Ends Mains Mon that are very timely. Also note how the hash on the Max and Mins are smaller on the Corner station monitors.
On the 3 seconds of full data, you can see the glitch also on the CS Mon, and that the Diff Pressure drop is delayed about a second. This differential pressure is at the far end of the run. When I looked at the pressure right after the pump, the pressure drop was much closer, within a 1/4 second or something.
History: EndX--26 July 2016 28653; EndY July 2015 19503; I only went back a couple pages with my kissel/radkins HEPI PUMP alog search.
I have added a check to the LocklossShutterCheck guardian that will add to the guardian log a note whether the HAM6 GS13s saw a large kick or not. Note that this does not in any way change the functionality of this guardian, in that it still requires an Operator to run and look at the plot, then manually reset this guardian.
However, we should be able to look through the guardian log to ensure that each time we lose lock, the log notes that the GS13s saw a big kick. When we are satisfied that this is a good measure of the shutter having closed at the last lockloss, we can consider using it as an automation metric in lieu of Operators manually resetting the guardian.
I have set the threshold for "big kick" for the GS13 to be 30,000 nm/s (1nm/s / count, so 3e4 counts). When the shutter closes fast, the GS13 sees a signal as large as 90,000 nm/s, but when the shutter closes slowly it only kicks the GS13s to about 10,000 nm/s. The threshold of 30,000 nm/s should differentiate between a fast vs. slow closing of the shutter. Normal GS13 signal is under 100 nm/s peak.
Attached is the code, which is also checked into the svn. The idea is (once we're ready to make this automated) that if we see the GS13 kick, the CheckShutter state will automatically go to the LowArmPower state, and allow the IFO to relock. If there is any problem, including inability to fetch the data, the guardian will stay in the CheckShutter state, and will require human intervention. One could also consider adding light level checks to this, but I think that's probably overkill given the other checks that we already have in place.
At least twice today we've run into the situation where we're still locked at 2W but lose lock, and the power going to the AS port is not enough to warrant a fast close of the fast shutter. This makes things a bit more confusing, since you must know that it's okay that the shutter didn't close in this case.
So, I've added another level of checks to the LocklossShutterCheck guardian to check if the trigger PD saw more than the threshold. If not, it logs that the shutter should not have fired. If the trigger PD goes above threshold, then it looks at the GS13 data and logs whether it saw a kick or not.
Again, these changes only affect what goes into the guardian log. Manual intervention is still required at this time to reset the guardian and allow the IFO to relock.
A different option that we can also implement is to alter the thresholds for the power level that sends the guardian into the CheckShutter state. Sheila points out that maybe we only should be running these checks when there's a chance that the shutter will have closed, so at more than 2W.
After talking with Daniel this morning I made a few more changes to this guardian. The problem that Jenne was trying to address with the check on the AS power was that for 2 Watts most locklosses from 0 CARM offset will trip the shutter, but not all of them should since sometimes not as much of the power goes to the AS port. We think that the second option Jenne mentioned, only requiring a check of the shutter after high power locklosses in which we nearly always expect it to fire, will let us avoid relying on the trigger PD in this logic.
This morning I redid the arm power checking a little bit, to include the calibration into watts, and increased the threshold for requiring a shutter check to 14 kWatts of circulating power in an arm. (This is just over 4 Watts input power). I also took out the check on the triger PD, and made the logic that Jenne had logging results last night be used to reset the state of the guardian. Now the gaurdian will rest itself to LOW_ARM_POWER if it sees that the shutter went off and the GS13s see a kick. If the GS13s do not pass the test it will go to a new SHUTTER_FAIL state.
We will test this when we get to relocking this afternoon.
Sheila Evan Jenne Terra
The IMC length drive has large glitches when only the IMC is locked. The first attachment shows the MC2 M2 length drive durring a quiet time on July 31st, and a glitchy time today. There are glitches present in both traces, but they are much louder in the trace from today. If you look back at data from late July, the glitches have intermittently been present even before we started having trouble locking, so this is probably not the main reason we can't lock.
The second attachment shows 15 minutes around the time of the July 31st trace, when the glitches transition from being quiet to loud.
Daniel, Sheila
These glitches appear when green light starts flashing in the arms. The COMM PLL sometimes see a green flash, and tries to lock as you can see in the control signal in the top panel of the attached plot. This would be fed back to the mode cleaner if ALS COMM was locked, but that feedback is disabled. The PLL trying to lock moves the COMM VCO, which through some kind of RF coupling similar to the cause of the whistle glitches causes the mode cleaner glitch.
Once we are fully locked we park the VCO so this should not be a problem in full lock and should not be related to the glitches we see after changing gains on the CM board
Short version: The new shutter seems to be closing harder than before the vent. This is causing the ISI to trip. So far we have gotten around this by running the ISI in a low gain state, but this makes the table motion much worse above 1 hz.
After the shutter work, we started having problems with trips of the HAM6 ISI when the new shutter trips. This morning we put the ISI in the "Robust" isolated state, which uses low gain isolation loops with UGFs around 10hz. The normal loops have UGFs around 30-40 hz. For now this seems to have eliminated the tripping of the ISI, but the performance of the ISI is much reduced above 1 hz. First plot shows a comparison Y GS13s and ground STS for HAM6 for the two configurations. Red traces are with the high gain loops, blue traces are with the low gain loops. Below 1 hz they look roughly the same (low frequency diffs seem to be mostly in the ground for blue), but at 1 hz the high gain loop is doing ~100x better. Not surprising.
I've looked the GS13s in Z at shutter triggers before and after the vent and the shutter seems to be moving the table a lot more now. Attached plot shows a trigger from today with the 10hz loops (blue), yesterday with 30hz loops ( orange/ spiced cider) and Aug 1st with 30hz loops (red). I've looked over at least 3 events in each configuration and they look remarkably consistent, with respect to ISI configuration. Shutter triggers before the vent seem to have kicked the table less than they do now. Maybe this will reduce as the shutter breaks in?
We have some medium gain loops installed with slightly less gain that the high gain loops, tomorrow during maintenance I'll try testing those, but I doubt they'll be enough. Otherwise, we can come up with some loops with some better compromises between UGF and high frequency roll-off than the low gain loops. But the best option would be to figure out what can be done to reduce the kick from the shutter.
tagging sys
Jim, there is a little known feature on the fast shutter driver wherein a signal is available to say the shutter has triggered. You could use this signal as a form of a blanking pulse to allow you to ignore the seismic signals that follow a shutter trigger. Let me know if you want details, but the spigot is a BNC on the front of the shutter labeled "Blanking Output"
TITLE: 08/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Commissioning work all day. HAM6 chamber manager is currently paused and the ISI is set to Robust Isolated.
LOG:
Am baking Vertex RGA whilst HAM6 pumps run this week. 1200 hrs. local -> Heating of Vertex RGA started
Comparison of Calibration Epics Values (Calibration parameters at t = t0) during O1 and ER9. Param. Description 2015/09/29 2016/08/03 2016/08/10 ---------------------------------------------------------------------------------------------------------
EP1 ~1/Atst0 (ftst) 2.2005e+16 2.1066e+16 4.9121e+17 EP2 ~Delta C/(1+G) 0.9895 0.9934 0.9934
EP3 1/Apu0 (fctrl) 2.1115e+16 2.1077e+16 2.1077e+16 EP4 Atst0 (fctrl) 5.4972e-17 5.6354e-17 5.6354e-17 EP5 Apu0 (fctrl) 4.7360e-17 4.7446e-17 4.7446e-17 EP6 Cres(fpcal2) 1.1307e+06 8.9859e+05 8.9859e+05
EP7 D0(fpcal2) 1.7599e+07 1.5871e+11 1.5871e+11 EP8 Atst0 (fpcal2) 1.2178e-18 1.13718e-18 1.13718e-18 EP9 Apu0 (fpcal2) 2.0898e-19 2.0235e-19 2.0235e-19
Evan H., Darkhan T.,
Details
SUS-ETMY_LKIN_P (see LHO alog 28164). Later we found that the amplitude of this line in the DARM spectrum recorded during ER9) was unsufficient for tracking the ESD actuation strength. So, the EPICS values for time-dependendent factors were recalculated on Aug 10 for using a calibration line injected from SUS-ETMY_L3_CAL_LINE.
The transfer function for the SUS-ETMY_L3_CAL_LINE line includes DRIVE_ALIGN filter, which is not the case for a line injected from SUS-ETMY_LKIN_P. The gain of this filter at cal. line frequency is ~22, this explains EP1 value change.
EP6).EP7 (D0 at 331.9 Hz), could be that during ER9 we, by mistake, did not turn on the notch at 331.9 Hz. In O1 run this notch was in LSC-DARM, FM7 (see LHO alog 20075). After adding a second set of filter modules (now LSC-DARM1 and LSC-DARM2) the notch was moved to LSC-DARM2, FM4, but was not turned on during ER9.
Note: LSC-DARM2, FM4 will be turned on during nominal low-noise operation in ER10/O2 (must not forget to include this into Matlab DARM model).
Kyle reported on Aug 12 that he thought there was a correlation between his RGA signal and locking attempts.
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=29079
On July 19 there was a programming change to the FPGAs;
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=28535
During this process there was a pressure burst in PT410 at the YEND. Dave believes that as the system was brought back, dacs railed while connected to the ISI. This caused all actuators in the chamber to go to their maximum output. We believe this caused some heating which resulted in the pressure bump.
Perhaps Kyle saw pressure changes related to ISI activity?
ISI spike on the 12th - don't think the time matches though.
This is just to document that the rampdown-on-abort feature works.