I forgot to alog this, its been usable for a few weeks now.
I made a script that will create a map of all the nodes and who is managed by whom. It uses the same packages as the normal Guardian maps for a similar look, and the same colors as the medms. The script will only map the current management, so if managers change, it will have to be re-ran for a new map. I will try to make this better later, but for now this will have to work.
Location: /opt/rtcds/userapps/release/guardian/manager_map.py
I attached a sample output.
T240 Centering script results: There are 18 T240 proof masses out of range ( > 0.3 [V] )! ITMX T240 1 DOF X/U = -3.297 [V] ITMX T240 1 DOF Z/W = 0.439 [V] ITMX T240 2 DOF Y/V = 0.505 [V] ITMX T240 3 DOF X/U = -3.301 [V] ITMX T240 3 DOF Z/W = -0.341 [V] ITMY T240 1 DOF X/U = -0.433 [V] ITMY T240 1 DOF Y/V = 0.359 [V] ITMY T240 1 DOF Z/W = 0.34 [V] ITMY T240 2 DOF Y/V = 0.43 [V] ITMY T240 2 DOF Z/W = -0.345 [V] ITMY T240 3 DOF X/U = -0.941 [V] ITMY T240 3 DOF Y/V = -0.319 [V] ITMY T240 3 DOF Z/W = -3.275 [V] BS T240 1 DOF Y/V = 0.904 [V] BS T240 1 DOF Z/W = 0.384 [V] BS T240 2 DOF X/U = 0.96 [V] BS T240 2 DOF Z/W = 0.481 [V] BS T240 3 DOF Z/W = 0.927 [V] All other proof masses are within range ( < 0.3 [V] ): ETMX T240 1 DOF X/U = 0.158 [V] ETMX T240 1 DOF Y/V = 0.143 [V] ETMX T240 1 DOF Z/W = 0.19 [V] ETMX T240 2 DOF X/U = 0.061 [V] ETMX T240 2 DOF Y/V = -0.052 [V] ETMX T240 2 DOF Z/W = 0.168 [V] ETMX T240 3 DOF X/U = 0.153 [V] ETMX T240 3 DOF Y/V = 0.008 [V] ETMX T240 3 DOF Z/W = 0.113 [V] ETMY T240 1 DOF X/U = 0.066 [V] ETMY T240 1 DOF Y/V = 0.006 [V] ETMY T240 1 DOF Z/W = 0.046 [V] ETMY T240 2 DOF X/U = -0.126 [V] ETMY T240 2 DOF Y/V = 0.059 [V] ETMY T240 2 DOF Z/W = 0.148 [V] ETMY T240 3 DOF X/U = 0.045 [V] ETMY T240 3 DOF Y/V = 0.038 [V] ETMY T240 3 DOF Z/W = 0.152 [V] ITMX T240 1 DOF Y/V = -0.16 [V] ITMX T240 2 DOF X/U = -0.227 [V] ITMX T240 2 DOF Z/W = 0.219 [V] ITMX T240 3 DOF Y/V = 0.18 [V] ITMY T240 2 DOF X/U = 0.219 [V] BS T240 1 DOF X/U = 0.207 [V] BS T240 2 DOF Y/V = 0.182 [V] BS T240 3 DOF X/U = 0.072 [V] BS T240 3 DOF Y/V = 0.241 [V]
ITMX & ITMY T240 Masses Centered WP 5719
JeffB's post of the Mass Positions prompted us to center the ITM ISI T240s. This was successful. The BS T240 masses are a bit above action level but not by much; we'll wait a few before we mess with it.
Feb. 3 2016 16:38 UTC Reconnected conlog-test-master to the same 99,560 channels are h1conlog1-master.
Added 350 ml of water to the diode chiller, as I noticed the red LED was on with the error message about low water level.
4f H4 looks clean except for a small wipe mark near 1 o'clock close to the retaining ring. 4f H1 has streaks visible with green light. Wipe spots between 3 and 6 o'clock. Possible spot/damage mark in centre of optic. Hard to see. Try cleanroom cotton tip to wipe pump light side. Drag wipe removed centre smudge. 4f H2 has a streak mark around 6 o'clock near the edge. Also has a hard to spot streak lined up with 12 o'clock and is ~3 mm wide. Not visible with white light, only seen with green light. 4f H3 has a spot around 7 o'clock near the edge. Does not blow off. Spot is ~2-3 mm from the edge. Possibly on the pump light side of the optic. Hard to get to with the quartz rotator in the way. Re-examined 4f H4 with green light. Smudge in middle visible. Vague spot visible under green light from the pump light side. Spots visible from the pump light side, not visible from the H3 side. Drag wiped surface, spot is now gone. Quartz rotator surfaces look good under white and green light. Spot on resonator side of output coupler, a little more than half way from the centre to the edge, around 2 o'clock. Drag wiped clean. Perhaps out of an abundance of caution it was good that we examined the surfaces with green light. Certainly a few wipe marks that were present and not visible under a bright light source, were just visible in green. It might be that the wipe marks are inconsequential but at this moment, it seemed prudent to deal with them. Replaced intra-cavity shutter blade with new model. The shutter blades shipped from Livingston do not fit the external shutter as there is a handedness problem. EXTERNAL SHUTTER STILL NEEDS UPGRADING. Front end laser beam centered through MM2 iris, slightly off to the left of centre as you look at the MM1 iris. It is not clear whether or not the beam was centred in the first place. 12:46 (PT) Turned on front end laser power watchdog. Original mode matching lens positions: L02 = 13.38, L03 = 6.42 L02 = 14.26, L03 = 6.35 PMC we have an offset somewhere as it does not lock to the maximum transmission level unless the power on the locking photodiode is turned down below the nominal 1.5V unlocked. Upon some navel gazing, the offset might be due to some light in the wrong polarisation falling on the photodiode. ISS - adjusted waveplate until PDA DC = -10.0V and PDB DC = -10.16V. Zeroed ISS QPD; new readings are dX = 0.0005 mm, dY = 0.0013 mm. Jason, Peter
Summary: Fairly straightforward recovery from maintenance day activities, which didn't end until after 4pm PST. Several locklosses due to commissioning effort.
Activity log:
0:04 Kyle back from MY
0:30 PSL crew done
0:39 Fil done
1:07 Jeff K to turn on EY ESD driver
3:37 cleared H1SUSETMY tim error
There seem to be two new rf oddities that appeared after maintenance today:
Nothing immediately obvious from either the PR or SR bottom-stage OSEMs during this time. Ditto the BS and ITM oplevs.
Nothing immediately obvious from distribution amp monitors or LO monitors.
A bit more methodically now: all the OSEM readbacks for DRMI optics, including the IMC mirrors and the input mirrors. No obvious correlation with POP90 fluctuations.
I am tagging detchar in this post. Betsy and I spent some more time looking at sus electronics channels, but nothing jumped out as problematic. (Although I attach the fast current monitors for the beamsplitter penultimate stage: UR looks like it has many fast glitches. I have not looked systematically at other current or voltage monitors on the suspensions.)
Most likely, noise hunting cannot continue until this problem is fixed.
We would greatly appreciate some help from detchar in identifying which sus electronics channels (if any) are suspect.
In this case, data during any of the ISC_LOCK guardian states 101 through 104 is good to look at (these correspond to DRMI locking with arms off resonance). Higher-numbered guardian states will also show this POP90 problem. This problem only started after Tuesday afternoon local time.
I said above that nothing can be seen in the osems, but that is based only on second-trends of the time series. Perhaps something will be revealed in spectrograms, as when we went through this exercise several months before.
Comparing MASTER and NOISEMON spectra from a nominal low noise time on Feb 3 with Jan 10, the most suspicious change is SR2 M3 UL. Previously, this noisemon looked similar to the other quadrants, but with an extra forest of lines above 100 Hz. Now, the noisemon looks dead. Attached are spectra of the UR quadrant, showing that it hasn't changed, and spectra of SR2 M3 UL, showing that something has failed - either the noisemon or the driver. Blue traces are from Feb 3 during a nominal low noise time, and red are a reference from science time on Jan 10. I'm also attaching two PDFs - the first is spectra of master and noisemon channels, and their coherence, from the reference time. The second is the same from the current bad time. Ignore the empty plots, they happen if the drive is zero. Also, it seems like the BS M2 noisemon channels have gone missing since the end of the run, so I had to take them out of the configuration. Also, I took out the ITMs, but I should probably check those too.
Jeff Kissel graciously made 4 StripTool templates for OPS use when having issues with WFS running away during green locking, as happened today. These templates are:
ALSX_GreenWFS_Pitch_Metrics.stp
ALSX_GreenWFS_Yaw_Metrics.stp
ALSY_GreenWFS_Pitch_Metrics.stp
ALSY_GreenWFS_Yaw_Metrics.stp
and can be found in the usual place for OPS templates: /ligo/home/ops/Templates/StripTool.
These will likely be used with the assistance of a commissioner since I didn't catch all the intricacies of manipulating the WFS enough to explain it here, but they are there to save some time when needed.
J. Kissel, J. Driggers A little more information / motivation here. There are 4 degrees of freedom when it comes to keeping the green arms aligned: X and Y arms, in both Pitch and Yaw. There are three optics involved in each of these systems -- each arm's ITM, ETM, and TMS. As such, we've constructed a sensor array that has three sensors for each arm: a green WFS A, a green WFS B, and that arm's ITM camera. We've arranged it such that WFS A (in the I phase) is "DOF 1" which is controlled by the ETM, WFS B (again in the I phase) is "DOF 2" controlled by the TMS, and the ITM camera is "DOF3," which is controlled by a little bit of all the optics; ETM, ITM, and TMS -- but mostly the ITM. As such, each of these templates contain the error signals for each of these loops for the 4 degrees of green alignment. The goal during their use: Minimize (bring the absolute value toward zero) each of the sensor, or "error," signals, while maximizing the last signal in each template: the (normalized) power stored in the arms. When you'll typically need these templates: - Only when trouble shooting the green WFS alignment step of initial alignment, where the "trouble" is that every time the green WFS engage, it steers the arm (X or Y) off into the weeds (in Pitch or Yaw) and breaks the green arm lock. - We've found that this will only typically be after a particularly rough event on one (or several) of the chamber's isolation system, for example, an ISI or HEPI platform has tripped. (Or a maintenance day when the SEI platforms are brought down for a model change, or if there's been a site-wide power outage, etc.) - In such a case where you have to invoke this technique, it's likely that only one of the error signals is too large for the alignment system to handle. Strategy: - Be sure to turn OFF the automatic alignment before getting started. It's the auto-alignment that's the problem, so you must be sure it's not fighting you while you're manually 6pushing the optics to the right place. Do this by heading to the ALS controls overview screen of which ever arm and angular DOF you're fighting (X or Y, Pitch or Yaw), open each of the WFS "DOF" filters, and turn ON a limit of 0.0 for each. (If you turn OFF the input or output, guardian will fight you and automatically turn these things back ON. No good.) - Keep the alignment adjustments small (0.1 micro radians on each optic's slider), and make sure to write down where you've started on each so you yourself don't get lost in the weeds. - Be mindful of the optic indicated by the DOF which has a particularly large error signal. If WFS A error signal is large, it's likely that the ETM should be adjusted. - The error signals only should be used when the cavity is locked on a zero-zero mode. That means you've got to get the alignment most of the way there *by hand* before using this system. Further, you have to make sure that the *automatic* green alignment system is OFF so that it's not fighting your manual alignment. - Most typically, it's the camera error signal that's large. Unfortunately that error signal corresponds to a degree of freedom that is control by all three of the optics, so use your best judgement as to which optic to try first (based on your knowledge of what happened to the chambers and optics prior to you starting the alignment process). However, if no major alignment changing events have occured, start with the ITMs. As usual, if you change the alignment of an optic in either direction and it only seems to make all metrics worse, or break the lock, then that's not the problem optic! - Once you have the error signals relatively small (Under +/- ~1000 - 2000 counts for the WFS, Under +/- ~0.1 to 0.2 counts for the Camera), re-engage the auto-alignment by releasing the limits of 0.0 in each of the "DOF" filter banks. - Rinse and repeat until the arm cavity stays locked on the 00 mode with the auto-alignment engaged.
In Jeff's first paragraph, he forgot to include the other 2 degrees of freedom: the input beam pointing, which is defined by the TMS for the green lasers. So, there are 6 degrees of freedom for each arm.
I have reset all WD saturation counters for the platforms with warning.
Also, I reset the HAM6 watchdog counter, which is not found on the OPS_HEPI_L4C_WD_SAT_CUSTOM.adl screen. However, I was notified that HAM6 watchdogs were beyond 75% full via DIAG_MAIN. Perhaps we should add HAM6 to this MEDM screen? Or maybe not, since this seems to have been an ACT saturation, not an L4C??
See attached pics for details of what was reset.
I had to reset the HAM6 WD again after a lockloss. Note: this happened the first time when I was clearing L4C counters, but I attributed it to the warning of 75% full. Maybe something else is going on here. Attached is another screenshot.
Hugh has noted in the past that this ISI seems to be holding an larger than normal offset in Z, based on actuator drives in alog 19528. The ISI tripped on a vertical actuator, so relieving this offset may make HAM6 trips less likely. Of course this trip was caused by drives way over the actuator limits, so something else may be going on.
[Jenne, JeffK, EvanH, Travis]
On our very first lock attempt (that made it past DRMI) after today's maintenence, we lost lock during the BS coil driver switching state. Thanks to today's data saving work (see alog 25329), we now have some data to look at!
Attached are 2 figures of the same lockloss, one of which shows the whole time we were in the COIL_DRIVERS state, and the other zoomed in on the lockloss itself. On the top row of these figures, the left 3 plots are just power buildups, to indicate when the lock was actually lost. The right-most plot is the ISC guardian state - state 505 is COIL_DRIVERS. The bottom row of plots are the new FASTIMONs for each OSEM of the BS M2 stage.
These channels are plotted in units of ADC counts, but we have 40V/2^16 counts over a 40 ohm resistance, and the BS M2 actuation strength is about 1N/A. The features in the LL channel are of order 100 counts, which corresponds to about 15 microNewtons. For longitudinal, this is of order 60 nm of a kick to the BS. The rms for MICH control above 10Hz is only 1e-14 m, so this is 6 orders of magnitude above the rms. For pitch and yaw, this corresponds to about a 2 urad kick to the BS. (Newtons to meters or radians from T1200404).
This seems like plenty to explain our locklosses. Next up: what to do about it?
When you have an xml file made by older diaggui, opening that file using the latest diaggui will cause some problem.
If you do that right after launching the new duaggui, the problem is most obvious because hitting the "Start" button will raise an error message: "No measurement channel defined: Unable to setup test channels."
You need to click and reselect each and every measurement channel. It seems as if these channels are already "selected", but select manually anyway. Hit "start" and this time the measurement starts.
OTOH, once you make any measurements using the new diaggui and open an old file later, hitting "start" will not cause an error, but sometimes not all channels are measured (but sometimes all channels are measured, not sure about the condition).
FRS ticket 4333 was filed.
J. Kissel I'd found it frustrating that the ALS control system overview screens were not linked on the overall ALS overview screen, so I've added links. See attached screenshot. Changes have been committed to the userapps repo, /opt/rtcds/userapps/release/als/common/medm/ALS_CUST_MAIN.adl
In fact, the ALS overview screen already contained links to the X and Y control screens.
ACTIVITY LOG: 15:15 UTC Chris S. headed down X arm for beam tube enclosure sealing 3 doors from end X 15:24 UTC Peter transitioning LVEA to laser safe 15:34 UTC Peter done transitioning LVEA to laser safe 15:35 UTC Peter to H1 PSL to start HPO characterization 16:10 UTC Dale opening OSB receiving rollup door 16:11 UTC Hugh restarting guardian on HAM6 ISI 16:18 UTC Joe D. to LVEA to check batteries, eye wash stations, etc. 16:21 UTC Hugh done guardian update on HAM4 and HAM5, moving to HAM3 16:27 UTC Jeff B. to LVEA to get data from desiccant cabinets and work in the area between the changing room and LVEA 16:30 UTC Filiberto and Richard to LVEA to pull TCS EtherCAT cables from beer garden to mezzanine 16:32 UTC Betsy starting charge measurements 16:46 UTC Jason to H1 PSL to work with Peter 17:01 UTC Received SNEWS alarm as GRB in Verbal Alarms. No indication in GraceDB that it was a test. Arrived at time scheduled for test. 17:03 UTC Cheryl misaligning MC1 to check for light on photodiodes 17:04 UTC Jeff B. done 17:08 UTC Bubba to LVEA to work on crane shimming 17:15 UTC Joe D. done for now 17:17 UTC Hugh tripping HAM5 ISI for guardian update test 17:30 UTC LN2 delivery to mid Y 17:40 UTC Betsy done charge measurements 17:47 UTC Richard removing safety terminals from MSR Beckhoff chassis 17:53 UTC Filiberto done pulling TCS EtherCAT cables by beer garden, moving over to HAM6, Richard helping 18:05 UTC Hugh tripping ITMX for guardian update test 18:14 UTC Hugh done guardian update testing 18:14 UTC Karen and Christina to end stations to clean 18:22 UTC Kyle going to Y2-8. 18:25 UTC Hugh and Jeff B. to mid X, end X, end Y. 3IF0 inventory at mid X. HEPI maintenance at end X and end Y. 18:28 UTC Richard and Filiberto done in LVEA. Filiberto starting modification of Beckhoff Corner Chassis 4. 18:32 UTC Paradise water delivery 18:33 UTC Filiberto powering down Corner Chassis 4, removing chassis, adding terminals 18:47 UTC Corner Chassis 4 down 18:51 UTC Joe D. back to LVEA 18:57 UTC LN2 delivery to gate, Kyle directing driver to CP6 19:11 UTC Evan to CER 19:14 UTC I untripped MC2, noted that nothing was going to the DAC, was in safe mode, Cheryl put it to aligned 19:14 UTC Cheryl moving IMs 19:23 UTC Evan measuring voltage out of ITMY PUM coil driver SUSAUX123 model restarted 19:29 UTC ASC model restarted 19:31 UTC Hugh back from end/mid stations, going to mechanical room 19:31 UTC Kyle to end X to pickup part 19:34 UTC SUS ITMX RMS UL WD tripped from Evan disconnecting coil driver 19:42 UTC Jim W. recompiling end X BSC ISI model (WP 5717) 19:49 UTC Jim W. taking ETMX to damped 19:51 UTC Coca Cola delivery through gate 19:54 UTC Kyle back to CS for lunch 19:58 UTC SUSAUX ETMX/Y models restarted 20:05 UTC Richard to roof to look at GPS antennas 20:12 UTC Richard back from roof 20:23 UTC TJ reloading ISC lock guardian 20:31 UTC DAQ restart 20:54 UTC Richard removing safety terminals from MSR Beckhoff chassis 20:58 UTC Bubba to LVEA to move cleanroom, continue working on crane shimming 21:01 UTC Richard done removing safety terminals 21:02 UTC ETMX working, Jim W. taking measurements 21:09 UTC Jason and Peter breaking for lunch 21:17 UTC Dave restarting ISS model 21:20 UTC Jim W. taking down all BSC ISIs except ETMX, preparing for model restart (ETMX already done) 21:35 UTC Dave restarting all BSC ISI models except ETMX 21:45 UTC DAQ restart 21:53 UTC Filiberto replacing Corner Chassis 4 21:56 UTC LSC filter module changes applied 22:05 UTC Filiberto done. Richard and Filiberto to end Y to look at chassis 22:12 UTC Jason to H1 PSL. Peter already there. 22:23 UTC Jenne taking graduate students from Australia to measure electronics by HAM6 22:34 UTC Kyle to LVEA TJ leading tour in CR 22:46 UTC Jenne done 23:19 UTC Filiberto to LVEA to run RF cabling for ASC triplexor 23:26 UTC Dave and Jim B. running timing cable for conlog-test-master 23:30 UTC Kyle to CP3 overfill 23:31 UTC Bubba and John done crane shimming 23:55 UTC Evan redoing measurement from this morning
Today I set up the LASER_UP state which monitors the system once the locking is done. The power output and PZT voltage output are monitored and if they exceed allowed values guardian goes back to the FIND_LOCK_POINT state to try to find a new locking point.
I also added another assert class that monitors the chiller temperature setpoint throughout the locking process including the final LASER_UP state, so that the temperature set point doesn't run away to a value that can cause the chiller to trip off. When a chiller setpoint that is outside of the allowed range is detected it goes back to the DOWN state, resets the chiller setpoint and then starts moving back up through to LASER_UP.
Guardian has now also been set up for the X-arm laser, tested, and is now working.
Dave has taken a safe snap of the system in the DOWN guardian state to record new gain settings and the filter settings for when the servos are off.
Dave, Cao, Evan
We wanted to check briefly how the electronics noise of the PUM coil drivers compares to the excess noise in DARM.
With an SR785, we measured the voltage noise spectra out of the IX and IY coil driver chassises (going to the satellite boxes), and below 80 Hz we found them all to be below to the noise floor of the SR785 (40 nV/rtHz at 10 Hz, 10 nV/rtHz at 100 Hz). On the other hand, in order to explain a 1/f4 DARM noise of 1×10−19 m/rtHz at 30 Hz, this would require a white voltage noise at the coil output of 190 nV/rtHz for a single coil, or 50 nV/rtHz for 16 coils with uncorrelated noise.
If we want a good measurement (not just an upper limit) of this coil driver noise, we should use a FET-buffered preamp for reading out the signal.
Today, Kissel and I worked through completing the following (taken from ECR E1600033) for the SUSAUX QUAD and BS models. This meant recompiling and restarting h1susauxb123, h1susauxex, and h1susauxey with CDS and building new medms screens.
1) Remove SUS NOISE and VOLT MONs from all UPPER stages of SUSes, and non-globally controlled LOWEST stages, and replace with FAST_I_MONs with same data rate.
2) Remove SUS NOISE and VOLT MONs from frames of 10 SUSes under global control and replace with FAST_I MONs (or Fast LV ESD MONs where applicable) with increased data rate of 16k.
3) We will add associated filter banks for every FAST_I_MON.
Data rate calculations show that there will be a zero sum gain to the Science frames by performing the above (see supporting spreadsheet on DCC page).
New FASTIMON screens are shown below. All model and medm changes have been committed to the SVN.
Due to this change, when you try to look at LVESDAMON further back in time than this afternoon, DTT says "synchronization error".
This is where you can use the new feature of DTT.