HWSX peak counts bad, I stopped the code, powercycled the frame grabber and the camera, then restarted the code. Peak counts are back to normal (~940).
20:14 Noticed HWSX has bad peak counts again. Restarted HWSX code.
I restarted the HWS code on H1HWSMSR this morning. It had stopped following the Beckhoff failure last night.
The attached 1 day trend plot shows the BRSY RX (tilt), wind speed, BRS Beam velocity, and the Y STS Seismometer motion. Yesterday at the 1800 hours mark, the tilt and velocity rapid decrease is from when we forced the damping on. Otherwise for the remainder of the plots, the BRS is running on its own. The shape of the trends don't scale with the wind velocity as one might expect...
Remember that BRS_IN is the raw angle of the beam-balance, which is an undamped ~7.6 mHz oscillator with Q of ~2700. The real-time signal will be completely dominated by the resonance - it is like looking at the raw DARM channel and seeing only the violin mode and trying to see if there is a GW signal based on the unfiltered amplitude. To see the real-time ground tilt, you have to looks at (10-100 mHz) BLRMS of BRS_OUT or some such filtered signal.
TITLE: 04/29 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: Plagued by earthquakes, a bit of wind, and then laser trouble. Wasn't a great night tonight.
SEI note: I struggled to get ISI ITMX to stay in fully_isolated, even after the 0.03-0.1Hz seemed to be back to normal. I'm leaving it fully_isolated with the T240s in Low gain now, but it has fooled me before. Also is there a way for us to tell if one of the sensors is already in low gain? Should it be added to the DIAG to make sure all are in high gain?
LOG:
Keita, PeterK, Rick
Keita and I went out to the laser room to make some RIN measurements.
Short summary:
Front end RIN seems normal both with the high power oscillator (HPO) running and with it off
High Power Oscillator RIN much higher (>factor of 10). Glitches (few cycles of 500 kHz) at abouyt 37 kHz on the hight power oscillator light, but not the front end light
More tomorrow regarding this.
After shutting the laser (front end and HPO) down to reset the Long Range Actuator (via the reset button on the back of the control box), the HPO only came back up with about 1/4 of the expected power. Troubleshooting and brainstorming on the phone with Peter, we tried increasing the diode currents by 1 A (left the laser in this condition). This due to the low power and because the head 4 output dropoed from about 96% to about 91%, i.e. it came back about 5% lower after switching the laser off to reset the LRA and turning it back on.
HPO power (with internal shutter open, front end off) still only came up to about 50 W (expecting close to 200).
Peter suspects that some internal optic has been damaged. We decided to shut the system completely down - lasers and chillers - and try a cold start in the morning.
I plan to talk with MattH and Olli Puncken at LLO first thing in the morning to see if they have any ideas or suggestions.
It's looking pretty likely at this point that we aren't going to have a high-power laser until we can go into the HPO to inspect the optics and assess the state of the HPO. This would start next week on Tuesday at the earliest, when PeterK returns from vacation.
Assuming that the cold start in the morning is not successful, we will have to decide is we want to try to get a the 15 watts reflected from the HPO aligned and modematched to the PMC so we will at least have a low power beam to work with until next week.
I'm leaving the rX-BRS sensor correction at ETMY on over night, as far as I can tell at the moment it is doing very little, but if someone gets suspicious about ETMY motion feel free to turn it off in ISI-ETMY_ST1_SENSCOR_GND_RX_Match
Patrick, Matt, Kiwamu, Vern We tried testing some code changes for the rotation stage. They didn't seem to work, so I reverted the change to the Laser Power library and went through the GUI to start afresh (copy new code from target directory, compile, run, etc.). Now everytime I do this I get a divide by zero error. (see attached). Did someone change the code in PLC1 that introduced a divide by zero error, and not run it until we tried to now? Or did I somehow do this?
Seem to be fixed by recompiling PLC1. I've burtrestored PLC1, PLC2 and PLC3 to 6:10 this morning (local time).
Here are two examples of locklosses when powering up from the last 24 hours. The first one shows the rotation stage moving in a jerky way, this was an example of a time when the velocity was changed before the request was made, but is worse than the normal "moving in the wrong direction" problem. You can see that the accelerometers on the PSL all have glitches when the rotation stage angle encoder records a change in angle.
In the second example the rotation stage velocity moves smoothly, and the power changes smoothly, but we have a lockloss which could be ASC related.
One thing to remember is that TwinCAT will try to reuse its previously stored values of variables, when you log in with a slightly modified code. Generally, this is a good thing, but it can fail with an internal variable restored to a value which leads to a divide by zero error. You need to login and use the Reset (clear all variables except the persistant ones), or Reset All (clear all variables). In the later case, you definitely need a SDF/burt restore.
I had tried logging in with the PLC and resetting the variables.
Kyle, Joe D. Our two aLIGO workhorse Vacuum Bake Ovens are getting repaired and upgraded. We expect VBOC to re-enter service next week followed by VBOD sometime thereafter.
Chris, Carlos, Jonathan, Jim, Dave:
today we installed Chris' medm_time_machine software on the CDS workstations. This permits the user to ask how an MEDM screen looked like in the past, provided the channel data is available from the selected NDS server.
To open the feature, right-mouse inside an MEDM and select Execute (last item) and TimeMachine (last item on pull-out window). See figure 1.
As an example, H1 was locked at noon today and is currently unlocked due to an earthquake. Figure 2 is requesting the OMC DC-PDA filter module screen at noon (5 hours and 30 minutes ago at the time of asking). Figure 3 shows the current running MEDM on top, and the timewarped screen from 12:02 PDT today below it. The time strings in the upper right corner show the times. Channels which are not in the DAQ show up as white rectangles (for example strings, momentary buttons, redundent outputs). When the IFO was locked, the input was many thousands of counts.
One feature we added to the launcher today was to allow the user to set the precision of the playback data. In some screens a precision of 3 is good, for others lower precision makes the screen more readable.
Jonathan showed that by changing the NDSSERVER enviromnent setting and opening a kerberos ticket, this system is able to get data from the LDAS NDS2 server.
Things to note:
For this to run on the control room workstations, we installed pcaspy (easy-install pcaspy) locally at:
/usr/local/lib/python2.7/dist-packages/pcaspy-0.5.1-py2.7-linux-x86_64.egg/pcaspy
pcaspy should already be installed for LLO CDS, if we wish to install the patch
Successfully pumped down emergency GV hood prototype! After vulcanizing two 1/4" diam. o-rings and sealing via one band clamp, the volume pumped down to fractions of a torr. Using aux cart, pressure at turbo read 9.1e-3 Torr with a hefty foreline pressure of 3.2 Torr. I will try to improve hole plugging. I want to test the 6" gap calculation incrementally to see if the polyurethane withstands the atmospheric pressure differential as predicted.
Jenne, Sheila, Chris, TJ, Evan
It seems that tonight we have been sabotaged by some code that we have been using for a long time (this haas only happened once that we caught it, although we have a lot of unexplained locklosses tonight).
In the attached screenshot you can see that the DRMI guardian was sitting at DRMI_3F_LOCKED (130) when it decided to go to LOCK_DRMI_1F (30). There is a decorator in DRMI_3F_LOCKED that apparently returned LOCK_DRMI_1F, because it though DRMI was unlocked (it was fine as you can see from the power build ups in the top row).
The code that checks for DRMI lock is:
It happened again at 6:53:07
Sheila, Jenne, Jamie, Chris, Evan, Dave
We still don't understand why this would have happened, although we should be able to debug it a little bit better if it happens again.
Jenne and Jamie edited the DRMI_Locked function so that there will be more information in the guardian log in the future:
def DRMI_locked():
Made measurements of the oscillator power noise with a few photodiodes.
PowerNoise.png shows the free-running oscillator relative power noise
measured before the acousto-optic modulator. This is more than 10 times
noisier than when the laser was installed in the H1 enclosure. The
other trace in the plot is the out of loop of the relative power noise.
It is also about a factor of 10 higher than it should be.
Whilst the power stabilisation was locked, I looked at the AC coupled
output of the photodiode and did not observe any oscillations. The maximum
peak-to-peak variations were ~40 mVpp.
If I am reading this plot correctly, the ~37kHz rep-rate seen in last week is probably represented in this spectrum by the peak which the ISS is adding at that frequency. (The 500kHz oscillation is too high to see here.) It might be very informative to see what is going on above 100kHz since the ISS seems to be adding a lot of noise at 100kHz (about a factor of 10 above its input).
Is this plot really calibrated into RIN?
The digital RIN readback for the OOL inner-loop sensor appears to be 20 dB lower than the trace shown here (26773). Same for the digital readback for the HPO transmission.
Look at alog 26893.
Peter's "relative power noise" agrees well with the raw voltage spectrum of Rick and mine. In other words, Peter's plot seems to overestimate RIN by 15 dB or so for the HPL monitor (DC level is about -6V), and about 20dB for 1st loop sensor (DC level -9 to -10 Volt).