kyle.ryan@LIGO.ORG - posted 11:26, Thursday 18 August 2016 (29173)
Extending bake of Vertex RGA 12 hours longer than initially claimed
Every hour helps - exploiting the water leak recovery
Every hour helps - exploiting the water leak recovery
Gerardo, Chandra Adjusted potentiometer on PT-140A pirani gauge to bring it back on scale. CCW 4 turns. Closed FRS 6061. NOTE other gauges that were adjusted a few months ago for calibration: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=27726
Per WP# 6072 -- Activity: PNNL will be coordinating an OTDR test of the fiber pair that feeds the observatory's data connection to make sure that signal strength is appropriate for upgrading to 10Gb/s up from 1Gb/s. Wireless, phones, and internal wired networks at the observatory will be unaffected. Internet access will be interrupted during this work. Work started at 08:00 PT, completed at approximately 09:10 PT. If I receive the OTDR data (as viewed from PNNL - ISB2 end), I will attach it to this post.
Title: 08/17/2016, Evening Shift 23:00 – 07:00 (16:00 – 00:00) All times in UTC (PT) State of H1: PSL down due to water leak in the HPO Commissioning: Outgoing Operator: Corey Activity Log: All Times in UTC (PT) 23:00 (16:00) Start of shift 23:15 (16:15) Terra – Going into the LVEA 23:51 (16:51) Jason – Out of the LVEA 23:52 (16:52) Ben A. – Cable work around HAM2 23:56 (16:56) Terra – Out of the LVEA 23:58 (16:58) Jason – Going into the LVEA to turn out PSL lights and recover laptop 00:10 (17:10) Jason – Out of the LVEA 00:31 (17:31) Ben – Out of the LVEA Title: 08/17/2016, Evening Shift 23:00 – 07:00 (16:00 – 07:00) All times in UTC (PT) Support: Incoming Operator: N/A Shift Detail Summary: No commissioner work planned due to PSL problems. Ending shift early.
Richard, Fil, Jim W, Jim B, Dave
The voltage outputs from the gust meter when it was installed at EX yesterday did not look correct and did not correlate to wind conditions. Today Jim W brought the unit back to the shop and we tested it with a fan. There was some variation in voltage output but it did not look correct when referencing the manual. Jim Bconfigured the system for RS232 serial communications and logged into the unit. The serial readbacks for wind speed, azimuth and elevation looked correct. Jim reprogrammed the three voltage outputs to be Speed, Azimuth and Elevation. We then set the system back to analog voltage mode, and now the outputs vary by many volts and are V1=speed, V2=azimuth, V3=elevation. We suspect the system lost its programming at some time and was perhaps outputting U,V,W analog voltages.
The unit is ready for re-install at EX.
Ben, Dave:
today I modified h1psliss to read the PD5-8 FAST and SLOW channels. On paper it looked like the h1psldbb model was not using the last eight channels of its ADC (the fourth adc, or ADC3). In practice the DB25 connector for the last eight channels of this AA chassis have a cable connected. (Note to team-psl, perhaps the model should be recording these channels). I had already made the model change and restarted the DAQ before we discovered this.
Ben suggests for plan-B we re-use eight of the ISS ADC channels for reading PD5-8. We are going to disconnect the production ISS cables from ADC0 chans 16-23 and replace them with the new ISS PD5-8 cables during tomorrow's test. Specifically ADC0 chans16-19 are PD5-8FAST, chans20-23 are PD5-8SLOW.
I've modified h1psliss, we will wait until tomorrow morning the restart the model and DAQ.
This morning Ed asked me to look at some trend plots he had just generated. Clearly noticeable was a sharp
rise in the humidity and a drop in the flow rates for the heads. I checked the crystal chiller and its
water level was below minimum and the controller displayed an error informing of such.
Attached is a picture of the burst hose attached to the head 3 water cooling circuit. This is the same
connection that failed ~2-1/2 years ago. I do not recall if it has been adjusted since that time. Although
care is/was taken to prevent twisting of the hose when the fitting is tightened, it is highly likely that
there was a small twist in the hose that exerted a small shear resulting in the tear seen in the picture.
Also attached is a picture of the 4f lens for head 4, clearly showing some water droplets on it.
Apart from the obvious step of replacing the hose, the turbine flow sensor for head 4 will also be
replaced. The laser will take the better part of a day to dry out - to be on the safe side. All optical
surfaces and the fibre bundle tips will need to be inspected for cleanliness before any operation of the
high power oscillator is contemplated.
(All times in PST)
H1's state this morning: DOWN. ITMx was swinging around, and Kissel cleared history for M0 Lock to calm it down.
Turned off locking servos for the night for ISS, FSS, & PMC.
LN2 at exhaust in 8 minutes
Raised LLCV from 19% to 20%
In preparation for the next CW hardware injection test, the /hinj/Details/pulsar/ subdirectories on h1hwinj1 have been updated. Here are the major changes:
CPS sensor checks for BSCs and HAMs - measurement taken around now-6hours, or about 14:30UTC today.
Nothing remarkable - spectras look good.
Nothing unusual to report. Typical noise at both end stations and data glitching at EY. Corner station pressure shows pump removal as described in Hugh's aLog. Corresponding control activity for this action is also notable in the plots.
Peter, Ed
Investigation currently underway regarding potential water leak in HPO.
Peter & Ed are addressing it in the H1 PSL Room. Vern notified.
Peter said we will be down rest of today and possibly tomorrow. When TCSy work is complete, we could transition to LASER SAFE....but that will have to be after the active TCSy work.
This is FAMIS task #6109
FRS has been filed for the water leak: FRS#6063
Terra, Sheila
Tonight we had trouble engaging the ASC again.
Losing optical gain in POP X
We rang up what we think is a PR3 bounce mode when engaging the ASC the same way as last night. We found that we could avoid ringing this mode up by keeping the PRC2 gain low (digital gains of -500). Right before the OMC damage/vent, the POP X path was reworked and the optical gain seemed suspiciously low.
Tonight we found that the optical gain has decreased even more. Terra changed the demod phase by dithering PR3 pit (500 counts to M3) and rotated the phase positive 65 degrees, (Q1, Q2, Q3, Q4 from 55, 53, 54, 51 to 120, 118, 119, 116 ) to maximize the signal in I (minimize the Q signal). The 2 attached figures show Terra's before and after OLG measurements (excitation gain of 50), both with Jenne's gain of -5000, showing a 10dB increase in optical gain which is about what we expected based on the dither amplitude change.
After optimizing the phase, we did not see the 28 Hz mode get rung up, but this seems to come and go because we also didn't see it yesterday. We quickly tried moving L2 on the POP X path, while watching the amplitude of the PR3 dither line in the POP X signal. We moved the lens about 4 inches closer to POP X and about 3 inches further away, and didn't find any location that had more signal for PR3 so we replaced it as we found it.
We are going to leave the IFO locked in DC readout 2 Watts with the request set to down so that it will not try to relock. The noise is bad as expected.
POPX whitening gain is 0dB but should be odd, see alog 26307. FRS 6057 filed.
The whitening gain on POP X was changed from a gain step of 7 (21 dB) to 0 (0dB) on August 12th. This whitening chassis has a problem and we must use odd gain settings, or else it will return an error and not set the gains equally on all quadratns, as Keita and Hang noted 26307
The change in gain probably happened during a beckhoff restart for the shutter code, but we could have been saved from this problem by SDF. I cannot find a record for these whitening chassis in any SDF table.
Also, this does not explain the drop in gain that Jenne saw, which happened before the whitening settings changed.
The stuck whitening gain bit is the LSB of the Q3 channel. In the past this was typically an indication of a cable problem (short).
Sheila Daniel Terra
Connected the AM laser to the POP X head, and saw that we have very similar response in the electronics to what Evan measured in 27069
we had 3.3 mW out of the AM laser with a whitening gain of 21 dB, used -40 dBm of RF drive at 45.501150 MHz. We saw about 600 counts on each quadrant (except quadrant 3 which had 350 counts and also the least amount of DC light because of way the laser was mis centered on the diode).
We saw that there are rather large offsets when we changed the whitening gain, so Daniel reset the offsets. The large offsets might have contributed to problems last night, along with confusion about the whitening gain.
Also, we remembered that a factor of 6.7 of the mystery gain loss was due to adding a beamsplitter and forgetting to comensate for it on July 11.
(Edit: Actually, Haocun and I did remember to correct for this gain change, we just compensated for it in the digital loop gain. )
So to summarize:
loops were intially commisioned with a whitening gain of 21, a digital gain of -21, a 1 Hz ugf, and electronics gain similar to what we have now. (late may)
Edit: loops were originally commisioned with a filter gain of -200 for pit, -0.1 in the input matrix, an analog gain of 21 dB, and the WFS head electronics performing in a way simlar to what we have now. This is when the reference that I think Jenne used was saved, and within a few days the pit input matrix was reduced by a factor of 2.
Edit: Around June 16th, we had difficulty staying locked when these loops were engaged, which was noted in the alog. Terra and I just looked at trends of the filter gains, and it seems like we also reduced the digital gain from -220 to -3.4 although this was not noted in the alog. This, together with the input matrix change explains most of the missing gain that Jenne found.
On July 11th I forgot to compensate for the beamsplitter causing a gain reduction of 6.7 that no one noticed.
On July 26th, Evan and Keita relocated POP X and Jenne noticed that the digital gain had to be increased by a factor of 250 (or 500 for yaw) to keep the ugf the same.
August 12th the whitening gain was reduced to 0 dB from 21 dB by mistake in a beckhoff reboot.
August 16th Terra and I noticed this further reduction in gain, which is explained by the whitening gain. We also changed the demod phase which increased the gain by about 10 dB. We checked that small movements of the L2 don't change the optical gain much, and moving it by a few inches can decrease the signal.
So, we are missing about a factor of 40 gain, which we cannot explain with electronics.
In the end only a factor of 2 of Jenne's gain change in unexplained. It seems that we have had stable high power locks with both the high gain and low gain settings for PRC2, so we can decide which we want to use. We also should have a factor of 3 increase in gain because of the phasing Terra and I did.
More complicated than that.
|
Whitening (dB) |
POPX digital gain before rotation |
Input matrix | PRC2_P_GAIN |
BS |
Overall gain relative to original |
alog | |
| Originally | 33 | 1 | -1 | -220 | none | NA | |
| May 24 ~1:02 | 33 | 1 | -0.05 | -220 | none | 0.5 | |
| Jun. 17 | 33 | 1 | -0.05 | -3 | none | 6.8E-3 | |
| Jun. 22 ~noon | 21 |
2.8 |
-0.05 | -3 | none | 4.8E-3 | 27901 |
| Jul. 11-12 | 21 | 2.8 | -0.05 | -21 | inserted | 5.0E-3 | 28324 |
| Jul. 27 ~4:20 | 21 | 2.8 | -0.05 | -5000 | inserted | 1.2 | 28666 |
No mystery optical/electronic gain reduction any more. Maybe a factor of 1.2 came from the rework on the table.
It's not clear to me why the PRC2 filter gain was reduced by a huge amount on Jun. 17 but I haven't searched through alog.
Typo in the above table, originally the input matrix was -0.1, not -1.
PEM power supplies in the CER mezzanine were swapped out for Kepco model JQE supplies. This model is a 1/4 rack version, and allows for four power supplies to be mounted on one shelf. This is to make room for the ±24V power supplies that will be used to power Beckhoff/Baffle PD Amplifier/ Spool X/Y camera ect. All PEM instrumentation microphones, magnotometers, ect. were powered down from around 10:30 am to 1:00 pm. All power is now restored.
Fil, Terra
I did a quick check of power spectra for all corner station mics and mags, before and after power supply swap. Attached spectra show pre-swap in darker colors as references, post-swap in lighter.
Found post-swap saturation in several mags, strongest in MAG_LVEA_INPUTOPTICS (first attachment at 21:03 UTC ). Fil switched this mag to battery power, which has temporarily fixed the problem (second attachment at 23:03 UTC). He took a look at the mag box in the lab but found nothing wrong. We left it battery powered for now. Will investigate more tomorrow.
Additionally, it looks like MIC_LVEA_HAM7 got disconnected around 23:03 UTC (top right of second attachment).
Channel 2 of AA Chassis S1300104 found to be bad.
MIC_LVEA_HAM7 started negatively railing yesterday ~3 hours after the power supply swap, trend attached (first drop then raise is from power supply swap). Turns out that AA channel has gone bad (bottom most AA chassis in PEM rack). Fil will pull it and have a look tomorrow morning.
AA chassis repaired, HAM7 mic back up and running. MAG_LVEA_INPUTOPTICS was also fixed. At this point, all corner station mics and mags are running well. Attached are power spectra for all.
After I and Evan relocated POP-X and POP-L2, Jenne increased the ASC gain of POP-X by a factor of 250 for PIT and 500 for YAW (alog 28666) just to get back to the old UGF, which sounded crazy to me as I didn't expect much change in Gouy phase.
Just to see if my assumption was wrong and we got super unlucky, I calculated the Gouy phase of the POPAIR path, and it seems like there shouldn't be much change.
In the first attachment, left column is the current configuration, right is the old one. Bottom is the entire POP path from ITM to the ISCT1, and top is the zoomed-in view from HAM1 to ISCT1.
POP-L2 was placed far enough from the waist originally. This lens was moved farther from POP-L1 by 4 inches later, which should have increased the Gouy phase, but it's only 9 degrees. The beam diameter is 2mm now instead of 4mm but that should be OK. These don't explain the crazy decrease of the optical gain.
Funny thing is that the POP-X spectrum itself looks almost the same before and after the relocation (second attachment), so I'm kind of dubious that the optical gain is lost.
It's not totally impossible, but very unlikely, that the Gouy phase was not great to start with, e.g. 81 degrees for the DOF we want to see, and after the change it became 90 degrees.
Today talking with Terra and Daniel, I remembered that we added a beamsplitter to the POP path to void saturating, and apparently I forgot about adding the factor of 6.7 to compensate for lost gain to the POP X RF loops. (alog 28324) So this explains a factor of 6.7 lost gain, but not all the gain that Jenne had to add to the loops.
