We saw the noise at XEnd ESD LL again at ~1.28 MHz, 15mV pk2pk. Rich A. came out to have a look; believe the noise be real (a real oscillation in the hardware).
On LR drive, we also saw the spectrum glitching (bouncing up and down) when driving at 15.5kHz; goes away with no drive signal (note we did not try other drive frequencies yet). Some beat making a low frequency component?
The source of the 1.2Mhz oscillation was identified by opening the spare chasis and looking for marginally stable opamp stages.
The stage that is marginally stable is U6 page 7 of D1500016. It can be made to oscillate at ~600khz or ~1.2Mhz. The stage is a configurable with the pole/zero bypass bit.
When for example the H1:SUS-ETMX_BIO_L3_UL_STATEREQ control word is set to 2 the stage has a pole at 2.2Hz. This is the normal low noise configuration. In this configuration there is no 1.2MHz oscillation.
When this control word is set to 1 the stage is nominally a unitly gain stage. In this configuration some channels (like UL UR LL and LR) have a Q of >5 at 1.2MHz and can be induced to freely oscillate. This oscillation may be damped with a 30pF capacitor across R21.
As this oscillation is not a problem in the low noise configuration no changes will be made. Testing PI channels should be performed with the H1:SUS-ETMX_BIO_L3_UL_STATEREQ control word set to 2.
Kissel started the charge measurements on the SUS ETMs this morning around 8am. However, I just discovered that both ITM and the ETMx ISI's have been tripped since ~4am, so the ETMx data isprolly junk. Boo. Not sure why this wasn't addressed this morning. ISI's are back up now.
Now that the Anthropogenic plot is updating correctly (thx Kissel who poked CDS, and thx CDS), there appears to be a large EQ around the ISI trip time - USGS says M6.2 in Mexico at 03:51 AM local time.
The new idle state, which doesn't do anything (idle), was connected to both DOWN and READY. When the ISC_LOCK Guardian had DOWN requested, it would jump from DOWN to READY as it normally would, and then it would move to IDLE and back to DOWN and repeat. I just removed the path from READY to IDLE to stop this loop, but this makes getting to IDLE a bit akward since it has to pass through DOWN. Whomever made the state may want to reconsider the organization of the path, but this works for now.
Topped off the PSL Crystal Chiller with 125ml of water. This is a normal top off after swapping the filters yesterday. Both chillers are running well.
Added H1:SUS-ETMY_PI_OSC_DAMP_OSC_LINE2_COSGAIN and H1:SUS-ETMY_PI_OSC_DAMP_OSC_LINE3_SINGAIN to the exclude list and updated the channel list. Added 138 channels and removed 56 channels (see attached).
All accumulators were in good shape with little to zero drop in pressure since last checked--26 Jan 2016.
Of course you can't do this task without impacting the accumulators. After checking the HAM5 return unit, the valve was leaking upon pulling the gauge and I could not get it to reseat. The schraeder valve was replaced. Several units lost more than the usual when the gauge did not come off the valve cleanly(quickly) and subsequently had to be rechanged. There were several units which were marginal in January and although they were still in okay range, they were charged up to the top of the acceptable range.
The values have been recorded in T1500280 and I will update this on the DCC after today's HEPI maintenance. Levels I suspect will be different after the full shutdown and additionally I'll do a thorough purge of the resistors which also could affect the Reservoir Levels. Also, motor greasing is about a year old.
This was done yesterday during outage recovery.
TITLE: 06/07 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY: Still recovering from power outage over the weekend. Got through Initial Alignment with only mild struggling. Digital video camera 18 connection error gave us grief during MICH_DARK locking (see screenshot). Also see screenshot of IFO Align for a starting place tomorrow as we expect the alignment to drift for some time due to thermalizing. DIAG_MAIN has been giving an error for H1:PEM-EX_WIND_ROOF_WEATHER_MPH all night.
LOG:
23:46 Darkhan to EY investigating PCal
0:34 Darkhan back, PCalY back up and running
1:40 Dave restarts camera script
2:00 Model restart for ITMx and ITMy for PI model changes
Talked to John about a CP7 Pump Level alarm we were getting. He advised me to turn the 'Manual Setting' to 72%.
Note: According to Gerardo, who called back after John, he did not get an alarm notification by phone about this as he should have. Someone needs to check the machine responsible for these calls.
I did not get a text or email alarm either.
No CP7 texts for me
GV7 has sagged since John and I reinstated normal operational state this morning by re-applying 50-60 psi pressure upward on piston and removing locking pin. Not sure how far down it has sagged. I suggest waiting to run laser until it's boosted back up Tuesday morning by someone from the vacuum group.
I'll fix tonight and come in an hour later in the morning
3 hour trend of CS instrument air range between 62 psi - 82 psi. As found, 55 psi < GV5, GV6, GV8 < 60 psi, GV7 = 38 psi -> Adjusted all for > 60 psi. Note: the LVEA was quiet and I could easily hear the displaced air exiting the pneumatic manifold vent port. I would guess that the gate had sagged several inches (foot?) based upon duration of audible venting and the convincing "THUNK" of it hitting the hard stops.
Thank you, Kyle!
I had to restart the TwinCAT code for h0vacex and h0vacey to add back the Inficon gauges that were disabled during the power outage. This reset the PID controls for CP8 and CP7. CP8 is back on PID but CP7 has been oscillating (see attached). At this point I am going to leave it on manual at 85% open before I leave. Someone from the vacuum group might want to monitor it.
CP7 setpoint was raised to 94% because it overfilled last Friday during transition to generator power. Reboots today reset it to nominal 92%, which causes the PID loop to close valve, warming up the transfer line, and then it has a hard time catching up.
J. Driggers, J. Kissel, J. Warner While beginning recovery of the IFO, Jenne noticed that the Y arm fiber polarization was high (showing ~24% of input light rejected, where < 5% is where we want to be). We followed instructions similar to what can be found in the OPS Wiki. Since the percent-of-rejection channels are buried deep in the heart of the ALS MEDM screen jungle, I pulled the channels, H1:ALS-X_FIBR_LOCK_FIBER_POLARIZATIONPERCENT H1:ALS-Y_FIBR_LOCK_FIBER_POLARIZATIONPERCENT into a StripTool, which made it easier to by-eyeball, hand-tune the value via the tiny CDS laptop we were using. Also, there's no rhyme or reason to which of the three knobs to use; we just slowly turned all the knobs in both directions until we started to see the rejected light value go down sub-5%. While there, we also brought the X-arm below 5%. The fiber polarization tuning box was turned off once we were done.
There are instructions on how to perform this task in the Ops Wiki, also. It can be found in the Troubleshooting section.
Per Kyle's request I have temporarily forced the PT100 Cold Cathode gauge on in the Beckhoff controls software (see attached).
Thanks!
Summary
On Friday, Pcals were turned off for the June 3-5 power outage (LHO alog 27558).
Details
After finding out that PcalY PDs are not measuring any power I went to the End-Y and checked following:
From this limited information and under the assumption that nothing is broken in the Pcal system, one of possible sources of the issue might be that control signals are not reaching the Pcal Interface module through an EtherCAT connector. The laser on/off and power level (5.0 V) signals should come through this interface (the Pcal Interface Chassis back-side board circuit is given in D1400149).
The issue in currently under investigation.
Filed FRS 5651
Filiberto, Darkhan,
Summary
Details
As it was reported in the original report above, there was an issue with turning on the PcalY laser and operating the shutter. The issue was discovered yesterday around 11am. After that I came back to get breaker circuits to proceed with investigations, but I found out that it is not needed.
At End Y there were issues with Beckoff system that were discovered independent from Pcal work. The Beckoff system feeds control signals to Pcal through EtherCAT cable into Pcal Interface Module. In the afternoon Filiberto went to EndY and replaced couple interface boards of the Beckoff system (see LHO alog 27583). After he let me know about the replacement work, I went back to EndY to turn on PcalY (needed to switch shutter control back to "remote" and double-check power switches on Pcal modules). Now the issue is resolved and PcalY is operating in its nominal configuration.