(Borja)

We have finally manage to solve the mistery regarding the unresponsive LL quadrant of the ETMY ESD (see this and this). So from now on we can add the 4 quadrants in our measurements.

Another measurement was take today (before seeing the effect of green light on the charging of the ETMY) to see repeatibility of measurements or stationarity of the charge.

I attach here the document including the new measurements as well as a comparison plot with the results of Veff and slopes between the data taken 2 days ago (this is 3 days after the 2nd discharge run) and the data taken here.

A summary table is given next with the current values in green (for each quadrant), and we compare them with the measurements before the discharge runs (in blue) and the measurements taken 2 days ago after discharge run2 (in red):

Conclusions: During the last two days the charge of all quadrants has changed considerably bringing it back for some quadrants to same level (or worse) than before the ionized gas discharge runs took place. Look particularly to quadrant UR in pitch where we manage to do a full discharge after the 2nd discharge run but the current measurement shows again a high charge. The only think related to the ESD that has been done today before the measurements is solving the issue with LL quadrant which required turning off the ESD drive and also moving around some cables at the ESD BIAS low pass filter box. In principle this should not have caused a change in the ETMY charge.

Jim, Dave

With so much going on this weekend and today, I thought it would be nice to provide a top level summary of events.

The LDAS raid issue is thought to be completely unrelated to the various FE issues, and was purely coincidental.

There is no obvious connection between the various FE issues, but we cannot imaging they are coincidental.

All times are PDT.

LDAS disk9 failure in RAID

h1sush2a CPU freeze

h1seib1 CPU freeze

h1susb123 CPU freeze

h1seih23 DACs not driving

Disconnecting, moving and reconnecting aux. pump cart to annulus systems

Kyle, Gerardo 

We found the bottom center viewport on HAM4 South door to be leaking at around 2-3 x 10-2 torr*L/sec! -> Tomorrow we will attempt to avoid this leak and resume leak hunting

All annulus volumes in the Vertex, YBM and XBM had been evacuated before helium leak testing began this afternoon.  GV1 was closed then opened and GV2 was closed to demonstrate that the leak was on the Vertex volume and not the YBM or XBM volumes.

Jim, Arnaud, Dave

at 13:46 PDT this afternoon h1susb123 (SUS ITMX,ITMY,BS) cpu froze up in the same way as the recent problems with h1sush2b and h1seib1.

we removed h1susb123 from the Dolphin remotely and power cycled the CPU. The IOP model did not show any IRIG-B error so we thought this was a clean recovery. Unfortunately the DACs were not driveable (similar to h1seih23 problem this weekend) and when the models were activated no DAC channels could be driven. We then stopped the user models, restarted the iop model, restarted the user models and now all is well.

The reason for this crash (the 3rd in 3 days) is unknown, investigations are continuing.

Added the ESD Bias Path Filter Box (D1400192) to both ETMY and ETMX.
Units filter voltage noise present on the ETM bias path of the ESD HV drive.

Due to wiring differences inside the chamber, the filter units were connected as follows:
ETMY connected to pin 1 (Flange F2-3)
ETMX connected to pin 3 (Flange F2-3)

Filiberto Clara

Haven't been messing with this system since doing pumping stuff etc. so this idled along like this.

Overvoltage 3 is our old friend on these VFDs of the Pump Station.  They do seem to be occurring more frequently now.

Corey restarted in the documented manner.

The Filter for the DC bias had been placed on connection 3 as it is for all other ESD drives but the person who installed it did not know ETMy is different the DC bias in on pin1.  This swap was done and the configuration is correct.  The next problem introduced was the lock up of the DAC channels.  I had to cycle the ESD off pull the DAC cable reinstall the DAC cable and power the ESD controller back up.  The system has been turned over for more testing.  

Here are the two vacuum gauges during the two gas injection events last week.

Text files also attached.

H. Radkins, J. Kissel

Similar to what we've seen before on front end computers for HAM2 -- both SEI and SUS -- (see LHO aLOGs 11481, 10849, 10375, 8964, 8424, 7385), we found the h1seiham23 computer inexplicably unable to drive past its IOP model this morning. As with each of the other 6 times, we had to kill the user front-end processes, restarted the IOP process, reset all watchdogs to regain actuation. A good guess of the source this time would be the problems with h1sush2b this weekend, but a trend of the CDS State Word DACKILL bit shows the indicative status changed at Aug 9 2014 22:30 UTC, 15:30 Saturday Afternoon (after Dave finished his bootfest with h1sush2b). So ... still no smoking guns for this problem, nor any better solutions than the time-honored, sledge-hammer fix, restart of all models on the computer.

We STILL need a better indication of this problem. The DK bit in the CDS word often gets ignored by CDS admins, because they assume it's the user or iop DACKILLs that have tripped after their reboots (as expected). SEI / SUS commissioners -- though CDS state word is now on everyone's overview screen -- lose track of what bits mean what, the only non-invasive power they have is to reset all watchdog that they have, and there're often things like IPC errors (which is the bit right next to it) that don't affect the performance of the platform so the word cries wolf often. Presumably, to better indicate the problem, we have to identify the problem to begin with...

Details:
-------------------
- Tried turning on ISI, saw that actuators trip. A plot of the trip shows a clear unstable ramp up of output -- but no sign of movement from the sensors.

- Found no DAC outputs reported by the IOP, even though user model has requested it.

- "DK" bit shows red, but all MEDM control of USER and IOP DACKILL buttons have been reset. Bit went red, Aug 9 2014 22:30 UTC, 15:30 Saturday Afternoon (after Dave finished his bootfest with h1sush2b).

- Checked h1seih23 dmesg, 
controls@opsws4$ ssh h1seih23
controls@h1seih23$ dmesg 
this reports some node errors, but unfortunately the time stamp is meaningless. In the "recent" history, it reports things like
"Session for node 52 is disabled - Status = 0x5"
"Heartbeat alive-check for node=52 failed (cnt=6614 state=0x1 deb=0 val=0)."
"Session for node 20 is disabled - Status = 0x5"
"Session for node 20 is disabled - Status = 0xf"
though, according to T1400026, no front-end process is assigned to the DCUID 52, and h1ascimc.mdl is using DCUID 20, should be totally unrelated.

- Checked /proc/h1iopseih23/status file,
controls@opsws4$ ssh h1seih23
controls@h1seih23$ vi /proc/h1iopseih23/status
reports DAC FIFO Status for both DACs is OK
DAC #0 16-bit fifo_status=2 (OK)
DAC #1 16-bit fifo_status=2 (OK)

- Everyone of previous aLOGs of this problem says "kill user models, restart iop process, restart user models."

- killed all user models, restarted iop model, restarted user models, cleared all watchdogs and problem clears up.

I raised Alarm Levels for all LVEA Dust Monitors (numbers:  2,3,4,13, & 15) except for those which are still monitoring clean areas (Dust Monitor #'s 5,6 & 16).  Raised 0.3um levels to 1200 & 1500 counts (that's minor & major) & 0.5um levels to 600 & 800 counts.

I was not able to change the EX Dust Monitors (this is because they are new monitors).  I changed the EY dust monitors the same as the LVEA ones above.

Laser Status

• Output power:  29.8W
• FE Watchdog ACTIVE (HPO Watchdog still RED)
• PSL SYS STAT:  Looks FAIRLY normal ("VB Program Oline" & "LRA Out Of Range" are BOTH red)

PMC

• Locked for last 1.5+ days
• Reflected Power (5.8W) is NOT optimal and is higher than 10% of Total Power [25.5W] (Bad Alignment??)

FSS

• Locked for last 1.5+ days
• On video Refcav Refl looks like it may be off in yaw; Transmitted Power ("TPD [V]") = 1.14V (should be atleast 0.9V)

ISS

• Locked; AOM Diffracted Power is around 9% (should be around 5-15%)
• Last saturation event was 1.5+ days ago

Ran purge air compressors for ~10 minutes @ ~0945 hrs. local

The h1seib1 front end computer was locked up since Aug. 10, 18:46 PDT.  Restarted computer this morning at 08:35 PDT.  IRIG-B timing was bad until 08:39.  No other computers were glitched restarting this dolphin-connected front-end.

model restarts logged for Sun 10/Aug/2014
2014_08_10 08:32 h1fw1

unexpected restart of h1fw1, other writer was running, no data loss.

• laser safe--->BS Oplev work in morn
• laser hazard at ~10am
• HAM6/ISCT6 alignment work
• LVEA pumping stalled:  HAM2 annulur still needs to be done (need to torque down HAM1 doors)
• ETMy:  back to low pressure.  May have a broken ESD channel (will investigate); will need Richard due to High Voltage
• EX:  doesn't look like it's pumping down normally
• Tiltmeter:  tightening up and sealing mini chamber today
• EY:  May require a 3rd Discharge test
• Bring in (1 of 3) empty storage container from high bay in the morning (Apollo)
• EY B&K hammering (Tim Mcdonald)
• H2 enclosure work:  PD Arrays (Peter King)
• Fire Dept here today
• Couple of quads moved to 3ifo storage in LVEA

(Borja)

This entry is a summary of the manual measurements results of the ETMY charge done before Rai's ionized gas injection discharge run1 and run2.

The measurement technique is the standard procedure of injecting a sinusoidal signal of 4Hz and 91.5Volt in amplitude to each quadrant of the ESD (with the exception of quadrant LL because it did not respond to the excitation, more about this on another entry). Then we monitor the deflection of the ETMY both in Pitch and Yaw by looking at the oplev which was carefully centre before any measurement was taken. The deflection was measured in diaggui as power spectrum plot of pitch and yaw, with a BW = 0.01Hz on the range between 1 – 5 Hz and averaged of 2 (for the before measurements) and 3 (for the after discharge measurements) times.

During the measurements the coherence between ‘excitation and Pitch’ and ‘excitation and Yaw’ was monitored to be sure that the excitation was observed. The phase (in degrees) of the transfer function between excitation and oplev pitch and yaw was measured to take into account the sign in the deflection.

Attached to this entry I provide a document with the results for each measurement. Also I provide the combined plots of the results in the standard way “Normalised deflection [µrad/V] vs V BIAS” where the deflection is normalised by dividing the deflection with the excitation amplitude (91.5Volt). The plot contains quite a lot of information (well explained by reading the plot legends) so a zoom version is also added around the Veff values (or deflection zero crossing).

Next I show a table with the summary, providing for each quadrant values of the pitch and yaw slope and Veff.

Looking at the Veff in the table above we notice that the first discharge run was effective by a factor between 2 and 4 depending on quadrant and type of deflection (pitch or yaw). However the second discharge did not have so big effects and in some cases showed an increased charge.

(R. Weiss)

Rai has suggested that the results may suggest that the gate valve, in the second ionized gas discharge run, may have not been opened into a clear aperture even though the measurements of the ion currents were larger. It would be advisable to learn the best injection method for which It all needs to be done a third time. Rai thinks that the results so far are useful enough to decide whether we want to build more of the ionizers. Another very important step in the final decision will be the effect that the green laser light will have in the charging of the ETMY.

G.Moreno, K. Ryan, B, Sorazu, J. Worden, R. Weiss
Preliminary results from the first discharging indicate the charge has been reduced to 1/4 of the originally
measured charge. As we were dissatisfied by the negative to positive ion ratio and the flow rates in the initial
attempt, we made a second injection on August 8 with 1/2 the flow rate and about 3 times the ion current
into the test mass chamber. The charge on the etmy has not been measured yet. The new flow conditions are closer
to those in the original experiment done at MIT which reduced the charge to 0 +-10% of its initial value.
Again, once all the data is together, a more informative posting of the results will be made.