Peter F, Evan, Kiwamu,

ETMY was not damping today. It sometimes damps OK and sometimes does not. When it does not damp, it grows an oscillation in pitch at 0.5 Hz. Also, steering the suspension often kicked the suspension for some reason. After many hours of investigation, we finaly found that the DAC outputs show some discrete steps in every ~ 2000 counts or so. This, of course, made the actuators highly nonlinear. This was very sneaky -- one would not notice until he or she crossed a discrete point because the actuators are locally linear. The attached is a screen shot of StripTool showing the nonlinearlity. The purple curve is the requested DAC counts and the yellow one is an actual analog voltage observed by VOLTMON. As seen in the screen shot, the analog signal has multiple discrete steps. This behavior was seen at all 6 coils on the M0 (top) stage of ETMY and very repeatable. We didn't have energy to check the lower stages and therefore we don't know if this is true at the lower stages.

We must fix this issue.

Jeff, Calum, Ben, Rich

Today we got quite a bit of the controls checked out on the Y-end ESD.  We verified the functionality of the following:

1.  All Quadrant analog drive and readback signals associated with the HV and LV ESD are working.  The readback channels for the LV ESD were verified at the channel level, but are not yet incorporated into the MEDM screens.
2.  All 15 binary control and readback channels are functional

We ran into trouble on the Bias drive channel, which we troubleshot down to the DAC card.  A new DAC card (SUS DAC 4) will have to be installed to make this functional.  We also need to finish taking the transfer functions of the installed system.  Jeff has left us specific instructions for doing that in the morning.

We had 3 or 4 trips of both HV supplies while we were working, leading us to be suspicious of the vacuum interlocks associated with the HV ESD.  This has not yet been explored much, but will prove annoying if it persists as you cannot remotely reset this particular trip.  It could be associated with us relocating the interlock chassis within the SUS field rack, but the chassis in question seems simple enough to have tolerated a small move.

Here is a summary for the use of the manual control test fixture (D1500130) associated with the LV ESD amplifier:

Function -> State -> Meaning
HVLV Switch -> LED Lit -> The output drive mode is selected for LV operation
QuadrantSelect -> LED Lit -> PI Drive aligned to UR or UL
HV ESD ON/OFF -> LED Lit -> Not in reset or toggle mode ie. nothing happening in reset land
Input Relay -> LED Lit -> Input quadrant drive signal is not connected to the HV ESD amplifier
PZ Bypass -> LED Lit ->  The associated Pole-Zero stage in the LV quadrant drive chain is bypassed

Cheryl, Nutsinee

07:00 Richard in and out number of times CER

08:17 Richard taking all SUS in HAM2 to SAFE. Fixing MC3 issue.

9:12 HAM2 SUS restored. IMC locking.

10:30 Jeff B. to cleaning area

10:45 Jeff back

11:50 Rick & Nutsinee to LVEA

12:06 Rick & Nutsinee back

12:33 Jeff K. restarts ETMY SUS

15:00 Elli to LVEA (HWS table)

16:00 Nutsinee to LVEA (HWS table)

18:20 Nutsinee & Elli out.

SUS ETMY is not happy today.

Nutsinee, Elli

We ahve aligned ITMY HWS so that it gets a sled return beam.  Attached is a photo from the HWS with no Hartmann plate on.   The alignment was done with the X-arm locked. The beam looks clipped, but we spent a few hours today looking for clipping in both the outgoing sled beam and in the incoming ALS beam and this is the cleanest image we could get.  We made sure both beams are centered on all optics.  I have put the Hartmann plate on the camera, so that we can try to get a measurement using the HWS camera.

On Monday & Tuesday the operator had to respond to many HEPI Pressure alarms.  They did not appear real to me as they seem to be glitches and steps in the pressure, things that really don't happen.  At the time I associated it with all the electronics changes EE was doing.  I surmised the powering off and on as things were replaced was causing ground state shifts on the pump station controller in turn changing the pressure transducers response or offsets.

Checked the HEPI Platforms and they were off during the pressure transients so it was not the platforms de-isolating or re-isolating.  We'd have seen this before if it was that.

Attached are a few trend plots, but nothing is definitive.

The first attachment is the corner station.  I'm plotting the return and supply pressures at BSC2, the difference between them for the servo set point.  The VOUT controls the motor/pump speed and the last trace is the MAINSMON, something to do with the building power supply.  A few glitches occurred before lunch.   But at 1325 it appears the Return Pressure droops more than Supply increasing the differential pressure and the servo drops the speed of the pump to keep the DP correct and this continues down and then back up in a series of steps.  Further after lunch more steps are seen.  Interesting to see the MAINSMON go flat for a few hours (the data is there) and then glitch back to looking live just as the final pressure jump puts everything back to normal at 1740 on Tuesday.

The second attachment is for EndY but showing you fewer channels and including the HPI OUTF to show that the platform isolation is not the cause.  This story for EndY is similar to the corner station except it is pretty much a single step in the pressures and motor speed.  Again, everything jumps back to normal when the MainsMon comes back alive, hmm at 1740pdt.

For EndX, this occurred on Monday.  The picture isn't as clear (3rd attachment,) the sensors are much noisier at EndX and the MainsMon came back on a couple times.  The first step also occurs about the time Ken the electrician is at EndX for access system work (alog 18496.)  There is also correlation at one occurrence of the MainsMon coming back alive.

Can't say exactly what was causing these pressure glitches but the pump station really responds to them and had the platform been isolating, it would have been seen on them as the true differential pressure across the HEPI Actuator would change.

Bottom line, when Richard tells you it won't bother you or it isn't his fault, grab your salt shaker!

Sudarshan, Kiwamu, Rich, Peter F, Jeff, Jim, Dave:

h1psliss was updated from Duncan's latest svn version, which makes the ODC channel an 32 bit unsigned interger DAQ data type (7)

[H1:PSL-ODC_CHANNEL_OUT_DQ]

acquire=3

datarate=16384

datatype=7

Model changes were made to h1susetmy and h1sustmsy.

The routing of the Duotone signal out of the last channel of the first DAC on h1iopsusey was turned off.

The DAQ was restarted to sync it back up the the PSL and SUS ETMY changes.

- continuation of hardware upgrade, things starting to come back, like PSL

- starting to relock - X arm went ok, Y arm was still not locking at 3PM

- ETMY got new code, JeffK and others looking into that optics issues, i.e. arm not locking

- Nutsinee, Cheryl

The purge air system was turned off at the end of the day today.

Every ISI and all HAM HEPIs except HAM4 SDF had channels not found in their list.  From 28 to 1476 not found channels.  These were all alarm fields and related to old guarding channels, Sensor Correction scheme changes, and Oplev channels going away.  A total of 7192 lines to remove from the safe_alarms.snap files, see 18275.  Hopefully, this is the last of these issues popping up in SDF.

Elli, Aidan

HWS code v1.1.9 is installed on hws computers h1hwsmsr, h1hwsex and h1hwsey.  Updated symlink to HWS directory.  Changed aliases for the Run_HWS_H!I/ETMX/Y command.  (Refer to LLO alog 18291 for a lsit of the changes made.)

There is now a run-time error when running Run_HWS_H1ETMY and stream_image_Y on h1hwsey, and a run-time error running stream_image_X on h1hwsex, which we will work to resolve.

Filiberto, Peter, Jeff, Rich

Over the last few days, we have successfully installed the Low Noise ESD Driver (D1500129) and its associated field wiring.  This unit has 15 individual control lines:

4 bits are associated with switches to connect or disconnect the quadrant drive signals to the HV ESD amplifier that pass through the Low Noise Driver for control
4 bits are associated with switches to select or bypass the pole-zero stages in each quadrant drive path.
4 bits are used to select the output drive mode between High Voltage mode and Low Voltage mode for each quadrant.
2 bits are used to steer the two Parametric Instability (PI) drive inputs to any of the four available quadrants.
1 bit is used to enable or disable the High Voltage ESD amplifier

A wiring diagram for the entire ESD system is preliminarily available under D1400177-v4 pending the required DCN process.
Details of the Low Noise Driver circuitry are available under D1500129
The serial number of the installed chassis is S1500066.  The test results from manufacture are available at this serial number.
A test fixture has been supplied that allows manual operation of the low noise driver.  Details of the test fixture are available under D1500130

This afternoon, Jeff and I are planning to take detailed transfer functions of the driver for better filter compensation.  Jeff has also been working on the needed model updates and control screens.

Richard, Filiberto, Jim, Dave:

This week we have installed the modified 18bit DAC cards in most H1 Suspension IO Chassis. All suspension systems have been upgraded except for h1sush56 (we have exhausted the first batch of cards).

Full list of systems which use 18bit DACs which are not presently upgraded:

• h1sush56
• h1oaf0
• h1iscex
• h1iscey
• h1pemmmx
• h1pemmy

We have four modified cards which are showing errors and need further investigation:

• SN101208-05  Does not autocal
• SN101208-44  Stuck output voltage (was pulled from h1sush2a this morning)
• SN110425-19  Suspected bad capacitor on solder side, marked by Manufacturing Services
• SN101208-59  FE code failed to run when this card was installed in h1susex

All of the IO chassis except the mid station PEM Chassis.  have been upgraded to the new 24V DC power supplies.  Caltech sent up kits that fit in the chassis seamlessly.  At this time the only problem we have to report is one on/off switch did not function.  Easily replaced. Again  Thank you to Steve Vass and Bob Taylor for the help.

Monday thru Wed. We took the time to replace all of the AA/AI chassis with chassis that had the TI chip replaced with an AD8622 or AD8672.  In the end we had hoped to have all boards installed with AD8622 but have a number with the AD8672.  The only technical difference we care about is the amount of current the 8672 draws as it is higher than AD8622.  We will have chassis that run a little warmer than others.  The entire site has been changed out now.  This would not have been possible to accomplish in this time without the help of Steve Vass and Bob Taylor who came up to assist.  Thank you. 

Stuart, Greg, Dan, Jim, Dave:

Since the LDAS tape library was moved from the OSB Computer Users Room (CUR) to the Warehouse (WH) on Tuesday 12th May h1fw0 became unstable. This came as a surprise, and we have been working over the past week and a half to understand the problem and implement a solution.

When the Tape Library was in the CUR, it was in fiber optics connection with LDAS in the LSB via the multimode patch panel in the CUR. In the WH, the tape library was added to the H1 DAQ Q-Logic Fiber Channel Switches (FCS). At this point the tape traffic along with h1fw0's traffic are sent via single mode to the LSB. We think this additional data traffic on these switches is the root cause of h1fw0's instability problems.

Upon further investigation, data errors were seen on one of the FCS in the LSB, which correlate to frame errors on the LDAS SATABOY, which then correlate to h1fw0 restarts.

For the pair of FCS (one in MSR, one in LSB) which are showing the error, we tried using a different single mode link between the buildings and a different patch cable in the LSB. The error rate was unchanged.

When LDAS stop all access to the Tape Library, h1fw0 becomes stable again. So we know its the additional traffic which is expressing the core problem.

So where do we stand now? Dan has several single mode SFPs on order which will be here for next Tuesday maintenance. We plan on replacing the SFPs in the FCS pair to get back to where we wanted to be on 5/12. To get through this weekend, Dan has done the following:

For the FCS pair which the glitchy link, the single mode SFPs have been turned off. To permit packets from this FCS to get to LSB, a multimode fiber optics link was made between the MSR FCS.

This change was made at 15:40 PDT Wed 5/20 and the tape library was ramped up to full operation, h1fw0 has been stable since then (20 hours), so it looks like a good short term solution for this upcoming long weekend.

The h1sush2a computer was shutdown to allow a DAC card (D0) to be replaced in the I/O chassis, due to reports of a stuck voltage.  See alog 18543.

It was not possible to log in to this computer remotely, as it was not accepting new connections due to a "nf_conntrack: table full" error viewable on the console.  I was able to shut the computer down in an orderly manner from the console.  We have not encountered the nf_conntrack error in many months, but it appears when there's too many network connections being made.

The stuck voltage on D0 disappeared after the card was replaced and the computer restarted.

John, Bubba

The 3 IFO storage containers are under a constant nitrogen gas purge taken from the CP2 LN2 storage dewar boiloff.

The dewpoint sensors are showing some erratic behavior so Bubba and I investigated. We have determined that the pressure regulators supplied with each storage container are not operating correctly at such low inlet pressures (< a few psi ) and actually alternately shut the flow off and then as the pressure builds they again allow flow.  The glitches shown do not reflect the air quality in the storage can but only at the sensor itself - which is directly exposed to LVEA ambient conditions on it's exhaust port.

We will replace these regulators with simple rotameters in the next week or two.

(R. Weiss, B. Abbott, C. Torrie, J. Worden, B. Gateley, G. Moreno)

We assembled one TMDS*, and is currently being tested in the corner station mechanical room.
We are using dry air from the corner station purge air system to test the unit.
The corner station purge air will remain on until testing is complete.

*TMDS = Test Mass Discharging System.