WP6263 SUS model updates

Jeff, Jim, Dave:

Every H1 SUS model was rebuilt and restarted with a new DAQ configuration. Note that the HAM SUSAUX models were sped up from 2K to 4K to permit writing their channels at 4096 to the DAQ (h1susauxh2, h1susauxh34, h1susauxh56)

WP6264 add new DAQ diag channels to the DAQ

Jonathan, Dave:

I upgraded the H1EDCU_DAQ.ini file to add all of Jonathan's new daqd diag channels to the DAQ for trending. I have a python script to generate this file (create_edcu_daq_ini_file.py).

WP6260 add more memory to virtual machine server

Carlos:

Additional memory was added to cdsproxmox. Required restart of matlab server, fmcs ioc, dust monitor ioc, autoburt.

WP6273 Additional Beckhoff diagnostics channels

Daniel, Dave:

Beckhoff PLC code change on all three systems to create new diag channels.

WP6275 New NDS2 client code in control room

Jim B:

New nds2 client code installed for U12 and U14 workstations.

WP6272 DAQD code tests on h1fw2

Jonathan, Jim, Dave:

Testing new cpu affinity/priority code on h1fw2 in order to remove retransmission requests.

DAQ Restart

Dave:

Restarted the daq to support the above tasks.

model restarts logged for Mon 24/Oct/2016 No restarts reported

model restarts logged for Sun 23/Oct/2016
2016_10_23 09:21 h1susitmx
2016_10_23 09:26 h1susbs
2016_10_23 09:26 h1susitmy
2016_10_23 09:28 h1suspr3
2016_10_23 09:28 h1susprm
2016_10_23 09:30 h1susmc2
2016_10_23 09:30 h1suspr2
2016_10_23 09:32 h1sussrm
2016_10_23 09:33 h1isibs
2016_10_23 09:35 h1susetmy
2016_10_23 09:37 h1iscey
2016_10_23 09:37 h1susetmx
2016_10_23 09:39 h1alsex
2016_10_23 09:39 h1asc
2016_10_23 09:39 h1iscex
2016_10_23 09:40 h1ascimc
2016_10_23 09:42 h1lscaux
2016_10_23 09:42 h1lsc
2016_10_23 09:42 h1omc
2016_10_23 09:44 h1psliss
2016_10_23 09:59 h1sysecatc1plc2sdf
2016_10_23 09:59 h1sysecatx1plc2sdf
2016_10_23 10:01 h1sysecaty1plc2sdf
2016_10_23 11:33 h1alsex
2016_10_23 11:33 h1calex
2016_10_23 11:33 h1iopiscex
2016_10_23 11:33 h1iscex
2016_10_23 11:33 h1pemex
2016_10_23 11:34 h1susetmx
2016_10_23 11:36 h1hpietmx
2016_10_23 11:36 h1sustmsx
2016_10_23 11:39 h1isietmx
2016_10_23 11:41 h1iopseiex
2016_10_23 11:43 h1hpietmx
2016_10_23 11:43 h1iopsusex
2016_10_23 11:43 h1isietmx
2016_10_23 11:44 h1susetmx
2016_10_23 11:44 h1susetmxpi
2016_10_23 11:44 h1sustmsx

full /opt/rtcds disk system caused 08:00 - 10:00 restarts. h1iscex ran out of network connections which prompted the restart of all Dolphin'ed models at EX.

model restarts logged for Sat 22/Oct/2016
2016_10_22 18:24 h1fw0
2016_10_22 18:26 h1fw1

unexpected restart of h1fw0 appears to have caused h1fw1 to restart, resulting in loss of frames.

model restarts logged for Fri 21/Oct/2016
2016_10_21 16:25 h1fw0
2016_10_21 16:57 h1fw0
2016_10_21 18:53 h1fw0

unexpected fw0 restart prompted a preemptive power cycle of h1ldasgw0 (not shown). Did get a second unexpected fw0 restart later in the evening.

We recorded AI output of PCAL injection as well as PCAL_DAC_FILT thing together with witness GPS 1pps using Tektronics MSO4043 (1 sec with 1Msample per channel, and 2 sec with 1Msample per channel).

We also recorded the digital output of the PCAL injection as well as digital input of the PCAL_DAC_FILT thing.

Evan will look at the timing comparator data.

These will be analyzed in the near future for further timing sanity check.

As a follow on from alog 30790 where RM and OM vertical modes are of interest, this morning we measured them for LHO - they are in fact at 6.1Hz.  Attached are spectra of these modes for RM1, RM2, OM1, OM2, and OM3.  The spectra are with the HEPI not isolating, while the black ref trace on all polots is from earlier in the morning when the HEPI was isolating.  At a glance, we also note that while the OMs look relatively healthy, there is some fishyness on the RM spectra from the 6.1Hz v-mode up to ~20Hz seen on a few OSEMs.  OSEM open light voltages look ok, however.

Data can be found in

/ligo/svncommon/SusSVN/sus/trunk/HTTS/Common/Data/

2016-10-25_1924_H1SUSOM_PSD.png
2016_10_25_1924_H1SUSOM_PSD.xml
2016-10-25_1924_H1SUSRM_PSD.png
2016_10_25_1924_H1SUSRM_PSD.xml

Compressors #3, #4, and #5 were greased and pressure tested.

Compression test findings: #3 at 120 psi, #4 at 130 psi, and #5 at 110 psi.
All compressors and electrical motors were greased.

All relief valves passed their test.

All compressor assemblies were run tested after service was performed.

Work performed under WP#6277.

WP 6267

No accumulator needed charging and differences from June suggest none have leaks.  T1500280 updated.  Famis 4594 closed.

J. Kissel

While browsing the new MEDM overview screens from Stuart (updated at LHO earlier today LHO aLOG 30844), I noticed that the ODC lights for the QUADs were red, because the top mass, main chain damping loops claimed to be in a bad state. 

The ODC bit indicating the status is informed by the comparison between a "known good state" as defined by EPICs records entered in by-hand (e.g. H1:SUS-ETMX_M0_DAMP_P_STATE_GOOD), and the current state as reported by the front end (e.g. H1:SUS-ETMX_M0_DAMP_P_STATE_NOW). Both are shown on the right-and-side of the DAMP screen (see second attachment). These "known good state" values were probably not updated when Rana and Evan had made changes to the QUAD damping loops back in May of 2016 (LHO aLOG 27464). 

I've now updated the good values, the comparator lights are green, all ODC lights are green, and I've accepted the new good values into the SDF system. 

Lot's of redundancy there, but it's just indicative of all three generations of state definition control that the SUS have seen that haven't been cleaned up, de-scoped, or standardized.

Andy, Jess, Josh, TJ

Following up alog 30790 and comments (range drop may be related to excess RM motion), we've found some very strange motion with apparently quantized jumps in frequency. This motion is seen in accelerometers near HAM1, in the nearby HAM HEPI L4Cs, in the RM mirror motion, and in some REFL and POP DC signals. Some of this motion seems to show up in DARM as well.

The first plot is a zoom in frequency on the kind of motion that we see in these channels. It looks like several lines which sometimes jump suddenly in frequency, then jump to another frequency, then another. It looks like a MIDI music file (or a music box). We've found this in many channels, but the clearest so far is in the HAM1 floor accelerometer. It's also in the ISCT1 accelerometer, and the RM OSEMs see it in some degrees of freedom (which is how we first noticed it). The HAM1 and HAM2 HEPI L4Cs see some of it, see second and third plots. Even the HAM6 accelerometer sees some of it, so it's not local just to HAM1/2 - but we haven't checked exactly how widespread it it. We've also checked at least in the HEPI that this was there at an earlier time in the day, and also two weeks prior.

The last plot is DARM, showing that this seems to couple at least in the 10 to 20 Hz region. That could be through the RMs somehow, or maybe through scatter from ISCT1. Since the beam diverter was closed (alog 30835), the next locks can check if this is through ISCT1.

What's causing this motion? It looks really peculiar. It's hard to pick it out in just a few minutes of data, because the lines are narrow and don't wander - they jump suddenly. So it may have gone un-noticed before, but it would be nice to understand it even if the coupling to DARM is easily fixed.

Jason Daniel (WP 6273)

We added a whitening filter to the PMC HV monitor in the PMC fieldbox, D1001619. The changes are:

• N12: Add OP27
• R19 & R21: Add 158K
• R22: Add 1.58K
• C7: Add 1u
• R20 & C8: Leave open
• J4: Remove jumper

This generates a whitening filter with a zero at 1 Hz, a pole at 100 Hz and a DC gain of –1. The inverse of this filter has been added to the corresponding PSL PMC filter bank, so that the slow readbacks are unchanged.

The attached spectrum shows the PZT high voltage which is 50 times the monitor readback and has an additional pole at 770 Hz which is formed by the output series resistor, see alog 30729. The noise level of the unwhitened HV monitor was at 0.2 mV/√Hz before the change, see alog 30648. The signal  is at least a factor of 10 above ADC noise at all frequencies and is coherent with the temporary channel hooked up to EXTRA_AI_1. The later is no longer needed.

J. Kissel

Integration Issues: 6463, 4694
ECRs: E1600316, E1600028
WP: 6263

I've completed the changes begun this past Friday (LHO aLOG 30728) and Monday (LHO aLOG 30821) by installing and restarting all SUS front-end models this morning. Again, changes are 
(1) the removal of now-redundant ISI_WIT channel path which projected ISI GS13s to the suspension point basis. These are now computed elsewhere.
(2) the revamp of the SUS science frame channel storage; updated to match T1600432
Regarding (1), I've also updated each SUS-type's MEDM overview screens to remove these paths (just an SVN update; thanks Stuart!), and I've resaved and reloaded all SUS safe.snaps in the SDF system to remove these channels which were registering as are no longer found.

We need only a DAQ restart, and then (2) takes affect. 

@Detchar: Please let us know if your down-stream / off-site software has been adversely affected (once maintenance is complete, of course).

WP 6275

Updated NDS2 client software package to nds2-client-0.13.0 for Ubuntu 12 and Ubuntu 14 control room workstations.

This update supports (WP 6259):

• Additional channels to monitor the health of the EtherCAT data transfer (frame rates, lost frames, etc)
• Improved error handling, so that end station ALS don't report an error when locked on red.

Distribution of hours at which scratchy glitches occurred according to the ML output from GravitySpy. In addition, a histogram of amount of O1 time spend in analysis ready mode is provided. I have uploaded omega scans and FFT spectrograms of what Scratchy glitch looked like in O1.

We repeatedly noticed that the current front-end calibration is slightly off - tonight all cal lines (low and high freq) in DARM were 11% above the PCAL read-back.

If I take the PCAL readback as reference and scale down the calibrated spectrum (as attached), I got about 80Mpc.

On the other hand, Evan Goetz reported that he thinks the PCAL is clipping (30827). We'll see whether these 11% are real...

[Stefan, Jenne]

We removed the PR2 length feedforward that removes the MICH signal in PRCL.  We did this by ramping the PR2 LOCK_L gain at the lowest stage to 0.  We didn't see any change in DARM.  We also tried increasing the gain by a factor of 3.  Again, we didn't see any change in DARM. 

However, since we discovered and mitigated some scattering effects earlier tonight (but after this PR2 test), we should try this again. 

TITLE: 10/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Locking DRMI/PRMI was not easy, it required large adjustments and waiting a good amount of time. useism is also getting pretty high so I tried the WINDY_USEISM state, brought it back to WINDY because I couldnt tell which was better. Aside form that the commissioners are working.

 

Summary:
The Pcal Y laser beam is likely clipping somewhere in the beam path. This will need to be addressed ASAP. In the future we need to keep a close eye on the Pcal summary spectra on the DetChar web pages.

Details:
Jeff K. and I noticed that the spectrum for the Y-end Pcal seemed particularly noisy. I plotted some TX and RX PD channels at different times since Oct. 11. Several days since Oct. 11, the Pcal team has been to EY to perform some Pcal maintenance. One of those times (I think Oct. 18, but we don't have an aLOG reference for this), we realigned the beams on the test mass. Potentially, this change caused some clipping.

Attached are the spectra for TX and RX. Notice that there are no dramatic changes in the TX spectra. In the RX spectra, there is structure becoming more apparent with time in the 15-30 Hz region and 90-140 Hz. Also, various other peaks are growing

Also attached is a minute trend of the TX and RX PD mean values. On Oct 18, after realignment (the step down), the RX PD starts to drift downward while the TX PD power holds steady. The decrease in RX PD is nearly 10% from the start of the realignment. 

The Pcal team should address this ASAP, hopefully during tomorrow's maintenance time.

At 5:20am local time we saw a significant range drop (from about 70Mpc to 60Mpc) that seems to be due to a signicant increase of the line structure in the bucket that always lingers around the noise floor.

Attached are two spectra - 1h12min apart (from 11:18:00 and 12:20:00 UTC on 2016/10/24), showing that structure clearly.

Plot two shows the seismic BLRMS from the three LEAs - the corner shows the clearest increase. We are now chasing any particularly bad correlations around 12:20 thgis morning in the hope that it will give us a hint where this scatter is from.