Jim, Dave:
We are seeing a DAC card in h1susey
which passes autocal on power up, but subsequently fails autocals if the IOP model is restarted. This is the same card (fifth card of five) which stopped driving its output to the ESD (see Richard's alog earlier this afternoon).
To check if autocal failure and lack of output are related, we used the card which always fails autocal (SN 101208-05
) and looked at its output. In summary, its output drive is good and not related to its autocal success.
Details: DAC is the only card in x1susex
. h1iopsuseywdt is the IOP model, it has a SWWD which required reset of its DACKILL part. A new user model called h1cds18bitdactest.mdl was created (DCUID=100) which drives all eight DAC channels using filtermodules. The DAC was connected to the DTS 18bit DAC AI chassis (lower eight channels) and a breakout board and DVM used to determine the output DC voltage. Voltages were driven by putting an offset on the filtermodules.
07:15 Richard and Peter to end Y 08:13 Sudarshan to end X to pick up PCAL equipment 08:56 TJ misaligning SR3 for brief period 08:57 Rich, Calum, Ben to end Y, ESD 08:58 Richard to end Y 09:17 Sudarshan, Darhan to end Y, PCAL calibration 09:18 Hugh moving BS HEPI 10:08 Richard to end Y 10:12 Nutsinee to HAM1 HWS table 12:23 Richard to end Y 12:47 Richard back Relocking got as far as CHECK_IR. Further progress has been interrupted by an earthquake.
My logs from LHO 18386 & 18468 showed some 'after' elevated noise on the CPS. Mainly on ITMX Stage2 H2, and now that I look again, the ITMX Stage1 V1 noise is a little elevated. The first look at these in the first log, was during the day. When I looked during the long lock stretch last Thursday, second log, I didn't explicitly look at the ITMX ST2 H2 but I said this channel looked good during the lock stretch. However, ST1 V1 on ITMX is elevated above about 25hz where the traces flatten--see attached. The ST2 V2 also is elevated but the Horizontals generally look fine.
The second attachment shows just the Corner 2 CPS and they look good.
Josh, Andy, TJ, Duncan, Detchar,
In 17791, Dan followed up the observation that COMM control was saturating by doing the most awesome thing - writing a script to identify all times when software limits are reached, by searching for all enabled software limits and then scanning data from those channels to see if 99% of the limit value was exceeded in a given time range.
I asked Duncan Macleod if he could optimize this code and he did. Dmac said: I have a working version that uses multiprocessing and is significantly faster than the original, and I’m pretty sure returns the same result. I have also added functionality to record the segments during which a given channel was saturating its software limit, and write them all to a LIGO_LW XML file.
To test this code, Duncan ran it over the long locked stretch from roughly 9-23 UTC on 5/17/15 (range plot shown in attachment 1). The output XML is attached below.
The primary limit saturations found were, a bunch near the end of the lock (probably after lock loss) and two in the midde of the lock: 1115927164.94, from H1:SUS-ITMX_L2_OLDAMP_P and 1115927158.19 from H1:PSL-ISS_DIGITAL_LOOP. Attachment 3 shows a normalized spectrogram of DARM during the saturation, 4 shows the timeseries of DARM, 5 shows timeseries for H1:PSL-ISS_DIGITAL_LOOP, 6 shows timeseries for the PSL PDB diode first loop intensity stabilization diode, 7 shows a saturation in H1:SUS-ITMX_L2_OLDAMP_P at nearly the same time (which is real, but doesn't have a big noticeable effect in DARM), 8 is a saturation counter from the PSL that finds this time equally well.
We plan to run this script for each lock stretch from now on to produce DQ segments and identify times when channels are hitting their software limits.
The Y arm had not locked since the new 18bit DACs were installed. One problem was a bad DAC. DAC 0 was not giving linear outputs. I have replaced this DAC. The other problem was channel 0 on DAC4 which would not output any signal. After a reboot/IO chassis power down the problem with DAC4 was gone. We then had to replace DAC0 which seems to have fixed the problem.
While installing the Low Voltage ESD drive chassis at EY. Rich A. needed to relocate the ESD pressure sensor interface chassis which required powering it down. This chassis provides the power to the combination pirani/Cold Cathode(CC)gauge that is used for the interlock on the ESD High Voltage(HV) Supplies. After powering the gauge back up the cold-cathode portion of the gauge has not restarted most likely due to the excellent vacuum at Ey. The pirani is reading 10-5 which is our cutoff point for allowing the HV supplies to operate thus preventing the HV supplies from working. Since the vacuum has been quite steady John Worden gave me permission to by pass this interlock until Tuesday. Hopefully the gauge will be on by then or I will connect my computer to it and try and for on the CC. I have put jumpers on pins 1-6 and 3-7 on the connector on the safety relay box at the power supply rack to allow the ESD to operate. Work permit 5225.
Richard, Jim, Dave:
there have been some swapping of modified 18bit DAC cards as problems were found during this week's upgrade, I wanted to give a summary of what happened and where we currently stand.
Prior to this week, 5 DACs (chosen at random) were tested in the DTS. It was found that card 101208-05 fails autocal, which it did prior to the upgrade, and has remained in the DTS throughout this week.
Monday, in h1susex, it was found that if card 101208-59 was in the chassis the code would not run. This card is waiting for further tests and not in any system.
Tuesday we installed EY and corner station. We thought we had enough cards to update all BSC and HAM in the corner station if we skipped h1sush2b (two cards) and we used all five cards from the DTS. This would entail using the card which fails autocal, but Jeff and Dan said that could be used for the OMC. But, we found that one of the new cards was flagged with a potential capacitor problem (card 110425-19) so we abandoned that plan and decided instead to not update h1sush56 (five cards) but instead update h1sush34 (six cards) and h1sush2b (two cards). This meant taking three cards from the DTS, leaving it with two (one good, one bad-autocal).
On Thursday we found a bad channel on a DAC in h1sush2a (stuck output). This card (101208-44) was pulled from h1sush2a and the one remaining good card from the DTS was used to replace it. The stuck output card is waiting further tests and not in any system.
On Friday we discovered a non-linear output problem with the first card in h1susey (card 101208-78). The suspect capacitor card was inspected and then used to replace it in h1susey. The non-linear cad is waiting further tests and not in any system.
So five cards with issues, they are
101208-59 | code wont run with this card installed | in hand waiting testing |
101208-05 | fails autocal | being tested in DTS x1susex |
110425-19 | suspect capacitor | running in h1susey |
101208-78 | non-linear output | in hand waiting testing |
101208-44 | stuck output voltage | in hand waiting testing |
Scott L. Ed P. Chris S. 5/21/15 Cleaned 9 meters and finished at HNW-4-058. Removed lights and rehung them in next section north. Vacuumed support tubes and sprayed diluted bleach/water solution on heavily soiled floor areas. 5/22/15 Scott L. & Ed P. Cleaned 2 of the large box fans for circulation in the tunnel and gave the vacuum machines a complete cleaning. The crew left at 11:00 A M today and will take the entire week of 5/25 off. We will resume cleaning operations on 6/1. To date, a total of 3127 meters of tube have been cleaned and all support tubes in that length have been cleaned and caps installed.
I looked at the following channels: H1:IOP-SUS_EX_ADC_DT_OUT_DQ H1:IOP-SUS_EY_ADC_DT_OUT_DQ H1:IOP-LSC0_ADC_DT_OUT_DQ H1:IOP-SUS_EX_DAC_DT_OUT_DQ H1:IOP-SUS_EY_DAC_DT_OUT_DQ H1:IOP-LSC0_DAC_DT_OUT_DQ Attached are 6 figures, broadband and close-ups to the 960/961Hz signals. Notable features: 1) The EY spectra don't look right at all. The doutone signal is missing (SUSEYcloseup.pdf) and the spectrum has a positive slope (SUSEY.pdf). 2) All of the plots show a comb of 1Hz harmonics. Probably not surprising. 3) All the DAC channels (in blue) show extra noise (see all the non-closeup).
Dave Barker informed me that there shouldn't be any signals on the SUS_EX/EY_DAC_DT_OUT_DQ channels so that the absence of a duotone signal in EY is fine. In fact we turned off the EX signal. Now the spectra for the EX/EY ADC/DAC look very similar so there's no reason to be alarmed. We still don't understand why (now both EX and EY) DAC signals show a positive slope.
We have a 0.4mm vertical drive on the BS HEPI Vertical Actuators. Long story but with the HEPI loops open, this platform sags low enough that the IPS are no longer in their linear range and behave poorly giving us tilt coupling. Our pump station system pressure still has some coherence with the BS platform so I wanted to check if this large drive on the BS Vertical was increasing that coherence.
Bottom line, I'm not sure. I took a reference coherence look and then lowered the vertical on the hepi loops by 0.2mm and took another set of data. Attached are the results. The reference traces are the normal offsets and the current plots are with the offsets reduced. I'm looking at the T240s on the ISI stage1. Conclusion, ehh, it all just looks noisy, some places better some worse. There is a band in the X dof that seems broadly better from 10 to 25mHz but I'm not sure if it is meaningful. I'll look at the HEPI platform channels rather than just the ISI channels too and maybe I need to drop it the full 0.4mm to really see an effect.
Ben, Calum, and Gerardo, We received the two Gas Delivery systems, and two TMDS units (S1500093 and S1500094). We assembled the remainder of the table, and then mounted the Gas Delivery system. Lacking the final mounts, we put the TMDS on a screw jack so we could easily connect it to the Gas Delivery system. Once all hooked up, we attached the electronics, and put both TMDS systems through their paces, and tested them for ion production. We had two issues with S1500094 that we fixed. The first issue was that we accidentally swapped the cables going to the Baratron and Pirani gauges, which we believe damaged the Pirani gauge somehow. The second was that the cutoff valve leading to the vacuum pump would not seal completely. Gerardo took the whole thing to the clean room removed the suspect parts and replaced them with new ones sent from Caltech. Both new parts were found to work fine. The valve ended up having a cut and frayed Viton o-ring, which explained its problem. We are going to get an RMA for all things, and see what we can have done. Currently, the two TMDSs are in perfect working order, and have been fully checked out. We shipped the TMDS Interface back to caltech for further refinement, and so I can continue testing the units that we have there. Pictures of the system, and videos showing the operation procedures can be found at: https://dcc.ligo.org/LIGO-E1500252 These instructions are in support of, and superseded by information found in Rai's procedure document here: https://dcc.ligo.org/LIGO-T1400497 Test data collected from all tested units will go into the DCC next week. More information, and drawings can be found here: Gas Delivery System drawing: https://dcc.ligo.org/LIGO-D1500078 TMDS drawing: https://dcc.ligo.org/LIGO-D1400331
Peter, Calum, Ben, Rich Took transfer functions (1Hz to 10kHz, 255 points in to the DAC Drive Input on the front panel of the ESD driver, out of the respective SHV connector going to the ETM) of all the quadrant paths of the newly installed LN Driver. The results are stored in: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Measurements/ElectronicsMeasurements/ under the following file suffixes UR -> TFSR785_22-05-2015_102054.txt LR -> TFSR785_22-05-2015_102804.txt UL -> TFSR785_22-05-2015_103546.txt LL -> TFSR785_22-05-2015_105053.txt Everything looks good as compared to the Spice model. Also, we measured the output noise of each channel at 20Hz with and without DC bias. Results are all consistent with the design (<40nV/rtHz at 20Hz and above) and Spice model. 0V bias/-7.4V bias UR -> 21nV/rtHz/32nV/rtHz LR -> 22nV/rtHz/30nV/rtHz UL -> 23nV/rtHz/37nV/rtHz LL -> 28nV/rtHz/37nV/rtHz Analyzer Noise Floor 10nV/rtHz @ 20Hz Typical output noise at higher frequencies is: 20nV/rtHz @ 50Hz, 22nV/rtHz @ 100Hz, 20nV/rtHz @ 150Hz Noticed that there is still a discrepancy in the DAC monitors for the user model vs. the IOP. Richard has the details and Jeff will likely follow up next week. Once Richard sorts out the somewhat touchy vacuum interlock (he's working on that now and knows the problem), the entire system is now functional and ready for use.
Plot attached.
For the purposes of figuring out the appropriate compensation for DARM, I also plotted a simple model which just contains the effect of the LPF stage (2 poles at 2.2 Hz, 2 zeros at 50 Hz) and the PI summing node (pole at 152 Hz, zero at 3.2 kHz). The effect of the summing node is not yet digitally compensated, so that could explain the loss of phase that we saw in the DARM OLTF last night (about 50° at 200 Hz).
I logged into H1HWSEY and restarted the EPICS IOC. A new channel (H1:TCS-ETMY_HWS_FREE_DISK_FRAC) was added in v1.1.8 of the HWS code and the IOC had to be restarted to incorporate it.
The DAQ EDCU connected to this channel and is now GREEN.
Note to operations staff: during ER7 please investigate if the EDCU is not GREEN.
I was asked to add to the SUS Guardian the ability to turn off the DarmDamp filter block outputs when the suspension goes into SAFE and turn them back on when the other outputs are turned back on (ENGAGE_DAMPING). To do this I added three methods in sustools2.py in the same style as the rest of the methods for filter blocks, and the appropriate dictionary values to susData. These filter banks are located on the quads in the M0 stage and act on the V,R degrees of freedom.
Added methods are:
# Methods for DARM_DAMP blocks
def darmDampPvs(self, levels=[], chans=[], suffix='', verbose=False, withprefix='bare', matlab=False):
return self.levelfilterblockpvs('darmdamp', levels=levels, chans=chans, suffix=suffix, verbose=verbose, withprefix=withprefix, matlab=matlab)
def darmDampOutputSwitchWrite(self, enable, levels=[], chans=[], verbose=False, pair='none', withprefix='bare', matlab=False):
return self.genSwitchWrite(self.darmDampPvs, 'OUTPUT', enable=enable, levels=levels, chans=chans, verbose=verbose, pair=pair, withprefix=withprefix, matlab=matlab)
def darmDampOutputSwitchRead(self, levels=[], chans=[], verbose=False, pair='value', withprefix='bare', matlab=False):
return self.genSwitchRead(self.darmDampPvs, 'OUTPUT', levels=levels, chans=chans, verbose=verbose, pair=pair, withprefix=withprefix, matlab=matlab)
I tested the new code on ITMY, and SR3 just in case, and it all seems to work well. The new code is committed to the svn and loaded into all of the quad guardians.
Richard, Jim The h1susey computer will be unusable for a while as we investigate 18 bit DAC problems. Initial investigation shows that D4 now fails to do it's auto calibration, where it initially succeeded when the computer was first started.
Richard replaced DAC D0, Kiwamu tested the problem channel and the effect noted in alog 18569 is not present.
Jim W.: work on FF and blend filters on HAMs 4 and 5 Hugh: possible check of vertical offset on BS HEPI Richard, Jim B: investigate problem with ETMY SUS DAC Richard: investigate ETMX ESD drive Jason: PSL alignment Sudarshan: end Y PCAL calibration Apollo: beam tube cleaning
Per Jeff Kissel's request Betsy and I (Xavi) looked at several SYS subsystem channels to perform timing checks on the IO chassis. These checks were in slides by Shivaraj and the slides should be added to this alog and references to the slides added. The channels we looked at (shown in the attached green medm screen shot) are: H1:SYS-TIMING_C_MA_A_PORT_2_SLAVE_CFC_TIMEDIFF_1 (Master1) H1:SYS-TIMING_C_MA_A_PORT_2_SLAVE_CFC_TIMEDIFF_2 (Master2) H1:SYS-TIMING_Y_FO_A_PORT_9_SLAVE_CFC_TIMEDIFF_1 (Y-arm) H1:SYS-TIMING_X_FO_A_PORT_9_SLAVE_CFC_TIMEDIFF_1 (X-arm) Two of them are in the Master, one in the y-arm, and the third in the x-arm. It does not appear that any other channels are active at this time. We took a week of data (in the past of today) for each of these channels and attach figures that show the time series (to check for trends) as well as histograms. All in all these plots look reassuring. Notable features: 1) The timing differences between adjacent points are integer multiples of 7.45ns. Mostly the integer is 1 (see Master1TimeseriesDiff.jpg, only posted it for Master 1, others look the same). 2) There do not appear to be significant drifts (e.g. < 50ns for 1 week in Master 1) 3) The spread is about 50ns (shown in figures labeled Histograms) 4) there is ~1 day periodicity in the time series for Master 2 5) There seem to be common glitches (small ones at the level of ~0.25 us, e.g. see all time series toward the end) Betsy and Xavi
For Livingston I was able to look at the following channels: L1:SYS-TIMING_C_MA_A_PORT_2_SLAVE_CFC_TIMEDIFF_1 (Master1) L1:SYS-TIMING_C_MA_A_PORT_2_SLAVE_CFC_TIMEDIFF_2 (Master2) L1:SYS-TIMING_Y_FO_A_PORT_9_SLAVE_CFC_TIMEDIFF_1 (Y-arm) L1:SYS-TIMING_X_FO_A_PORT_9_SLAVE_CFC_TIMEDIFF_1 (X-arm) The channels exists but the time series are populated with zeros. X