Kiwamu, Peter, Dan, Evan
We made it back to dc readout tonight. 3 W PSL power, DARM feedback to ETMX.
We are in the process of trying to engineer the new transition to ETMY. We were hampered by the intrusion of a large amount of 1/f3 noise in DARM, with ASD 1×10−14 m/Hz1/2 at 10 Hz. The initial portion of the lock was fine, and then the noise came. We trended various channels and looked at coherences, but could not nail down the cause. The noisy time starts at 2015-05-23 9:41:00 UTC. After about an hour, the noise went away again.
The ASC was a bit touchy. It could be engaged by hand, but with the guardian it seemed that engaging the PRM pointing loop caused several of the other loops to go unstable. Also, we gave up on the ITM pointing loops. They increased the recycling gain, but made the arm and sideband buildups less stable.
We found that there was no tidal signal being sent to ETMY. At first blush it appears to be an IPC issue.
Following Xavier's timing measurements, the following FECs have their DAC Duotone* still turned on:
During parts of this week this was also turned on for h1susex and h1susey. These have been turned off as these DAC channels are used to drive the R0_COIL_RT signal. For the systems listed above, neither the DAC channel nor its ADC readback counterpart are being used by the models (PCAL reads the ADC channel but only archives it to DAQ).
* - reminder: if the DAC Duotone drive is enabled (via the GDS_TP MEDM screen for the IOP model), relays in the IO Chassis interface cards for the first DAC and the first ADC route the output of the last channel of the first DAC to the second-to-last channel of the first ADC. The IOP model takes the Duotone (digitized from the last channel of the first ADC) and writes this to the last channel of the first DAC.
Jim, Dave:
We are seeing a DAC card in h1susey
which passes autocal on power up, but subsequently fails autocals if the IOP model is restarted. This is the same card (fifth card of five) which stopped driving its output to the ESD (see Richard's alog earlier this afternoon).
To check if autocal failure and lack of output are related, we used the card which always fails autocal (SN 101208-05
) and looked at its output. In summary, its output drive is good and not related to its autocal success.
Details: DAC is the only card in x1susex
. h1iopsuseywdt is the IOP model, it has a SWWD which required reset of its DACKILL part. A new user model called h1cds18bitdactest.mdl was created (DCUID=100) which drives all eight DAC channels using filtermodules. The DAC was connected to the DTS 18bit DAC AI chassis (lower eight channels) and a breakout board and DVM used to determine the output DC voltage. Voltages were driven by putting an offset on the filtermodules.
07:15 Richard and Peter to end Y 08:13 Sudarshan to end X to pick up PCAL equipment 08:56 TJ misaligning SR3 for brief period 08:57 Rich, Calum, Ben to end Y, ESD 08:58 Richard to end Y 09:17 Sudarshan, Darhan to end Y, PCAL calibration 09:18 Hugh moving BS HEPI 10:08 Richard to end Y 10:12 Nutsinee to HAM1 HWS table 12:23 Richard to end Y 12:47 Richard back Relocking got as far as CHECK_IR. Further progress has been interrupted by an earthquake.
My logs from LHO 18386 & 18468 showed some 'after' elevated noise on the CPS. Mainly on ITMX Stage2 H2, and now that I look again, the ITMX Stage1 V1 noise is a little elevated. The first look at these in the first log, was during the day. When I looked during the long lock stretch last Thursday, second log, I didn't explicitly look at the ITMX ST2 H2 but I said this channel looked good during the lock stretch. However, ST1 V1 on ITMX is elevated above about 25hz where the traces flatten--see attached. The ST2 V2 also is elevated but the Horizontals generally look fine.
The second attachment shows just the Corner 2 CPS and they look good.
Josh, Andy, TJ, Duncan, Detchar,
In 17791, Dan followed up the observation that COMM control was saturating by doing the most awesome thing - writing a script to identify all times when software limits are reached, by searching for all enabled software limits and then scanning data from those channels to see if 99% of the limit value was exceeded in a given time range.
I asked Duncan Macleod if he could optimize this code and he did. Dmac said: I have a working version that uses multiprocessing and is significantly faster than the original, and I’m pretty sure returns the same result. I have also added functionality to record the segments during which a given channel was saturating its software limit, and write them all to a LIGO_LW XML file.
To test this code, Duncan ran it over the long locked stretch from roughly 9-23 UTC on 5/17/15 (range plot shown in attachment 1). The output XML is attached below.
The primary limit saturations found were, a bunch near the end of the lock (probably after lock loss) and two in the midde of the lock: 1115927164.94, from H1:SUS-ITMX_L2_OLDAMP_P and 1115927158.19 from H1:PSL-ISS_DIGITAL_LOOP. Attachment 3 shows a normalized spectrogram of DARM during the saturation, 4 shows the timeseries of DARM, 5 shows timeseries for H1:PSL-ISS_DIGITAL_LOOP, 6 shows timeseries for the PSL PDB diode first loop intensity stabilization diode, 7 shows a saturation in H1:SUS-ITMX_L2_OLDAMP_P at nearly the same time (which is real, but doesn't have a big noticeable effect in DARM), 8 is a saturation counter from the PSL that finds this time equally well.
We plan to run this script for each lock stretch from now on to produce DQ segments and identify times when channels are hitting their software limits.
The Y arm had not locked since the new 18bit DACs were installed. One problem was a bad DAC. DAC 0 was not giving linear outputs. I have replaced this DAC. The other problem was channel 0 on DAC4 which would not output any signal. After a reboot/IO chassis power down the problem with DAC4 was gone. We then had to replace DAC0 which seems to have fixed the problem.
While installing the Low Voltage ESD drive chassis at EY. Rich A. needed to relocate the ESD pressure sensor interface chassis which required powering it down. This chassis provides the power to the combination pirani/Cold Cathode(CC)gauge that is used for the interlock on the ESD High Voltage(HV) Supplies. After powering the gauge back up the cold-cathode portion of the gauge has not restarted most likely due to the excellent vacuum at Ey. The pirani is reading 10-5 which is our cutoff point for allowing the HV supplies to operate thus preventing the HV supplies from working. Since the vacuum has been quite steady John Worden gave me permission to by pass this interlock until Tuesday. Hopefully the gauge will be on by then or I will connect my computer to it and try and for on the CC. I have put jumpers on pins 1-6 and 3-7 on the connector on the safety relay box at the power supply rack to allow the ESD to operate. Work permit 5225.
Richard, Jim, Dave:
there have been some swapping of modified 18bit DAC cards as problems were found during this week's upgrade, I wanted to give a summary of what happened and where we currently stand.
Prior to this week, 5 DACs (chosen at random) were tested in the DTS. It was found that card 101208-05 fails autocal, which it did prior to the upgrade, and has remained in the DTS throughout this week.
Monday, in h1susex, it was found that if card 101208-59 was in the chassis the code would not run. This card is waiting for further tests and not in any system.
Tuesday we installed EY and corner station. We thought we had enough cards to update all BSC and HAM in the corner station if we skipped h1sush2b (two cards) and we used all five cards from the DTS. This would entail using the card which fails autocal, but Jeff and Dan said that could be used for the OMC. But, we found that one of the new cards was flagged with a potential capacitor problem (card 110425-19) so we abandoned that plan and decided instead to not update h1sush56 (five cards) but instead update h1sush34 (six cards) and h1sush2b (two cards). This meant taking three cards from the DTS, leaving it with two (one good, one bad-autocal).
On Thursday we found a bad channel on a DAC in h1sush2a (stuck output). This card (101208-44) was pulled from h1sush2a and the one remaining good card from the DTS was used to replace it. The stuck output card is waiting further tests and not in any system.
On Friday we discovered a non-linear output problem with the first card in h1susey (card 101208-78). The suspect capacitor card was inspected and then used to replace it in h1susey. The non-linear cad is waiting further tests and not in any system.
So five cards with issues, they are
101208-59 | code wont run with this card installed | in hand waiting testing |
101208-05 | fails autocal | being tested in DTS x1susex |
110425-19 | suspect capacitor | running in h1susey |
101208-78 | non-linear output | in hand waiting testing |
101208-44 | stuck output voltage | in hand waiting testing |
Scott L. Ed P. Chris S. 5/21/15 Cleaned 9 meters and finished at HNW-4-058. Removed lights and rehung them in next section north. Vacuumed support tubes and sprayed diluted bleach/water solution on heavily soiled floor areas. 5/22/15 Scott L. & Ed P. Cleaned 2 of the large box fans for circulation in the tunnel and gave the vacuum machines a complete cleaning. The crew left at 11:00 A M today and will take the entire week of 5/25 off. We will resume cleaning operations on 6/1. To date, a total of 3127 meters of tube have been cleaned and all support tubes in that length have been cleaned and caps installed.
I looked at the following channels: H1:IOP-SUS_EX_ADC_DT_OUT_DQ H1:IOP-SUS_EY_ADC_DT_OUT_DQ H1:IOP-LSC0_ADC_DT_OUT_DQ H1:IOP-SUS_EX_DAC_DT_OUT_DQ H1:IOP-SUS_EY_DAC_DT_OUT_DQ H1:IOP-LSC0_DAC_DT_OUT_DQ Attached are 6 figures, broadband and close-ups to the 960/961Hz signals. Notable features: 1) The EY spectra don't look right at all. The doutone signal is missing (SUSEYcloseup.pdf) and the spectrum has a positive slope (SUSEY.pdf). 2) All of the plots show a comb of 1Hz harmonics. Probably not surprising. 3) All the DAC channels (in blue) show extra noise (see all the non-closeup).
Dave Barker informed me that there shouldn't be any signals on the SUS_EX/EY_DAC_DT_OUT_DQ channels so that the absence of a duotone signal in EY is fine. In fact we turned off the EX signal. Now the spectra for the EX/EY ADC/DAC look very similar so there's no reason to be alarmed. We still don't understand why (now both EX and EY) DAC signals show a positive slope.
We have a 0.4mm vertical drive on the BS HEPI Vertical Actuators. Long story but with the HEPI loops open, this platform sags low enough that the IPS are no longer in their linear range and behave poorly giving us tilt coupling. Our pump station system pressure still has some coherence with the BS platform so I wanted to check if this large drive on the BS Vertical was increasing that coherence.
Bottom line, I'm not sure. I took a reference coherence look and then lowered the vertical on the hepi loops by 0.2mm and took another set of data. Attached are the results. The reference traces are the normal offsets and the current plots are with the offsets reduced. I'm looking at the T240s on the ISI stage1. Conclusion, ehh, it all just looks noisy, some places better some worse. There is a band in the X dof that seems broadly better from 10 to 25mHz but I'm not sure if it is meaningful. I'll look at the HEPI platform channels rather than just the ISI channels too and maybe I need to drop it the full 0.4mm to really see an effect.
Ben, Calum, and Gerardo, We received the two Gas Delivery systems, and two TMDS units (S1500093 and S1500094). We assembled the remainder of the table, and then mounted the Gas Delivery system. Lacking the final mounts, we put the TMDS on a screw jack so we could easily connect it to the Gas Delivery system. Once all hooked up, we attached the electronics, and put both TMDS systems through their paces, and tested them for ion production. We had two issues with S1500094 that we fixed. The first issue was that we accidentally swapped the cables going to the Baratron and Pirani gauges, which we believe damaged the Pirani gauge somehow. The second was that the cutoff valve leading to the vacuum pump would not seal completely. Gerardo took the whole thing to the clean room removed the suspect parts and replaced them with new ones sent from Caltech. Both new parts were found to work fine. The valve ended up having a cut and frayed Viton o-ring, which explained its problem. We are going to get an RMA for all things, and see what we can have done. Currently, the two TMDSs are in perfect working order, and have been fully checked out. We shipped the TMDS Interface back to caltech for further refinement, and so I can continue testing the units that we have there. Pictures of the system, and videos showing the operation procedures can be found at: https://dcc.ligo.org/LIGO-E1500252 These instructions are in support of, and superseded by information found in Rai's procedure document here: https://dcc.ligo.org/LIGO-T1400497 Test data collected from all tested units will go into the DCC next week. More information, and drawings can be found here: Gas Delivery System drawing: https://dcc.ligo.org/LIGO-D1500078 TMDS drawing: https://dcc.ligo.org/LIGO-D1400331
Peter, Calum, Ben, Rich Took transfer functions (1Hz to 10kHz, 255 points in to the DAC Drive Input on the front panel of the ESD driver, out of the respective SHV connector going to the ETM) of all the quadrant paths of the newly installed LN Driver. The results are stored in: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Measurements/ElectronicsMeasurements/ under the following file suffixes UR -> TFSR785_22-05-2015_102054.txt LR -> TFSR785_22-05-2015_102804.txt UL -> TFSR785_22-05-2015_103546.txt LL -> TFSR785_22-05-2015_105053.txt Everything looks good as compared to the Spice model. Also, we measured the output noise of each channel at 20Hz with and without DC bias. Results are all consistent with the design (<40nV/rtHz at 20Hz and above) and Spice model. 0V bias/-7.4V bias UR -> 21nV/rtHz/32nV/rtHz LR -> 22nV/rtHz/30nV/rtHz UL -> 23nV/rtHz/37nV/rtHz LL -> 28nV/rtHz/37nV/rtHz Analyzer Noise Floor 10nV/rtHz @ 20Hz Typical output noise at higher frequencies is: 20nV/rtHz @ 50Hz, 22nV/rtHz @ 100Hz, 20nV/rtHz @ 150Hz Noticed that there is still a discrepancy in the DAC monitors for the user model vs. the IOP. Richard has the details and Jeff will likely follow up next week. Once Richard sorts out the somewhat touchy vacuum interlock (he's working on that now and knows the problem), the entire system is now functional and ready for use.
Plot attached.
For the purposes of figuring out the appropriate compensation for DARM, I also plotted a simple model which just contains the effect of the LPF stage (2 poles at 2.2 Hz, 2 zeros at 50 Hz) and the PI summing node (pole at 152 Hz, zero at 3.2 kHz). The effect of the summing node is not yet digitally compensated, so that could explain the loss of phase that we saw in the DARM OLTF last night (about 50° at 200 Hz).
Summary
We were able to transition control of DARM to ETMY with feedback to the UIM, PUM, and test masses, with the ESD controlled by the low-noise driver. The low-pass filtering was off, but the rms drive to the ESD is low enough that we should be able to switch it on without inducing saturation.
The philosophy was to first adjust the gain and filtering of ETMY L3 and ETMY L2 so that they each actuate identically to ETMX L3. Then we engineered an L2/L3 crossover by inserting a 30 Hz lowpass into L2, and a 30 Hz highpass into L3. From 10 Hz to 100 Hz, this makes the combined ETMY L2/L3 actuator look similar to the ETMX L3 actuator. Then we transitioned control from ETMX to ETMY in the usual way; i.e., ramping on the gain in ETMY L3 LOCK L (with the offloading engaged), and then ramping down the gain on ETMX L3 LOCK L.
Details
Although we set the binary sus switches so as to have lowpass filtering on, it was evident by comparing the TFs of ETMX L3 and ETMY L3 that the filtering was off. We didn't want to spend time debugging this BIO issue, so we undid the digital LPF compensation (see below) and proceeded with the transition anyway.
The procedure was as follows:
In L3 L2L there is also a highpass filter at 1 Hz which cuts out the appearance of the microseism in the ESD drive. This reduces the rms ESD drive from 104 ct to more like 103 ct.
In L3 L2L, FM2 is used to compensate for the antiLP filters which are engaged in the ESDOUT FMs. This FM2 was installed only because ESDOUT is currently not correctly compensating the analog driver transfer function (since the analog LPFs are off; see above). FM2 should be removed once this is fixed.
The DARM OLTF is attached. Blue shows our "best" DARM OLTF configuration (i.e., what we use in full low-noise lock), with about 50 Hz ugf. Orange shows the OLTF we measured tonight on ETMX, before doing transitioning. The gain is intentionally low at this stage, because it facilitates ramping on ETMY feedback. Red shows what we measured with ETMY controlled by the LVLN driver. Here the DARM gain has been increased slightly. But the most notable feature here is the apparent loss of phase after transitioning. It is not immediately clear to us where this phase loss is coming from.
Crossover measurements have not been taken, and certainly there is some tweaking that we can do.
There were some severe thunderstorms and downpours last night - Any chance this could be a cause of the noise at 9:41?
Came in to give a tour and found that it's been losing lock at Find IR. So I took the ISC_LOCK to DOWN.
Great to see you got DARM control to ETMY working!
Regarding the phase loss: take a look at one of the transfer functions pointed to in entry 18579 Besides the 2.2/50Hz pole/zero pair, Rich said there is another pole above/around 100 Hz due to the output capacitance. This will need to be at least partially compensated.
John, we don't think that the thunderstom have caused the 1/f3 noise in DARM. In the periode we had this mysterious noise, there were no significant vibration on the ground according to the seismometers.