Kiwamu and Dan reported that there is no potable water in the OSB. It looks like the RO system went into fault on May 22 around 5:30pm and this morning the potable water tank went dry.
Kiwamu visited the water room for me and reset the RO system so it now appears to be operating normally and making water. However, the building is not yet pressurized.
I suspect the Naimco water skid will need to be reset.
Water is flowing somewhere..After the RO unit was reset the tank level started to climb but when the building was pressurized the level began to fall at an unusual rate.
We think the water system is back to normal.
Gerardo happened to be at the site when I got there so we visited the water room together to look into the problems.
We found that the water pumps were very hot and after breaking some fittings we found steam and hot water. After bleeding the system and cooling it down we restarted it and were able to immediatly build pressure.
The cause was likely a pressure switch which did not get reset during the initial startup. We'll investigate next week.
Dan, Kiwamu, Evan
List of things done today:
This attachment shows the comparison between ETMX and ETMY L3 drives. Here the interferometer is locked using ETMX. For ETMY, offloading is disabled, and all shaping is disabled except for the LPF and summing network compensation. A gain of −50 has been inserted in ETMY L3 L2L.
The CW hardware injection code psinject now runs without crashing at LHO. (Thanks, Peter Shawhan, for debugging the launch script.) The injection pathway is turned off, but the injection code has been sending 13 standard CW signals into the CAL model since approximately 21:30 PDT. Psinject is still running many hours later, so the code seem stable. Additionally, all injection code has now been migrated to svn here: https://svn.ligo.caltech.edu/svn/dac/hwinj/Details I've tested the svn version of tinj (for transient injections) at LHO and it seems to work properly, though, additional tests are planned. The latest version includes additional interaction with EPICS channels. During the course of today's debugging, I determined that GRB alerts do not appear to be recorded in the channel H1:CAL-INJ_EXTTRIG_ALERT_TIME. I have contacted Andrew Williamson to sort out why.
Kiwamu, Peter, Dan, Evan
We made it back to dc readout tonight. 3 W PSL power, DARM feedback to ETMX.
We are in the process of trying to engineer the new transition to ETMY. We were hampered by the intrusion of a large amount of 1/f3 noise in DARM, with ASD 1×10−14 m/Hz1/2 at 10 Hz. The initial portion of the lock was fine, and then the noise came. We trended various channels and looked at coherences, but could not nail down the cause. The noisy time starts at 2015-05-23 9:41:00 UTC. After about an hour, the noise went away again.
The ASC was a bit touchy. It could be engaged by hand, but with the guardian it seemed that engaging the PRM pointing loop caused several of the other loops to go unstable. Also, we gave up on the ITM pointing loops. They increased the recycling gain, but made the arm and sideband buildups less stable.
We found that there was no tidal signal being sent to ETMY. At first blush it appears to be an IPC issue.
We were able to transition control of DARM to ETMY with feedback to the UIM, PUM, and test masses, with the ESD controlled by the low-noise driver. The low-pass filtering was off, but the rms drive to the ESD is low enough that we should be able to switch it on without inducing saturation.
The philosophy was to first adjust the gain and filtering of ETMY L3 and ETMY L2 so that they each actuate identically to ETMX L3. Then we engineered an L2/L3 crossover by inserting a 30 Hz lowpass into L2, and a 30 Hz highpass into L3. From 10 Hz to 100 Hz, this makes the combined ETMY L2/L3 actuator look similar to the ETMX L3 actuator. Then we transitioned control from ETMX to ETMY in the usual way; i.e., ramping on the gain in ETMY L3 LOCK L (with the offloading engaged), and then ramping down the gain on ETMX L3 LOCK L.
Although we set the binary sus switches so as to have lowpass filtering on, it was evident by comparing the TFs of ETMX L3 and ETMY L3 that the filtering was off. We didn't want to spend time debugging this BIO issue, so we undid the digital LPF compensation (see below) and proceeded with the transition anyway.
The procedure was as follows:
In L3 L2L there is also a highpass filter at 1 Hz which cuts out the appearance of the microseism in the ESD drive. This reduces the rms ESD drive from 104 ct to more like 103 ct.
In L3 L2L, FM2 is used to compensate for the antiLP filters which are engaged in the ESDOUT FMs. This FM2 was installed only because ESDOUT is currently not correctly compensating the analog driver transfer function (since the analog LPFs are off; see above). FM2 should be removed once this is fixed.
The DARM OLTF is attached. Blue shows our "best" DARM OLTF configuration (i.e., what we use in full low-noise lock), with about 50 Hz ugf. Orange shows the OLTF we measured tonight on ETMX, before doing transitioning. The gain is intentionally low at this stage, because it facilitates ramping on ETMY feedback. Red shows what we measured with ETMY controlled by the LVLN driver. Here the DARM gain has been increased slightly. But the most notable feature here is the apparent loss of phase after transitioning. It is not immediately clear to us where this phase loss is coming from.
Crossover measurements have not been taken, and certainly there is some tweaking that we can do.
There were some severe thunderstorms and downpours last night - Any chance this could be a cause of the noise at 9:41?
Came in to give a tour and found that it's been losing lock at Find IR. So I took the ISC_LOCK to DOWN.
Great to see you got DARM control to ETMY working!
Regarding the phase loss: take a look at one of the transfer functions pointed to in entry 18579 Besides the 2.2/50Hz pole/zero pair, Rich said there is another pole above/around 100 Hz due to the output capacitance. This will need to be at least partially compensated.
John, we don't think that the thunderstom have caused the 1/f3 noise in DARM. In the periode we had this mysterious noise, there were no significant vibration on the ground according to the seismometers.
Following Xavier's timing measurements, the following FECs have their DAC Duotone* still turned on:
During parts of this week this was also turned on for h1susex and h1susey. These have been turned off as these DAC channels are used to drive the R0_COIL_RT signal. For the systems listed above, neither the DAC channel nor its ADC readback counterpart are being used by the models (PCAL reads the ADC channel but only archives it to DAQ).
* - reminder: if the DAC Duotone drive is enabled (via the GDS_TP MEDM screen for the IOP model), relays in the IO Chassis interface cards for the first DAC and the first ADC route the output of the last channel of the first DAC to the second-to-last channel of the first ADC. The IOP model takes the Duotone (digitized from the last channel of the first ADC) and writes this to the last channel of the first DAC.
Jim, Dave:
We are seeing a DAC card in h1susey which passes autocal on power up, but subsequently fails autocals if the IOP model is restarted. This is the same card (fifth card of five) which stopped driving its output to the ESD (see Richard's alog earlier this afternoon).
To check if autocal failure and lack of output are related, we used the card which always fails autocal (SN 101208-05) and looked at its output. In summary, its output drive is good and not related to its autocal success.
Details: DAC is the only card in x1susex. h1iopsuseywdt is the IOP model, it has a SWWD which required reset of its DACKILL part. A new user model called h1cds18bitdactest.mdl was created (DCUID=100) which drives all eight DAC channels using filtermodules. The DAC was connected to the DTS 18bit DAC AI chassis (lower eight channels) and a breakout board and DVM used to determine the output DC voltage. Voltages were driven by putting an offset on the filtermodules.
07:15 Richard and Peter to end Y 08:13 Sudarshan to end X to pick up PCAL equipment 08:56 TJ misaligning SR3 for brief period 08:57 Rich, Calum, Ben to end Y, ESD 08:58 Richard to end Y 09:17 Sudarshan, Darhan to end Y, PCAL calibration 09:18 Hugh moving BS HEPI 10:08 Richard to end Y 10:12 Nutsinee to HAM1 HWS table 12:23 Richard to end Y 12:47 Richard back Relocking got as far as CHECK_IR. Further progress has been interrupted by an earthquake.
My logs from LHO 18386 & 18468 showed some 'after' elevated noise on the CPS. Mainly on ITMX Stage2 H2, and now that I look again, the ITMX Stage1 V1 noise is a little elevated. The first look at these in the first log, was during the day. When I looked during the long lock stretch last Thursday, second log, I didn't explicitly look at the ITMX ST2 H2 but I said this channel looked good during the lock stretch. However, ST1 V1 on ITMX is elevated above about 25hz where the traces flatten--see attached. The ST2 V2 also is elevated but the Horizontals generally look fine.
The second attachment shows just the Corner 2 CPS and they look good.
Josh, Andy, TJ, Duncan, Detchar,
In 17791, Dan followed up the observation that COMM control was saturating by doing the most awesome thing - writing a script to identify all times when software limits are reached, by searching for all enabled software limits and then scanning data from those channels to see if 99% of the limit value was exceeded in a given time range.
I asked Duncan Macleod if he could optimize this code and he did. Dmac said: I have a working version that uses multiprocessing and is significantly faster than the original, and I’m pretty sure returns the same result. I have also added functionality to record the segments during which a given channel was saturating its software limit, and write them all to a LIGO_LW XML file.
To test this code, Duncan ran it over the long locked stretch from roughly 9-23 UTC on 5/17/15 (range plot shown in attachment 1). The output XML is attached below.
The primary limit saturations found were, a bunch near the end of the lock (probably after lock loss) and two in the midde of the lock: 1115927164.94, from H1:SUS-ITMX_L2_OLDAMP_P and 1115927158.19 from H1:PSL-ISS_DIGITAL_LOOP. Attachment 3 shows a normalized spectrogram of DARM during the saturation, 4 shows the timeseries of DARM, 5 shows timeseries for H1:PSL-ISS_DIGITAL_LOOP, 6 shows timeseries for the PSL PDB diode first loop intensity stabilization diode, 7 shows a saturation in H1:SUS-ITMX_L2_OLDAMP_P at nearly the same time (which is real, but doesn't have a big noticeable effect in DARM), 8 is a saturation counter from the PSL that finds this time equally well.
We plan to run this script for each lock stretch from now on to produce DQ segments and identify times when channels are hitting their software limits.
The Y arm had not locked since the new 18bit DACs were installed. One problem was a bad DAC. DAC 0 was not giving linear outputs. I have replaced this DAC. The other problem was channel 0 on DAC4 which would not output any signal. After a reboot/IO chassis power down the problem with DAC4 was gone. We then had to replace DAC0 which seems to have fixed the problem.
While installing the Low Voltage ESD drive chassis at EY. Rich A. needed to relocate the ESD pressure sensor interface chassis which required powering it down. This chassis provides the power to the combination pirani/Cold Cathode(CC)gauge that is used for the interlock on the ESD High Voltage(HV) Supplies. After powering the gauge back up the cold-cathode portion of the gauge has not restarted most likely due to the excellent vacuum at Ey. The pirani is reading 10-5 which is our cutoff point for allowing the HV supplies to operate thus preventing the HV supplies from working. Since the vacuum has been quite steady John Worden gave me permission to by pass this interlock until Tuesday. Hopefully the gauge will be on by then or I will connect my computer to it and try and for on the CC. I have put jumpers on pins 1-6 and 3-7 on the connector on the safety relay box at the power supply rack to allow the ESD to operate. Work permit 5225.
Richard, Jim, Dave:
there have been some swapping of modified 18bit DAC cards as problems were found during this week's upgrade, I wanted to give a summary of what happened and where we currently stand.
Prior to this week, 5 DACs (chosen at random) were tested in the DTS. It was found that card 101208-05 fails autocal, which it did prior to the upgrade, and has remained in the DTS throughout this week.
Monday, in h1susex, it was found that if card 101208-59 was in the chassis the code would not run. This card is waiting for further tests and not in any system.
Tuesday we installed EY and corner station. We thought we had enough cards to update all BSC and HAM in the corner station if we skipped h1sush2b (two cards) and we used all five cards from the DTS. This would entail using the card which fails autocal, but Jeff and Dan said that could be used for the OMC. But, we found that one of the new cards was flagged with a potential capacitor problem (card 110425-19) so we abandoned that plan and decided instead to not update h1sush56 (five cards) but instead update h1sush34 (six cards) and h1sush2b (two cards). This meant taking three cards from the DTS, leaving it with two (one good, one bad-autocal).
On Thursday we found a bad channel on a DAC in h1sush2a (stuck output). This card (101208-44) was pulled from h1sush2a and the one remaining good card from the DTS was used to replace it. The stuck output card is waiting further tests and not in any system.
On Friday we discovered a non-linear output problem with the first card in h1susey (card 101208-78). The suspect capacitor card was inspected and then used to replace it in h1susey. The non-linear cad is waiting further tests and not in any system.
So five cards with issues, they are
| 101208-59 | code wont run with this card installed | in hand waiting testing |
| 101208-05 | fails autocal | being tested in DTS x1susex |
| 110425-19 | suspect capacitor | running in h1susey |
| 101208-78 | non-linear output | in hand waiting testing |
| 101208-44 | stuck output voltage | in hand waiting testing |
I looked at the following channels: H1:IOP-SUS_EX_ADC_DT_OUT_DQ H1:IOP-SUS_EY_ADC_DT_OUT_DQ H1:IOP-LSC0_ADC_DT_OUT_DQ H1:IOP-SUS_EX_DAC_DT_OUT_DQ H1:IOP-SUS_EY_DAC_DT_OUT_DQ H1:IOP-LSC0_DAC_DT_OUT_DQ Attached are 6 figures, broadband and close-ups to the 960/961Hz signals. Notable features: 1) The EY spectra don't look right at all. The doutone signal is missing (SUSEYcloseup.pdf) and the spectrum has a positive slope (SUSEY.pdf). 2) All of the plots show a comb of 1Hz harmonics. Probably not surprising. 3) All the DAC channels (in blue) show extra noise (see all the non-closeup).
Dave Barker informed me that there shouldn't be any signals on the SUS_EX/EY_DAC_DT_OUT_DQ channels so that the absence of a duotone signal in EY is fine. In fact we turned off the EX signal. Now the spectra for the EX/EY ADC/DAC look very similar so there's no reason to be alarmed. We still don't understand why (now both EX and EY) DAC signals show a positive slope.
Peter, Calum, Ben, Rich Took transfer functions (1Hz to 10kHz, 255 points in to the DAC Drive Input on the front panel of the ESD driver, out of the respective SHV connector going to the ETM) of all the quadrant paths of the newly installed LN Driver. The results are stored in: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER7/H1/Measurements/ElectronicsMeasurements/ under the following file suffixes UR -> TFSR785_22-05-2015_102054.txt LR -> TFSR785_22-05-2015_102804.txt UL -> TFSR785_22-05-2015_103546.txt LL -> TFSR785_22-05-2015_105053.txt Everything looks good as compared to the Spice model. Also, we measured the output noise of each channel at 20Hz with and without DC bias. Results are all consistent with the design (<40nV/rtHz at 20Hz and above) and Spice model. 0V bias/-7.4V bias UR -> 21nV/rtHz/32nV/rtHz LR -> 22nV/rtHz/30nV/rtHz UL -> 23nV/rtHz/37nV/rtHz LL -> 28nV/rtHz/37nV/rtHz Analyzer Noise Floor 10nV/rtHz @ 20Hz Typical output noise at higher frequencies is: 20nV/rtHz @ 50Hz, 22nV/rtHz @ 100Hz, 20nV/rtHz @ 150Hz Noticed that there is still a discrepancy in the DAC monitors for the user model vs. the IOP. Richard has the details and Jeff will likely follow up next week. Once Richard sorts out the somewhat touchy vacuum interlock (he's working on that now and knows the problem), the entire system is now functional and ready for use.
Plot attached.
For the purposes of figuring out the appropriate compensation for DARM, I also plotted a simple model which just contains the effect of the LPF stage (2 poles at 2.2 Hz, 2 zeros at 50 Hz) and the PI summing node (pole at 152 Hz, zero at 3.2 kHz). The effect of the summing node is not yet digitally compensated, so that could explain the loss of phase that we saw in the DARM OLTF last night (about 50° at 200 Hz).