Maintenance completed just after noon. Ryan and Jason then ran a calibration with the PSL rotation stage and then I started an initial alignment.
We are moving onto DC readout now.
WP 11971
The following cables were disconnected at the feedthrough:
1. H1:ISC-X9-82 (EX)
2. H1:ISC-X10-82 (EY)
Cables are for the in-vacuum picomotors. Picomotors are not used. A shortening plug was installed on the each flange (all pins tied to backshell, backshell tied to chamber ground via mounting screws). Will monitor DARM, PEM ESD VMON,and FScan for improvements.
We've consistently seens PRCL coherence with DARM that is higher than MICH and SRCL, and the coupling doesn't seem to be reduced well by decoupling PRCL from SRCL and MICH (see 77416 and 77289). Here are some plots based on old LSC injections made when OM2 was cold, on May 9th. Using the PRCL excitation, I've made an exess power projection to DARM, SRCL and MICH. Then using the MICH (or SRCL) excitation I've projected where the PRCL contribution to MICH (or SRCL) noise that's from PRCL should be in DARM. Both are much lower than the PRCL projection to DARM, indicating that this PRCL projection isn't a double counting of MICH and SRCL noise, and confirming that we won't be able to reduce the PRCL to DARM noise by improving the decoupling of PRCL from SRCL and MICH.
Camilla has taken a more recent set of these injections 78782, so I will try to run this script for those injections. I'd expect to get a similar result. Camilla has now taken measurements to implement a PRCL feedforward.
These measurements and script can be found at /ligo/home/sheila.dwyer/Noise/PRCL/PRCL_projection
We cannot make a reasonable assessment of energy deposited in HAM6 when we had the pressure spikes (the spikes themselves are reported in alogs 78346, 78310 and 78323, Sheila's analysis is in alog 78432), or even during regular lock losses.
This is because all of the relevant sensors saturate badly, and the ASC-AS_C is the worst in this respect because of heavy whitening. This happens each and every time the lock is lost. This is our limitation in configuration. I made a temporary change to partly mitigate this in a hope that we might obtain useful knowledge for regular lock losses (but I'm not entirely hopeful), which will be explained later.
Anyway, look at the 1st attachment, which is the trend at around the pressure spike incident at 10W (other spikes were at 60W, so this is the mildest of all). You cannot see the pressure spike because it takes some time for the puffs of gass molecules to reach the pirani.
Important points to take:
This is understandable. Look at the second attachment for a very rough power budget and electronics description of all of these sensors. QPDs (AS_C and OMC QPDs) have 1kOhm raw transimpedance, 0.4:40 whitening that is not switchable on top of two stages of 1:10 that are switchable. WFSs (AS_A and AS_B) have 0.5k transimpedance with a factor of 10 gain that is switchable, and they don't have whitening.
This happens with regular lock losses, and even with 2W RF lock losses (third attachment), so it's hard to make a good assessment of the power deposited for anything. At the moment, we have to accept that we don't know.
We can use AS_B or AS_A data even though they're railed and make the lower bound of the power, thus energy. That's what I'll do later.
(Added later)
After TJ locked the IFO, we saw strange noise bump ffrom ~20 to ~80 or so Hz. Since nobody had any idea, and since my ASC SUM connection to the PEM rack is an analog connection from the ISC rack that also has the DCPD interface chassis, I ran to the LVEA and disconnected that.
Seems like that wasn't it (it didn't get any better right after the disconnection), but I'm leaving it disconnected for now. I'll connect it back when I can.
In a hope to make a better assesment of the regular lock losses, I made the following changes.
My gut feeling is that these things still rail, but we'll see. I'll probably revert these on Tuesday next week.
SDF screenshot of accepted values.
Low voltage operation of the fast shutter: It still bounces.
Before we started locking IFO, I used available light coming from IMC and closed/opened the fast shutter using the "Close" and "Open" button on the MEDM screen. Since this doesn't involve the trigger voltage crossing the threshold, this only seems to drive the low voltage output of the shutter driver which is used to hold the shutter in closed position for a prolonged time.
In the attached, the first marker shows the time the shutter started moving, witnessed by GS-13.
About 19ms after the shutter started moving, the shutter is fully shut. About 25 ms after the shutter was closed, it started opening, got open or half-open for about 10ms and then closed for good.
Nothing was even close to railing. I repeated the same thing three times and it was like this every time.
Apparently the mirror is bouncing down or maybe moving sideways. During the last vent we haven't taken the picture of the beam on the fast shutter mirror, but it's hard to imagine that it's close the the end of the mirror's travel.
I thought that it's not supposed to do that. See the second movie in G1902365, even though the movie is capturing the HV action, not the LV, it's supposed to stay in the closed position.
ASC-AS_C analog sum signal at the back of the QPD interface chassis was put back on at around 18:30 UTC on Jul/11.
Unfortunately, I forgot that the input range of some of these PEM ADCs are +-2V, and so the signal still railed when the analog output of ASC-AS_SUM didn't (2V happens to be the trigger threshold of the fast shutter), so this was still not good enough.
I installed 1/11 resistive divider (nominally 909Ohm - 9.1k) on the output of the ASC-AS_C analog SUM output on the chassis (not on the input of the PEM patch panel) at around 18:30 UTC on Jul/12 2024 while IFO was out of lock.
WP11970 h1susex 28AO32 DAC
Fil, Marc, Erik:
Fil connected the upper set of 16 DAC channels to the first 16 ADC channels and verified there were no bad channels in this block. At this point there were two bad channels; chan4 (5th chan) and chan11 (12th chan).
Later Marc and Erik powered the system down and replaced the interface card, its main ribbon cable back to the DAC and the first header plate including its ribbon to the interface card. What was not replaced was the DAC card itself and the top two header plates (Fil had shown the upper 16 channels had no issues). At this point there were no bad channels, showing the problem was most probably in the interface card.
No DAQ restart was required.
WP11969 h1iopomc0 addition of matrix and filters
Jeff, Erik, Dave:
We installed a new h1iopomc0 model on h1omc0. This added a mux matrix and filters to the model, which in turn added slow channels to the DAQ INI file. DAQ restart was required.
WP11972 HEPI HAM3
Jim, Dave:
A new h1hpiham3 model was installed. The new model wired up some ADC channels. No DAQ restart was required.
DAQ Restart
Erik, Jeff, Dave:
The DAQ was restarted soon after the new h1iopomc0 model was installed. We held off the DAQ restart until the new filters were populated to verify the IOP did not run out of processing time, which it didn't. It went from 9uS to 12uS.
The DAQ restart had several issues:
both GDS needed a second restart for channel configuration
FW1 spontaneously restarted itself after running for 9.5 minutes.
WP11965 DTS login machine OS upgrade
Erik:
Erik upgraded x1dtslogin. When it was back in operation the DTS environment channels were restored to CDS by restarting dts_tunnel.service and dts_env.service on cdsioc0.
Tue09Jul2024
LOC TIME HOSTNAME MODEL/REBOOT
09:50:32 h1omc0 h1iopomc0 <<< Jeff's new IOP model
09:50:46 h1omc0 h1omc
09:51:00 h1omc0 h1omcpi
09:52:18 h1seih23 h1hpiham3 <<< Jim's new HEPI model
10:10:55 h1daqdc0 [DAQ] <<< 0-leg restart for h1iopomc0 model
10:11:08 h1daqfw0 [DAQ]
10:11:09 h1daqnds0 [DAQ]
10:11:09 h1daqtw0 [DAQ]
10:11:17 h1daqgds0 [DAQ]
10:11:48 h1daqgds0 [DAQ] <<< 2nd restart needed
10:14:02 h1daqdc1 [DAQ] <<< 1-leg restart
10:14:15 h1daqfw1 [DAQ]
10:14:15 h1daqtw1 [DAQ]
10:14:16 h1daqnds1 [DAQ]
10:14:24 h1daqgds1 [DAQ]
10:14:57 h1daqgds1 [DAQ] <<< 2n restart needed
10:23:07 h1daqfw1 [DAQ] <<< FW1 spontaneous restart
11:54:35 h1susex h1iopsusex <<< 28AO32 DAC work in IO Chassis
11:54:48 h1susex h1susetmx
11:55:01 h1susex h1sustmsx
11:55:14 h1susex h1susetmxpi
Power Spectrum of channels 0 through 15. No common mode issues detected.
Channel 3 & 9 are elevated below 10Hz
It is unclear if these are due to the PEM ADC or the output of the DAC. More testing is needed.
New plot of first 16 channels, with offsets added to center the output to zero. When offsets were turned on, the 6Hz lines went away, I believe these were due to uninitialized DAC channels. This plot also contains the empty upper 16 channels on the PEM ADC chassis as a noise comparison with nothing attached to the ADC. Channel 3 is still noisy below 10Hz.
New plot of second 16 channels (ports C & D), with offsets added to center the output to zero. This plot also contains the empty lower 16 channels on the PEM ADC chassis as a noise comparison with nothing attached to the ADC. Channel 3 is still noisy below 10Hz, signifying this to be an ADC issue, not necissarily a DAC issue. These plots seem to imply that the DAC noise desnity while driving zero volts is well below the ADC noise floor in this frequency range.
All the Dust Monitors in use passed the zero count and flow rate test for this cycle. Closes FAMIS27659, this was last done by me last December alog74610.
I ran the zero count and flow rate test for all the dust monitors in use around the site. Passing is having zero counts of 0.3 and 0.5 micron particles and a flow rate of 2.8 L/min.
We also have 4 working spares, 3 pumpless & 1 pumped, 3 (the 3 PL) of which have been calibrated by METONE earlier this year.
EX: Passed zero count on the first test and flow rate test.
EY: 2 0.3 particle counts on the first test, zero on the second which is a pass, and the flow rate was good.
The LVEA as been swept, no items to note.
Jennie, Jenne, Sheila, Keita
While the PSL team were preparing for a PSL in cursion, I started moving MC3 based on our plan to re-align onto the ISS array and IM4 TRANS QPD after the PMC swap last week.
We tried going up in yaw, saw some increase in power on IM4 trans QPD but then the ISS second loop QPD started to become further from the centre position as did the IM4_TRANS QPD (this is well centred enough that we could see the power drop whereas the ISS QPD started quite far off centre so that the nsum is quite close to zero).
We went back to the maximum value then tried IM3 alignment is both DOFs but this did not improve things in both QPDs so we tried MC3 pitch. Sheila then noticed that we were mazking the IMC relflected spot some horrible higher order mode so we decided to stop and think.
MC1 and MC3 changed in common yaw due to the WFS as we changed the uncorntolled differential DOF (MC1 - MC3), which we did not expect, to recover our alignment we would need MC1 and MC3 to change differentially as they did after the PMC swap, in this image.
The ndscope channels I used for this alignment walking session are attached with cusors before we changed anything and after we recovered the IMC alignment to pre-maintenance.
In conclusion we have reverted the IMC alignment back to before our changes as we think some action before the in-vacuum mirrors is needed.
Plan: To revert periscope PZT and MC1 back to their positions before the PMC swap and get Jason/Ryan to move steering mirrors in the PCAL enclosure to recover the prompt reflection from the IMC as witnessed by the MC WFS behind MC1. Then we can re-align the IMC.
J. Kissel, D. Barker, E. von Reis WP 11969 The h1iopomc0 model changes I prepped this morning LHO:78956 have been installed. Further, one at at time, I've installed copies of the primary OMC DCPD A0 and B0 filter bank filters into the A1, A2, and B1, B2 filters in order to test the consumption of computational turn around time (see preliminary discussions in LHO:78958). Finally, I've added a reasonably high-order high-pass filter into the A1 and B1 banks to test whether we can add even *more* filters enough to accomplish the tests discussed in LHO:78956. Attached is an annotated time series of the CPU meter channel (H1:FEC-179_CPU_METER) as each of these things happened. In short -- even with new the filters "fully loaded," the model computational turn around time runs at 12 [usec], occasionally popping up to 13 [usec] -- under the ideal limit of 15.3 [usec]. In addition, we don't see any IPC timing errors in either the OMC user model or the end station PI models that are receiving IPC from the h1iopomc0 model. So, we'll leave this diagnostic infrastructure in place for a few weeks, until we decide we don't need it anymore.
J. Kissel, E. von Reis, D. Sigg Circa March 2023, Daniel had installed some never-used first attempts at further filtering the high frequency noise present in the 524 kHz OMC DCPD channels (never aLOGged because it was never used, but I call them out after I found the work in LHO:68098). Now, because I'd like to characterize the existing, in-use, digital AA filtering, running into some unknown noise (LHO:78516), and hoping to install 2 to 4 more parallel filter banks that would also be quite full of filters (LHO:78956), there is worry that there won't be enough computation time in the 524 kHz system. Remember, that "the 524 kHz system" is actually a modified "standard" 65 kHz system, which is reading out 8 samples from the 524 kHz ADC each 65 kHz clock cycle and computing everything at 65 kHz. Thus, in principle, the max turn around time is 1 / (2^16 Hz) = (1 / 65536) [sec] = 1.5258789e-5 [sec] = 15.3 [usec] However, we *think* the practical limit is somewhat less that this. I don't think I understand what those limitations are enough to say definitively and/or to quote a limit quantitatively, but I think they're related to "the copy of the OMC DCPD channels which are demodulated at high frequency to create PI channels that are shipped to the end station -- in other words, the IPC sending demands a bit of computational time, and if there isn't enough turnaround time left in the OP to write to the IPC network, then the end-station SUS PI models throw an IPC timing error." Anyways -- this morning, I looked at the 524 kHz system's computation time as is before doing anything (via the channel H1:FEC-179_CPU_METER), and it's sitting at 9 [usec] (out of the ideal 15 [usec]), occasionally popping up to 10 [usec]. But -- this led me to remember that -- regardless of whether the filter is turned ON -- the front-end computes the output of the filter -- sucking up computation time. So, I've removed these unused prototype notch filters from the DCPD A0 and B0 filter banks. In addition, I've also removed the old "V2A" filter from a previous version of the digital compensation for the OMC DCPD transimpedance amplifier response. Removing the notch filters drops the computation time from "9 [sec] occasionally bopping up to 10 [usec]." See attached time series of the CPU meter. These filters are, of course, available in the filter_archive, under the latest previous archived file before today's work: /opt/rtcds/lho/h1/chans/filter_archive/h1iopomc0/ H1IOPOMC0_1401558760.txt but for ease of use, I copy them here. FM3 :: Notches1 ellip("BandStop",3,0.5,30,12800,13200) notch(10216,50,30) ellip("BandStop",3,0.5,30,10380,10465) ellip("BandStop",3,0.5,30,12900,13100) FM5 :: Notches2 ellip("BandStop",3,0.5,30,8100,8200) notch(9101,50,30)notch(9337,200,30) notch(9463,50,20) ellip("BandStop",3,0.5,30,9750,9950) FM8 :: Notches3 ellip("BandStop",5,0.5,40,14384,18384) ellip("BandStop",5,0.5,40,30768,34768) FM6 :: V2A zpk([5.699+i*22.223;5.699-i*22.223;32.73],[2.549;2.117;6.555],0.00501187,"n")gain(0.971763) I've also posted a plot of the magnitude of these notch filters -- mostly just to demonstrate how many second order sections that these filters had -- sucking up computation time.
Keita, Sheila
We were looking for explanations for the drops in range we've seen since Tuesday. Attached is a plot of the CPU meter, it seems that this jumped up shortly after Jeff's plot was made. It is still below 13usec, and doesn't look correlated with our range problems.
Variation of CPU time in that range shouldn't by itself have any effect on the control loops running on that model until they get to a sustained time above 15 us or an individual cycle time somewhat more than 15 us depending on the model.
The effects of a model that's run too long are DAC buffer starvation, i.e. the IOP didn't keep up with the DAC clocks, or IPC communication between models arriving to late.
Both of these errors would appear immediately on the CDS overview MEDM.
J. Kissel, with help from E. von Reis, D. Barker, D. Sigg WP 11969 Continuing investigations of high frequency limitations of characterizing the anti-aliasing processing performance for the 524 kHz OMC DCPD system (LHO:78516), there are a few unconfirmed suspicions about whether the flat noise "limit" above ~20 kHz -- seen in only in the NOMINAL_LOW_NOISE data -- is ADC noise or numerical precision issues. There are two suggested measurements that would confirm one or the other empirically: (1) "Compare the high frequency noise of 4 channel sum / average against the noise in a single channel" If the noise changes by sqrt(4) = 2 (with the underlying assumption that the noise of the ADC is identical for each channel), then the noise floor is ADC noise not precision limit. (2) "High pass the data in the real-time to remove the large DC component" If the low frequency DC signal is removed in the real time system -- which is running at 32-bit double precision, and *then* we compute the ASD via a grabbing the data once turned into a test point (which is done at single precision) -- then the ASD should *not* be limited by the single precision noise floor. However, the current infrastructure doesn't support either tests. For (1), while we do have access to the the ADC channels are summed immediately off the ADC as test points (i.e. H1:IOP-OMC0_MADC0_CH[0,4,8,12]), they're not piped into any filter banks, so we can't make any of the high-dynamic-range testing of the Digital AA filtering comparisons with the output of the primary DCPD banks. For (2), these are the gravitational wave channels, and the in-loop DARM loop error signals. "Just toss in a high-pass" is not an option (the DARM loop would go unstable and/or at least lose a bunch of low frequency gain; if if the IFO would funciton it would mean drastic impact on the calibration of the detector). As such, I'm document here the front-end model changes I hope to install that creates pick-off paths for the eight ADC channels (four copies of the voltage from DCPD A and four for DCPD B), piping them in to an ADC_INMTRX matrix (along with the 4 channel sum) for a total of 10 inputs, and then piping that output of that matrix into 4 filter banks whose output is terminated. The design intent is for these filter banks to mostly contain exact copies of the primary DCPD_A0 and B0 filter banks, but we're free to modify them and play around with, say, adding a high pass, whenever we want without consequences for the detector. I'm also a big fan of "clean as you go," so I've also taken the opportunity to make the simulink diagram a (little cleaner / legible / less redundant) for the primary existing paths. Attached are screenshots showing the model changes: - Top level before vs. after Comparing these two, you see immediately the new input matrix. But also, you see that the PI path is now no longer a equal parallel copy of the sum of the channels, but instead the sum is done once, and the output of the sum is fed into the OMC block to be split out in there. - One layer down, into the top_named OMC block, before vs. after Here is where the OMC DCPD 4 channel sum is picked off for the PI path with a goto tag, rather than re-computing the sum. The new matrix output channels are fed into the DCPD block. - A further layer deeper, into the OMC_DCPD block, before vs. after Here the new matrix output channels are fed into 4 filter banks. At the moment, we're unsure whether the 524 kHz system can handle this many more filter banks without running out of computation time to ship the PI data to the end stations. So we may end up reducing the amount of test filters from 4 to 2. Further, once we gather enough data, there may be a chance that we no longer need these test pickoff filters. Thus, at the moment, this a "prototype" change, with work permit only, and not an ECR -- since it may not be needed for more that a week or two.
Link to report
Tue Jul 09 08:16:30 2024 INFO: Fill completed in 16min 25secs
Jordan confirmed a good fill curbside.
TITLE: 07/09 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 4mph Gusts, 2mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.04 μm/s
QUICK SUMMARY: Locked for 6.5 hours. PEM_MAG_INJ node is in error, I'll wait to fix it so it dont start to conflict with the charge measurements. Maintenance day today.
DriptaB, FranciscoL
After one week of having the inner beam up on the Rx sensor (alog 78807), on July 2, we moved the beam from up to center.
The last page of EndStationLog.pdf lists the voltage values after each significant step of procedure T2400163. The steps represent writing down a voltage value after a particular change to the beam position. Some highlights from the recorded values:
The 'Initial' measurement is *equal* to the last voltage measurement from the previous movement, done on June 25 (alog 78807). The initial and final voltage measurement during today procedure did not change.
Title should be "eight", not "seventh". This movement was a week after the seventh move, which was moved up (alog 78807).