Patrick, Gerardo Entered command through Beckhoff to start degas at 19:27 UTC. It appears the command did not succeed because the pressure is currently too high. The manual states that it must be below 7.2×10-6 mbar (5.4×10^-6 torr). It is currently 1.72e-05 torr. I entered the command to turn it off.
Both chillers are good. Added no water to either chiller. The Crystal chiller filters were replace during Tuesday's maintenance period. The diode chiller filters are clean and clear. Closing FAMIS #10485
Using the following script,
${CALSVN}/trunk/Runs/O3/Scripts/CALCS_FE/CALCSvsDARMmodel.py
I've made a first direct comparison of the installed CAL-CS model with the new pyDARM model. This was not that easy to do because some standard scipy functions to make ZPK filters from sos coefficients have some very restrictive tolerances that don't make a lot of sense to me. I've kludged my way around this by adding some of the same functions scipy uses for the ZPK filters without such tight tolerances.
Attached are three figures showing the comparisons:
1) Full sensing function and actuation functions. This reveals the limitations of the CAL-CS model, especially at high frequencies. There are issues in magnitude below 10 Hz--probably an issue of large dynamic range of the suspension model--but not very important because it is below 10 Hz. In addition there look to be issues around 300 Hz (missing filter?). Needing further investigation is the phase: it is unclear what is going wrong with phase offsets at low frequency in sensing. Again, further investigation needed
2) Individual stages are compared between CAL-CS and pyDARM. The L1 stage is the cause of low frequency (<10 Hz) discrepancy, and L3 stage for the 300 Hz discrepancy. The effective time delay in L1, L2, and L3, does not seem consistent--also worth investigating.
3) Given the current model, we can find a relative time delay between sensing and actuation in units of 61 usec clock cycles. The orange dot marks the point closest to the UGF.
As mentioned, there is further investigation needed to make sure we fully understand the differences between stages in the CAL-CS vs pyDARM model.
J. Kissel ECR E1800246 IIET 11305 I've (nearly*) completed several MEDM screens that are reflecting the new upgraded time-dependent correction factor calculation in the fron-end calibration pipeline, in support of splitting the calculation of kappa_pu into kappa_p and kappa_u, plus the new supporting calibration line, as described in T1700106. The changes were installed into the CAL-CS front-end model moons ago (see LHO aLOG 44459, and G1801594), and thus I'm just completing the effort started in LHO aLOG 44752 to develop a user interface. This system doesn't work yet (we also don't have an IFO back yet to test it), but now that the use interface is complete enough to debug the parameters and settings and we can finally move forward. *There're a few bug fixes and a bit of aesthetic cleanup let to be done, but we'll get to that in due time. These new screens live in the following corner userapps repo: /opt/rtcds/userapps/release/cal/common/medm/ with names akin to the file names of the respective screenshots attached below. Also in due of time, I'll update G1801594 to reflect the new changes. WP 7968 Over the course of developing these screens, I'd identified a few tweaks, extra EPICs records, and bug-fixes to the front code along the way, so Dave graciously agreed to support a couple of h1calcs model, and subsequent DAQ, restarts yesterday evening, a. la. LHO aLOGs 45539 and 45543. These model changes have been committed to /opt/rtcds/userapps/release/cal/common/models/ as a part of the clean-up.
I've made a few minor tweeks to the TDEP_OVERVIEW screens (see new attachment), but more importantly, I've spruced up the EPICS RECORDS screen such that it's MUCH better labeled. See attached updates. Changes have been committed to the userapps repo.
Following on from Keita's alog entry from yesterday this morning I looked at the mixer monitor signal. The MEDM screen screen indicates that the mixer output is typically ~0.5 V. With a digital multimeter I measured 1.2 mV when the MEDM screen indicated that the voltage was ~0.7 V. An oscilloscope trace of the signal is shown in tp3_1.png. CG20FG9.tif shows the measured open loop transfer function from this morning, with the marker placed at the UGF. The transfer function does not appear as smooth as ones I can recall from before. Phase margin looks okay. Whilst I was in the enclosure, I tweaked the alignment to the reference cavity a little and re-centered the beam onto the locking photodiode. The FSS still has some trouble locking even when the transmission voltage indicated on the MEDM screen exceeds the resonant threshold. I do not know why this would be, although sometimes either the runtime model and/or guardian takes over the common gain slider and moves it at what I think is an inappropriate time - like when there is no light flashing in the reference cavity, for example.
The discrepancy in the mixer signal is not explained by the schematics of the TTFSS field box. Next step is to look at the front panel output of the field box.
A few moments ago (~1:35 pm local) I measured the mixer monitor voltage at the front panel of the TTFSS field box.
It measured -0.1 mV. Coincidentally I looked at the value on the MEDM screen and it reported a value consistent
with the voltmeter. Pulling up the trend data for the past 10 hours, the mixer voltage dropped close to 0 V since
about 8:10 am local.
Not sure what caused the big change.
I replaced the IO lens, L1, that's downstream of the EOM by about 3 inches. The new lens needed to shift +Y direction from the old lens, to recover yaw, and recovery of the IMC WFS with an unlocked IMC revealed that the IMC_IN beam was still off in yaw, so I also made a very small adjustment with M3, unnoticeable on the rotation stage, and the iris, but seen by the cameras and IMC.
I realigned the IMC, and relocked, and checked against previous locks. My target lock was the Nov 26, 17:00 lock, and current values for the IMC mirrors are within -15.7urad to +10.9urad from the target. H1:IMC-REFL_DC_OUT16 is now at 0.072.
| Nov 26 | Nov 28 | diff | |
| 17:00 | 07:42 | ||
| lens change | before | after | |
| 'H1:SUS-MC1_M3_WIT_PMON' | -275.9 | -272.0 | 3.9 |
| 'H1:SUS-MC1_M3_WIT_YMON' | -983.1 | -972.3 | 10.9 |
| 'H1:SUS-MC2_M3_WIT_PMON' | 156.2 | 140.5 | -15.7 |
| 'H1:SUS-MC2_M3_WIT_YMON' | 627.2 | 630.7 | 3.5 |
| 'H1:SUS-MC3_M3_WIT_PMON' | -685.6 | -675.6 | 10.0 |
| 'H1:SUS-MC3_M3_WIT_YMON' | -1255.8 | -1260.6 | -4.8 |
WP7968 h1calcs channel name fix
Jeff K, Dave:
New h1calcs model for name fixing with DAQ restart.
The changes made in the restart are motivated in LHO aLOG 45548.
16:30 Fil to HAM2
17:00 Cheryl to PSL
17:00 Chandra to HAM2
19:00 TJ to PSL
21:00 Sheila to EY
21:15 Fil to EY
22:00 Fil to EY
22:30 Kyle to HAM1
22:30 Marc to LVEA
23:30 Gerardo to OMC area
23:45 Sheila to ISCT1
Incursion activity to fix the problem: alog 45391
I checked if the ASC REFL_A DC segment 4 problem described above was fixed for good by walking the bean on the WFS using RM1 from one segment to another so the beam is contained by only one of the four segments. At first I was glad to find that the problem was gone (SEG1, 2, 3 and 4 showed about 8200, 8300, 8100 and 9000) but later found that the digital gain of 2 was somehow put back in after our work in the above alog.
This means that the problem came back after we thought that we fixed the problem.
During the EQ this morning I went to the floor and checked the analog signal. I switched the DC interface off, waited for a minute or so, and turned it on.
At first both legs of the H1:ASC-REFL_A_DC_SEG4 were active, but after a minute or so the negative leg failed (attached).
(Sorry that "negative" leg which is a cyan trace is physically positive and vice versa, this is consistently so for all LIGO WFSs. For the record I'm measuring pin 4 relative to pin 15 in ch1, and pin 12 to pin 15 for ch2, see D1300467.)
I also observed a behavior after a longer "cool off" of about 10 minutes that the output switches between good and bad states several times before finally settling down to bad state.
Looking back at my alog from 2014 (alog 14017),
"However, it's not totally impossible that the feedthrough is OK but the seg4 in-vac circuit works only for a few minutes after it is powered on because of slowly developing oscillation or thermal problem or whatever."
that petty "not totally impossible" "thermal problem or whatever" looks much more plausible.
I am beginning to wonder if somehow this problem is inside the detector. In series with each of the differential DC outputs there is a 10 ohm resistor. It is sometimes possible that during manufacture only one side of these resistors is soldered on. The unsoldered side will initially (weeks or months) make contact, so the testing and casual inspection will not reveal any problem. After some time goes by, the unsoldered junctions develop oxide layers and eventually open up. Thermal cycling as a result of powering off the device for a significant time may cause mechanical motion associated with the CTE of the circuit board material which can cause the circuit to once again make contact. After a heatup period, it might well go back to an open circuit. Verification of the open circuit can be made by an ohmmeter check of the offending wire while the circuit is powered up. Unfortunately, this will look the same as a connector pin opening and closing, but there's really no reason for a connector to do this as a function of being powered up and down, so my money is on the detector itself. The output impedance of the device can be measured while it is in the "working" state, and it should be 10 ohms. If you measure a substantially higher resistance that eventually opens up, then I would think it's the poor solder joint scenario.
After the maintenance the FSS was found to be unlocked and the auto locker had a hard time relocking.
We knew that the fringe was supposed to be at around 0.37 to 0.38 K but the scan setting was [0.2,0.4] K, and it seemed like the auto locker finds something scanning up at around 0.37, but gives up after some failed attempts and starts to go down without scanning up further.
I changed the scan setting from [0.2,0.4] K to [0.32,0.42] K so the scan is slower and the right temperature is at around the center of the scan range. Also I temporarily lowered the resonant threshold from 1 to 0.2.
I've noticed that FSS mixer output readback was about 0.5V when FSS was locked, which sounded very large.
Peter and I looked at the trend going back 180 days and it seems like the mixer output has never been small but it got bigger on Tuesday last week (attached right).
When the refcav crosses fringe it goes from 0.5 to 1.5V roughly as of now (attached left) so it seems that the refcav is still locked to the center. But I wanted to make sure that this is just the readback problem and Peter offered to take a look at analog signal tomorrow morning.
Attached are the files that were generated in /ligo/svncommon/SusSVN/sus/trunk/QUAD/H1/ETMX/SAGL3/Results/ESD_Results. I ran the Long_Trend.m script, but it generated a lot of errors.
WP7929 Sensor Correction ISI end stations
Jim W, Dave:
New h1isietmx and h1isietmy models were installed, DAQ was restarted soon after.
WP7968 h1calcs channel name fix
Jeff K, Dave:
New h1calcs model was installed, DAQ was restarted soon afterwards.
The changes made in the h1calcs model restart are motivated in LHO aLOG 45548.
GV 1,2,18 were opened (slowly). EY and vertex turbos are spinning down (stations are still energized for magnetic bearing). EY air compressors are OFF.
Vertex pressure curve + RGA scans before and after opening to arms.
Big improvement already in corner pressure after fixing leak. Nitrogen peak back to normal.
EY pressure curve and RGA scans before and after opening to y-arm.
Following on from the EY ESD repair work: Once the pressure at End Y was low enough, Patrick T and I went out to turn on the high voltage and check that the ESD is operational.
We turned the HV supplies (outside the VEA) back on at 5:03pm local time, we then turned on the switches on the back of the low voltage ESD chassis (inside the VEA).
We did the same test that Fil and I ran in May - alog 41772 - driving the ESD quadrants at the OUTF stage (4.3 Hz, 30000 Cnts drive at the excitation point), and looking at the response on the optical lever. We didn't see a response to the LR quadrant drive, compare red trace to other traces in the attached screenshot, very perplexing. Could the problem be at the feedthrough?
We did no further tests this evening.
This is disheartening. It is possible to tell pretty much exactly where the break is. Look at T1800199 for a note I wrote detailing the method for finding the open circuit in a cable run. You will certainly know where in the chain the flaw exists. Richard and Fil are familiar with the technique. All things are obvious in retrospect, but we should have done this analysis method prior to going into the chamber. I was so taken by the possibility of the failure at the end of the cable nearest the optic, that I didn't think about alternative possibilities.
The feed through is tested when the continuity test is done. The pigtail that is in place is not removed so it is not the feed through.
The ETMY Oplev segments seem to be responding when I repoint the optic, so doesn't look like the Oplev readback signals are a problem.
We looked at the cables using the FieldFox method recommended by Rich, T1800199 and here are the results. Our Field Fox is limited to 2MHz on the low end.
Bias cable ends at 64' from the ESD chassis
UR, LR, UL, and LL all end at 60' from the ESD chassis.
We noted a difference in the LR reflections so we took screen shots of each quadrant. I believe the length of the loop corresponds to the length of reflection on the test mass. The loop on the first reflection of the bias trace is quite large. The loop on the working quadrants are seemingly equal and smaller than the bias. The loop on the LR quadrant is very small.
Note: Fil did the same open circuit test in May. The results of the electrode lengths are here, alog 41861.
It makes sense that LL is longer than the upper quadrant electrodes (UL and UR) given the extra length of electrode around the barrel of the AERM. LR being shorter seems to suggest a break close to the optic.
After speaking with some of the team, and reviewing Marc's data: 1. It would be a good idea to take the transfer function at RF (say 2 to 10 MHz sweep) from the air-side coax leading to the bias, out to each of the 4 quadrants. By examining these 4 transfer functions for symmetry, we can strengthen the case for there being a break in the gold mask on the LR quadrant of this optic. The connection from the incoming wire to each of the 4 quadrants is made by little soldered gold tabs. Were one of these tabs to break free, or if the pin that's soldered to the first tab on the top of the barrel of the reaction mass to come undone, it may account for the existing symptom. 2. When closely examined, the RF data taken by Marc does have 2 asymmetries in the LR plot vs the other 3 quadrants. The fact that the residual impedance at the first marker frequency (~2.69MHz) is different (capacitive for LR and slightly inductive for the others) is noteworthy, but not stunning. The precision of this type of measurement relies on knowing the characteristic impedance of the entire cable assembly. Given that these assemblies end in a single wire strung into space, it's not immediately compelling to see slight differences, and indeed there is variation in the other "good" quadrants. However if you couple this observation with the funny looking loops seen on the right side of the plots, the story gets more interesting. The funny loops are likely to be parasitic couplings to another resonant element (bias electrode?) in the cable/ESD system. A smaller loop (as seen in LR) indicates less coupling. This would fit the model of there being a break somewhere in the gold pattern distribution that exists on the reaction mass. The coupling is likely to be a cross coupling to the bias element through parasitic electro-magnetic coupling. when taking these transfer functions, it is likely that there will be enhanced coupling evident at the frequency of the loops as seen in Marc's data (manifesting in a lower loss in the RF transfer function). 3. Continuity tests are done to the top electrodes on the reaction mass barrel at the 12 o-clock position. If there was a break further down the chain (like the gold bond wires that are soldered on), then the continuity test would not catch that. Calum thinks we used to examine the gold bond wires when we did incursions relating to ESD troubles. I don't know if that was done during this vent cycle. I ran a simulation of a coaxial cable with an open lossy resonant termination, and was able to mimic the loops and overall response seen in Marc's data.
Fil, Marc, Georgia
We went back down to End-Y to run some more tests on the ESD, including that mentioned in the first point of Rich's comment.
Following tests were redone yesterday afternoon.
Looking at feedthrough with pigtail connector attached checked for shorts across:
1. Pin to Shield on each individual SHV connector
2. Pin to Pin on all SHV connectors
3. Shield to Shield on all SHV connectors
4. Pin to Chamber GND on each individual SHV connector
5. Shield to Chamber GND on each individual SHV connector
All tests passed.
Place a T adapter in line with the LR segment and monitored voltage going into chamber. Same voltage was observed when connected to chamber vs not connected to chamber.
The AERM solder joints at the optic were intact in Jan 2018 during the install (alog 40336) - although this picture doesn't show the side shot for the LR.
It's hard to image that the solder joints (large and look very good in picture) have come undone. I know for a fact that upon my inspection of the top 5 pins while in-chamber this last Tuesday, the pins were still landed well and the solder/pin joint look the same as in this picture from Jan. It is very difficult to inspect the "bridges" that connect the barrel gold traces to the face traces, and I did not look specifically at those on Tuesday.
We have some new scans from ETMY Annular End Reaction Mass (AERM), LR quadrant. We tried scanning the LR quadrant while changing the other cable configurations to determine where the coupling is strongest. On the plot we can see the LR signal as it was in prior scans. We then disconnected the Bias at the ESD which shifted the trace slightly but not significantly. Next we disconnected the Bias at the feed through and saw a much larger shift. Next we reconnected the Bias at the feed-through, and disconnected the UR signal. This made the most difference to our trace which leads me to believe that there is more coupling between LR and UR, than there is between LR and Bias, which should be the case if we are disconnected at the joint on the side of the AERM. Disconnecting the UL signal made little difference to the coupling.
We scanned the LL quadrant in the same way we scanned the LR quadrant, I will post it here as a way to compare the known good lower quadrant with the sketchy lower quadrant.
