WP6120
Daniel, Keita, Jim, Dave:
We installed a spare Dolphin card into the h1psl0 front end computer. The sequence was:
We found that the PSL-OPC shutter still closed, investigating this offline.
I checked that the received data on the two new dolphin channels were correct, and they are.
I noticed that the channels are called TR_[X,Y]_NORM but they are picked off before the NORM filter modules in the LSC model, so perhaps not normalized or incorrectly named.
Running a Worden experiment: -CP4 at 88% full -Set LLCV to nominal in manual mode (41% open) -Increase LLCV by 5% (43% open) and leave it at this value -Monitor increase in fill -Allow fill to exceed 100% and monitor exhaust flow meter (expect a sharp increase in flow when liquid starts to hit warm exhaust) -After this sharp increase, reduce LLCV by 10% to below nominal (<41%) -Hope that no LN2 reaches the flow meter this time (when flow approaches 100% I will monitor while sitting next to exhaust to be ready to open bypass exhaust valve and remove flow meter with LN2 gloves on) **THIS WILL GENERATE CP4 ALARM ON PUMP LEVEL**
DIAG_MAIN was reporting that IM2 P and Y were out of normal range, and after trending them I saw that they were WAY out of range (Normal for H1:SUS_IM2_M1_DAMP_P_INMON is 603 and -205 for Y). The values now are at P = 0 and Y=-19. The trend shows a sudden move at around 11:15UTC (4:15PST), when no one was here and the IFO was unlocked. The most recent lockloss before that was at 8:03UTC. The suspension has been aligned with no trips since the 18th, and HAM2 has been Isolated since the 19th, no trips.
The other IM's also seem to show signs of movenment then, but nothing as extreme as IM2. 1&3 seemed to get noisy after 11:15. (see attachment)
What could have caused this move?
Chandra, Kyle Attached are scans following the most recent bake of the Vertex RGA. The pressure in the Vertex at the time of these scans was 3.4 x 10-9 torr as indicated by the nearest CC gauge. The bake out of this RGA was resumed following these scans.
[Jenne, Terra]
We've been pretty frustratingly plagued by the CSOFT / dPdTheta instability today. Earlier today with Sheila, we were able to use the offsets from yesterday, but then later in the evening those offsets make things unstable when we go up in power. I'm starting to wonder if an initial alignment needs to be done, since the green arm powers (measured at LockingArmsGreen, or anything early in the sequence) have been decreasing with each lock. Maybe that will help?
Sheila migrated the Soft offsets into the trans QPDs, so the IFO now aligns to that placewhen the soft loops come on. Also, the POPA offsets are engaged during ASC_Part2, so the IFO is aligned to yesterday's place. However, with this alignment at 2W, we cannot engage the roll mode damping. The last few hours we've been skipping over EngageRollModeDamping, and I've commented out the final roll mode gain setting that used to happen in DC readout. We don't seem to usually have much of a problem just leaving the damping off, but we can turn it on once we're at at least a semi-high power (maybe 20+ W?).
Since we kept being troubled by the CSOFT instability, we went back to trying new offsets. We think there's a little more that could be done, but we have a place where the recycling gain stays fairly constant as we power up. The recycling gain looks terrible at 2W (30ish), but then only decays to 28ish. This is in comparison with the usual decay to 22ish. Also with this alignment, when we're at high power, the green arm transmissions are both high. This seems like something that we want, since we think that the green and red beams are pretty well co-aligned after the Soft loops come on, so hopefully this means that we're closer to finding a place that maintains the alignment of the IFO from 2W to 50W.
In the SDF screenshot, the POP_A offsets in the setpoint are what Sheila and Terra found last night, while the current values are the place that we like best so far tonight. The setpoint offsets for the soft loops aren't meaningful, but the current values are the ones that we like from tonight, which are on top of last night's offsets (which have already been put into the QPDs and accepted in SDF).
Attached also is a screenshot showing two striptools (and a bunch of other stuff that's basically ignorable). Right around -100 minutes is the inital power-up and corresponding power recycling gain drop. You can see that as we move offsets around, we're obviously changing the red and green arm powers, as well as the recycling gain. Note though the time around -15 min where both green arm transmissions (blue and teal) are high, and the PRC gain is high. Xarm green seems most strongly affected by Pop_A_pit, while Yarm green seems most strongly affected by DSOFT.
To help maintain these locks, we were increasing the ISS 3rd loop gain from -1 to -10. Also, we increased the SOFT Yaw gains from 3 to 7, and the Soft Pit gains from 0.5 to 0.7. This seemed to ameliorate most instabilities, although we still are sometimes struggling to hold the lock - I think it's perhaps an indicator of needing an initial alignment.
H1 Commissioning
Jenne, Terra, & Sheila were at the helm for H1 higher power work.
MEDM work
Made a corrections to the sitemap (IFO Common Mode Servo on pull-down), and to the Digital Video Overview (added name for POP Air camera at the CAM30 postion).
Stranger Things Happening Around Gate Tonight
Happened to catch a vehicle rolling on site around 9:45 (it turned toward the LSB, turned around, & lingered before leaving). The other oddity here was this car illuminated another vehicle as it approached. This vehicle was at the gate with lights off. Monitored the care for about 30min, called out to the Gate phone, and then three of us went to investigate. There were about 4-6people, and they were looking at the stars. They said they had talked to someone on the phone about coming out to watch the sky.
Upon returning to the OSB, we checked the external front door lock to make sure it hadn't been unlocked (luckily it was locked). The group eventually left (so they were here for atleast an hour.
Per FAMIS #7068, Saw that two ISIs needed their L4C WD counters to be cleared.
Jeff K, Darkhan T,
Last Tuesday we updated the infrastructure for injecting calibration lines (see LHO alogs 29245, 29249). Below is the table of currently active calibration lines:
Channel Names _FREQ (Hz) _SINGAIN (ct)
H1:CAL-PCALY_PCALOSC1_OSC 36.7 125 O1-scheme kappa_TST / kappa_PU
H1:CAL-PCALY_PCALOSC2_OSC 331.9 2900 O1-scheme kappa_C / f_C
H1:CAL-PCALY_PCALOSC3_OSC 1083.7 15000 high-frequency calibration check ("bonus" line)
H1:CAL-PCALX_PCALOSC1_OSC 3501.3 39322 high-frequency sensing function characterization ("mobile" line)
_FREQ (Hz) _CLKGAIN (ct)
H1:CAL-CS_TDEP_DARM_LINE1_DEMOD 37.3 0.1 O1-scheme kappa_PU
H1:SUS-ETMY_L3_CAL_LINE 35.9 0.11 O1-scheme kappa_TST / kappa_PU
H1:SUS-ETMY_L1_CAL_LINE 33.7 11 O2-scheme synched oscillator for kappa_U
H1:SUS-ETMY_L2_CAL_LINE 34.7 1.1 O2-scheme synched oscillator for kappa_P
H1:SUS-ETMY_L3_CAL2_LINE 35.3 0.11 O2-scheme synched oscillator for kappa_T
We plan to adjust the three O2-scheme line frequencies and amplitudes, and cancel them out with PCALY (they will not appear in the reconstructed DARM spectrum), following synchronized oscillators will be utilized for this purpose H1:CAL-PCALY_PCALOSC{4-6}_OSC.
We were locked last night at 50 W for ~2.5 hours and lost lock from an ITMX 15522 Hz PI. This is a known PI seen months ago; I had purposefully left damping settings off to see if it rang up. See first attachment showing lockloss.
This afternoon we were locked at 50 W for ~2 hours and I let the mode ring up so I could demonstrate successful damping. See second attachment showing damping.
All PIs that were previously observed have now been seen and damped post OMC vent. We have 5 PIs at 50 W (at least up to ~3 hour locks): ITMX 15520 Hz; ETMX 15541 Hz; ETMY 15542 Hz, 15009 Hz, 18041 Hz (aliased from 47495 Hz). All are successfully damped via the guardian and have had their damping phase and gain optimized.
Last week, Keith posted the results of a study of folded magnetometer channel data (alog 29166) aimed at understanding the results of recent changes to the timing system (primarily LED reprogramming and power supply switching). This is a follow-up, looking at the spectra of the same channels, and tracking the behavior of the two combs which the timing system interventions were intended to mitigate.
Detailed plots
Overview table (daily spectra, selected dates)
Full data set (daily spectra)
Full data set (cumulative spectra since Jul 1 2016, covering date ranges where Fscan SFTs were available)
These plots were generated from Fscans + spec_avg_long + my own plotting tools.
Timeline
July 14-21 comparison: before and after initial updates to timing slave card firmware (blinking LEDs turned off in many places, but not on timing fanouts)
July 21-Aug 6 comparison: firmware updated for EX fanout; CPS timing fanout power supply changed
Aug 6 - Aug 18 comparison: firmware updated for CER, MSR, EY fanouts
Notable features
This afternoon I ramped CP4's LLCV in 5% increments every 2 minutes, from 39% to 88% open, from 88% full to 100% full, for more data from exhaust flow meter. Fill level SP has been reset to 92% and level is slowly coming back down at 20% open on LLCV. Kudos to Patrick for writing an effective PI-code. It works very well for overfill scenarios!
Gerardo, Chandra On Tuesday, Aug. 23rd, we adjusted potentiometer on PT-140a (pirani) again - this time CCW 11 turns. Since Gerardo terminated cables for AIPs, gauge voltage has changed again and needs to be adjusted again so CC does not keep tripping due to set point interlock.
One more adjustment to the potentiometer since the CC interlock tripped a couple of times since the last change. 6 more turns CCW.
TITLE: 08/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Commissiong work continues.
LOG:
1425 - 1440 hrs. local -> To and from Y-mid Opened exhaust check-valve bypass-valve, opened LLCV bypass valve 1/2 turn -> LN2 @ exhaust in 1 minute 20 seconds -> Restored valves to as found configuration. Next CP3 overfill to be Friday, August 26th.
I was asked to summarize the SWWD (software watchdog) timing sequence as a reminder.
t=0: SUS IOP detects top OSEM RMS exceeds trip level, starts its 1st countdown (5 mins)
t=5mins: SUS IOP 1st countdown expired, its IPC output goes to BAD and it starts its 2nd countdown (15 mins). SEI IOP receives the BAD IPC, and starts its 1st countdown (4 mins)
t=9mins: SEI IOP 1st countdown expired, its IPC output goes to BAD and it starts its 60 second 2nd countdown. SEI user models get the 60 second warning IPC so they can cleanly shutdown before the DACs are killed
t=10mins: SEI IOP 2nd countdown expired, DAC cards associated with the chamber the SUS is located in are killed
t=20mins: SUS IOP 2nd countdown expired, SUS DAC cards are killed
For the hardware watchdog (HWWD) the times are doubled. The power to the ISI Coil Driver chassis is removed after 20 mins of continuous SUS shaking.
Tagging SEI and SUS.
S. Dwyer, J. Kissel, C. Gray After successfully recovering the IMC's VCO and recovering the IMC (29264), we were able to get up through LOCKING_ARMS_GREEN in the lock acquisition sequence. However, we found that ALS COMM failed caused lock losses during the next step (LOCKING_ALS), when the input for IMC length control in its Common Mode Chassis was switched from the IMC's PDH output to the ALS COMM PLL output. The ALS COMM PLL output is connected to IN2 of the IMC chassis that had a new daughter board installed in the star-crossed ISC rack H1-ISC-R1 today (LHO aLOG 29250). After fighting through MEDM screen confusion* at the racks, we found that OUT2 (an analog pickoff pick-off just after the input gain circuit) indicated a ~2.5 [V] offset, even with IN2 terminated with 50 [Ohms]. Suspecting that this symptom was indicative that the input gain circuit (circled in red in MEDM screen capture) was yet another causality of the unfortunate rack power mishap today (LHO aLOG 29253), we've replaced the entire chassis (which lives in U14 of H1-ISC-R1) with a spare we found in the EE shop -- S/N S1102627 (or Board S/N S1102627MC). Notably, this spare does not have one of the new daughter boards on which Chris has worked so hard. We're not suggesting this swap be permanent, but we make the swap for tonight at least, so we can hopefully make forward progress. We suggest that IN2 and/or the input gain stage of SN S1102626 be fixed tomorrow, and the chassis restored so we can employ the new daughter board. Other Details: - Before removing the chassis, we powered down the entire rack using the voltage sequencer around the back at the top of the rack. - After installing the rack, we were sure to have all cables connected appropriately before turning the rack power on again (via the sequencer again). - We added a few labels to the IMC's PDH output and the ALS COMM PLL output cables such that they're easier to follow and reconnect in the future. *MEDM Screen Confusion -- whether IN1 or IN2 is fed into OUT2 of all common mode chassis is selectable on their MEDM screens. For the IMC's common mode board (at least for SN S1102626), the MEDM screen's indication of the status of that switch is exactly backwards. When the screen indicates that IN1 is feeding OUT2, IN2 is feeding OUT2, and vice versa. #facepalm
With Sheila's help, the OUT 2 switch should now be correct for the MC Common Mode Servo medm (H1IMC_SERVO.adl). This change was committed to the svn.
M. Pirello (reported by J. Kissel from verbal discussion with F. Clara) Marc has inspected the Common Mode Board chassis we've removed (SN S1102626), and indeed found several blown transistors and opamps -- and is not even through the chassis test procedure. Unfortunately, the EE shop needs a restocking of surface mount components before we can make the repairs, but the plan is to shoot for a re-install of this board by next Tuesday (Aug 30th).
Repairs to S1102626 are complete and the chassis has been tested with the 200kHz low pass filter. The chassis performance is similar to the previous test performed September 2011.
When the 200kHz low pass filter is activated we detected a 3mV dc offset which should be noted. The low pass filter works as designed with -3dB gain at 200kHz and rolls off nicely. I have attached files from the testing. File details can be found in the readme.txt included in the zip.
Summary: Repeating the Pcal timing signals measurements made at LHO (aLOG 28942) and LLO (aLOG 27207) with more test point channels in the 65k IOP model, we now have a more complete picture of the Pcal timing signals and where there are time delays. Bottom line: 61 usec delay from user model (16 kHz) to IOP model (65 kHz); no delay from IOP model to user model; 7.5 usec zero-order-hold delay in the DAC; and 61 usec delay in the DAC or the ADC or a combination of the two. Unfortunately, we cannot determine from these measurements on which of the ADC or DAC has the delay. Details: I turned off the nominal high frequency Pcal x-arm excitation and the CW injections for the duration of this measurement. I injected a 960 Hz sine wave, 5000 counts amplitude in H1:CAL-PCALX_SWEPT_SINE_EXC. Then I made transfer function measurements from H1:IOP-ISC_EX_ADC_DT_OUT to H1:CAL-PCALX_DAC_FILT_DTONE_IN1, H1:IOP-ISC_EX_MADC0_TP_CH30 to H1:CAL-PCALX_DAC_NONFILT_DTONE_IN1, and H1:CAL-PCALX_SWEPT_SINE_OUT to H1:CAL-PCALX_TX_PD_VOLTS_IN1, as well as points in between (see attached diagram, and plots) The measurements match the expectation, except there is one confusing point: the transfer function H1:IOP-ISC_EX_MADC0_TP_CH30 to H1:CAL-PCALX_DAC_NONFILT_DTONE_IN1 does not see the 7.5 usec zero-order-hold DAC delay. Why? There is a 61 usec delay from just after the digital AI and just before the digital AA (after accounting for the known phase loss by the DAC zero-order-hold, and the analog AI and AA filters). From these measurements, we cannot determine if the delay is in the ADC or DAC or a combination of both. For now, we have timing documentation such as LIGO-G-1501195 to suggest that there are 3 IOP clock cycles delay in the DAC and 1 IOP clock cycle delay at the ADC. It is important to note that there is no delay in the channels measured in the user model acquired by the ADC. In addition, the measurements show that there is a 61 usec delay when going from the user model to the IOP model. All this being said, I'm still a little confused from various other timing measurements. See, for example, LLO aLOG 22227 and LHO aLOG 22117. I'll need a little time to digest this and try to reconcile the different results.
By looking at the phase of the DuoTone signals we can constrain whether there is any delay in ADC side (like Keita's analysis here). The DuoTone signals are desgined such that the two sinusoidal signals 960 Hz and 961 Hz will be maximum at the start of a GPS second (and also in phase with each other). To be presice, the maximum will be 6.7 µs delayed from the integer GPS boundary (T1500513). The phase of 960 Hz signal at IOP (L1:IOP-ISC_EX_ADC_DT_OUT) is -92.52 degrees with respect to GPS integer boundary (LLO a-log 27207). Since the DuoTone signal is supposed to be maximum at GPS integer boundary i.e, it is a cosine function, this corresponds to -2.52 degrees (estimate of 92.52 assumes it is a sine function) phase change. Converting this phase change to time delay we get 7.3 µs. Since there is an inherent 6.7µs delay by the time the DuoTone signals reaches the ADC, we are left with only 0.6 µs delay possibly from ADC process (or some small systematic we haven't accounted for yet). This is what Keita's measurements were showing. Combing this measurment and above transfer function measurments we can say that we understand the ADC chain and there are no time delays more than 0.6 µ in that chain. This also suggest that the 61 µs delay we see in ADC-DAC combination exist completely in DAC side.
The DuoTone signals are sine waves, so a minor correction to Shivaraj's comment above, the zero-crossing corresponds to the supposed GPS integer second. I looked at a time series and observe that the zero-crossing occurs at ~7.2 usec. Since the analog DuoTone signal lags behind the GPS second by ~6.7 usec, I can confirm that the ADC side has essentially no delay. Thus, the 61 usec seen through the DAC-ADC loop is entirely on the DAC side. Attached is a time series zoom showing the zero crossing of the DuoTone signal.
When using dtt to make a transfer function measurement between an IOP model and a user model, one has to keep in mind that dtt does another decimation silently. This is due to dtt trying to match the number of data points between two models. Fortunately, this does not seem to affect the phase, see my note at https://dcc.ligo.org/T1600454.
Updated the timing diagram for consistency with other timing measurements (LHO aLOG 30965). See attached PDF to this comment.