Plots of the water flow rates, after yesterday's work on the flow sensors, are attached. The early behaviour looks promising.
Jenne, Sheila, Jeff, Kiwamu, Keita, Terra
Once the laser was back and Corey had finished inital alignment today, we made some progress:
Note that the DARM spectrum here is not in the low noise state, it is only meant to show that the intensity noise coulping (without the 2nd loop on) changes dramatically with the thermal state of the interferometer. For now we are leaving the inteferometer in the coil_drivers state, to see what happens with PI. This means we are at 50 Watts input power, recycling gain of about 23, no ISS second loop, high noise ESD driver, low noise pum states.
TITLE: 08/30 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: None
SHIFT SUMMARY: Commissioning work continues. The violin modes were rung up a bit from a large eathquake yesterday and Jeff K worked to damp them out. Sheila and I ran into some TCS rotation stage issues that we "fixed" by logging into the system manager and temporarily changing the warning flag values. Meanwhile the TCSX was putting 2W into the IFO so ITMX is a bit hot and still cooling down.
There were two versions of the RS screens out there with some differences between the two that has caused some minor confusions today. The older one was mainly just lacking information compared to the new one, so I just made sure that all the links go to the new one now.
Old: (userapps)/tcs/common/medm/TCS_LASER_POWER_CONTROL.adl
New: (userapps)/ioo/common/medm/LASER_POWER.adl
Sheila, TJ
After requesting a power for TCSX from the rotation stage medm, Sheila noticed that the RS was just spinning and not going to the correct power. She promptly hit abort where it stopped at 2W going into the IFO. Requesting a new power did not seem to work, nor did clicking abort one or five more times. This happened last week as well, but last time it mysteriously fixed itself. We weren't so lucky this time. We logged onto h1ecatx1 and pulled open some Beckhoff manuals and tried reading through the errors to diagnose why the RS got stuck. After about an hour or so of digging through errors, we traced it down the RotationstageIn.Warning flag coming from the hardware that may be stuck somehow, so we decided to force the warning to be False in the system manager and then all we well (after remembering to turn the autoabort back on). RS seemed to go to requested powers again without complaint.
When Sheila went to investigate the initial incident, she found that the RS had actually starting moving before a request right before this. She'll give more info on that later.
I was adjusting the differential TCS power in 100 mW steps and seeing no impact on the interferometer, and as a final step deicded to try 0 Watts of ITMX power and 0.2 Watts of ITMY power. This had a much more dramatic impact that the previous few similarly sized steps had, and caused the sideband powers to drop. I reset the request for ITMY CO2 to 0.1 (the previous setting) and reset it, then looke at X which was moving around to about 5 Watts. The attached screenshot shows when I set the request to 0 for ITMX CO2, and then the encoder and power meter readback start moving before the request is changed. When the interferometer looses lock the down state sets the request to 0,5 Watts for pre heating, but the rotation stage continues zooming up to almost 5 Watts of input power.
For FRSes, this cost us 2 hours so far, but it will probably take another hour or so for the test mass to cool off enough that our ASC works.
J. Kissel After the large Earthquake ~24 hours ago (see LHO aLOG 29359) that tripped all BSCs, the fundamental violin modes have been quite rung up. So I've spent the evening behind the scenes babysitting the gains of the Violin Mode damping -- trying to push the gains us in amplitude to accelerate the damping without saturating the DAC, only slowed by a few failed attempts at switching the PUM coil drivers to a lower noise (and less actuation range) state. I've basically been following the instructions as defined in Violin Mode Damping 101, and using the table and parameters defined in LHO's Violin Mode Resonances table. Occasionally, I'd get bold and nudge the phase of the filter by 60 [deg] here or there, but for the most part the filter settings as defined by guardian are good. If anyone needs them, I've saved some strip tool templates for the modes that were problematic tonight, they can be found in /ligo/home/jeffrey.kissel/Templates/StripTool/ H1SUSETMX_ViolinFundamentals.stp H1SUSETMY_ViolinFundamentals.stp H1SUSITMX_ViolinFundamentals.stp H1SUSITMY_ViolinFundamentals.stp Of course, the ranges must be tuned in proportion to whatever gain the damping loops are set, but it's a few less MEDM screens to open and buttons to click to get started.
J. Kissel While grabbing RMS Current monitors for every suspension to further research the PUM / L2 Current Limiting Watchdogs, I noticed that most of the monitors for the UIM / L1 stage of the H1 SUS ETMY suspension show an offset of -4200 [ct]. This is typically the case when a single leg of the SCSI input cable to the ADC channel is bent and therefore not connected. This problem type was originally documented in LLO aLOG 1857, and found on these particular ADC channels in Nov 2014; see LHO aLOG 14930. It had been added to the long standing Integration Issue #9 (II #9, now FRS #5135) then, but because it's likely overwhelming generic, I'll open up an FRS ticket for this specific issue. Should be a quick fix. This is now FRS Ticket #6114.
over 120 days
3:30pm local 36 min. 26 sec. to overfill CP3 with 1/2 turn open on LLCVBV. Based on CP4 tests, the pump was probably right at 100% full. I'm going to leave LLCV at 19%.
Observatory Mode: taken from Commissioning to Corrective Maintenance (for PSL work) for most of the morning
Atlantic/Africa 7.1 earthquake last night around 10pm. It tripped Test Masses & TMSy.
Once the PSL was brought back (around noon), I performed an Initial Alignment (lost the MC for a bit during the PRM portion of procedure). After the Alignment was complete, H1 was handed over to the Commissioners & they've had a few 50W locks.
Jeff K damped down some rung up violin modes (most likely due to the earthquake).
Following on from Jason's log from yesterday (29356). The crystal chiller was engaged without turning the high power oscillator. The flow rates for heads [1-4] were monitored. Every now and then the flow rate for head 4 would drop. Whilst monitoring the flow rates, a trend plot of the relative humidity was generated. It indicated a slow rise in the humidity towards the alarm level of 60%, which from previous experience is the foretelling of a leak. The high power oscillator cover was removed and the insides of the laser was inspected. No signs of a leak were observed. No unusual bulges in the cooling hoses were observed either. The output of the flow sensors for heads 2 and 4 were switched. The signal noise followed the switch in flow sensor, leaving to the conclusion that the output electronics for head 4 was bad. The LM2907-N frequency to voltage converter chip for that channel was replaced with one from an old board (the one closest to the edge of the table, labelled N4). The chip labelled N3 was also replaced due to noise observed in the flow of head 3. Since replacing the chips seemed to have improved the noise situation, the decision was taken not to remove flow sensor 4. The moisture sensitivity level (MSL) of the LM2907-N is 1 according to Texas Instruments, meaning that its lifetime is unlimited provided the relative humidity is less than 85% and the temperature is less than 303 Kelvin. The chips that were replaced happen to be the two closest to the head 3 sensor leak that occurred recently. It is not impossible that these chips got wet when the hose burst, and hence the erratic behaviour since. As an aside, the diode current for diode box 3 went back to reading ~50 A after the flow sensors were reconnected. However since that time it has gone back to reading over 100 A. Jason/Jeff/Peter
The new prototype board was modified to satisfy some of the things in the requirement (T1600064) as listed in 1., 3. and 5. below. I also listed things that were not modified but would have been good if recommendations in T1600064 were implemented.
I wouldn't claim that we won't modify it further, but this is just the current state.
In this entry you'll be able to see the electronics modifications in the attached, but the photos are too small to read the original component values, so read the original drawing D1600298 as necessary.
1. Readback of the PD array signal after the servo loop switch ("SUM PE MON") needed to be DC coupled.
The "whitening" was originally AC-coupled (second attachment) and therefore it was not compatible with digital AC coupling described in the requirement.
We bypassed the big caps in the input of the opamps with some resistors so it acts as one single pole at 2.7k with DC gain of 50 (second attachment, mod 1).
To make the OLTF measurement easier, we made the same change to the whitening downstream of the summation for the excitation DAC output.
The servo board output readback was also AC-coupled, but we want to see if everything is working fine including DC injected into the inner loop board.
We made the output whitening a true whitening (second attachment, mod 2). 0.35Hz zero might be too low, though.
2. The board doesn't implement the analog compensation for the in the inner loop filtering upstream of the error point.
The inner loop error point "whitening" works as a very steep boost seen from the outer loop, roughly an equivalent of (z, p) = ([3; 3; 130], [0.07, 0.07]). T1600064 reccomends to implement the same thing in the outer loop somewhere to cancel this effect to make the loop design simpler.
Since this is not compensated in the prototype board, the outer loop OLTF would become huge for f<3Hz. This means that the digital AC coupling servo should work REALLY hard to effectively AC-couple the entire outer loop at, say, f<1Hz, without reducing the outer loop gain at 10Hz and up.
We do not implement this by analog cut and paste job (no opeamp available for this on the board), but we'll try to make it work by an aggressive digital AC-coupling filter.
3. Integrator as a boost doesn't go well with digital AC coupling so it was converted to a usual boost with finite DC gain.
The error point of the digital AC coupling servo is kept zero at DC (i.e. a pole at zero Hz). The outer loop prototype had an integrator as a boost.
Of course they don't go well as any tiny DC difference between the AC coupling error point and the integrator input will be integrated and eventually the servo runs away. But we don't really need a pole at zero Hz.
We changed it such that the first boost acts as 50Hz LPF with DC gain of 10, and when added to a flat unity gain it becomes 500:50 boost (third attachment) (but see the next entry).
4. Boosts were implemented as three parallel integrators added to a flat unity gain.
We had a choice of unity gain, zp=50:0, 100:0 or 150:0. See the first and the third attachment.
As described in 3. above, we already changed one integrator to []:50 LPF so the boost becomes 500:50.
We didn't fix the parallel summation, it might be OK to go without any boost or using just one.
But T1600064 shows our standard practice of connecting boosts in series.
5. Outer loop servo filters on the board were also changed.
Originally the servo filtering was something like (z,p)=(100, [20^2;40]) (fourth attachment). As described in 2. above, 1st loop effectively adds ([3^2; 130], [0.07^2]). IMC adds another pole at ~8kHz.
Everything added together it's like ([3^2; 100; 130], [0.07^2; 20^2; 40; 8k]), and this might be OK.
But we made a change so the board became ([380;9k],[20^2;600]) by changing one of zero-Ohm resistors in the second stage and one C-R pair in the third stage (fourth attachment). Everything added together it would be ([3^2;130;380;9k], [0.07^2; 20^2;600; 8k]).
This change might not have been necessary. We just felt that letting a big 4.7uF capacitor and a zero-Ohm resistor to determine AC gain at around 10kHz in the first as well as the second stage was maybe too much of an uncertainty in the AC gain.
6. Sallen-Key at 0.1Hz for Digital AC coupling path might be too aggressive
For the moment the Sallen-Key is disabled for the digital AC coupling path (so the only analog filtering is a 10Hz LPF) to make things easier as we try to make it work.
Outer loop and digital AC coupling topology
Error point for the digital AC coupling is upstream of the summation point for the outer loop and the third loop (ERR1 in the attached medm). This comes from two requirements:
If the error point is downstream, the 3rd loop gain will be eaten by the AC coupling loop.
This means that, as we change the VGA gain, the offset coming between the AC coupling error point and the input to the VGA will be amplified and injected downstream, and if this becomes the problem we need to compensate for that using the analog excitation offset.
Second attachment shows the AC coupling OLTF, with the setting shown in the first attachment.
I was looking (again) at the ISS inner loop circuit diagram to make sure that I understand the system correctly and found that my alog 26879 was wrong about inner loop diode in that one pole and one zero were somehow swapped (see the new entry I've just made as an attachment to the alog 26879.
Anyway, since the amplitude actuation of the outer loop summation point of the inner loop is the inverse of the inner loop sensing "whitening", 2. should read
"The inner loop error point "whitening" works as a very steep boost seen from the outer loop, roughly an equivalent of (z, p) = ([3; 3; 2.34k;2.7k], [0.07;0.07;130])".
Sorry for a confusion, it seems like it's too late to fix the original alog entry above.
Correction of correction (sorry).
Original alog 26879 seems to have been correct, therefore the origial entry of this alog should be correct.
The inner loop error point "whitening" works as a very steep boost seen from the outer loop, roughly an equivalent of (z,p)=([3; 3; 130], [0.07, 0.07])
Sorry for further confusion. Again I cannot correct the above original entry nor the above correction any more, thus this entry.
This morning, I started running ETMX and ETMY charge measurements in an effort to utilize today's PSL downtime window instead of tomorrow. While looking around at SUS SDF settings during the measurements, I found all 4 test mass L2 EUL2OSEM matricies in a funny state. The snapshot attached shows the ITMY settings as left from Saturday - note all 4 test masses show the same matrices settings (and therefore have the same SDF screen diffs). Jenne points out that these setting diffs are from a relatively new implementation of the 3-coil switching alog 28959 and had not been put into the Guardian DOWN because they hadn't been fully exercised yet. She has now added them to the appropriate place in Guardian to be restored back to nominal.
ITMX seemed to be extra sensitive to larger ground motions--it was often the only ISI to trip. Of course somebody had to be the most sensitive so could be a wild chase. FRS 5948.
LHO aLog 28567 -- Investigation first report finds wierd reversal of current in Stage2 H2 CD. Some more data from EQ (only ITMX tripped) 28621.
LHO aLog 28649 -- Direct excitation of CD leads to finding bad voltage regulator on coil driver. Entire unit replaced (no spare voltage regulator available.) No funny behavior seen on Current monitor.
LHO aLog 28948 -- 9 Aug: Original Coil Driver replaced (with replaced voltage regulator) but no excitation drive experiment done to confirm good behavior.
Today, drove Coil Driver w/ 3000 Ct 0.05 Hz sine, I_INMON responds to greater than 110 counts with no ill behavior observed, see attached. Before repair, the I_INMON reversed and spiked to very large output (340cts) when the current reading hit 50-60 counts. This suggests the coil driver is back to nominal behavior. And, there have not been any ground motion events where ITMX was the only platform which tripped.
Deem this problem fixed and will move FRS 5948 to pending and recommend closure.
New ISS outer loop prototype board was modifed and put in. Modification details will be in a separate alog.
PD5, 6, 7, and 8 anode and cathode cables were disconnected from the old transimpedance box under HAM2 and was connected to the PD5-8 cable for the new prototype board. One strange thing about the old setup was that the "LR C" and "LR A" cable that come from the chamber feedthrough were connected to the anode and cathode input of the old box, respectively. Since apparently they gave us a correct signal (no forward bias across the diode), I connected "LR C" chamber side cable to "LR A" rack side cable for the new setup.
I connected PD5-8 cable to PD1-4 input in the new prototype box. We can conveniently switch back and forth between the old and the new by just switching one cable that goes to the "outer loop input" connector on the 1st loop chassis.
I briefly connected the output of the new box to 1st loop, and enabled the servo, but it didn't work. It's not clear if the sign is correct, but there's no way to change the sign externally, I need to open the box. I'll make more measurements next week to see if the sign is indeed wrong.
I connected the old second loop back to the 1st loop.
Cold solder joint in the short circuit protection box.
Initially I was confused with the gain control of the VGA chip in the new box as there seemed to be a factor of 2 missing in the gain control chain. It turns out that PSL people put a small box on the anti-imaging chassis front panel in CER. These boxes are just DB9 pass-through except that all traces have 100 Ohm resistor in series.
Anyway, pin1 of this box, which corerespond to the positive leg of the VGA gain output, didn't have any signal coming though. When I opened the box I was baffled to find no error, but it turns out that just a tiny amount of bending would cut the connection. I re-soldered the resistor and the connectors, and now it seems to be good.
But this is not the reason why the new servo didn't work.
This is the box that was fixed.
There appears to be an entry in DCC for S1201761, but I don't have permission to view that.
Update:
Vern has the permission to view S1201761 and has found that the corresponding D document is D1102351, aLIGO PSL DAQ DB9 breakout. I can view that D document without problem.
The summary on that page doesn't sound like it was for short circuit protection as I have initially guessed, but an in-line damper against high frequency oscillation of AI output amplifier. I don't know if we need this, but if we do, I'd strongly suggest to improve the mechanical design (or move them inside the AI chassis).
It appears that something goes wrong when zooming in that creates random lines across the plots.
Pyplot does some strange things with the data when you zoom in, sometimes. Maybe this is a result of data gaps being handled poorly by pyplot? I've been able to get these artifacts to go away by resetting the plots and zooming in slightly less, but Patrick and I weren't able to get these particular ones to clean up. I'll see if I can make this a little nicer on Monday.
It looks like using NaN to fill in the gaps in the data was not the right thing to do. Filling with POF_INF seems to eliminate the glitching in the plots. I've also set some hard coded Y-axis limits on the pit and yaw plots and scaled the sum plots to the max in each dataset, so the plots should start out closer to a finished product.
I've also added the HAM2 oplev, which doesn't have any DQ channels, so I've used the OUT16 channel. Shouldn't matter much, since the plots are of m-trends.
Terra and I used the PI damping infrastructure to excite the butterfly and drumhead modes on EY, and then ring them down.
We excited the butterfly mode (6053.9 Hz) during a 50 W lock. The observed ringdown time was 23.5 minutes (= 1410 s), giving a Q of 27×106.
We excited the drumhead mode (8158.0 Hz) during at 2 W lock. The observed ringdown time was 13.5 minutes (= 810 s), giving a Q of 20×106.
The templates containing the spectrum data for these ringdowns live in my directory under Public/Templates/SUS/BodyModes.
In PI model:
MODE 29 = ETMX Drumhead
MODE 30 = ETMX Butterfly
MODE 31 = ETMY Butterfly
MODE 32 = ETMY Drumhead
The attached plot shows the expected ratio of the surface strain energy (in J/m) on the test mass face to the total strain energy (in J) in the test mass for the body modes between 5 kHz and 11 kHz. This is a simple Comsol model with a perfect silica cylinder.
Evidently, the drumhead and butterfly modes have similar energy ratios, so we should not expect their Q factors to be too different. It might be good to try the 9.2 kHz modes, since their energy ratio is rather different from the drumhead and butterfly modes, and they produce test mass strain in the beamline direction (the modes at 8.25 kHz and 9.4 kHz do not).
