Upon being made aware that the Corner Station instrument air compressor(s) were creating a measureable noise in the IFO (see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=53248), we have made various attempts to reduce or eliminate this.
Recall that 50 -100 psi is required to hold the 44" pneumatic gate valves GV5, GV6, GV7 and GV8 open. To do this, we use air compressors and dryers remotely located in the chiller yard and then route this to the LVEA via piping. The compressors are isolated from the ground via spring mounts and are isolated from the building via a flexible connection to the piping. Our first attempt at noise reduction was to shut-down the mechanical compressors and dryer, isolate them from the building piping and to then substitute regulated compressed bottled gas (N2) on the building side of the isolation. This is the preffered solution as it is the quietest but the numerous small leaks-to-atmosphere currently present in the building piping rendered this a maintenance nuisance - requiring frequent bottle changes (see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=53271).
Next, we realized that the piping leaks which proved a significant loss when using the bottle-supplied gas were quite insignificant when compared to the constant loss-to-atmosphere associated with the continuous regeneration of the "swing style" desiccant dryer when using the compressor system. For this system, one desiccant tower dries the a majority fraction of the air supplied from the compressors and then routes this dry air the LVEA. Simultaneously, a minority fraction of the air supplied from the compressors gets routed through the 2nd tower during its regeneration phase. This "regeneration" air exhausts to atmosphere. This system works excellent as a drier but is wasteful and requires that a constant supply of compressed air be exhausted to atmosphere.
The recent configuration changes incude the addition of check valves downstream of the dryer (prevents building air from backflowing to atmospher via regenerating drying tower when the compressor is off), moving the pressure switch sampling from the compressor receiver tank to downstream of the newly installed check valves (prevents compressors from starting if the auto drain depressurizes the compressor receiver tank), the addition of a NC solenoid valve between the compressors and the dryer (prevents compressor receiver tank from supplying regerating drying tower when the compressor is off) and the rewiring of the auto drain solenoid valve such that it only becomes energized when the compressor is running (prevents unnecessarily depressurization of the compressor receiver tank during off periods).
Good work, and nice visual!
Still recovering from the slew of Pacific EQs. The most recent one hit about an hour ago, and finally am able to lock both green arms (intervened on the Y-arm). Seismic noise is still high, so will leave in EARTHQUAKE state and hang out in LOCKING ARMS GREEN for a little while while I go heat up lunch.
Dave, Corey, Rahul
Yesterday, Cheryl made changes on the ETMX mode 6 damping filter settings (alog 54043), which she did not load to make sure that we don't go out of observing. This morning I loaded these changes (alog 54063) . However, it didnt go well and etmx mode 6 got excited very quickly (monitor level 5.5ish), after IFO was relocked (lockloss due to an earthquake) this morning and new settings were initiated. I manually adjusted the gain (reduced it from +5 to +1) which worked and the peak excitation went below 1.0. However according to Cheryl's alog the changes would also affect ETMY mode 6 (510.7264Hz) - which I still have to understand how.
In the meanwhile to keep things simple and buy some time, I requested Dave to revert this specific change and he will be posting a separate alog on this.
I checked Cheryl's MODE6 modified H1SUSTEMX.txt as svn r20778. I then restored the filter file from last Wednesday which was loaded into h1susetmx. This file has been checked in as r20779.
This morning we had a pair of earthquakes roll through from south and north of us! Each of the earthquakes reached the 2um/s in the 0.03-0.1Hz band. Here they are:
We almost made it through this one! But it ended H1's 65hr lock & while relocking....
18:18: Back to NOMINAL LOW NOISE, but Had Couple of Issues To Address
1) H1 SUS ETMx Violin MODE-6 Filter Change from over weekend Loaded during Down Time, but ETMx violin MODE-6 immediately rung up
h1susetmx on the CDS Overview had a CFC diff from changes to violin mode filters over the weekend. This morning Rahul reviewed & hit LOAD COEFFICIENTS for this cpu. Once we were locked & the new filter changes were engaged, ETMx MODE 6 immediately rung up. Rahul & Dave addressed this by reverting the changes. Rahul will alog details.
2) H1SUSTMSx SDF Diffs
REVERT was selected for M1 TEST P & Y (this was related to some INCREASE FLASHES work TJ was doing for Green Arm Locking during down time). The thought was he was going to revert these P & Y channels back after his work, but he thinks he just selected REVERT on the SDF page, and this was the diff which came up. He unselected this and Diffs went away!
19:12 Back to OBSERVING, but....
19:17 EQ Alert & Lockloss due to 5.6 off of Vancouver Island!!
This was a close (& much bigger EQ), and I tried transitioning to EARTHQUAKE, and moments later LARGE_EQ_NOBRSXY, but this large EQ caused several Watchdog trips to:
We'll be DOWN for a while!
19:54 Neargy EQ Alert: 5.8 off of Vancouver Island! More Watchdogs trips!!
The changes made to the violin mode damping filters were not loaded, hence CDS was flagging this for hi1susrproc and h1susetmx. I have loaded the coefficients and now the message is gone. I have attached a screenshot of the differences which CDS picked up (band pass filter changes in the ITMX mode 7, mode 8, ETMX mode 6).
TITLE: 12/23 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
SEI_CONF state: EARTH_QUAKE
Wind: 6mph Gusts, 4mph 5min avg
Primary useism: 0.16 μm/s
Secondary useism: 0.28 μm/s
QUICK SUMMARY:
Just had a 5.7 Central America EQ end our 64hr lock---if we could have only lasted a min longer, we probably could have ridden through it, too! Anyway--back to locking!
TITLE: 12/23 Owl Shift 08:00 – 16:00 (00:00-08:00), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Quiet shift, no range dip until morning increase in 3-30 Hz ground motion. Microseism still declining. Locked 64.5 hours, Observing 18.5 hours.
LOG:
ITMY modes 12, 17, and 18 have been rising throughout the lock, mode 12 being the fastest at a slope of ~0.5/day
15:53 (07:53) Incoming 5.7 mag EQ from Guatemala
15:55 (07:55) Going to EQ seismic configuration
Ops Shift Transition: 12/23/2019, Owl Shift 08:00–16:00 (00:00-08:00) - UTC (PT)
State of H1: Locked
Intent Bit: Observing
Weather: 0-5 mph wind
Primary 0.03 – 0.1Hz: 0.01 um/s
Secondary 0.1 – 0.3Hz: 0.3 um/s
Outgoing Operator: Ed
Quick Summary: Locked for 56.5 hours, Observing for 10.5. Microseism declining, wind down from earlier.
TITLE: 12/23 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 103Mpc
INCOMING OPERATOR: Niko
SHIFT SUMMARY:
H1 Locked and Observing for entire shift - 56 hours so far
There were a couple of DCPD glitches and a flurry of EX and EY saturations (2 each that occurred between 04:29 and 04:35
Guam Earthquake didn't shake a lot. Niko will switch back to Windy at his discretion. Handing off.
LOG:
06:42UTC got a couple of verbal alerts for a mag 6+ EQ from th Guam area. I switched H1 SEI_CONF to EARTHQUAKE. Seismon says it's still about 38 minutes out (R3.5).
TITLE: 12/23 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 18mph Gusts, 13mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.31 μm/s
QUICK SUMMARY:
H1 locked for 49 hours and still chugging along!
TITLE: 12/22 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 119Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:
H1's been locked for 48.5+hrs.
Odd shift with a couple of mysteries: (1) SEI_CONF transitioned to EARTHQUAKE, and (2) Dropped from OBSERVING for unknown reason.
Had wind up around 12-14mph for good part of shift, but it loooks like it could be calming down.
LOG:
At 21:31:21utc, H1 was dropped out of OBSERVING. I did a quick scan and did not see anything obvious. There have been several Guardian Nodes which are NOT green: ISI (for a TRamp, Violins (from violin activities), Inj Trans (it's waiting to go back to Success state), etc, but these remained as they have been. A few notes:
Will keep the Guardian and SDF medm windows open in case there's another drop---so I can hopefully catch what flashes and sheds ligth as to the culprit here.
21:33:04utc Went back to OBSERVING
You can always look at the guardian log of any node using guardctrl command (see below).
In this case, TCS ETMX ring heater Beckhof was unresponsive for a few seconds though I don't know why.
~$ guardctrl log --after '2019/12/22 21:20:00 UTC' --before '2019/12/22 21:36:00 UTC' IFO
2019-12-22_21:31:21.524583Z IFO [OBSERVE.run] USERMSG 0: TCS_ETMX_RH_PWR: has notification
2019-12-22_21:31:21.534630Z IFO [OBSERVE.run] USERMSG 5: waiting for node: TCS_ETMX_RH_PWR
2019-12-22_21:31:21.568637Z IFO JUMP target: DROP_OBSERVE
2019-12-22_21:31:21.569218Z IFO [OBSERVE.exit]
2019-12-22_21:31:21.569218Z IFO STALLED
2019-12-22_21:31:21.630609Z IFO JUMP: OBSERVE->DROP_OBSERVE
2019-12-22_21:31:21.631284Z IFO calculating path: DROP_OBSERVE->OBSERVE
2019-12-22_21:31:21.631284Z IFO new target: WAITING_FOR_NODES
2019-12-22_21:31:21.644872Z IFO executing state: DROP_OBSERVE (5)
2019-12-22_21:31:21.715324Z IFO [DROP_OBSERVE.run] USERMSG 0: TCS_ETMX_RH_PWR: has notification
2019-12-22_21:31:21.716172Z IFO [DROP_OBSERVE.run] USERMSG 1: VIOLIN_DAMPING: has notification
2019-12-22_21:31:21.719072Z IFO [DROP_OBSERVE.run] USERMSG 2: SEI_ITMX: has notification
2019-12-22_21:31:21.720278Z IFO [DROP_OBSERVE.run] USERMSG 3: ISC_LOCK: has notification
2019-12-22_21:31:21.720707Z IFO [DROP_OBSERVE.run] USERMSG 4: ISI_ITMX_ST2: has notification
2019-12-22_21:31:21.772147Z IFO JUMP target: WAITING_FOR_NODES
2019-12-22_21:31:21.772828Z IFO [DROP_OBSERVE.exit]
2019-12-22_21:31:21.772828Z IFO STALLED
2019-12-22_21:31:21.822498Z IFO JUMP: DROP_OBSERVE->WAITING_FOR_NODES
2019-12-22_21:31:21.823140Z IFO calculating path: WAITING_FOR_NODES->OBSERVE
2019-12-22_21:31:21.823140Z IFO new target: READY
2019-12-22_21:31:21.824140Z IFO executing state: WAITING_FOR_NODES (20)
2019-12-22_21:31:21.898601Z IFO [WAITING_FOR_NODES.run] USERMSG 0: TCS_ETMX_RH_PWR: has notification
2019-12-22_21:31:21.899459Z IFO [WAITING_FOR_NODES.run] USERMSG 1: VIOLIN_DAMPING: has notification
2019-12-22_21:31:21.903659Z IFO [WAITING_FOR_NODES.run] USERMSG 2: SEI_ITMX: has notification
2019-12-22_21:31:21.904770Z IFO [WAITING_FOR_NODES.run] USERMSG 3: ISC_LOCK: has notification
2019-12-22_21:31:21.905071Z IFO [WAITING_FOR_NODES.run] USERMSG 4: ISI_ITMX_ST2: has notification
2019-12-22_21:31:21.908626Z IFO [WAITING_FOR_NODES.run] USERMSG 5: waiting for node: TCS_ETMX_RH_PWR
2019-12-22_21:31:23.380593Z IFO JUMP target: READY
2019-12-22_21:31:23.381379Z IFO [WAITING_FOR_NODES.exit]
2019-12-22_21:31:23.382120Z IFO STALLED
2019-12-22_21:31:23.453002Z IFO JUMP: WAITING_FOR_NODES->READY
2019-12-22_21:31:23.454064Z IFO calculating path: READY->OBSERVE
2019-12-22_21:31:23.454512Z IFO new target: OBSERVE
2019-12-22_21:31:23.455691Z IFO executing state: READY (50)
2019-12-22_21:31:23.522106Z IFO [READY.run] USERMSG 0: VIOLIN_DAMPING: has notification
2019-12-22_21:31:23.527690Z IFO [READY.run] USERMSG 1: SEI_ITMX: has notification
2019-12-22_21:31:23.529655Z IFO [READY.run] USERMSG 2: ISC_LOCK: has notification
2019-12-22_21:31:23.530093Z IFO [READY.run] USERMSG 3: ISI_ITMX_ST2: has notification
2019-12-22_21:33:03.893311Z IFO REQUEST: OBSERVE
2019-12-22_21:33:03.894002Z IFO STALL cleared
2019-12-22_21:33:03.894002Z IFO calculating path: READY->OBSERVE
2019-12-22_21:33:03.961278Z IFO EDGE: READY->OBSERVE
2019-12-22_21:33:04.018526Z IFO calculating path: OBSERVE->OBSERVE
2019-12-22_21:33:04.018526Z IFO executing state: OBSERVE (100)
2019-12-22_21:33:04.032461Z IFO [OBSERVE.run] USERMSG 0: VIOLIN_DAMPING: has notification
2019-12-22_21:33:04.036775Z IFO [OBSERVE.run] USERMSG 1: SEI_ITMX: has notification
2019-12-22_21:33:04.037957Z IFO [OBSERVE.run] USERMSG 2: ISC_LOCK: has notification
2019-12-22_21:33:04.038344Z IFO [OBSERVE.run] USERMSG 3: ISI_ITMX_ST2: has notification~$ guardctrl log --after '2019/12/22 21:20:00 UTC' --before '2019/12/22 21:36:00 UTC' TCS_ETMX_RH_PWR
2019-12-22_21:31:21.391292Z CA.Client.Exception...............................................
2019-12-22_21:31:21.392061Z Warning: "Virtual circuit unresponsive"
2019-12-22_21:31:21.392607Z Context: "h1ecatx1.cds.ligo-wa.caltech.edu:5064"
2019-12-22_21:31:21.393068Z Source File: ../tcpiiu.cpp line 947
2019-12-22_21:31:21.393504Z Current Time: Sun Dec 22 2019 13:31:21.391261577
2019-12-22_21:31:21.393926Z ..................................................................
2019-12-22_21:31:21.394372Z TCS_ETMX_RH_PWR [NOMINAL.run] USERMSG 0: CONNECTION ERRORS. see SPM DIFFS for dead channels
2019-12-22_21:31:21.463499Z TCS_ETMX_RH_PWR EZCA CONNECTION ERROR. attempting to reestablish...
2019-12-22_21:31:21.463930Z TCS_ETMX_RH_PWR CERROR: State method raised an EzcaConnectionError exception.
2019-12-22_21:31:21.463930Z TCS_ETMX_RH_PWR CERROR: Current state method will be rerun until the connection error clears.
2019-12-22_21:31:21.463930Z TCS_ETMX_RH_PWR CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume.
2019-12-22_21:31:23.267678Z TCS_ETMX_RH_PWR connections reestablished
Edit later: If you're in front of the workstation when something happens, you don't have to use command line, you can open MEDM screen for whatever node (in this case IFO node by pressing "GRD IFO" button at the top of the guardian overview screen) and then press the "log" button to see what's going on in real time.
This morning (17:08utc) we had an EQ alert for an earthquake on the small side (according to the SEISMON EQ Response plot). On the other operator work station, I run with Seismon, SEI_CONF, and SEI channels up on ndscope to monitor EQs; I generally only transition to the EARTHQUAKE state when I see H1 begin to show effects from the EQ (via IMC, ASC, and seismometers).
The earthqake turned out to in fact be a small one with seismicity being elevated from 17:30-18:04utc as seen on the STSs. So I ignored this EQ.
H1 Transitioned to EARTHQUAKE, But By Whom?
At about 17:40utc received call from Eyal S. at LLO via TeamSpeak letting us know that we were still in the EARTHQUAKE state (Thanks for catching this, Eyal!). I checked the SEI_CONF log and it transitioned from WINDY to EARTHQUAKE at 17:12utc. I'm certain I did not transition it, and Cheryl was here, but she did not tell me she transitioned, so I'm going to guess an operator didn't transition. There are plans to automate how we handle earthquakes (see Jim's alog from last Thursday), but I don't think this is in effect yet.
18:09: Since this EQ was negligible & the small seismic waves were long gone, I transitioned H1 back to the WINDY state.
Timeline Summary:
Unless TJ has worked on this in the last couple of days, there is no automatic transition of SEI_CONF.
None of this is automated yet, this was done by a person.
A lot has happened since my last alog on this topic.
I presented a little presentation to the Pcal team some weeks ago, where I claimed I was able to remove all observable temperature dependence in the the Pcal Calibration tools. Unfortunately this result is valid only for the "old" style detectors, which feature a board that only has a photodiode and no other on-board electronics.
Since the WSL as been returned here, and is effectively a spare until a replacement is put together, I have been using it to investigate temperature dependence of the in-use-boards, which have a number of extra circuitry components that get really quite hot when these things are being used... Just turning on the instruments heats up the circuit board by 6.5 degC!
I will summarise contributions to observed temperature dependence as I currently know it, in order of importance (in units of HOPs/degC = hundredths of a percent/degC):
Putting all of these aspects together, both experimentally (with reduced FOV) and numerically (subtracting known temperature dependent factors), I managed to reduce observed temperature dependence in the WSL from ~5.5 HOPs/degC (Resp_vs_Temp_Intial.png) to -0.14 HOPs/degC (Resp_vs_Temp_Final.png). There is a small change left over that goes the opposite way. I suspect this is an effective increase in responsivity due to the height dimension of the hole changing (see the presentation for an illustration).
Having spoken with Niko,
One component we have not considered yet is the transimpedance amplifier: We previously ignored it as it should contribute nothing to the output at DC voltages.
There are some questionable things in it's spec sheet.
I'll investigate this by trying to heat it only in isolation.
I did a pseudo-experiment before leaving today.
I attached the test board with no PD to a clamp, and gave it a current source. With a dissasembled pen I would blow on components to cool them, to see what gives me considerable drops in output voltage.
When cooling the capasitor that is in parallell with the main transimpedance feedback resistor, I got the largest voltage change - the transimpedance amplifer itself gave no real change despite my attempts to cool it.
I have a very strong hunch that this is the "unknown temperature dependance" that I am looking for. I will investigate this more scientifically in the near future.
I got hold of a soldering iron that goes down to 65 degC from Mark.
Checked the output voltage effect from heainting individual components - The R2 resistor is the biggest contributor on the entire board that I can find, and gives <0.1 HOPS/degC (~0.02) (this is a better resistor than the ones in the working standards).
The C5 capasior in parallel gieves about 1/3 of the response change when heated, compared to the resistor.
The transimpedance amplifier, and buffer out amplifier do not appear to constitute ant changes, nor does any other component (I tested them all).
I tried heating the whole board in the oven, and found changes consistent with just the R2 resistor (maybe a little bit more). While the board was cooling, I found that cooling the Transimpedance amplifier by blowing on it lenth-wise had the biggest effect on the output. I can't apply heat to it in that way and may be why I missed in on the first try. The offset voltage (or current) drift at inpout and output per deg/C is largely consistant with what I am seeing.
I've run into a bit of a brick wall in quantifying contributions however (could explain the Transimpedance Amplifier acting up). I noticed voltage fluctations on the scale of the effects I am studying apparently coming and going for now reason. While unplugging my setup I noticed that the earthing of my current source *is* coupling to my output, to the tune of the effects that I am studying. Basically the optics lab appear to be riddled with ground loops - and probably needs an overhaul of grounding.