Wed Nov 08 10:11:10 2023 INFO: Fill completed in 11min 6secs
Travis confirmed a good fill curbside. TCs were displaced by ice ball.
Back to Observing after a rough night of earthquakes. Lock acquisition was fully automated which is fantastic to see after large earthquakes!
Our range looks a bit lower over the past 24 hours. Power recycling gain seems to be lower than normal as well. Doesn't seem to be correlated with the higher microseisms that we've been seeing (see attached).
The peak-to-peak values of the power recycling gain are also almost 3x what they are when the recycling gain is higher.
The last several locks, the PRG has been about 47. We're usually closer to 51, so this is certainly something to look at.
TITLE: 11/08 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.10 μm/s
Secondary useism: 0.25 μm/s
QUICK SUMMARY: A few large earthquakes have been keeping us out of lock overnight, but lock acquisition has recently started. The earth is still moving a bit so we'll see how this goes.
CDS OK, no alarms.
In the last 3 overnight locks, our SQZ quickly degraded from 4.4dB to ~3.3dB as the IFO thermalized. Plot attached. The CLF RF6 slightly increased. Sheila discusses these changes in 74059.
After the last OPO temperature change, it looks like the SQZ angle should be adjusted, see attached.
This morning when we got to NLN, I changed it form 144.66deg to 138.02 deg plot, but I expect we should retune it when the IFO is thermalized at the start of commissioning.
Was alerted at 9:09UTC
IFO was in NLN but could not move into observing due to 2 (attached screenshot) SDF diffs in ALS Y, which I accepted.
IFO reached OBSERVING at 9:21UTC
TITLE: 11/08 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:
The last few hours of the shift turned out to be eventful with a large earthquake hovering at high seismic levels for 2.5hrs which made locking Green Arms impossible. In the last 30min of the shift, seismic finally looked like it was sort of coming down, so I attempted an Initial Alignment. X-arm showed signs of life and eventually was aligned, but the Y-arm gave an error about the Camera Centering being too far off. I ran a DOWN out of the Initial Alignment due to the Y-arm, to see if I could do anything. I might have messed up the Xarm because when I tried locking again, X-arm was no longer aligned and had no light in the arm (I returned ETMx & TMSx to where they were when it was aligned). At this point, I'm letting ISC LOCK take over (both X & Y arms are in INCREASE FLASHES).
Unfortunate timing with the earthquake and frustrating last 30min of the shift.
Addendum: as I log out, atleast the Y-arm now has life after INCREASE FLASHES and can finally OFFLOAD with a quieter ground. Xarm....looks a little glitchy, but is trying to offload.....and now it is moving on to FINDING IR.
LOG:
H1 taken to Earthquake mode at 0509utc; started seeing seismic signal on Picket Fence and seismic blrms. Squeezer unlocked taking H1 out of Observing and then lockloss 1-min later. It's been about 15min since the EQ waves hit us and seismic is still increasing (ISC LOCK has x&y arms at INCREASE FLASHES). Marking time as EARTHQUAKE.
Since earthquake motion is still increasing, I'm going to take ISC_LOCK to DOWN and wait 30min.
UPDATE After An Hour Since The Earthquake Arrived: Picket Fence is still orange (or more) and BLRMS hasn't quite started to turn downward yet....assess in another 30min (I tried to lock a green arm, but they are wobbling all over the place anyways.)
With 30min left in the shift and seismic elevated 2.5orders of magnitude due to the earthquake (for almost 2.5hrs!), going to run an Initial Alignment. Mainly because Y-arm has OK flashes, but now it won't attempt to lock its green arm.
At the moment,
Seismically, the "Earthquake Band" is finally touching the 1um/s mark after being above that for the last 2.5hrs.
H1's been locked 4.25hrs, but the range is 8-9Mpc lower than 24hrs ago (and this is with quiet breezes now vs high winds last night); it took about 3hrs for the range to slowly ramp up to the range of 151Mpc (where last night we were at about 160Mpc with high winds. useism has been somewhat high and constant the last 24hrs.
TITLE: 11/08 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Maintenance day with higher winds. There have been a few changes that should help lock acquisition - COMM beatnote improvement and ALS_DIFF IR finding. We are back at observing, though our range is a bit lower than it has been recently.
LOG:
0001 Observing
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 16:05 | PCAL | Tony | PCAL Lab | y(local) | Getting stuff for tests | 16:15 |
| 16:06 | PCAL | Tony, Rick | EY | YES | PCAL measurements | 19:54 |
| 16:12 | FAC | Mitch | EY/EX | YES(EY) | FAMIS Tasks | 17:22 |
| 16:12 | FAC | Karen | EY | YES | Tech clean | 17:28 |
| 16:14 | IMC | Jason | CR | n | RefCav Adjustment | 16:23 |
| 16:17 | TCS | Camilla | CR | n | CO2 Laser Power adjustment (started 16:15) | 17:33 |
| 16:18 | FAC | Chris, Eric | LVEA | n | FAMIS Tasks | 16:40 |
| 16:26 | FAC | Kim | EX | n | Tech clean (done at 17:20) | 18:11 |
| 16:39 | EE | Fil | CER | n | Slow Controls Chassis 5 installation | 17:12 |
| 16:40 | FAC | Chris, Eric | FCES | n | FAMIS tasks | 16:51 |
| 16:50 | PEM | Robert | LVEA | n | Moving stuff/PEM Injections | 20:13 |
| 16:51 | S&R | Christina | Wood shop | n | Throwing away stuff | 17:44 |
| 16:52 | FAC | Eric, Chris | EY,EX | YES(EY) | FAMIS tasks | 17:27 |
| 16:57 | SEI | Jim | CR | n | Troubleshooting HAM8 | 17:12 |
| 17:05 | CDS | Erik | EY | n | Replacing HWS server | 19:51 |
| 17:11 | SAF | Safety walkthough | LVEA, FCES | n | Safety walkaround | 18:41 |
| 17:13 | SEI | Jim, Fil | FCES | n | HAM8 GS13 | 18:12 |
| 17:20 | SQZ | Daniel, Fernando | CR | n | Beckoff (started 8:30ish) | 17:38 |
| 17:21 | FAC | Randy | EY | YES | Moving cleanrooms | 19:52 |
| 17:26 | FAC | Cindi | FCES | n | Tech clean | 18:54 |
| 17:27 | ISC | Jason, Ryan C | LVEA | n | Moving equipment | 18:13 |
| 17:35 | Tour | Camilla, Martha | LVEA, FCES | n | Tour | 18:09 |
| 17:41 | ISC | Daniel, Fernando | CER | n | Removing Beckoff chassis | 18:10 |
| 17:41 | FAC | Mitch | LVEA | n | Moving equipment | 19:30 |
| 17:51 | FAC | Chris, Pest Control | LVEA | n | Check traps | 19:49 |
| 18:10 | Tour | Camilla, Martha | Roof | n | Admiring the view | 18:20 |
| 18:12 | FAC | Karen | LVEA | n | Tech clean | 19:31 |
| 18:12 | FAC | Kim | LVEA, HighBay | n | Tech clean | 19:31 |
| 18:13 | EE | Fil | EX | n | Getting equipment | 19:11 |
| 18:20 | GRD | TJ | CR | n | Testing Guardian on ITMY | 19:32 |
| 18:21 | Tour | Camilla, Martha | EY | n | Tour of outside of EY | 19:12 |
| 18:55 | FAC | Cindi | FCET | n | Tech clean | 19:00 |
| 19:00 | FAC | Cindi | OSB Receiving | n | Getting cardboard | 19:32 |
| 19:02 | SAF | Safety walkthrough | Mid/End stations | YES(EY) | Safety walkaround | 19:53 |
| 19:09 | ISC | TJ, Sheila | LVEA | YES(local) | Adjusting COMM beatnote | 19:40 |
| 19:12 | EE | Fil | HAM6 | n | Getting cable lengths | 19:50 |
| 19:15 | SEI | Jim | HAM7 | n | Looking at HAM7 | 19:55 |
| 19:29 | TCS | Camilla | LVEA | n | Looking through TCS cabinets | 19:50 |
| 19:56 | SEI | Jim | CR | n | Testing SC recovery | 20:20 |
| 21:36 | CDS | Jonathan, Erik | MSR | n | Removing unused switch | 21:37 |
| 21:52 | PCAL | Tony | PCAL lab | local | PCAl meas wrapup | 01:52 |
I wrote a new search in ALS_DIFF to hopefully help with our issues finding IR resonance during this step in locking. Previously the node would go to 3 or 4 past saved offsets, then if there was a little bit of signal it would try to find tune. Successful offsets seem to always be around 400 and 3250 +/- 100, so there's no need to search the entire range. I have it get near these saved offsets +/- 150, then step 30 at a time to look for for any power. If no power is found around this saved offset, then it will move on to the next saved offset range. As long as I move in the same direction, I didn't run into any odd hysterysis.
This ran successfully a handful of times and seems to be decently fast.
TITLE: 11/07 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 13mph Gusts, 7mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.38 μm/s
QUICK SUMMARY:
Maintenance & Winds had H1 out of Observing when I arrived, but H1 was ready to return to Observing
Struggled a bit with high winds but it calmed down recently and we are back to Observing. There were some SDFs related to Keita's alog74070 and the CO2 rotation stage work.
I disabled 1Hz LPF which was somehow present in the SUM (not NSUM) of the QPDs and WFS DCs that are used for normalization of PIT and YAW. These LPFs ruin the rejection of the intensity noise for QPDs and WFS DCs even at 1Hz.
Here's a list of the filter modules where the 1Hz LPF (all FM1) was disabled: H1:ASC-OMC_[AB]_SUM, H1:ASC-REFL_[AB]_DC_SUM, H1:ASC-AS_[AB]_DC_SUM, H1:ASC-AS_C_SUM, H1:ASC-POP_[AB]_SUM, H1:ASC-[XY]_TR_[AB]_SUM.
Note: Without LPF, when large clipping action is going on and SUM goes very small, I can imagine that PIT and YAW become somehow worse than with LPF. If that ever becomes a problem, enable 1Hz LPF.
Note 2: Green WFS DCs and QPDs don't have the BPF for SUM.
Turns out that disabling the LPF for PIT and YAW normalization has a side effect of making RIN channels 1 (one, unity), but only for ASC-OMC_[AB]. Others don't seem to have RIN channel.
I don't see this as an urgent problem, but if this RIN function is needed, we have to add a dedicated filter module just for RIN normalization purpose.
Also, this will make the sensitivity of ASC-OMC_RIN to the detchar safety injections go away, but this is just an unintended side effect of the change in the normalization. (See alog 74056.)
Well, turns out that H1:ASC-[XY]_TR_[AB]_SUM setting was reverted back right after I changed it, but the rest stood (1st attachment). I might turn the LPF for the TR sensors off later, but not now.
Anyway, 2nd attachment shows that disabling the LPF for OMC QPD SUM for normalization made a big reduction in AM to ASC leakage (2nd attachment right column, though only OMC_A is shown here). NSUM is calibrated to RIN. Current traces are from last night after the LPFs were disabled (Nov/08/2023 10:00:00 UTC), and references are before (Nov/07/2023 02:30:00 UTC), both in nominal low noise. As you can see, disabling LPF reduced the intensity noise coupling to OMC ASC by a large amount especially for f>10Hz. (I'm sure that the coupling was reduced at lower frequency too but it's hard to see the effect w/o injection.)
3rd and 4th attachment show similar plots for ASC-AS_[ABC], ASC-REFL and in-vac POP. I haven't done any deep analysis but there doesn't seem to be much improvement if any. In general, the larger the centering offset is, the bigger the intensity noise coupling is with LPF on, so it's understandable that the OMC qpds show the obvious improvements (because they have crazy centering offsets and the coupling was high with the LPF ON).
TJ, Camilla. After we found both CO2 output powers have been changing since the timing master swap 74025, we recalibrated the CO2 rotation stages this morning ~ 16:20UTC. Last done in September 72961.
Followed the README.txt file in /opt/rtcds/userapps/release/tcs/h1/scripts/RS_calibration/. We did this with annular masks in. I'd previously tried this without using the matlab fitting code and hadn't been able to make a good fit.
Maximum power measured out of each laser:
| Annular Mask | Central Mask | |
| CO2X | 6.6W | 3.8W |
| CO2Y | 5.5W | 3.7W |
Plot attached, Aidan noted that we expect to get more power through the central mask than annular. We plan to check the mode matching going into the masks in the next couple of weeks.
TJ noticed some strangeness with the CO2X rotation stage not going back to the same place, over ~minutes. Turned H1:TCS-ITM{X,Y}_CO2_LASERPOWER_FINEADJUST to False, as PSL RS uses. And re-tuned the attached calibration, this could do with more work but will leave alone as we are relocking.
I've added some bootstrapping to the CO2 power guardians to approximate the adjusted power request after it reaches the initial request. This will only bootstrap on the way up, not when finding minimum power. Something that should be added is to not bootstrap if close enough, we've noticed the rotation stage will do some odd moves when very small angle changes are requested.
adjusted power request = (initial request)^2 / current power
This lock loss had another one of the LSC-DARM wiggles ~100ms before the lock loss.
There was HVAC work going on at EX at the time, but I don't see much ground motion at that time, so I'm inclined to rule them out.
Summary:
It seems that the supply voltage for thermistors is oscillating, the frequency depends on whether or not the supply is loaded with thermistors (830mHz open to 1.66Hz fully connected to the in-chamber thermistors), the amplitude of the oscillation jumps seemingly randomly, and this is also happening for the unused Beckhoff module for the yet-to-be-installed second T-SAMS unit, all measured at the back of the Beckhoff chassis. (Can't we measure this from Beckhoff itself, without me going to the floor?)
Fil and Fernando quickly set up the same Beckhoff module in the lab and didn't observe this. Could we swap or maybe restart the modules in the chassis?
As of now, Beckhoff cable is disconnected from the back of the heater driver chassis. (I'm applying DetChar-Request tag just so people know, but we're just changing from one no-comb configuration to the other.)
Detaisls:
Since the past findings about OM2 and 1.66Hz comb (alogs 73367, 73233, 72967 72241 and 72061) didn't make sense, I went to the floor and remeasured the comb in the Beckhoff heater output (which goes to the heater driver input) as well as thermistor pins in the back of the heater driver chassis while Beckhoff connection was intact.
Turns out that all of these things share the same frequency but the voltage across thermistor pins was ~3 orders of magnitude larger than Beckhoff heater driver output pins (pin 9 and 19) (the latter were referenced from the driver board ground as it's common mode for both pins). I used a scope on battery and the thermistor voltage was like roughly 1Vpp 1.66Hz rectangular wave (1st pic). Yellow is the voltage across pin10 and 23 (across thermistor 1), blue is pin9 and 22 (thermistor 2) of the DB25 at the back of the driver chassis when the Beckhoff cable was still connected. Voltage difference seemed to have come from the temperature difference of the thermistors (I disconnected the Beckhoff cable and measured the thermistor 1 and 2 resistance incl. cables to be 7.41k and 4.08k, respectively). When I disconnected the cable from the chassis and just measured the pin10-23 and pin9-22 voltage coming from the cable (picture 2), they were both about 1.2V pp. This is supposed to be the source voltage for thermisters. The frequency slowed down by about a factor of 2 (832mHz) when the thermistors were disconnected.
For your convenience, below is a table of which pins are what (see e.g. E1100530 and D2000212). Note that thermistors themselves only have two pins, therefore supply and readback pins are bundled together in chamber as shown. Supply is presumably a reference voltage supplied through a reference resistor.
| which thermistor? | DB25 pin | Beckhoff | in chamber |
| 1 | 10 | Temperature Supply 1A+ | thermistor 1 pin 1 (10&12 bundled together in chamber) |
| 12 | Temperature Readback 1A+ | ||
| 23 | Temperature Supply 1A- | thermistor 1 pin 2 (23&25 bundled together in chamber) |
|
| 25 | Temperature Readback 1A- | ||
| 2 | 9 | Temperature Supply 2A+ | thermistor 2 pin 1 (9&11 bundled together in chamber) |
| 11 | Temperature Readback 2A+ | ||
| 22 | Temperature Supply 2A- | thermistor 2 pin 2 (22&24 bundled together in chamber) |
|
| 24 | Temperature Readback 2A- |
Went to the CER, disconnected the cable from the back of the Beckhoff chassis and did the same measurement. Frequency didn't change but the amplitude was much smaller (~280mVpp instead of 1.2Vpp) for a while, but suddenly the amplitude of the thermistor 1 supply changed back to 1.2V (pic 3). Nuts. When the beckhoff cable was reconnected (and the connection to in-chamber thermistor was restored) the frequency went back to 1.66Hz (picture 4).
Picture 5 shows pin 10-23 (thermistor 1 supply) and pin12-25 (thermistor 1 readback, which is not connected to anything). Picture 6 is the same thing but for the unused Beckhoff unit for the second T-SAMS. It's strange that the same thing is happening in two independent units. Picture 7 is the thermistor 1 supply and pin 6-19 (voltage output for the heater driver). It really seems that this is a problem of the supply voltage.
I checked the 24V power strip for the Beckhoff chassis but it was good (pic 8 and 9).
Fil and Fernando set up EL3692, which is the Beckhoff unit used for Thermistors. They didn't observe this oscillation behavior.
I wanted to do some injections into thermistors to see how this couples to DARM but didn't have time.
Well, this seems to be a feature! The EL3692 terminal has 2 measurement inputs for resistance but only one ADC and current source. In alternating mode it switches between the 2 channels continuously. From the scope traces, the measurement time per channel looks like about ~400 ms. This is consistent with the data sheet. We expect about ~0.16 mA of current in the range between 10 Ω and 10 kΩ.
Fil, Marc, Keita, Daniel, Fernando Fil and I set a test bank in the lab and verified the square pulses found are part of the features of the EL3692 terminal when both channel reading is set. Then we implemented a configuration using a continuous reading over the channel 1 and one shot reading over the channel 2 (under request by a raising edge on the Start Conversion input in the module, that will be never used). Finally the scopes show the continuous signal in the channel 1 with some minor noise component that is still in analysis (basically 60Hz and 12Hz) however this virtually solves the main problem with the square pulses. One ECR should be open to double the quantity of EL3692 in the places where reading in both channels are necessary since just one channel provides continuous reading at this point. Note: the autorange feature was left intact so the new configuration will not cause any limitation in the resistor range to be measured and also will keep the same PDO to not break the Epics configuration. Attached: scopes and the EL3692, configuration applied to the EL3962 and spectral analysis for the noise signals.
WP 11501 Daniel Keita Fernando Today we configured the one channel reading on the two Beckhoff EL3692 modules for the PSL IO Chassis. After restarting the system Keita Daniel and I were to the rack to scope the thermistor channels we verified the absence of the square pulses reported initially. Finally the module R20 CH2 was rewired to R21 CH1 and configured in the system accordingly. Attached the picture including the R20 and R21 EL3692 modules for reference.
After having a solution for the issue duplicating the number of EL3692 modules, and ECR and FRS ticket have been created: ECR: https://dcc.ligo.org/E2300408-v1 FRS ticket: https://services1.ligo-la.caltech.edu/FRS/show_bug.cgi?id=29563
Daniel, Fernando
As part of the WP11506 a new configuration was loaded into the Beckhoff system which includes the 1-channel continuos reading for the EL3692 terminals. The change included the rewiring in the TCS Corner EtherCAT chassis TSAMS consisting of connecting the second channel in the EL3692 (R20) to the first channel in the terminal EL3692 (R21) to match the TwinCAT configuration added. The disconnected wires are not currently connected ot any thermistor on the floor.
