Vlad and I noticed a set of 20Hz peaks in DARM at 16:50UTC, see attached. Maybe similar to the peaks Robert and Tyler mitigated by changing AC settings in 73430
Vicky, Camilla
IFO was Out of Observing 15:40 to 15:56UTC as SQZ FC Unlocked. After 5 failed automatic SQZ_FC TRANSION_IR_LOCKING attempts, Vicky and I kept the FC GRD in DOWN, closed the Input 1 loop (SQZ Overview > FC Servo > H1:SQZ-FC_SERVO_IN1EN) to manually lock FC on green and aligned FC2 using the GR camera and maximizing H1:SQZ-FC_TRANS_C_LF_OUTPUT. Accepted FC2 alignment sdfs and went back to observing. Updated "Issues with SQZ FC Guardian" wiki
Think the issue could have been due to alignment drifts over the long 21 hour lock and as the M3 length was cleared, this couples in with pitch and yaw alignments, plot attached. Enough that the FC2 alignment needed to be slightly tweaked to relock. From plot, can see that M2 Pit (grey trace, bottom left) changed ~40urad on FC unlock?! This is a lot of drift over 20 hours, zoomed out plot here.
We have previously decoupled L2A for FC2's M1 stage, see LHO:67957, March 2023, most L2A decoupling was needed in pitch. But we have not decoupled L2A for FC2's lower M3 stage (in principle we should not have to do this as most DC gain is on M1). More relevant is probably that since March 2023, the beam spots on FC1/2 mirrors have since drifted away from center so the previous L2A decoupling may be less valid now. May be worth checking and re-centering FC cavity axis on the optic centers now, where our previous L2A decoupling should work better, and may overall be more stable.
I think that if L2A decoupling coefficients are very different now, that could impact FC alignment and relocking after long lock stretches. That's basically what happened before, which led us to do the decoupling FC2 M1 L2A in the first place (i.e., after long locks, the FC alignment drifts a lot due to the length-to-pitch coupling, such that when FC unlocks and we stop pushing on FC2 in length, then FC2 pitch changes suddenly and we lose FC alignment). Something to keep an eye on as Camilla's fixing of FC2 pitch alignment seemed to recover the filter cavity lock transition from green to red, which feels similar to before.
TITLE: 10/27 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 162Mpc
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 13mph Gusts, 10mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.09 μm/s
QUICK SUMMARY:
IFO has been locked 20h30. There is plans for commissioning time 1-3pm this afternoon.
TITLE: 10/27 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 163Mpc
INCOMING OPERATOR: Austin
SHIFT SUMMARY:
Very Quiet night. Violins look great! Everything running smoothly.
H1 Current Status: Locked in NOMINAL_LOW _NOISE & OBSERVING for 12.5 hours.
LOG:
No Log
TITLE: 10/26 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 163Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY: Earlier lockloss due to earthquake, then we needed to do an Initial Alignment before being able to lock agian. Currently inn Observing and have been Locked for 4.5hours.
LOG:
15:00UTC Detector Locked for 2.5hours
16:20 Earthquake mode activated due to local earthquake
16:25 Earthquake incoming from Japan
16:38 Lockloss due to earthquake
- Holding in DOWN for ~10mins
16:53 Started relocking
17:21 Took us to DOWN to start an initial alignment
- While I was waiting for PREP_FOR_LOCKING to complete, IMC kept going between OFFLINE and FAULT
17:23 Started Initial Alignment
17:43 INITIAL_ALIGNMENT complete, started relocking
-At PREP_ASC_FOR_FULL_IFO got lots of IFO_OUT call-outs, similar to what Tony had noticed (73753)
18:29 Reached NOMINAL_LOW_NOISE
18:49 Observing
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 15:28 | FAC | Tyler, Randy, Eric | EX/EY | n | Chillers | 16:44 |
| 15:36 | FAC | Karen | OptLab,VacPrep | n | Tech clean | 16:00 |
| 16:13 | FAC | Karen | WS | n | Tech clean | 16:38 |
| 16:39 | VAC | Travis | LVEA | n | Checking ion pump | 16:47 |
| 18:21 | FAC | Cindi | MechRoom | n | Tech clean | 19:51 |
| 20:09 | VAC | Travis | MY | n | Looking for tools | 20:51 |
| 20:36 | EE | Fil | MY | n | Moving things to/from | 21:01 |
TITLE: 10/26 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 163Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 5mph Gusts, 4mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.11 μm/s
QUICK SUMMARY:
IFO is currently Locked for 4 .5 hours and is no longer in earthquake mode.
Based on the unexpectedly high gain of this PI (~260), if LHO is in a position where they wish to avoid inducing this PI at any cost, without requiring to have active actuation runnning to damp it, they have some breathing room in the settings of the X-arm ring heaters (RHs) before the mode-spacing crosses the threshold of parametric gain, R > 1. Here I give a rough estimate on the magnitude these RH settings can be, whilst remaining in this "safe space", based on Diopter per Watt estimates here.
Presently modespacing for the relevant interacting mode in a well thermalised state sits at (see black curves in figures of this aLog): 5240 Hz in the 0'th FSR, and 80287 Hz in the 2'nd FSR.
To reach the condition for R > 1, modespacing can go up as high as 5244.5 Hz in the 0'th FSR, giving us 4.5 Hz of breathing room right now.
3 basic scenarios to reach that threshold:
Effective PI damping actuation would relax this further.
The last time LHO lost lock to this PI was 22nd May 2023. I fetched tape data to derive the observed paramertic gain (R_obs) for that event, and fit exponentials to periods of exponential growth, to get the time constants:
I will take the conservative estimate and take R_obs = 16.
Given an estimate of the mechanical Q factor of 1.7 million, all I need to know is what was the difference between the optical mode frequency and the mechanical mode itself, in order to deduce the underlying Parametric Gain (R0), which would occur at an exact frequency match of these parameters.
To fetch these paramters, I fetched spectra of the OMC_DCPD_SUM (to see optical mode spacing) and MODE28_BP_IN1 (to see exact PI frequency which is demodulated at 80000 Hz). I use the frequency mapping I established earlier for the optical mode to project what the frequency is 2 FRSs away:
Now I take these juicy details and I crank the algebraic handle with this simple script.
R0 = 260
Highlights:
All in all, a pretty quiet week, data qualiity wise. Full shift report here: https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20231016
Thu Oct 26 10:14:54 2023 INFO: Fill completed in 14min 49secs
Travis confirmed a good fill curbside.
Note that now the outside temperatures have dropped (and therefore also the thermocouple reference junctions) the TCs no longer drop down to -200C.
I increased the trip temperature from their summer setting of -130C to their winter setting of -100C this morning. The TCs min temps for today's fill were TCA=-125C, TCB=-119C.
The plot y-axis marker has been set to the new trip temp.
Lockloss @ 16:38 UTC due to earthquake
Holding in DOWN for a bit while the ground motion settles.
16:53 Started relocking
18:49UTC Observing
TITLE: 10/26 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 9mph Gusts, 6mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.12 μm/s
QUICK SUMMARY:
We've been Locked for 2.5 hours and are Observing
TITLE: 10/26 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
Since the Mid shift report relocking from that incoming earthquake went well. BUT
04:04 UTC:
IFO OUT saturations happned again During Prep_ASC_FOR_FULL_IFO. And usually that does not tend to happen. Thats the second lock in a row that this has happened, in the same ISC_LOCK State. So maybe it's a coinicidense because both locks happened on the same night, after the same initial alignment. I'd be interested in seeing if this happens again after another initial alignment.
3:48 UTC Lockloss
4:44 UTC Nominal_LOW_NOISE reached
5:05 UTC OBSERVING reached
6:02 UTC Incoming 5.5M Earthquake from Timor-Leste
6:57 UTC Seismic to CALM, we made it.
Current H1 Status: Locked in NOMINAL_LOW_NOISE and OBSERVING.
LOG:
None
Lockloss from 5.4M Earthquake from Tonga from ISC_LOCK STATE : INJECT SQUEEZING.
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1382327323
Aquired H1 while it was in comissioning and had been locked for 20.5 hrs.
Comissioning finished at 23:32 UTC.
00:29 UTC Dropped out of observing and back in comissioning due to:
CAMERA_SERVO Guardian log has the following message:
USERMSG 0: ASC-CAM_PIT1_INMON is stuck! Going back to ADS
2023-10-26_00:34:24.014687Z
CAMERA_SERVO [TURN_CAMERA_SERVO_ON.run]
USERMSG 1: ASC-CAM_YAW1_INMON is stuck! Going back to ADS
Naioki Patrick and I got on to team speak to try and restart the camera service on h1digivideo2.
At first using the monit system h1digivideo2:2812/h1cam26
Which didn't seem to be benificial.
And again by opening a ssh console to h1digivideo2 and manually restarting it.
And also trying to Kill the process using top commands.
We are now trying to powercycle the camera via removal of POE on its port
Contacting Dave to power cycle switch sw-lvea-aux port 0/35
Lockloss at 01:51 UTC https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=73750
Dave was able to get the camera to reboot and I can now see a black camera image instead of blue screen. Link to his alog.
Relocking had ALS-Y arm alignment issues. Once that was taken care of by green arms manual and touchin up Y arm, I ended up in Check Mich Fringes twice and couldn't get DRMI locked.
So I started an Initial_Alignment. Which completed at 3:03 UTC.
3:19 UTC During Prep_ASC_FOR_FULL_IFO, there were saturations signaled by verbal's IFO out call out.
I dont usually hear saturations during this stage of locking. So I made a note of it.
3:22 UTC Incoming 5.4 Earthquake from Tonga...
Current status: Relocking at ISC_LOCK_STATE: MOVE_SPOTS.
Lockloss from NOMINAL_LOW_NOISE
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1382320306
This happened while Dave , Patrick , And Nioki were trying to resolve the camera issue that was preventing H! from going into OBSERVING.
We did get a small Earthquake roll through at the time, though I'm not sure if the earthquake was what took us down, as the earthquake was very small and wind was mildly elevated as well.
Summary:
It seems that the supply voltage for thermistors is oscillating, the frequency depends on whether or not the supply is loaded with thermistors (830mHz open to 1.66Hz fully connected to the in-chamber thermistors), the amplitude of the oscillation jumps seemingly randomly, and this is also happening for the unused Beckhoff module for the yet-to-be-installed second T-SAMS unit, all measured at the back of the Beckhoff chassis. (Can't we measure this from Beckhoff itself, without me going to the floor?)
Fil and Fernando quickly set up the same Beckhoff module in the lab and didn't observe this. Could we swap or maybe restart the modules in the chassis?
As of now, Beckhoff cable is disconnected from the back of the heater driver chassis. (I'm applying DetChar-Request tag just so people know, but we're just changing from one no-comb configuration to the other.)
Detaisls:
Since the past findings about OM2 and 1.66Hz comb (alogs 73367, 73233, 72967 72241 and 72061) didn't make sense, I went to the floor and remeasured the comb in the Beckhoff heater output (which goes to the heater driver input) as well as thermistor pins in the back of the heater driver chassis while Beckhoff connection was intact.
Turns out that all of these things share the same frequency but the voltage across thermistor pins was ~3 orders of magnitude larger than Beckhoff heater driver output pins (pin 9 and 19) (the latter were referenced from the driver board ground as it's common mode for both pins). I used a scope on battery and the thermistor voltage was like roughly 1Vpp 1.66Hz rectangular wave (1st pic). Yellow is the voltage across pin10 and 23 (across thermistor 1), blue is pin9 and 22 (thermistor 2) of the DB25 at the back of the driver chassis when the Beckhoff cable was still connected. Voltage difference seemed to have come from the temperature difference of the thermistors (I disconnected the Beckhoff cable and measured the thermistor 1 and 2 resistance incl. cables to be 7.41k and 4.08k, respectively). When I disconnected the cable from the chassis and just measured the pin10-23 and pin9-22 voltage coming from the cable (picture 2), they were both about 1.2V pp. This is supposed to be the source voltage for thermisters. The frequency slowed down by about a factor of 2 (832mHz) when the thermistors were disconnected.
For your convenience, below is a table of which pins are what (see e.g. E1100530 and D2000212). Note that thermistors themselves only have two pins, therefore supply and readback pins are bundled together in chamber as shown. Supply is presumably a reference voltage supplied through a reference resistor.
| which thermistor? | DB25 pin | Beckhoff | in chamber |
| 1 | 10 | Temperature Supply 1A+ | thermistor 1 pin 1 (10&12 bundled together in chamber) |
| 12 | Temperature Readback 1A+ | ||
| 23 | Temperature Supply 1A- | thermistor 1 pin 2 (23&25 bundled together in chamber) |
|
| 25 | Temperature Readback 1A- | ||
| 2 | 9 | Temperature Supply 2A+ | thermistor 2 pin 1 (9&11 bundled together in chamber) |
| 11 | Temperature Readback 2A+ | ||
| 22 | Temperature Supply 2A- | thermistor 2 pin 2 (22&24 bundled together in chamber) |
|
| 24 | Temperature Readback 2A- |
Went to the CER, disconnected the cable from the back of the Beckhoff chassis and did the same measurement. Frequency didn't change but the amplitude was much smaller (~280mVpp instead of 1.2Vpp) for a while, but suddenly the amplitude of the thermistor 1 supply changed back to 1.2V (pic 3). Nuts. When the beckhoff cable was reconnected (and the connection to in-chamber thermistor was restored) the frequency went back to 1.66Hz (picture 4).
Picture 5 shows pin 10-23 (thermistor 1 supply) and pin12-25 (thermistor 1 readback, which is not connected to anything). Picture 6 is the same thing but for the unused Beckhoff unit for the second T-SAMS. It's strange that the same thing is happening in two independent units. Picture 7 is the thermistor 1 supply and pin 6-19 (voltage output for the heater driver). It really seems that this is a problem of the supply voltage.
I checked the 24V power strip for the Beckhoff chassis but it was good (pic 8 and 9).
Fil and Fernando set up EL3692, which is the Beckhoff unit used for Thermistors. They didn't observe this oscillation behavior.
I wanted to do some injections into thermistors to see how this couples to DARM but didn't have time.
Well, this seems to be a feature! The EL3692 terminal has 2 measurement inputs for resistance but only one ADC and current source. In alternating mode it switches between the 2 channels continuously. From the scope traces, the measurement time per channel looks like about ~400 ms. This is consistent with the data sheet. We expect about ~0.16 mA of current in the range between 10 Ω and 10 kΩ.
Fil, Marc, Keita, Daniel, Fernando Fil and I set a test bank in the lab and verified the square pulses found are part of the features of the EL3692 terminal when both channel reading is set. Then we implemented a configuration using a continuous reading over the channel 1 and one shot reading over the channel 2 (under request by a raising edge on the Start Conversion input in the module, that will be never used). Finally the scopes show the continuous signal in the channel 1 with some minor noise component that is still in analysis (basically 60Hz and 12Hz) however this virtually solves the main problem with the square pulses. One ECR should be open to double the quantity of EL3692 in the places where reading in both channels are necessary since just one channel provides continuous reading at this point. Note: the autorange feature was left intact so the new configuration will not cause any limitation in the resistor range to be measured and also will keep the same PDO to not break the Epics configuration. Attached: scopes and the EL3692, configuration applied to the EL3962 and spectral analysis for the noise signals.
WP 11501 Daniel Keita Fernando Today we configured the one channel reading on the two Beckhoff EL3692 modules for the PSL IO Chassis. After restarting the system Keita Daniel and I were to the rack to scope the thermistor channels we verified the absence of the square pulses reported initially. Finally the module R20 CH2 was rewired to R21 CH1 and configured in the system accordingly. Attached the picture including the R20 and R21 EL3692 modules for reference.
After having a solution for the issue duplicating the number of EL3692 modules, and ECR and FRS ticket have been created: ECR: https://dcc.ligo.org/E2300408-v1 FRS ticket: https://services1.ligo-la.caltech.edu/FRS/show_bug.cgi?id=29563
Daniel, Fernando
As part of the WP11506 a new configuration was loaded into the Beckhoff system which includes the 1-channel continuos reading for the EL3692 terminals. The change included the rewiring in the TCS Corner EtherCAT chassis TSAMS consisting of connecting the second channel in the EL3692 (R20) to the first channel in the terminal EL3692 (R21) to match the TwinCAT configuration added. The disconnected wires are not currently connected ot any thermistor on the floor.
