Naoki, Camilla
We found that the TTFSS autolock did not work while the IMC was locked as shown in the attached ndscope and guardian log. The two cursors in the ndscope show when the SQZ MANAGER guardian checked the IMC. Although the IMC was locked, the SQZ MANAGER guardian thought that the IMC is not locked and kept waiting. We manually turned off the ramp and turned on the booster on the fast path and then TTFSS recovered.
We think we fixed this in snv commit #25404. There was a small typo that stopped the fiber enable being toggled and made the guardian incorrectly report "IMC unlocked". IFO has not been in a state to test the fix yet.
J. Kissel, D. Bhattacharjee, T. Sanchez, J. Betzwieser WP 11138 From LLO:64213, I'm importing three changes to, and one feature removal from, the PCAL end-station calculations to compute the representative force and corresponding displacement for each channel, updating them for the latest methodology the team wishes to use in O4. Within the force calculation block this includes: (1) Correcting an inconvenience an EPICs record set in the computation of force coefficient, where the reference optical efficiency for the RX and TX PDs are stored. Where before the RX and TX optical efficiency EPICs records (TX_OPT_EFF_CORR and RX_OPT_EFF_CORR) were divided into each channels path, now, the TX record is multiplied in and the RX record remains divided in to each respective channel. (2) The calculation of optical efficiency *ratio* (comparing the ratio of TX to RX, then normalized by the above reference values) has been re-arranged, with an additional EPICs monitor of the unnormalized ratio for better understanding. Note: this causes a channel name change -- formerly, the normalized ratio was called OPT_EFF_RATIO_MON, and now it's called OPT_EFF_LIVE_OVER_REF. The new channel monitoring the unnormalized ratio is called OPT_EFF_LIVE_MON. (3) The former monitor of the ratio between channels was taken *after* this optical efficiency correction, with an EPICs and test point channel called ETM_PWR_RATIO_MON and ETM_PWR_RATIO_OUT. These have been unceremoniously removed. See before vs. after screenshots. Outside the force calculation, just prior to the conversion to displacement in the TX_PD and RX_PD filter banks, there's now a new EPICs record that applies the multiplicative correction to account for the differences in displacement between the X and Y arm, informed by the side-by-side comparison between the answer as seen in DARM. This new record is XY_COMPARE_CORR_FACT. See before vs. after screenshots. To import these changes was relatively simple, just an svn up to /opt/rtcds/userapps/release/cal/common/models PCAL_MASTER.mdl The h1calex and h1caley models, with the following updated library part, have been built successfully in prep for tomorrow's install and restart. I'll update MEDM screens tomorrow.
The CDS team informs me that it's been recently decreed that "no channels shall be removed without ECR," so I've reverted the removal of the OPT_EFF_RATIO_MON, OPT_EFF_RATIO_OUT, ETM_PWR_RATIO_MON, ETM_PWR_RATIO_OUT channels since the PCAL team doesn't have an ECR for these changes. All other infrastructure that rearranges the calculation and adds new channels will still go in as described above. See new screenshot of the FORCE_COEFF block. Here's the official channel change list for these changes jeffrey.kissel@cdsws03:~$ check_model_daq_configuration h1calex --------------------- file times ---------------------- Tue Apr 25 07:54:43 2023 = Model build time Tue Apr 18 10:21:21 2023 = Current configuration load time DAQ configuration is changed, processing... ++: slow channel H1:CAL-PCALX_FORCE_COEFF_OPT_EFF_LIVE_OVER_REF added to the DAQ ++: slow channel H1:CAL-PCALX_FORCE_COEFF_OPT_EFF_LIVE_MON added to the DAQ ++: slow channel H1:CAL-PCALX_XY_COMPARE_CORR_FACT added to the DAQ Total number of DAQ changes = 3 (3 additions, 0 deletions) jeffrey.kissel@cdsws03:~$ check_model_daq_configuration h1caley --------------------- file times ---------------------- Tue Apr 25 07:59:02 2023 = Model build time Tue Apr 18 10:21:21 2023 = Current configuration load time DAQ configuration is changed, processing... ++: slow channel H1:CAL-PCALY_FORCE_COEFF_OPT_EFF_LIVE_OVER_REF added to the DAQ ++: slow channel H1:CAL-PCALY_FORCE_COEFF_OPT_EFF_LIVE_MON added to the DAQ ++: slow channel H1:CAL-PCALY_XY_COMPARE_CORR_FACT added to the DAQ Total number of DAQ changes = 3 (3 additions, 0 deletions) which all match the functional changes expected from above.
R. Short, J. Kissel
After Marc and Fernando replaced the ISC-R4 power supply this afternoon, the IMC couldn't lock and appeared poorly aligned. We checked that the MC and IMC PZT sliders/OSEM values hadn't changed, so I tried moving the PZTs to realign the cavity. Around that time we realized that the IMC had not been taken offline before the rack was taken down, so the IMC WFS had just been trying to follow noise. I then cleared the history on the WFS filter banks and brought the PZT sliders back, and the IMC was able to lock successfully.
The -18V power supply located in the Power Mezanine above the CER in rack C5 slot U13-U15 on the Right Hand Side (RHS) was measured over 130F+ from the front of the supply. This is indicative of a failed fan, so this supply was pulled and replaced. We took the opportunity to do this due to the earthquake.
Old Supply S1201952 was set to -18.97V
New Supply S1201930 was set to -18.93V
We used the sequencer aboev the rack ISC-R4 to make sure that we did not damage any low noise power boards.
We noted that the power supply is routed to the junction box above PSL-R2 and then jumps across to power ISC-R4.
M. Pirello, F, Mera
Nutsinee and I looked at LLO alog #64483 from Masayuki where LLO had a found 2050Hz peak in DARM when ZM4 PSAMS are at 200V.
We do have this 2050Hz peak in DARM but only when the PSAMS are actively changing. See attached. These PSAMS will not be adjusted while Observing. Nutsinee suggests it could be a jitter coupling with an alignment change. When The SQZ ASC are changing without PSAMS there seems to be a smaller 2033Hz peak.
I compared three times (2023/04/10 02:35 - 03:05 UTC; 2023/04/14 00:18 - 1:53UTC; 2023/04/20 08:53 - 10:23UTC) where we changed PSAMS between <150V and 200V and didn't see any peaks in the 2000Hz region before/after the change, only during.
TITLE: 04/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 16mph Gusts, 11mph 5min avg
Primary useism: 0.18 μm/s
Secondary useism: 0.16 μm/s
QUICK SUMMARY:
- IFO still recovering from a 7.x mag EQ that kicked a bunch of sus
- Things have started calming down a bit so we're going to try and relock now (might be starting with an initial alignment)
- DMs ok/CDS ok
TITLE: 04/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
SHIFT SUMMARY:
Locks:
Lockloss at 20:22 UTC due to a 7.1 magnitude Indonesia earthquake (10hr 15 min lock).
Put IFO in DOWN while the earthquake rings down
IFO went into SEI_CONF EARTHQUAKE mode for ~15s before losing lock
Achieved NLN after shift - started locking as shift ended.
Manual Touching:
SQZ was touched up at 16:17 UTC →16:45 UTC
Done due to live squeezing looking worse than the 4/20 reference - Camilla realized this and took a new reference.
Changing the SQZ phase, I started at 164.03 Deg and ended at 135.59 Deg
Resulted in FDS lower than the last 4/20 reference as well as Camilla’s reference - both as seen on the FDS FOM.
Turned off sensor correction at 20:44 UTC by changing SEI_ENV to maintenance. This put SEI_CONF from EARTHQUAKE to SC_OFF. This was done due to the Earthquake.
SEI_ENV and SEI_CONF returned to EARTHQUAKE Mode with sensor correction on. This happened at 21:51 UTC.
Turning back to SEI_ENV is CALM (SEI_CONF is WINDY) at 22:43 UTC
Turned on ALS Polarizer through ALS Guardian (to fix some polarization ring-ups) at 21:23 UTC.
SUS_RM1 and SUS_RM2 turned to SAFE at 22:02 UTC.
SUS_RM1 and SUS_RM2 returned to ALIGNED at 22:35 UTC.
Done due to a power supply switch.
Automatic Early Morning Lock:
This morning’s running 5-hour lock was automatic. There seem to have been three NLN attempts during Sunday night, two of which were successful!
Attempt 1: Started trying at about 5:07 UTC (22:07 PDT) and eventually achieved NLN at around 6:00 UTC (23:00 PDT) before losing lock 1 hour and 48 minutes later at 07:45 UTC (00:45 PDT). Relocking was attempted immediately after lockloss.
Attempt 2: Reached and stayed CAMR_TO_TR at 8:31 UTC (1:31PDT) for ~9 minutes before losing lock at 8:40 UTC (1:40 PDT). Relocking was attempted immediately after lockloss.
Attempt 3: Reached NLN at around 10:06 UTC (3:06 PDT), which was the state I found it in.
Other:
- Dust monitor investigation at 16:55UTC -
- The locked period of the day had a couple of state changes between Cal Meas and NLN for measurement taking.
Cal Meas at 18:27 UTC
Back to NLN at 19:02 UTC
Cal Meas at 19:52 UTC
Back to NLN at 20:11 UTC
- At 19:33 UTC - Monthly Hanford Site scheduled test alert successful (and at the right, right phone)
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 15:03 | FAC | Karen | Optics & Vacuum Prep Labs | N | Technical Cleaning | 15:28 |
| 15:28 | FAC | Karen | MY | N | Technical Cleaning | 16:26 |
| 16:18 | FAC | Kim | MX | N | Technical Cleaning | 17:18 |
| 16:49 | FAC | Tyler | EX (Chiller Yard) | N | Dropping Parts | 17:19 |
| 17:34 | CDS | Marc | CER Mezzanine | N | Writing down values | 17:54 |
| 17:35 | FAC | Karen & Kim | FCES | N | Technical Cleaning | 18:11 |
| 19:51 | CDS | Fil | EX & EY | N | PEM Injection Work | 20:51 |
| 20:49 | VAC | Gerardo & Travis | LVEA | N | Ion Pump Measurements | 21:09 |
| 21:01 | CDS | Fil | EY & EX | N | Electronics Work | 23:30 |
| 21:20 | EE | Marc. Jason, Fernando, Betsy | CER Mezzanine | N | Stagnig for Tues Maint | 21:49 |
| 21:26 | VAC | Vac Team | MX | N | Purge AIR Check | 22:03 |
| 21:46 | SYS | Gabriele | X/Y Arms | N | DSR (Daily Schedule Run) | 23:08 |
| 22:05 | CDS | Marc, Fernando | CER Mezzanine | N | Switch Power Supply | 22:29 |
| 22:21 | SUS | Betsy | LVEA | N | Walkabout | 23:08 |
Gabriele ran a recent bruco in CHARD Y, and noticed there was large coherence between CHARD Y and the IM damping loops between 4-10 Hz. The IM damping loops have been updated before to improve noise reinjection into CHARD and therefore DARM. Take a look at alogs 64007 and 64017 for more information. Over the summer, I updated the damping loop cutoff to a high Q elliptical low pass at 7 Hz. There is room for improvement, by reducing the Q of the elliptical filter and moving the cutoff to 5 Hz. There is quite a lot of phase margin in the loops to do this. The new cutoff loses about 11 degrees of phase, but all three damping dofs have 70 or more degrees of phase margin. This new cutoff allows 23 dB less gain at 8 Hz and no change in the Q. Comparison of the damping design is here.
I placed the new cutoff in all IMs and all dofs. I made measurements of IM1 in L, P and Y. The new measurements are saved in "/ligo/svncommon/SusSVN/sus/trunk/HAUX/H1/IM1/SAGM1/Data" as "2023-04-24_2206_H1SUSIM1_M1_WhiteNoise_{dof}_0p05to100Hz.xml".
The new cutoff filters are SDFed.
Changed tcs_nom_annular_pwr for CO2X in lscparams.py from 1.15 to 1.5W and re-loaded the TCS_ITMX_CO2_PWR guardian. On our next power-up, the CO2X will turn on with 2.2W rather than the nominal 1.7W. This is so that Elenna can cotinine her CO2 measurements that were stopped when we lost lock (she increased CO2X from 1.7 to 2.2W at 18:47UTC). We should revert this change later after the CO2 tests are finished.
I have edited LSCparams to take CO2X to 4W annular. Again, this is just for testing, so that as soon as we have lock we can start thermalizing these settings and running measurements. This is not necessarily the nominal operating point.
FAMIS 23803
The ndscopes Look for the vibrometers look reasonable, nothing unreasonable.
J. Kissel, T. Shaffer, R. Short
I've instantiated a new guardian node CAL_AWG_LINES that's a reduced and re-purposed copy of AWG_LINES. Both nodes use Craig's SineMultiple code to generate an arbitrary of lines and amplitudes through a single excitation channel.
The code now lives in, and has been committed to
/opt/rtcds/userapps/release/cal/h1/guardian/
CAL_AWG_LINES.py
Attached are some screenshots of node manager, state graph, and state ladder.
The intent of the state evolution is quite simple:
- request LINES_ON, and it'll transition from IDLE (not doing anything), through TURN_ON_LINES, to LINES_ON.
- request IDL, and it'll transition from LINES_ON, through TURN_LINES_OFF, to IDLE.
The node as been instantiated, via
guardutil print CAL_AWG_LINES # initial check to test whether there are any obvious bugs.
guardctrl enable CAL_AWG_LINES # creates the infrastructure for the new node
guardctrl start CAL_AWG_LINES # actually turns on the guardian
guardmedm CAL_AWG_LINES # pops up medm screen for early testing direct access to the node manager
To do:
- actually test out the node
- Add to guardian overview screen
- add channels to DAQ
All that'll be done by tomorrow afternoon.
Wrote a new guardian called THERMALIZATION. FOr now it will only change the PRCL gain following a tyrend close to what we measured over past locks.
The plan is that ISC_LOCK will request the "THERMALIZED" state after LOWNOISE_LENGTH_CONTROL, instead of turning on the PRCL UGF servo.
The THERMNALIZATION guardian will transition to "SERVO_PRCL_GAIN" and start increasing the PRCL gain (with 30s ramp) following the equation:
H1:LSC-PRCL1_GAIN = [Gain set in LOWNOISE_LENGTH_CONTROL] * SCALE
SCALE = 1 + 3*(1 - exp(- time / 7800))
After 6 hours the THERMALIZATION guardian will stop adjusting the gain and go to "THERMALIZED"
We still need to test this new guardian at the next lock acquisition
The new hard-coded adjustment of the PRCL gain worked almost correctly. During one lock the initial gain was incorrectly set to 8 instead of 6. This probably happenend becvause the THERMALIZATION guardian read the PRCL1 gain too early after ISC_LOCK changed from 8 to 6.
I added a 10 seconds wait in the main() function of the SERVO_PRCL_GAIN to avoid this problem in the future.
E. Goetz, J. Kissel
We're getting serious about measuring the thermalization during the engineering run to better understand how the systematic error in calibration evolve during the first ~6+ hours after power up to 75W, so I've taken the time this morning to settle on number of lines, their frequencies, and their amplitudes before I push them into a guardian node using awg. In short -- two changes from when I last resurrected the lines,
(1) I'm adding two more line pairs, and
(2) I'm driving PCALY instead of PCALX because after the reduction of the TDCF line amplitudes there's now much more available range on PCALY than PCALX.
(3) I'm choosing line frequencies based on being bin-centered in a 40 sec FFT (the same as we did in O3)
H1:CAL-PCALY_PCALOSC4_OSC_FREQ 8.925 3000
H1:CAL-PCALY_PCALOSC5_OSC_FREQ 11.575 1000
H1:CAL-PCALY_PCALOSC6_OSC_FREQ 15.275 120
H1:CAL-PCALY_PCALOSC7_OSC_FREQ 24.500 80
H1:LSC-DARM1_EXC 8.825 5.0e-9
11.475 1.2e-9
15.175 3.0e-10
24.400 1.0e-10
The PCAL and DARM line amplitudes are tuned to the "coherence be consistently above 0.99" using DTT's closest thing to a 40 second FTT, which one might think means entering 0.025 in the BW (and I've done so), but really that yields at 32 sec FFT. Nothing fancy here, and I didn't spend more than 20 minutes on doing this, so nothing's perfect. I've not yet considered the impact on BNS range. We should probably have subtraction teams add this to their list though.
Note, these tests today were driven by individual oscillators for the PCAL system. When things are coded up into guardian using AWG I'll need a standard filter module's excitation channel (like the DARM_EXC channel). As such I'm going to chose H1:CAL-PCALY_DARM_EXC (the old filter bank that was creating for testing whether we had the range to control DARM by driving the QUAD with PCAL instead of its ESD at the lowest stage). As with the oscillators, this channel is not independently stored in the frames, but everything is summed into H1:CAL-PCALY_EXC_SUM_DQ before heading out to the DAC. In reality, when the analysis is performed, we use the RX_PD readback channels, which are nicely already calibrated into DARM displacement [meters].
The H1 DAQ full-frame and GDS-frame channel lists are available on the above web page.
A cronjob running every 6 hours starting at 2am compliles the DAQ full-frame channel listings for h1daqfw0 and h1daqfw1 frame writers.
The DAQ chan list files are ordered alphabetically by channel name. Lines are space delimited with the format:
CHANNAME sample_size(bytes) data_rate(Hz)
The cronjob also compiles the GDS channel listings for h1daqgds0 and h1daqgds1. The GDS chan files list the channel names only, ordered alphabetically.
Note that this system compiles the working files for the DAQ and GDS. It is not dependent upon code repository files being committed.
HAMs
HAM 7 and 8 Look different than the rest of the HAMs and comparing 7 to 8, & seems noiser.
Looking at the previous alog this seems to be expected.
BSCs
All of the BCS look fairly reasonable but there is an increase in noise around 60hz on ETMX_ST2_CPSINF_V2 &3 .
FAMIS 19949
PMC reflected power is trending slightly upwards. The brief laser shutdown from Thursday morning (4 days ago) appears as it should.
Measured and retuned the PRCL feedforward with a measurment with the IFO about 5 hours into a lock.
Good performance measured.
Everything is in guardian.
The interesting observation is that we are now limited by non-linear / non-stationary coupling of PRCL to DARM: while injecting PRCL noise with the new FF on, there is little coherence betwen DARM and PRCL, but DARM noise is increased broadband by ~10.
The residual PRCL noise ocupling to DARM is non-statioanry, as shown in the attached spectrogram and BLRMS plots.
One cal also look at the time with a PRCL noise injection and the PRCL FF on. A linear and stationary subtraction can improve DARM just a little, showing that the FF is already doing a good job. Allowing the subtraction to use ASC signals for coupling modulation allows us to subtract a lot of the residual PRCL coupling from DARM.
However, this is not necessary to clean DARM in normal condition: we had to inject PRLC noise to see the effect.
At 04:56 it appears we lost communication to the CER Beckhoff system. Timing MEDMs are attached.
Gabriele, DanielS, Jenne
Indeed it looks like the power supply has failed. Alog 68911 indicated it was very much not healthy, and Richard pointed out in a comment that plans were already in place to replace it early next week, but it seems that it didn't last that long.
Fil and Fernando are discussing whether it'll be possible to replace it this weekend.
Corner station ethercat device3 Slavecountactual went from 310 to 1 at 04:55:54 PDT
Supply replaced per WP11147
Power looks good, old supply fan bushing was completely shot. Turning off the lights and heading home.
Device3 Slavecountactual increased from 1 to 301, but still 9 shy of 310 needed. We are contacting Daniel and Patrick to see what is needed to complete the recovery.
Logging into 10.105.0.26 I first saw that the Anybus X-gateway terminals were not in OP, so I toggled the requested state of the EtherCAT device. This brought these back but then I noticed that the Corner Chassis 4 L10 EL9410 was in error (see first two screenshots), so I tried toggling some more. Now that terminal is back, but a few of the terminals after it are still in error (see last screenshot, yellow above terminals instead of green).
Toggling isn't bringing these back. Next step would be to power cycle corner chassis 4. If that fails, pull the chassis.
Power cycled Corner 4 at 1:25 pm
Power cycling appears to have brought things back online.
Supply replaced was S1201930, this supply had a failed fan. We will replace the fan and put this supply into the spares inventory. I have it recorded that thie old supply was sourcing 4A at 24V.
Supply installed was S1201903. This supply is on mezanine rack C2, U25-U27 LHS powering ISC C2 Slow Controls with 24V. This supply seems to be delivering 8A, unclear why this is.
We now had several locks with the PRCL UGF on. In each lock there were different TCS settings and different thermalization going on. Nevertheless the behavior is quite consistent.
The PRCL gain needs to be increased by a factor 3.8 in about 400 seconds.
A fit of one of the trends shows a good match with an exponential thermalization with a time constant of 100 minutes.
We could switch off the PRCL UGF servo (and the 52.1 Hz line) and hard code a gain change as above.