Back to Observing after planned commissioning and calibration.
Locked for 7 hours.
Andrei, Sheila
This morning we are taking the commissioning time to get a sqz data set similar to 77133 with longer averaging time and no other changes to the IFO happening in parrallel.
Data times:
After this data collection, we ran SCAN_SQZANG, which resulted in a demod phase of 184.35
In the attached screenshots the units are different in the top and bottom plots, I filed a dtt bug because the plot disappears if I try to change the units on the bottom plot: 403
Wed May 08 10:11:08 2024 INFO: Fill completed in 11min 5secs
Gerardo confirmed a good fill curbside.
Erik, TJ, Camilla. 16:00UTC to 16:45UTC
Continued troubleshooting awg issues in ITM in-lock charge measurements 77059. Erik had new code to monitor awg and we took ESD_EXC_ITMX to COMPLETE with amplitude of excitation 1000 smaller (so it will not effect SQZ data taking).
First time the excitation didn't come though, Erik suggested restarting the GRD code 'guardctrl restart ESD_EXC_ITMX'. After this, it worked fine. Tested with both the production awg (tested on ITMY) and Erik's awg version (tested on ITMX).
Unsure why we didn't have this issue on ETMs. Had to take the nodes back to manged when complete by taking SUS_CHARGE though INIT. Reverted amplitudes to nominal values.
h1susitmx and itmy were restarted on April 9 because of a Dolphin crash see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=77040
h1susetmy and h1susetmx didn't need to be restarted, so have been running since March 12. I think if a Guardian node opens an excitation on a model that's been restarted, the node must also be restarted.
There's one bit of confusing evidence: Last tuesday the excitation status bit for h1susitmx was set during the charge measurement and cleared after, even though excitation output was zero (see attached image). This is inconsistent with a guardian node merely needing a restart, in which case the excitation bit isn't set. Camilla pointed out that there were other excitations sent during the charge measurement that could explain the status bit.
FAMIS27788
I ended up not adding water to either chiller as Y was still overfilled and X was just getting to the ruler markings. The Y filter is starting to green and we should think about replacing it soon. This was all updated in the google sheet at T2200289.
Ran our usual calibration sweep with a broadband and simulines following the instructions on the TakingCalibrationMeasurements wiki.
Simulines start:
PDT: 2024-05-08 08:37:21.772712 PDT
UTC: 2024-05-08 15:37:21.772712 UTC
GPS: 1399217859.772712
Simulines end:
2024-05-08 15:58:52,248 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L3_SS/SUSETMX_L3_SS_20240508T153722Z.hdf5
ICE default IO error handler doing an exit(), pid = 571885, errno = 32
PDT: 2024-05-08 08:58:52.297522 PDT
UTC: 2024-05-08 15:58:52.297522 UTC
GPS: 1399219150.297522
TITLE: 05/08 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 3mph Gusts, 2mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.19 μm/s
QUICK SUMMARY: Observing for 2 hours after an automated relock. Commissioning planned for today.
TITLE: 05/07 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:
H1 has been running with a range of ~150Mpc (lower than normal) & rung up violins. Additionally, there were high winds for half of the shift, so H1 was not down for most of the shift. But once the sun set, the winds died down and finally allowed for an opportunity to lock and get back to Observing!
LOG:
H1 had a lockloss within the 1st hour of shift. Winds were rising at this time (gusts touching 25mph), and for the next few hours the winds have increased with gusts just under 50mph. Needless to say, have spent several hours in the IDLE state due to winds. The sun just set and wind gusts have been slowly coming down to 25mph and returning to locking.
Although the last couple of attempts for PRMI looked horrible and have immediately triggered CHECK MICH FRINGES. May run an alignment next.
After an Initial Alignment, had a lockloss at LOWNOISE ASC and this was with fully automated locking and DRMI locking fine. For attempt #2 after IA, currently at Power UP 10W.
Since the A2L script that measures a transfer function isn't quite minimizing our ASC noise in DARM, Sheila suggested re-finding the old A2L script that we used to use, which just looks at the height of the peak in DARM.
I think I've found it, in /opt/rtcds/userapps/release/isc/common/scripts/decoup/al2_min_LHO.py . I made sure that it, and other scripts in that directory, were checked in to svn (they were last modified somewhere between 2018 and 2019, depending on the script).
The script (as wrtitten) uses the ADS to actuate each of the quads, and uses the demods to find the size of the signal (it does a sum of the squares of the I and Q of the demodulated signal). The script just guess-and-checks several A2L gain values for each optic, makes a step, and checks again. Once it's finished, it does a linear fit to find the A2L value at which the peak height is expected to be zero. The script runs 8 dither lines (pit and yaw from each quad) around 20 Hz, so that it can do this guess-and-check for all the optics at the same time.
The values in the script are out of date (we've slightly modified the frequencies we use for ADS while locking), so those values and the filter numbers for the matching bandpasses need to be checked. Also, the excitation amplitude in the script is probably higher than we need (they are set to be 300 counts, but we use 30 counts at full power right now, but we may want a little better SNR so we might want to find a value that is between those two.
Also, we may find that we want to instead do the optics one at a time, at 30 Hz, where the coupling of ASC to DARM is more important to our range.
We can try this out during one of our commissioning windows later this week, to see how it goes.
Hey Jenne, I did a total rework of the A2L scirpt for LLO [70246, 69555, 70219].
We never did all of the quads at the same time. We only ran 1 QUAD at a time.
Despite that, the biggest issue I found was that running 2 lines (pitch and yaw) at the same time was a big no-no because the sideband noise they create bleeds into eachother's demod bandwidhts, making the data rubbish and consequently the result rubbish.
My new script that now lives in, and is up to date in the svn:
/opt/rtcds/userapps/release/isc/common/scripts/decoup/a2l_min_generic.py
Warning: despite the script living in a common directory, it is infact only for LLO, because it makes calls to our Calibration Guardian to turn off all of the calibration lines, since we chose to run the demod line at the worst decoupled frequency (exactly where our calibration lines are).
Feel free to draw upon my hours of testing this approach :)
I pulled Vlad's LLO script via the svn (thanks!!), and made a copy /opt/rtcds/userapps/release/isc/common/scripts/decoup/a2l_min_generic_LHO.py . The name is not so good, but at least by knowing that it's the most recently modified file in the folder, we might have some memory of it being the latest.
I lowered the amplitude of excitation to be 30 counts (the same as what we use for ADS after Lownoise_ASC), and when the A2L gains are at the extremal values that the script measures (+/-0.3 from nominal) the line is clear in DARM but not scary-big. I am using the same 30 Hz that Sheila had been using, so this 30 count amplitude is actually a bit smaller in meters than 30 counts at ~20 Hz for ADS.
I set up filters in PIT7 / YAW7 and changed the script to use #7 rather than LLO's #2 set of ADS infrastructure, since #7 was unused for us. I used the same 0.1 Hz half-width for the bandpass that I think Vlad has in place at LLO.
I commented out the "setting the matrices" section and just did those steps by hand today, since I need to import the correct guardian matrices and double-check the indices, and doing it by hand got me going faster to actually trying the script.
I added _SPOT to the test mass drivealign channel names, since we use those here.
I added a few measurement steps so that it's not just jumping by an A2L gain of 0.3. The IFO can handle it, and I may take these extra steps back out (where I pause at a value of 0.15 from nominal before finishing to the 0.3 step), but for testing I didn't want to be too risky.
I also added a calculation to Vlad's script (and this is what could easily be the source of the error that I'll talk about next has come from), to take the sqrt of the sum of the squares of the I and Q demod signals, so that I don't have to worry about setting the demod phase of the ADS demodulator.
However, the answer from the linear fit doesn't seem to make much sense at all. Again, this could be due to my modification of Vlad's script, so I'll need to come back to this and make sure I'm doing what I think I'm doing, before testing again on the IFO. I was only testing on ETMX P so far, and it's clear by watching the line height in DARM and watching the I and Q demod outputs that the current nominal that Sheila has set of 3.35 P2L gain for ETMX is about as good as we can do. However, after trying values between 3.05 and 3.65, somehow the fitting function seems to think that it should be set to 1.76(!!). Thankfully I had forgotten to add the _SPOT to the line that would have written the value and jumped the gain straight to there, so that value didn't actually get written to the IFO. I've commented out the writing of the value from the script for now, until I figure out what's going on with the fitting.
Next up is to see if I can understand why the fit tried to send me to such a strange A2L gain, then check that this amplitude is okay for other quads both pit and yaw, then actually run it to see it minimize coupling.
Below here is just notes to self, for figuring out what's going on with the fitting. I should have had the script print out more so I could double check it's sqrt-sum-squares, but I can at least check for the last value. The two long arrays are what are being fit to. Reminder to self that I got the same gain it wanted to send me to of 1.7-ish, even before I changed / added more steps and re-measuring some values. But, that was after I added in the summing of the squares. I didn't ever run the script with just looking at the I output of the demod.
Result in terminal from (lines 279-282 in the script)
print(gainList)
print(meanI)
print(meanQ)
print(meanList)
is
[3.2 3.05 3.2 3.35 3.5 3.65 3.5 3.35]
0.0004724146701240291
2.2764012343638282e-05
[0.00614473 0.01331277 0.0065377 0.00075812 0.00755947 0.01463074
0.00760131 0.00047296]
Want to change gain from 3.35 to 1.779, rounded to 2 decimal places. St.Div is 0.054
take the sqrt of the sum of the squares of the I and Q demod signals, so that I don't have to worry about setting the demod phase of the ADS demodulator.
I only take the I; I am not sure if the sqrt of I and Q might confise signs: and therefore break the linear fit of the function. Might make it non-linear; which breaks things. I quickly plotted (attachment) what you wrote, and it looks like it isnt going negative because sqrt will only give positive results. It tries to solve for zero crossing, but here zero looks to be near 3.35, but then goes back up.
Actually the reason I only look the I-phase is because I dont want go rephasing the demodulation, and just assume there is some signal in I. Then just solving for zero crossing should not care about actual amplitudes.
However, after trying values between 3.05 and 3.65, somehow the fitting function seems to think that it should be set to 1.76(!!)
Do not trust this! After fixing it such that you preserve sign (so that linear fit solves for zero crossing), always check the output to make sure it is not extrapolating a linear result far away, because at far distances you can't comepletly trust that it is linear.
Want to change gain from 3.35 to 1.779, rounded to 2 decimal places. St.Div is 0.054
For reference: the fitted zero crossing st.div for us is about 0.003. Adjust amplitudes accordingly, after you are getting logical results.
TITLE: 05/07 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Lighter maintenance today but we managed to do some more SR3/2 moving to investigate our recent required alignment shift. On the relock we had some lock losses that had to be fixed with some filter changes (alog77698). Violins have also come up during these relocks, but they are damping nicely. We are now observing for almost an hour.
LOG:
2240 Observing
Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
---|---|---|---|---|---|---|
14:57 | VAC | Gerardo | LVEA | n | Tubro pump functionality tests | 15:34 |
14:57 | FAC | Ken | LVEA, FCTE | n | Cable trays | 20:35 |
15:03 | VAC | Jordan | MX | n | Setup CP6 dewar pump | 15:31 |
15:07 | FAC | Karen, Kim | FCES | n | Tech clean | 15:40 |
15:12 | - | Mitchell | EX, EY | n | Hepi, dust pump checks | 15:54 |
15:17 | SQZ | Fil, Camilla | LVEA | local | TTFSS chassis swap | 15:46 |
15:17 | CDS | Jonathan | - | n | nds1 restart | 15:32 |
15:34 | VAC | Janos, Jordan, Travis, Fil | FCES | n | HV cable install for ion pumps | 18:34 |
15:34 | FAC | Chris | LVEA, outbuildings | m | Pest escort and FAMIS | 17:53 |
15:41 | FAC | Kim | EX | n | Tech clean | 17:17 |
15:41 | FAC | Karen | EY | n | Tech clean | 16:39 |
15:48 | CDS | Fil | EX | n | PCAL interlock inspection | 17:00 |
15:54 | SQZ | Terry | Opt Lab | local | SHG work | 17:51 |
16:04 | - | Jeff, Josh, Kar Meng | LVEA | n | Tour | 17:24 |
16:13 | FAC | Tyler | EX | n | Mac Miller fan bearing repair | 23:25 |
16:40 | VAC | Gerardo | LVEA | n | Turbo pump check | 16:58 |
16:49 | - | Richard | LVEA | n | Walk about | 17:08 |
17:16 | CDS | Marc | EY, MY, MX, EX | n | Power supply listening | 18:15 |
17:52 | FAC | Karen, Kim | LVEA | n | Tech clean | 18:46 |
17:56 | VAC | Gerardo | LVEA | n | Turn off turbos | 18:04 |
18:48 | TCS | Camilla | LVEA | n | Grabbing mirror mounts | 18:58 |
18:59 | SQZ | Terry, Kar Meng | Opt Lab | local | SHG work | 21:03 |
20:53 | TCS | Camilla | Opt Lab | local | CO2 testing | 22:53 |
21:51 | SQZ | Kar Meng | Opt Lab | local | SHG work | 23:04 |
22:08 | VAC | Janos, contractor | MX | n | Tour of vac system at MX | 00:08 |
Jenne D, Ryan S, Camilla C, TJ S
Today in a single bounce 10W configuration we set SR2 to center AS_C with the SRC2 ASC loops via the ALIGN_IFO guardian in the SR2_ALIGN state, then moved SR3 in steps to see how our AS_C power would change in different positions. We found that there is an area, or spot, that we lose significant throughput to AS_C, but moving around this spot in any direction recovers our nominal single bounce power. This is followup from Jenne Driggers SR3/2 move (alog77388) to bring back our power at the AS port on April 24th, and Jennie Wright, Minhyo, and Sheila's efforts searching in yaw to find where we are losing the power (alog77513). This time we went further than Jennie W and team went and managed to get back almost all of the AS_C sum back. Almost all because just before we got there we starting running into a different, perhaps clipping, issue that changed the shape of the AS air camera in a much different way than when around this spot.
More details:
We started out by reverting the alignment to a time during DRMI locking from the Tuesday morning of April 23. We then brought up tried to move SR2 and SR3 as was done previously, then Jenne suggested that we turn on SRC2 to have SR2 follow our SR3 moves and keep AS_C centered. This worked great along with a little extra SRC2 gain to help move things along faster. Moving in pitch in either direction yielded fairly quick power gains (trends SR3 -P move & SR3 +P move). Jenne suggested that we try to go further in yaw than the Jennie W. team did to see if it ever improves, and we were able to get AS_C almost back after ~550urad in SR3 (trend SR3 -Y move today, and SR3 +Y move April 24 ). If I had the time I'd love to make a nice graphic to show where we can't go but we'll have to deal with this table for now.
Pre April 24 | May 07 10:39 | May 07 11:03 | May 07 11:39 | Post April 24 | |
Start | -P move | +P move | -Y move | +Y move | |
SR3 P slider value | 438 | 81.9 | 798 | 438 | 438 |
SR3 Y slider value | -148.8 | -148.8 | -148.8 | -686.3 | 120 |
Here's a screenshot of the AS air camera when we were at the edge of our SR3 -Y move. The beam spot began to squish and spread across the screen a bit more.
Here are some trends with SR2 and SR3 M1 osems during these moves.
EDIT: Added a revamped table with the SR2 and SR3 OSEM changes. The deltas from these could all be found by subtracting the "Pre April 24" value from each column for each move. These values are the final position when AS_C power was recovered to nominal values.
Pre April 24 | May 07 10:39 | May 07 11:03 | May 07 11:39 | Post April 24 | |
Start | -P move | +P move | -Y move | +Y move | |
SR3 P slider value | 438 | 81.9 | 798 | 438 | 438 |
SR3 Y slider value | -148.8 | -148.8 | -148.8 | -686.3 | 120 |
SR3 P M1 osem | -290 |
-674 |
112 | -263 | -292 |
SR3 Y M1 osem | -616 | -592 | -640 | -1018 | -405 |
SR2 P M1 osem | -570 | 3040 | -1955 | 503 | 596 |
SR2 Y M1 osem | 11 | 34 | 28 | -2164 |
1160 |
Sheila, Jennie, Alena
Alena realised during her calculations of the spot positions on the OFI at different SR2 and SR3 alignments, that the osem value quoted in the table above for SR2 P M1 osem should be +570.
Sheila alos saw this when looking at osem trends.
TITLE: 05/07 Eve Shift: 23:00-08:00 UTC (16:00-01:00 PST), all times posted in UTC
STATE of H1: Observing at 66Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 16mph Gusts, 12mph 5min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.22 μm/s
QUICK SUMMARY:
TJ has H1 back to observing w/ a range just under 150Mpc and passed on some changes to keep an eye on for next-locks. Will also keep an eye on the rung up violins we have been dealing with the last few days.
It's a bit breezy, with gusts just touching 25mph (so hopefully not an issue). useism has slowly been ramping up from the 50th percentile for the last ~18hrs.
Maintenance recovered, back to Observing at 2240 UTC.
We struggled with a handful of lock losses at inconsistent spots and two lock losses just after Transition_from_ETMX. Jenne and Ryan S tracked this down to some changes made yesterday and changed a filter (alog77698), then we made it through one time. Unsure if this will be ok net relock or if it will interfere with locking though.
Sheila, Camilla.
After this morning's windy lockloss from TRANSISITON _FROM_ETMX, we continued previous 77366 troubleshooting.
Sheila manually stepped though TRANSISITON _FROM_ETMX:
[RyanS, TJ, Jenne]
We've had 2 locklosses today that seem to be about when that MadHatter Darm FM2 gets turned off. But, since Sheila and Camilla moved the filter turning off, now the locklosses are happening a little later.
As Camilla points out, the ramptime of that filter inside Foton is very short. I've increased it to 3 seconds and we're about to give it a try (since I don't have a better plan to try right now). Something we haven't seen (since we're already locked right now) is whether increasing the ramp of that filter inside Foton causes trouble for the engagement of that filter, in DARM_OFFSET.
EDIT: Indeed we made it through the turning-off of the MadHatter filter with this 3 second ramp time (rather than the previous 0.1 sec ramp). I am hopeful that it won't matter for the turning-on of the filter, since that happens in a quite stable part of the locking sequence. But, we'll just have to see over the next few lock acquisitions.
Also EDIT: If we are unable to relock with this ramp time, attached is a screenshot of the previous ramp time, that we should revert to.
We've now relocked twice with the longer ramp time in the MadHatter filter in place for the turn-on and turn-off actions of that filter, so it seems fine to leave in place.
In the observing stretch that started Monday around 8 pm, there was a drop in optical gain (by 4%), and power at AS port PDs (4% drop in AS_A sum, AS_B sum, AS_C sum, and OMC REFL). There not at that time any change in circulating power in the arms, PRG, coupled cavity pole, or alignment of suspensions (shown are SR2, SRM, OM1+2, but we have also looked at many other suspension, ISI and HEPI related channels for this time 77382 ). This time is shown by the first vertical cursor in the attachment.
We were unable to relock the interferometer after that Monday evening lock (a new lockloss type appeared early Tuesday morning, 77359), and after the maintence window we were uable to lock with difficulties powering up (77363) and an AS camera imagine that had lobes whenever the beam was well centered on AS_C. We locked that evening by adding large offsets to AS_C's set point, (77368). This is the cause of the large alignment shifts seen in the screenshot on Tuesday evening to Wed morning, which allowed us to lock the IFO but created a large scatter shelf and resulted in a lower coupled cavity pole, and lower optical gain, and we did not have much squeezing that night.
We do not think that Tuesday maintence activities are the cause of our problems, although some of the Tuesday work has been reverted (77350 77369).
Yesterday we spent much of the day with SRY locked, single bounce or using the squeezer beam reflected off SRM to investigate our AS port alignment. 77392, We are able to recover the same throughput of a single bounce beam to HAM6 by moving the alignment of SR2 + SR3 by a huge amount 77388, which also produced a round looking beam on the AS camera. We haven't since tried to explore this aperture, to see if we could have also recovered this thoughput with a pitch move, for example. We also saw that the squeezer beam does not arrive in HAM6 with a good transmission when injected with the alignment used previously, but that we could recover good transmission to HAM6 by moving ZM5 by 150-200 urad, in pitch or yaw. With this shift in SRC axis, and squeezer alignment we were able to relock and not have the large scattering issues we had Tuesday night.
Today's time was spent recovering from a seemingly unrelated problem in the SQZ racks, and some commissoning aimed at recovering our previous (165Mpc) sensitivity with this new alignment. This will continue during tomorow's commissoning time.
additional related information:
Adding some trends, which have otherwise been mentioned. The optiocal levers and top mass osems suggest that there hasn't been any shift in the PRC, Michelson, or arms. There was also not a shift in alignment of the SRC in the first low optical gain lock on the 22nd, that alignment didn't shift until the manual move of the SRC in the recovery effort.
Camilla notes that the OFI temperature controler had unusual behavoir in the Tuesday relocking attempts: 78399
comparisons of single bounce throughputs: 77441