In alog_81849 Sheila and Camilla found a bug in the picomotor controller code, whereby if the motor to be controlled is changed while the controller is busy with the current move command, the motor change is not accepted resulting in the wrong motor being driven during subsequent commands.
As a quick work around until such time as the code can be fixed, I've made a change to the MEDMs which hides the motor selection buttons when the controller is busy.
Attached screenshots show a test case, left shows the normal situation where the controller is not busy and the selection buttons are all visible. The right screen shows a simulation of the controller being busy (INUSE=1), the selection buttons are replaced by rectangular blocks.
Wed Dec 18 10:12:11 2024 INFO: Fill completed in 12min 7secs
Today we had low TC mins [-156C,-155C] because of an unusually warm outside air temp of 56F, 14C.
To hopefully help combat our recent bouts of PI24 ring ups (alog81890), we've bummped the ETMY ring heater power from 1.1W per segment to 1.2W. The safe.snap and observe.snap are updated.
We also have a plan to update the PI guardian to temporarily increase the ETM bias to hopefully give us more range to damp the PI. We will need to be out of Observing for this, but we could hopefully save the lock. TBD.
Sheila, TJ, Camilla
We've had 3 locklosses since 9th Dec from PI24 ringing up (10.4kHz on ETMY): last night 81883, Tuesday AM 81862 and the 9th Dec, plot of all.
We had locklosses like this in September after the vent, Oli has a timeline in 80299. We increased the ETMY RH on September 26th 80320 and didn't have any PI locklosses since.
We normally damp this PI during the start of thermalization and successfully damped this after a 13 hours locked on 10th Dec, plot.
We can see a 25Hz wiggle in DARM just before the lockloss, e.g. 81862. The lockloss tool isn't tagging these as DCPD Saturation so we should implement LLO's PI code: Locklost Issue #198
Plotted is the location of the arm higher order modes around 10kHz. They have moved so that the central frequency is the same as the spikes around 10.4kHz, this excites the PI's more.
Maybe an unrelated point is that since the start of October, the circulating arm power has drifted down ~4kW (~1%) from 389 to 285kW when thermalized, plot attached. It's hard to know what's caused this, the IM4 TRANS drifted ~2% but was misaligned so would see alignment shifts 81735, the POP also drifted ~1% before it was recentered 81329. Input power measured by IMC_PWR_IN remained constant.
You cannot just use L1's lockloss code for the tagging universally [should work for the 10kHz PI though].
Because of the way LHO changed their readouts for the OMC, digging into the simulink model will reveal the PI is reading in only the first 32kHz band, for the RMS calculations
You simply have no aliasing down of the full band to utilise it in the way we do; and it will never see 80kHz PI.
Also your scaling is using calibrated DARM units by the look of things; while we use a counts scale; so the numbers in the RMS readouts are different many orders of magnitude.
Very quick shot of the 10.4k homs 2 hours into the first lock with the EY ring heater bumped up to 1.2W from 1.1W. This will need to be checked again later, but it's promising so far.
We had two short locks under an hour last night and both seemed to have lost lock from the ETMX glitches that we see (1418536304 and 1418541959). The former was tagged as an ETM glitch and is clearly much more obvious, the latter was moore minor but still looks like there was something odd going on a few tenths of a second before the lock loss.
TITLE: 12/18 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Wind
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 42mph Gusts, 32mph 3min avg
Primary useism: 0.09 μm/s
Secondary useism: 0.45 μm/s
QUICK SUMMARY: I just brought the IFO to IDLE since we are getting wind gusts into the 40s and ALS won't stay locked.
TITLE: 12/18 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:
Unknown lockloss at 5:51 UTC.
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1418536304
Screenshot attached below
Current Locking state: Carm_Offset_reduction.
TITLE: 12/18 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 3mph Gusts, 1mph 3min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.35 μm/s
QUICK SUMMARY:
Lockloss from a PI24 Ring up
https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1418526747
Pi monitor tried to Damp it but couldn't turn it around at all for over 15 minuts, where I attempted to mannually damp it.... this was not a successfully activity.
TITLE: 12/18 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 150Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY: Just got back to Observing with the POP Beam diverter open. We plan to leave this beam diverter open until the end of Tony's shift at 0600UTC. Relocking had three bits of testing, slowing the process down. There was some ALS testing (alog81870), SRCL OLG (alog81872), and DRMI locking (alog81880), then a lock loss closing the beam diverter...I pressed the wrong the button. Relocking was straight forward after that.
LOG:
TITLE: 12/18 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 11mph Gusts, 5mph 3min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.35 μm/s
QUICK SUMMARY:
IFO is locked, and Observing for 26 Minute now.
Note about the Configuration tagging DetChar & PEM:
The CORNER_POP Beam Diverter is currently OPEN and When I leave tonight I will make sure to close the beam diverters, after saying "The Beam Diverters Are About To Close" in my best imitation of the verbalarams voice.
Beam Diverter close instructions for future self:
Sitemap -> LSC -> Beam_Diverters -> Corner -> Corner_POP close button.
The Dust mon in the Vac prep lab is still not working.
Sheila D, TJ
Recently we've been having very long DRMI acquisition times. Sheila remembered that this might have to do with her PR3 changes on Nov 18 (alog81329). Tony confirmed this with some sleuthing (alog81879). Looking at the height of the POP air inputs to the MICH, SRCL, and PRCL triggers, this has almost doubled since the change on Nov 18th. So, we bumped up the triggers for these in lscparams.py by double.
Here's a reference from 2018 of what trigger levels were as a percentage of the locked build ups for DRMI at a time when it was locking well. 44348
I added a comment here, 81885, pointing to Tony's alog. The main point is that we probably may also need to update the check that tells the guardian to run a PRMI alignment for the higher flashes.
WP12249
Jonathan, Dave:
The offloading of the past 6 months of raw minute trend files from h1daqtw0 SSD-RAID to permanent archive on HDD-RAID is completed.
nds0 restart | 11:41 Mon 16dec2024 | |
file copy | 12:06 Mon 16dec2024 - 12:59 Tue 17dec2024 | 24hr 53min |
nds0 restart | 13:27 Tue 17dec2024 | |
old file deletion | 13:38 - 15:56 Tue 17dec2024 | 2hr 18min |
TW0 raid usage went from 91% to 2%. Jonathan made the DAQ puppet change to configure nds0's daqdrc file with the new archive.
Currently Dust LAB2 is not working, 0.3u count is NaN, 0.5u count is flatline zero.
Looked at the spectum of th 50W CO2 lasers on the fast VIGO PVM 10.6 ISS detectors when the CO2 is locked (using the laser's PZT) and unlocked/ free running: time series attached. Small differences <6Hz, see spectrum attached.
Gabriele, Camilla.
We are not sure if this measurement makes sense.
Attached is the same spectrum with the CO2X laser turned off to see the dark noise. It appears that measurement is limited to dark noise of the diode above 40Hz. The ITMX_CO2_ISS_IN_AC channel dark noise is actually above the level when the laser is on, this doesn't make sense to me.
Gabriele and I checked that the H1:TCS-ITM{X,Y}_CO2_ISS_{IN/OUT}_AC filter: de-whiten zpk([20], [0.01], 0.011322, "n"), is as expected from the PD electronics D1201111, undoing the gain of 105dB with the turning point around 20Hz, foton bode plot attached.
This means that both the AC and DC outputs should be the voltage out of the photodetector before the electronics, where the PD shunt resistance was measured to be 66 Ohms.
Sheila, Ibrahim, Jenne, Vicky, Camilla
After Ibrahim told Sheila that DRMI has been struggling to lock, she checked the POP signals and found that somethings been drifting and POP now appears to be clipping, see POP_A_LF trending down. We use this PD in full lock so don''t want any clipping.
We has similar issues last year that gave us stability issues. These issues last year didn't have "IMC" lock-losses so we think this isn't the main issue we've having now but maybe effecting stability.
Trends showing the POP clipping last year, and now. Last year we moved PR3 to removing this slipping while looking at the coherence between ASC-POP_A_NSUM (one of the QPD on the sled in HAM3) and LSC-POP_A (LSC sensor in HAM1): 74578, 74580, 74581.
Similar coherence plots to 74580, for now are show the coherence is bad:
Sheila is moving the PRC cavity now which is improving POPAIR signals, plot attached is the ASC-POP_A_NSUM to LSC-POP_A coherence with DRMI only locked before and during the move. See improvements. Sheila is checking she's in the best position now.
We have been holding 2W with ASC on before powering up, and using the guardian state PR2_SPOT move, which lets us move the sliders on PR3 and moves PR2, IM4 and PRM sliders to follow PR3.
Moving PR3 by -1.7 urad (slider counts) increased the power on LSC POP, and POPAIR 18, but slightly misaligned the ALS comm beatnote. We continued moving PR3 to see how wide the plateau is on LSC POP, we moved it another -3urad without seeing power drop on LSC POP, but the ALS paths started to have trouble staying locked so we stopped there. (POP18 was still improving, but I went to ISCT1 after the OFI vent and adjusted alignment onto that diode 79883 so it isn't a nice reference). I moved PR3 yaw back to 96 urad on the yaw slider (we started at 100urad), a location where we were near the top for POP and POPAIR 18, so in total we started with PR3 yaw at 100 on the slider and ended with 96 on the slider.
Please see Tony's comment about DRMI lock acquisition times here: 81879
When we moved PR3, we inceased the level of light on the POPAIR B diode. That does two things, first it means that the LSC controls were triggered at a lower percentage of the full buildups, which could cause difficulty locking DRMI (we want to maintain the trigger thresholds similar to what was documented in 44348. ). Also, the alignment check that Tony describes won't work well, because we didn't update the threshold the guardian used to decide that the alignment was poor and we needed to do PRMI.
From Tony's historgrams I'm not sure which of these is the main impact on DRMI locking times, if it's the triggering or the alignment check. We updated the trigger levels today, but not the threshold for the alignment check.
In the future we should check all these levels again 44348 when we have a change in power on POPAIR.
Sheila, Jenne, Tony, Camilla
We've had locklosses in DRMI because the PRCL gain has been to high when locked on REFL1F. Tony looked and thinks that this started on 77583, the day of our big shift in the output alignment.
Today we acquired DRMI with half the gain in the PRCL input matrix for 1F, this one acquisition was fast. I've attached the OLG measurements for PRCL and MICH after the change.
Tony is working on making histograms of the DRMI acquisition times, before the 23rd, from the 23rd to today, and eventually a histogram from today for the next few weeks to evaluate if this change has an impact on the DRMI acquisition times.
Jenne also found that it seems out POP18 build up seems higher in DRMI since the 23rd.
I'm no longer quite so sure about the conclusion that Pop18 is higher, or at least enough to really matter.
Here are 2 screenshots that I made extremely quickly, so they are not very awesome, but they can be a placeholder until Tony's much more awesome version arrives. They both have the same data, displayed 2 ways.
The first plot is pop18 and kappaC versus time. The x-axis is gpstime, but that's hard to interpret, so I made a note on the screenshot that it ranges from about April 20th (before The Big Shift) to today. Certainly during times when the optical gain was low, Pop18 was also low. But, Pop18 is sometimes high even before the drop in optical gain. So, probably it's unrelated to The Big Shift. That means that the big shift in the output arm is not responsible for the change in PRCL gain (which makes sense, since they should be largely separate).
The second plot is just one value versus the other, to see that there does seem to be a bit of a trend that if kappaC is low, then definitely Pop18 is low. But the opposite is not true - if pop18 is low kappaC isn't necessarily low.
The last attachment is the jupyter notebook (you'd have to download it and fix up the suffix to remove .txt and make it again a .ipynb), with my hand-typed data and the plots.
I actually didn't load the guardian at the time of this change, so it didn't take effect until today.
So, we'd like histograms of DRMI acquitisiton times from before April 23rd, from April 23rd until today, and for a few weeks from today.
Using the Summary pages I was able to get a quick google sheet to give me before and after Histograms of how long ISC_LOCK was in DRMI 1F.
First Sheet's data is before Nov 18th 2024, consisting of 100 gpstimes and durations where ISC_LOCK was in AQUIRE_DRMI_1F.
Second Sheet's data is After Nov 18th 2024. Consisting of 100 gpstimes and durations where ISC_LOCK was in AQUIRE_DRMI_1F
Interesting notes about ISC_LOCK.
ISC_Lock will request PRMI or Check MITCH Fringes some where between 180 seconds and 600 seconds, depending on how much light is seen on AS_AIR.
If AS_AIR sees flashes above 80 then ISC_LOCK will not kick us out of DRMI until 600 seconds.
So it looks like one of the changes that happened on or around Nov18th made the Flashes on AS_Air higher but we are still not actually locking DRMI.
We had fewer Aquire DRMI durations, over 180 Seconds before Nov 18th's changes.
The previous timing master which was again running out of range on the voltage to the OCXO, see alogs 68000 and 61988, has been retuned using the mechanical adjustment of the OCXO.
Today's readback voltage is at +3.88V. We will keep it running over the next few months to see, if it eventually settles.
Today's readback voltage is at +3.55V.
Today's readback voltage is at +3.116V.
Today's readback voltage is at +1.857V.
Today's readback voltage is at +0.951V.
Today's readback voltage is at -2.511V