Today we had two locklosses from the 80 kHz PI on ETMX (69800) in addition to one last Friday (69759), all occuring at Hour 2:30 into the lock. I compared today's 10.4 kHz HOMs with a previous long lock from May 13, when I think we had finished TCS tuning and stayed on at 76W. I didn't see another solid 10+ hour lock since then. Times used for comparison here.
Previously, we have ID'd the pointier HOM peak as ETMY, and the blunter peak as ETMX. Comparing today to 1.5 weeks ago, I think the HOM on ETMX (blunter) is now noticeably lower in frequency -- maybe that is why this 80kHz PI is now ringing up? Previously, it looks like we lowered the EX RH slightly to avoid this 80 kHz PI (eg. 68166, 68165). Maybe we ended up just barely stable after the last EX RH TCS change. While the EX VEA temperature looks pretty stable, there have been some temperature excursions in the last couple week, maybe this is related to the 80 khz PI showing up and the lowered ETMX HOM peaks? And now, possibly with the EX VEA is heating up b/c of summer (even though its temp looks stable), maybe it's worth trying to lower the ETMX RH a bit again, to further avoid this PI.
TITLE: 05/23 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
SEI_ENV state: SEISMON_ALERT
Wind: 8mph Gusts, 5mph 5min avg
Primary useism: 0.10 μm/s
Secondary useism: 0.21 μm/s
QUICK SUMMARY:
TITLE: 05/22 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY:
Winds & Dolphin crash caused grief on this shift.
LOG:
~0256 While locking had a Dolphin Crash. (See other alogs) This sadly took up the remainder of the shift. I have a separate draft with recovery notes and will post that later tonight.
IPC was found to be flashing red/green. There were DACKILLs on h1sus[b123, h2a, h34, h56] h1asc0, h1lsc0. The long range dolphin CS-ends were all red (ends to corner all green). I started to dolphin disable switch ports for omc, asc, lsc, oaf, then all the sus. Finally I disabled h1cdsrfm 3 ports (corner, ex, ey) which finally stopped the flashing
Prior to rebooting any SUS, I set all the SEI SWWDs into bypass mode.
As a first recovery, I rebooted h1susb123. It came back with no problems.
I then rebooted h1cdsrfm, which started with no issues, and at this point still showing no CS->ES traffic because all the CS senders are disabled.
I then rebooted all the remaining CS FE which had been dolphin disabled. All came back OK, all the h1cdsrfm CS->ES went green.
I untripped the SWWD and handed the system over to Corey, Jim and Rahul to complete the recovery.
h1cdsrfm dmesg reports a Dolphin IX adapter error at the time this IPC problem started.
[Tue Apr 11 11:14:37 2023] IXH Adapter 2 : Enable option to create session to all available nodes.
[Mon May 22 19:57:30 2023] IXH Adapter 2 : Uncorrectable error on adapter PCIe SLOT detected - Event_type=0x3 Event_data=0x20
[Mon May 22 19:57:30 2023] IXH Adapter 2 : Port 0 is not operational -- UpTime: 0 sec - Event = 3
[Mon May 22 20:11:51 2023] IXH Adapter 1 : Port 0 is not operational -- UpTime: 0 sec - Event = 0
[Mon May 22 20:11:57 2023] IXH Adapter 0 : Port 0 is not operational -- UpTime: 0 sec - Event = 0
All SUS are back online.
here is an example of the IPC flashing on h1iopseib2
Mon22May2023
LOC TIME HOSTNAME MODEL/REBOOT
20:13:51 h1susb123 ***REBOOT***
20:16:15 h1susb123 h1iopsusb123
20:16:28 h1susb123 h1susitmy
20:16:41 h1susb123 h1susbs
20:16:54 h1susb123 h1susitmx
20:17:07 h1susb123 h1susitmpi
20:20:47 h1cdsrfm ***REBOOT***
20:21:59 h1sush2a ***REBOOT***
20:23:36 h1sush34 ***REBOOT***
20:24:12 h1sush2a h1iopsush2a
20:24:25 h1sush2a h1susmc1
20:24:38 h1sush2a h1susmc3
20:24:51 h1sush2a h1susprm
20:25:04 h1sush2a h1suspr3
20:25:11 h1sush56 ***REBOOT***
20:25:46 h1sush34 h1iopsush34
20:25:59 h1sush34 h1susmc2
20:26:12 h1sush34 h1suspr2
20:26:25 h1sush34 h1sussr2
20:26:43 h1asc0 ***REBOOT***
20:27:15 h1sush56 h1iopsush56
20:27:28 h1sush56 h1sussrm
20:27:41 h1sush56 h1sussr3
20:27:54 h1lsc0 ***REBOOT***
20:27:54 h1sush56 h1susifoout
20:28:07 h1sush56 h1sussqzout
20:28:19 h1asc0 h1iopasc0
20:28:32 h1asc0 h1asc
20:28:45 h1asc0 h1ascimc
20:28:58 h1asc0 h1ascsqzifo
20:29:08 h1oaf0 ***REBOOT***
20:29:29 h1lsc0 h1ioplsc0
20:29:30 h1omc0 ***REBOOT***
20:29:42 h1lsc0 h1lsc
20:29:55 h1lsc0 h1lscaux
20:30:08 h1lsc0 h1sqz
20:30:21 h1lsc0 h1ascsqzfc
20:30:49 h1oaf0 h1iopoaf0
20:30:58 h1omc0 h1iopomc0
20:31:02 h1oaf0 h1pemcs
20:31:11 h1omc0 h1omc
20:31:15 h1oaf0 h1tcscs
20:31:24 h1omc0 h1omcpi
20:31:28 h1oaf0 h1susprocpi
20:31:41 h1oaf0 h1seiproc
20:31:54 h1oaf0 h1oaf
20:32:07 h1oaf0 h1calcs
20:32:20 h1oaf0 h1susproc
20:32:33 h1oaf0 h1calinj
20:32:46 h1oaf0 h1bos
At around 0256utc received Verbal saying HEPI HAM1 WD trip followed a few seonds later with verbals for TCS Chillers. As I looked on the Call List for TCS and was about to phone TJ I noticed the big Christmas Tree with the CDS Overview & OPS Overview flashing RED & GREEN.
At any rate, on TeamSpeak with Dave (whom has said "looks like a Dolphin Issue" & "not an easy fix".
Dave is already drafting his own alog, but wanted to get this out now. Sent a text to TJ w/ status.
Had one lock almost 2hrs and then had a lockloss (most likely due to winds). This lock happened to occur when therte was a lull in winds which were below 20mph, but then in the last 90min, the winds have picked up again (saw one gust up to 40mph).
Currently ISC LOCK is working on getting back to NLN.
Took OBSERVATORY MODE back to WIND state.
Attached is the filter change for MICH/SRCL FF filters that Elenna posted about earlier (alog 69785); this change was at 0008utc--about 5min after NLN. We are now in OBSERVING and thermalizing.
I've added increase flashes to the INIT_ALIGN guardian into the LOCKING_GREEN_ARMS state. For now its running mostly just a copy of the code from ISC_LOCK, but it will need to be changed around a bit tomorrow to be optimized for this node. I did a test while we were down for wind and it seemed to work with the current version of the code that's in there. Increase flashes will come on after 120 seconds with no locking (same as ISC_LOCK for now).
TITLE: 05/22 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 15mph Gusts, 10mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.14 μm/s
Environmental NOTEs:
QUICK SUMMARY:
(My first EVE shifts since Mar 2020!)
H1 was in the middle of an InitialAlignment when I arrived (currently locking at ENGAGE ASC FOR FULL IFO). Received full rundown from Ryan on how locking has been today (and over the weekend for him). Items Of Note:
Sheila, Vicky, Naoki
Recently we had several lockloss due to PI mode 29 ring up (80kHz PI). The PI guardian tried to damp it, but the PI kept ringing up and caused lockloss in ~5 minutes. We tried both common/diff ETMX actuation (LL:UR=1:1 and LL:UR=1:-1), but both of them failed.
We modified the PI guardian for mode 29. We increased the damp gain to 50000. To have more time to evaluate the correct damping phase, we increased the phase step to 60 deg, the average time to calculate the new rms to 5s, and the timer for each step to 10s. We will see if this change helps in the next PI ring up.
J. Kissel, J. Rollins, E. Goetz (with ideas, support and inspiration from others) Picking up where I left off on the saga of the IFO's thermalization vs. the DARM sensing function (LHO:69593), the open actions were: (a) more, lower, frequency points (b) synchronizing the thermalization period to a point in time (c) more data sets all to better understand if the sensing function during each thermalization period evolved consistently -- since we saw that the first three examples looked very much not consistent. Namely, we're still craving a single metric of thermalization that we can use to create a look-up table for "when the metric is value Z, apply this XX% / YY deg frequency dependent transfer function amount of extra modeled systematic error to the sensing function's budget of systematic error." I've not yet been able to execute (a), and probably won't before the start of the run (and thus maybe never). So we'll have to model what we can from these four frequency points. This aLOG focuses on (b) and (c). Regarding the number of data sets, (c), where LHO:69593 only characterized 3 data sets, here I've gathered 10, all since May 10th 2023. The times and durations are: UTC Start UTC Stop GPS Start GPS Stop Duration Ref. ID Notes 2023-05-20 22:00 2023-05-21 02:00 1368655218 1368669618 4.00 1 (has systematic error lines amp increase) 2023-05-20 15:40 2023-05-20 19:40 1368632418 1368646818 4.00 2 (has systematic error lines amp increase) 2023-05-20 07:45 2023-05-20 11:45 1368603918 1368618318 4.00 3 (has systematic error lines amp increase) 2023-05-19 19:30 2023-05-19 23:30 1368559818 1368574218 4.00 4 (has systematic error lines amp increase) 2023-05-18 11:00 2023-05-18 15:00 1368442818 1368457218 4.00 5 2023-05-17 18:48 2023-05-17 22:00 1368384498 1368396018 3.20 6 2023-05-16 04:48 2023-05-16 08:47 1368247698 1368262038 3.98 7 (has glitch) 2023-05-15 04:50 2023-05-15 08:50 1368161418 1368175818 4.00 8 2023-05-14 08:15 2023-05-14 11:45 1368087318 1368099918 3.50 9 2023-05-10 11:18 2023-05-10 15:18 1367752698 1367767098 4.00 10 Note, here, unlike the few datasets before, we've (sadly) had enough lock losses that I can select for lock stretches that are 4 hours or longer. This way, if we go with *time* synchronization, we have enough data within each stretch for the data to comfortably be declared "thermalized." Regarding synchronization of data sets, Jamie had the idea of synchronizing the sensing function to IFO Arm Power rather than time, since the varies ways of looking at the data from LHO:69593 indicate that the sensing function evolves on several, mixed, exponential, if not random functions of time (especially with the first 30 minutes of the IFO achieving nominal input power). On the other hand, at least by eye and with what limited study we had, the arm power seemed to be evolving in the same way as the sensing function (and this intuitively made sense given that the optical spring present in the sensing function is some relationship between the carrier power in the arm cavities vs. the power in the SRC). The pdf attachment, sensingFunction_syserror_vs_power.pdf, shows the comparison between these 10 data sets. (i) Page 1: shows all 10 data sets' arm power as a function of time, and then (ii) Pages 2-11: show 3D bode plots, where the frequency, magnitude, and phase of the sensing function systematic error transfer function are show against arm power (on the left) and time (on the right). Note each "arm power" data point over the 4 hour stretch is the median power value over each two minute stride of 16 Hz data, and the data is created by the average of the two arm powers at each 16 Hz data point, from the channels H1:ASC-X_PWR_CIRC_OUT16 H1:ASC-Y_PWR_CIRC_OUT16. Here's what I observe from this collection of data sets: (1) These additional data sets' sensing function is just as inconsistent in the first 30 minutes after achieving the nominal 76W PSL input power as the previous three from LHO:69593. So, that means, if we're going to create a model of this systematic error to add to the collection of time-dependent response function's systematic error like we did in O3, it's going to *have* to have large uncertainty, at least in the first 30 minutes. (2) The arm power vs. time evolution is also inconsistent from lock-stretch to lock strength, both in rate of change as well as final thermalized arm power. (x) During the discussion of the plots in LHO:69593, during last week's LHO commissioning meeting, the suggestion was that the thermalization may be dependent on how cold the optics are at the start of heating them back up, or -- phrased differently -- if it's been a while since the last well-thermalized, high power lock segment, then the heating up may look different. This is consistent with what's seen here -- the orange data set (set 9, starting at 2023-05-14 08:15 UTC) is one of the lower starting powers, and achieves a higher power than the other 9 sets, at a faster rate. Unlike the other 9 data sets, set 9 had 13 hours of "dead time" prior to re-lock-acquisition, vs. most other data sets had "only" 3 hours or less of "dead time," so the supposition is that the optics have not yet full cooled down. (y) Also, the thermalized power is different, but as much as 4-5 kW. This should be less of a big deal, because the SRC detuning will have landed on the SRCL offset value that *retunes* the response, which means the only thing *left* changing with time, or between lock stretch to lock stretch in the sensing function is the optical gain and cavity pole, which are constantly being measured and corrected for. (3) The arm power will be about as good a metric for thermalization as log(time) -- not perfect good, but better than linear time. So, we'll go with the arm power as the metric for thermalization, 'cause ...why not? We're winging this anyways, and we don't have time to research other channels or ideas (or even go back to other older ideas). Now it's on to manipulating this data into arm power (as a proxy for time) slices, and them fitting the *collection* of 10 transfer function values at each arm power, such that we have a representative sensing function systematic error transfer function -- with quantified uncertainty -- at each arm power level. I.e. we'll "stack" these arm-power slices and feed them to the GPR fitting monster and then create a the desired look-up table of fits per arm power.
I've finally wrangled python's handling of 3D data structures enough to re-arrange the above data sets in terms of a "stacked" large meta-collection of arm power slices from all 10 thermalization stretches rather than in terms of segregated the 3D bode plots as a function of time from each thermalization stretch.
Remember the goal is to bin these sensing function systematic error transfer functions synchronized with similar arm power values, such that we can create a representative model what the sensing function systematic error is doing for a given arm power at *any* time.
In other words, in the *response* function systematic error budget, which is computed at an arbitrary time, we trigger that application of "this" systematic error model given the arm cavity power of "that" value at that time.
I've conditioned the data I put into these meta-collections in the following way:
(1) How fine a grain do we need in terms of arm power? 1 kW steps? 2 kW steps?
Given that the data collections contain arm powers that range from ~396 kW to ~435 kW, 1 kW steps means 40 different models.
1 kW steps like a good balance between
- "a huge number of options for the look-up table" and
- "don't bin too many arm power values together, because in the beginning of thermalization, the arm power can span 10 kW in 15 minutes."
This choice of 1 kW is admittedly arbitrary, or at least the metric for choosing the step size is human at the moment. "Perfect is the enemy of good enough."
(2) On the low-end of the arm powers, at the earliest parts of the thermalization, we don't have a lot of data, and the transfer function answer varies wildly.
So, I group all powers below 404 kW into one grouping.
While this "all TFs from the 10 data sets, when the power is below 404 kW" collection spans a large magnitude and phase, a fit to this data set is legit, given
(a) it's an accurate, semi-statistical representation of what's going on, but even if not,
(b) this group will be used quite rarely, given that the start of the 4-hour data sets is always *before* we hit nominal low noise. (The temporary thermalization
lines were loud enough, and the low frequency noise was low enough, that we can start getting information about the detuning before we hit NOMINAL_LOW_NOISE.)
So that knocks the number of power groups to 32.
(3) The final data conditioning is to remove the data set Ref ID 7 (starting on 2023-05-16 04:48 UTC) because the glitch in the IFO near the end of the thermalization
stretch spoils the fidelity of the sensing function systematic error measurement. So, I only use the thermalization from 9 data sets instead of 10. Still leaves
*plenty* of transfer functions at a given power to inform a GPR fit.
OK, so let's show the results of this sorting and data conditioning.
sensingFunction_syserror_vs_power.pdf is an updated version of the same collection of 3D bode plots from the above aLOG (LHO:69796), but with the Ref ID 7 data set removed, and an updated first page which is labeled a bit better.
sensingFunction_syserror_vs_powerslices.pdf is the newly sorted and conditioned data, showing
Page 1: a repeat of the first page to show the power trend vs. time (this really helps be guide the decisions I made regarding power bin step sizes and upper and lower limits of the power bins, so I think it's useful to re-include)
Pages 2 thru 33: the 32 groups of sensing function systematic error transfer functions, grouped by arm power.
As you page through them, you get the gratifying sense that the span of TF magnitude and phase of the data really reflects the range of TF magnitude and phase that had occurred in these nine data sets.
Further, you see, gratifyingly, that as the arm power increases, the sensing function's systematic error transfer function magnitude and phase spread gets smaller, tightening up, and the value of the error sets closer and closer to 1.0 and 0 deg phase, or at a least landing on the thermalized systematic error that we've seen from, e.g. from the sensing_gpr.png from the GPR collection in report 20230510T062635Z which is currently being used in the response function uncertainty budget.
So, we'll use each of these pages of transfer function meta-collections as the input to the GPR fits that will inform the look-up table.
TITLE: 05/22 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY:
Locking:
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 09:36 | ifo | Jenne | Italy | n | locking | 10:31 |
| 15:25 | FAC | Kim | Opt Lab | - | Technical cleaning | 17:37 |
| 16:02 | FAC | Mitch, Randy | Ends | - | Wind fence checks | 19:01 |
| 17:38 | FAC | Kim | MX | - | Technical cleaning | 18:19 |
| 17:38 | FAC | Cindi | MY | - | Technical cleaning | 18:38 |
| 20:09 | FAC | Ken | Carpenter Shop | - | Replacing lights | 22:09 |
| 20:50 | EE | Fil | EY | - | Staging equipment | 21:31 |
| 20:52 | SQZ | Vicky, Naoki | LVEA | - | Clean up SQZ area | 21:03 |
| 20:56 | VAC | Travis, Janos | CER | - | Grabbing particle counter | 21:04 |
Randy and Mitchell
This morning after checking how the EY wind fence held up we were asked to inspect damage at EX. This damage has been known and remained constant for some time. A plan was developed to repair this damage with hardware and tools from the EY fence build.
The second cable from the bottom had broken at the connection point on two different uprights. In one of those cases the cable had unlaced from the hogrings, leaving only about 10 feet still connected to the panel, the rest was hanging to the ground.
The fix was to trim and tape the frayed cable. Then a piece of cable long enough to overlap the two broken ends was secured through the clamp at the upright. Then the new and damaged cables were tentioned and secured with cable clamps. The cable that was laying on the ground was laced back through the existing hogrings and connected in the same way.
All the CP dewar jackets need to be pumped down under 10 micron, in order to improve their insulation capacity, and so to reduce the Nitrogen consumption, which was found 10 % higher compared to last year's same period. We've started with the CP1 dewar jacket, and will advance with the others as well. For this particular instance, we'll leave the pump there overnight, and throughout tomorrow (it can be changed in function of the results). On the 19th (last Friday) we've been pumping it for ~5 hours, and we managed to push down the pressure from 30 to 19 microns.
TJ, Rahul, Austin
Today, we have decided to remove the MONITOR_PAUSE state in the VIOLIN_DAMPING guardian and combine the code into the main damping state - DAMP_VIOLINS_FULL_POWER. The motivation of this was to prevent the waiting in the monitor state and go directly into the damp violins state where the damping settings are set.
I have committed the above changes in the svn.
Naoki, Vicky - we've walked through and cleaned up the area around the squeezer tables SQZT0+7 and racks and removed dangling unused cables. There were some viewport covers under SQZT7, which we've now put back to the racks at the LVEA entrance, in bottom right corner of this photo.