Displaying reports 19921-19940 of 87669.Go to page Start 993 994 995 996 997 998 999 1000 1001 End
Reports until 00:14, Tuesday 23 May 2023
H1 ISC (ISC, TCS)
victoriaa.xu@LIGO.ORG - posted 00:14, Tuesday 23 May 2023 (69811)
Comparing HOMs triggering 80kHz PI locklosses from today vs 2 weeks ago

Today we had two locklosses from the 80 kHz PI on ETMX (69800) in addition to one last Friday (69759), all occuring at Hour 2:30 into the lock. I compared today's 10.4 kHz HOMs with a previous long lock from May 13, when I think we had finished TCS tuning and stayed on at 76W. I didn't see another solid 10+ hour lock since then. Times used for comparison here.

Previously, we have ID'd the pointier HOM peak as ETMY, and the blunter peak as ETMX. Comparing today to 1.5 weeks ago, I think the HOM on ETMX (blunter) is now noticeably lower in frequency -- maybe that is why this 80kHz PI is now ringing up? Previously, it looks like we lowered the EX RH slightly to avoid this 80 kHz PI (eg. 68166, 68165). Maybe we ended up just barely stable after the last EX RH TCS change. While the EX VEA temperature looks pretty stable, there have been some temperature excursions in the last couple week, maybe this is related to the 80 khz PI showing up and the lowered ETMX HOM peaks? And now, possibly with the EX VEA is heating up b/c of summer (even though its temp looks stable), maybe it's worth trying to lower the ETMX RH a bit again, to further avoid this PI.

Images attached to this report
H1 General
ryan.crouch@LIGO.ORG - posted 00:12, Tuesday 23 May 2023 (69812)
OPS Tuesday OWL shift start

TITLE: 05/23 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
    SEI_ENV state: SEISMON_ALERT
    Wind: 8mph Gusts, 5mph 5min avg
    Primary useism: 0.10 μm/s
    Secondary useism: 0.21 μm/s
QUICK SUMMARY:

LHO General
corey.gray@LIGO.ORG - posted 00:11, Tuesday 23 May 2023 (69801)
Monday EVE Operator Summary

TITLE: 05/22 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY:

Winds & Dolphin crash caused grief on this shift.
LOG:

~0256 While locking had a Dolphin Crash.  (See other alogs)  This sadly took up the remainder of the shift.  I have a separate draft with recovery notes  and will post that later tonight. 

H1 CDS
david.barker@LIGO.ORG - posted 20:47, Monday 22 May 2023 - last comment - 20:51, Monday 22 May 2023(69805)
Dolphin problems, several frontends are down

FRS28008

IPC was found to be flashing red/green. There were DACKILLs on h1sus[b123, h2a, h34, h56] h1asc0, h1lsc0. The long range dolphin CS-ends were all red (ends to corner all green). I started to dolphin disable switch ports for omc, asc, lsc, oaf, then all the sus. Finally I disabled h1cdsrfm 3 ports (corner, ex, ey) which finally stopped the flashing

Prior to rebooting any SUS, I set all the SEI SWWDs into bypass mode.

As a first recovery, I rebooted h1susb123. It came back with no problems.

I then rebooted h1cdsrfm, which started with no issues, and at this point still showing no CS->ES traffic because all the CS senders are disabled.

I then rebooted all the remaining CS FE which had been dolphin disabled. All came back OK, all the h1cdsrfm CS->ES went green.

I untripped the SWWD and handed the system over to Corey, Jim and Rahul to complete the recovery.

h1cdsrfm dmesg reports a Dolphin IX adapter error at the time this IPC problem started.


[Tue Apr 11 11:14:37 2023] IXH Adapter 2 : Enable option to create session to all available nodes.
[Mon May 22 19:57:30 2023] IXH Adapter 2 : Uncorrectable error on adapter PCIe SLOT detected - Event_type=0x3 Event_data=0x20
[Mon May 22 19:57:30 2023] IXH Adapter 2 : Port 0 is not operational -- UpTime: 0 sec - Event = 3
[Mon May 22 20:11:51 2023] IXH Adapter 1 : Port 0 is not operational -- UpTime: 0 sec - Event = 0
[Mon May 22 20:11:57 2023] IXH Adapter 0 : Port 0 is not operational -- UpTime: 0 sec - Event = 0
 

Images attached to this report
Comments related to this report
rahul.kumar@LIGO.ORG - 20:42, Monday 22 May 2023 (69807)SUS

All SUS are back online.

david.barker@LIGO.ORG - 20:50, Monday 22 May 2023 (69808)

here is an example of the IPC flashing on h1iopseib2

Images attached to this comment
david.barker@LIGO.ORG - 20:51, Monday 22 May 2023 (69809)

Mon22May2023
LOC TIME HOSTNAME     MODEL/REBOOT
20:13:51 h1susb123    ***REBOOT***
20:16:15 h1susb123    h1iopsusb123
20:16:28 h1susb123    h1susitmy   
20:16:41 h1susb123    h1susbs     
20:16:54 h1susb123    h1susitmx   
20:17:07 h1susb123    h1susitmpi  
20:20:47 h1cdsrfm     ***REBOOT***
20:21:59 h1sush2a     ***REBOOT***
20:23:36 h1sush34     ***REBOOT***
20:24:12 h1sush2a     h1iopsush2a 
20:24:25 h1sush2a     h1susmc1    
20:24:38 h1sush2a     h1susmc3    
20:24:51 h1sush2a     h1susprm    
20:25:04 h1sush2a     h1suspr3    
20:25:11 h1sush56     ***REBOOT***
20:25:46 h1sush34     h1iopsush34 
20:25:59 h1sush34     h1susmc2    
20:26:12 h1sush34     h1suspr2    
20:26:25 h1sush34     h1sussr2    
20:26:43 h1asc0       ***REBOOT***
20:27:15 h1sush56     h1iopsush56 
20:27:28 h1sush56     h1sussrm    
20:27:41 h1sush56     h1sussr3    
20:27:54 h1lsc0       ***REBOOT***
20:27:54 h1sush56     h1susifoout 
20:28:07 h1sush56     h1sussqzout 
20:28:19 h1asc0       h1iopasc0   
20:28:32 h1asc0       h1asc       
20:28:45 h1asc0       h1ascimc    
20:28:58 h1asc0       h1ascsqzifo 
20:29:08 h1oaf0       ***REBOOT***
20:29:29 h1lsc0       h1ioplsc0   
20:29:30 h1omc0       ***REBOOT***
20:29:42 h1lsc0       h1lsc       
20:29:55 h1lsc0       h1lscaux    
20:30:08 h1lsc0       h1sqz       
20:30:21 h1lsc0       h1ascsqzfc  
20:30:49 h1oaf0       h1iopoaf0   
20:30:58 h1omc0       h1iopomc0   
20:31:02 h1oaf0       h1pemcs     
20:31:11 h1omc0       h1omc       
20:31:15 h1oaf0       h1tcscs     
20:31:24 h1omc0       h1omcpi     
20:31:28 h1oaf0       h1susprocpi 
20:31:41 h1oaf0       h1seiproc   
20:31:54 h1oaf0       h1oaf       
20:32:07 h1oaf0       h1calcs     
20:32:20 h1oaf0       h1susproc   
20:32:33 h1oaf0       h1calinj    
20:32:46 h1oaf0       h1bos       
 

H1 CDS (CDS)
corey.gray@LIGO.ORG - posted 20:15, Monday 22 May 2023 - last comment - 20:17, Monday 22 May 2023(69804)
Major CDS Crash

At around 0256utc received Verbal saying HEPI HAM1 WD trip followed a few seonds later with verbals for TCS Chillers.  As I looked on the Call List for TCS and was about to phone TJ I noticed the big Christmas Tree with the CDS Overview & OPS Overview flashing RED & GREEN. 

At any rate, on TeamSpeak with Dave (whom has said "looks like a Dolphin Issue" & "not an easy fix".

Dave is already drafting his own alog, but wanted to get this out now.  Sent a text to TJ w/ status.

Images attached to this report
Comments related to this report
corey.gray@LIGO.ORG - 20:17, Monday 22 May 2023 (69806)CDS, PSL
  • End Stations seem OK at a glance.
  • IMC is gone (but FSS & PMC look ok).
  • I have closed the Verbal Alarm window because I could not hear Dave.
LHO General
corey.gray@LIGO.ORG - posted 19:53, Monday 22 May 2023 (69803)
EVE Mid Shift Status

Had one lock almost 2hrs and then had a lockloss (most likely due to winds).  This lock happened to occur when therte was a lull in winds which were below 20mph, but then in the last 90min, the winds have picked up again (saw one gust up to 40mph).  

Currently ISC LOCK is working on getting back to NLN.

Took OBSERVATORY MODE back to WIND state.

H1 ISC (ISC)
corey.gray@LIGO.ORG - posted 17:39, Monday 22 May 2023 (69802)
ACCEPTED LSC SDF Diffs For NEW LSC FeedForward Filters Change

Attached is the filter change for MICH/SRCL FF filters that Elenna posted about earlier (alog 69785); this change was at 0008utc--about 5min after NLN.  We are now in OBSERVING and thermalizing.

Images attached to this report
H1 OpsInfo (ISC)
thomas.shaffer@LIGO.ORG - posted 16:42, Monday 22 May 2023 (69798)
Increase flashes added to initial alignment

I've added increase flashes to the INIT_ALIGN guardian into the LOCKING_GREEN_ARMS state. For now its running mostly just a copy of the code from ISC_LOCK, but it will need to be changed around a bit tomorrow to be optimized for this node. I did a test while we were down for wind and it seemed to work with the current version of the code that's in there. Increase flashes will come on after 120 seconds with no locking (same as ISC_LOCK for now).

LHO General
corey.gray@LIGO.ORG - posted 16:42, Monday 22 May 2023 (69799)
Mon Eve Ops Transition Status

TITLE: 05/22 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 15mph Gusts, 10mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.14 μm/s

Environmental NOTEs

QUICK SUMMARY:

(My first EVE shifts since Mar 2020!)

H1 was in the middle of an InitialAlignment when I arrived (currently locking at ENGAGE ASC FOR FULL IFO).  Received full rundown from Ryan on how locking has been today (and over the weekend for him).  Items Of Note:

H1 ISC (Lockloss)
naoki.aritomi@LIGO.ORG - posted 16:38, Monday 22 May 2023 (69800)
Lockloss due to 80 kHz PI

Sheila, Vicky, Naoki

Recently we had several lockloss due to PI mode 29 ring up (80kHz PI). The PI guardian tried to damp it, but the PI kept ringing up and caused lockloss in ~5 minutes. We tried both common/diff ETMX actuation (LL:UR=1:1 and LL:UR=1:-1), but both of them failed. 

We modified the PI guardian for mode 29. We increased the damp gain to 50000. To have more time to evaluate the correct damping phase, we increased the phase step to 60 deg, the average time to calculate the new rms to 5s, and the timer for each step to 10s. We will see if this change helps in the next PI ring up.

H1 CAL
jeffrey.kissel@LIGO.ORG - posted 16:27, Monday 22 May 2023 - last comment - 17:54, Wednesday 24 May 2023(69796)
On Which Metric to Group / Trigger Sensing Function Systematic Error During Thermalization
J. Kissel, J. Rollins, E. Goetz (with ideas, support and inspiration from others)

Picking up where I left off on the saga of the IFO's thermalization vs. the DARM sensing function (LHO:69593), the open actions were:
    (a) more, lower, frequency points
    (b) synchronizing the thermalization period to a point in time 
    (c) more data sets
all to better understand if the sensing function during each thermalization period evolved consistently -- since we saw that the first three examples looked very much not consistent.

Namely, we're still craving a single metric of thermalization that we can use to create a look-up table for "when the metric is value Z, apply this XX% / YY deg frequency dependent transfer function amount of extra modeled systematic error to the sensing function's budget of systematic error."

I've not yet been able to execute (a), and probably won't before the start of the run (and thus maybe never). So we'll have to model what we can from these four frequency points. 

This aLOG focuses on (b) and (c). 

Regarding the number of data sets, (c), where LHO:69593 only characterized 3 data sets, here I've gathered 10, all since May 10th 2023. The times and durations are:
    UTC Start           UTC Stop            GPS Start     GPS Stop      Duration    Ref. ID     Notes
    2023-05-20 22:00    2023-05-21 02:00    1368655218    1368669618    4.00        1          (has systematic error lines amp increase)
    2023-05-20 15:40    2023-05-20 19:40    1368632418    1368646818    4.00        2          (has systematic error lines amp increase)
    2023-05-20 07:45    2023-05-20 11:45    1368603918    1368618318    4.00        3          (has systematic error lines amp increase)
    2023-05-19 19:30    2023-05-19 23:30    1368559818    1368574218    4.00        4          (has systematic error lines amp increase)
    2023-05-18 11:00    2023-05-18 15:00    1368442818    1368457218    4.00        5
    2023-05-17 18:48    2023-05-17 22:00    1368384498    1368396018    3.20        6
    2023-05-16 04:48    2023-05-16 08:47    1368247698    1368262038    3.98        7          (has glitch)
    2023-05-15 04:50    2023-05-15 08:50    1368161418    1368175818    4.00        8
    2023-05-14 08:15    2023-05-14 11:45    1368087318    1368099918    3.50        9
    2023-05-10 11:18    2023-05-10 15:18    1367752698    1367767098    4.00        10

Note, here, unlike the few datasets before, we've (sadly) had enough lock losses that I can select for lock stretches that are 4 hours or longer.
This way, if we go with *time* synchronization, we have enough data within each stretch for the data to comfortably be declared "thermalized."
 
Regarding synchronization of data sets, Jamie had the idea of synchronizing the sensing function to IFO Arm Power rather than time, since the varies ways of looking at the data from LHO:69593 indicate that the sensing function evolves on several, mixed, exponential, if not random functions of time (especially with the first 30 minutes of the IFO achieving nominal input power). On the other hand, at least by eye and with what limited study we had, the arm power seemed to be evolving in the same way as the sensing function (and this intuitively made sense given that the optical spring present in the sensing function is some relationship between the carrier power in the arm cavities vs. the power in the SRC).

The pdf attachment, sensingFunction_syserror_vs_power.pdf, shows the comparison between these 10 data sets.
    (i) Page 1: shows all 10 data sets' arm power as a function of time, and then
    (ii) Pages 2-11: show 3D bode plots, where the frequency, magnitude, and phase of the sensing function systematic error transfer function are show against arm power (on the left) and time (on the right).

Note each "arm power" data point over the 4 hour stretch is the median power value over each two minute stride of 16 Hz data, and the data is created by the average of the two arm powers at each 16 Hz data point, from the channels
    H1:ASC-X_PWR_CIRC_OUT16 
    H1:ASC-Y_PWR_CIRC_OUT16.

Here's what I observe from this collection of data sets:
    (1) These additional data sets' sensing function is just as inconsistent in the first 30 minutes after achieving the nominal 76W PSL input power as the previous three from LHO:69593.
    So, that means, if we're going to create a model of this systematic error to add to the collection of time-dependent response function's systematic error like we did in O3, it's going to *have* to have large uncertainty, at least in the first 30 minutes.

    (2) The arm power vs. time evolution is also inconsistent from lock-stretch to lock strength, both in rate of change as well as final thermalized arm power.
        (x) During the discussion of the plots in LHO:69593, during last week's LHO commissioning meeting, the suggestion was that the thermalization may be dependent on how cold the optics are at the start of heating them back up, or -- phrased differently -- if it's been a while since the last well-thermalized, high power lock segment, then the heating up may look different.
        This is consistent with what's seen here -- the orange data set (set 9, starting at 2023-05-14 08:15 UTC) is one of the lower starting powers, and achieves a higher power than the other 9 sets, at a faster rate. Unlike the other 9 data sets, set 9 had 13 hours of "dead time" prior to re-lock-acquisition, vs. most other data sets had "only" 3 hours or less of "dead time," so the supposition is that the optics have not yet full cooled down.
        (y) Also, the thermalized power is different, but as much as 4-5 kW. This should be less of a big deal, because the SRC detuning will have landed on the SRCL offset value that *retunes* the response, which means the only thing *left* changing with time, or between lock stretch to lock stretch in the sensing function is the optical gain and cavity pole, which are constantly being measured and corrected for.
    (3) The arm power will be about as good a metric for thermalization as log(time) -- not perfect good, but better than linear time.

So, we'll go with the arm power as the metric for thermalization, 'cause ...why not? We're winging this anyways, and we don't have time to research other channels or ideas (or even go back to other older ideas).

Now it's on to manipulating this data into arm power (as a proxy for time) slices, and them fitting the *collection* of 10 transfer function values at each arm power, such that we have a representative sensing function systematic error transfer function -- with quantified uncertainty -- at each arm power level. I.e. we'll "stack" these arm-power slices and feed them to the GPR fitting monster and then create a the desired look-up table of fits per arm power. 

Non-image files attached to this report
Comments related to this report
jeffrey.kissel@LIGO.ORG - 17:54, Wednesday 24 May 2023 (69898)
I've finally wrangled python's handling of 3D data structures enough to re-arrange the above data sets in terms of a "stacked" large meta-collection of arm power slices from all 10 thermalization stretches rather than in terms of segregated the 3D bode plots as a function of time from each thermalization stretch.

Remember the goal is to bin these sensing function systematic error transfer functions synchronized with similar arm power values, such that we can create a representative model what the sensing function systematic error is doing for a given arm power at *any* time.
In other words, in the *response* function systematic error budget, which is computed at an arbitrary time, we trigger that application of "this" systematic error model given the arm cavity power of "that" value at that time.

I've conditioned the data I put into these meta-collections in the following way:
(1) How fine a grain do we need in terms of arm power? 1 kW steps? 2 kW steps?

    Given that the data collections contain arm powers that range from ~396 kW to ~435 kW, 1 kW steps means 40 different models. 
    1 kW steps like a good balance between 
        - "a huge number of options for the look-up table" and 
        - "don't bin too many arm power values together, because in the beginning of thermalization, the arm power can span 10 kW in 15 minutes."
    This choice of 1 kW is admittedly arbitrary, or at least the metric for choosing the step size is human at the moment. "Perfect is the enemy of good enough."

(2) On the low-end of the arm powers, at the earliest parts of the thermalization, we don't have a lot of data, and the transfer function answer varies wildly. 
    So, I group all powers below 404 kW into one grouping.
    While this "all TFs from the 10 data sets, when the power is below 404 kW" collection spans a large magnitude and phase, a fit to this data set is legit, given
    (a) it's an accurate, semi-statistical representation of what's going on, but even if not,
    (b) this group will be used quite rarely, given that the start of the 4-hour data sets is always *before* we hit nominal low noise. (The temporary thermalization 
        lines were loud enough, and the low frequency noise was low enough, that we can start getting information about the detuning before we hit NOMINAL_LOW_NOISE.)

    So that knocks the number of power groups to 32.

(3) The final data conditioning is to remove the data set Ref ID 7 (starting on 2023-05-16 04:48 UTC) because the glitch in the IFO near the end of the thermalization 
    stretch spoils the fidelity of the sensing function systematic error measurement. So, I only use the thermalization from 9 data sets instead of 10. Still leaves 
    *plenty* of transfer functions at a given power to inform a GPR fit. 

OK, so let's show the results of this sorting and data conditioning.

sensingFunction_syserror_vs_power.pdf is an updated version of the same collection of 3D bode plots from the above aLOG (LHO:69796), but with the Ref ID 7 data set removed, and an updated first page which is labeled a bit better.

sensingFunction_syserror_vs_powerslices.pdf is the newly sorted and conditioned data, showing 
    Page 1: a repeat of the first page to show the power trend vs. time (this really helps be guide the decisions I made regarding power bin step sizes and upper and lower limits of the power bins, so I think it's useful to re-include)
    Pages 2 thru 33: the 32 groups of sensing function systematic error transfer functions, grouped by arm power.

As you page through them, you get the gratifying sense that the span of TF magnitude and phase of the data really reflects the range of TF magnitude and phase that had occurred in these nine data sets.
Further, you see, gratifyingly, that as the arm power increases, the sensing function's systematic error transfer function magnitude and phase spread gets smaller, tightening up, and the value of the error sets closer and closer to 1.0 and 0 deg phase, or at a least landing on the thermalized systematic error that we've seen from, e.g. from the sensing_gpr.png from the GPR collection in report 20230510T062635Z which is currently being used in the response function uncertainty budget.

So, we'll use each of these pages of transfer function meta-collections as the input to the GPR fits that will inform the look-up table.




Non-image files attached to this comment
LHO General
ryan.short@LIGO.ORG - posted 16:01, Monday 22 May 2023 (69787)
Ops Day Shift Summary

TITLE: 05/22 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY:

Locking:


LOG:                                                                                                                                      

Start Time System Name Location Lazer_Haz Task Time End
09:36 ifo Jenne Italy n locking 10:31
15:25 FAC Kim Opt Lab - Technical cleaning 17:37
16:02 FAC Mitch, Randy Ends - Wind fence checks 19:01
17:38 FAC Kim MX - Technical cleaning 18:19
17:38 FAC Cindi MY - Technical cleaning 18:38
20:09 FAC Ken Carpenter Shop - Replacing lights 22:09
20:50 EE Fil EY - Staging equipment 21:31
20:52 SQZ Vicky, Naoki LVEA - Clean up SQZ area 21:03
20:56 VAC Travis, Janos CER - Grabbing particle counter 21:04
H1 General
mitchell.robinson@LIGO.ORG - posted 15:33, Monday 22 May 2023 (69797)
End X Wind fence repairs

Randy and Mitchell

 

This morning after checking how the EY wind fence held up we were asked to inspect damage at EX. This damage has been known and remained constant for some time. A plan was developed to repair this damage with hardware and tools from the EY fence build.

The second cable from the bottom had broken at the connection point on two different uprights. In one of those cases the cable had unlaced from the hogrings, leaving only about 10 feet still connected to the panel, the rest was hanging to the ground.

The fix was to trim and tape the frayed cable. Then a piece of cable long enough to overlap the two broken ends was secured through the clamp at the upright. Then the new and damaged cables were tentioned and secured with cable clamps. The cable that was laying on the ground was laced back through the existing hogrings and connected in the same way.

 

Images attached to this report
LHO EPO (VE)
janos.csizmazia@LIGO.ORG - posted 15:16, Monday 22 May 2023 (69795)
CP1 dewar jacket pumpdown
All the CP dewar jackets need to be pumped down under 10 micron, in order to improve their insulation capacity, and so to reduce the Nitrogen consumption, which was found 10 % higher compared to last year's same period.
We've started with the CP1 dewar jacket, and will advance with the others as well. For this particular instance, we'll leave the pump there overnight, and throughout tomorrow (it can be changed in function of the results). On the 19th (last Friday) we've been pumping it for ~5 hours, and we managed to push down the pressure from 30 to 19 microns.
Images attached to this report
H1 SUS (GRD)
austin.jennings@LIGO.ORG - posted 14:31, Monday 22 May 2023 - last comment - 12:57, Tuesday 23 May 2023(69793)
Monitor Pause Removed from VIOLIN_DAMPING grd

TJ, Rahul, Austin

Today, we have decided to remove the MONITOR_PAUSE state in the VIOLIN_DAMPING guardian and combine the code into the main damping state - DAMP_VIOLINS_FULL_POWER. The motivation of this was to prevent the waiting in the monitor state and go directly into the damp violins state where the damping settings are set.

Comments related to this report
rahul.kumar@LIGO.ORG - 12:57, Tuesday 23 May 2023 (69842)SUS

I have committed the above changes in the svn.

H1 SQZ
victoriaa.xu@LIGO.ORG - posted 14:10, Monday 22 May 2023 (69792)
Cleaned up SQZ racks/tables for O4

Naoki, Vicky - we've walked through and cleaned up the area around the squeezer tables SQZT0+7 and racks and removed dangling unused cables. There were some viewport covers under SQZT7, which we've now put back to the racks at the LVEA entrance, in bottom right corner of this photo.

Images attached to this report
Displaying reports 19921-19940 of 87669.Go to page Start 993 994 995 996 997 998 999 1000 1001 End