TITLE: 05/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 11mph Gusts, 8mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY:
TJ was at early steps of locking (noticed some odd offset sweeps for COMM/DIFF while FINDING IR). PRMI does not look very good sadly and he's moving to CHECK MICH FRINGES.
Since the table work this morning we have been struggling to get past DRMI or engage DRMI ASC. Catching PRMI can even prove to be challenging and I've been skipping over PRMI ASC for now. We will next try to get the ASC to work and zero the error signals if we can get back to a stable PRMI.
The problem was that the HAM-ISIs were all drifting off in Yaw (RZ) slowly, but surely, for weeks -- see LHO:69934.
I have optimized generate_cds_overview.py to better handle the special cases where the nominal settings differ from the rest of the models. From recent settings used when H1 was in observe mode:
| model | warn if num TP exceeds | EXC allowed |
| h1calinj | 1 | YES |
| h1lsc | 2 | NO |
| h1iopomc0 | 1 | NO |
| all other models | 0 | NO |
Sheila, Ryan S, TJ
This morning we continued the ISCT1 and PR3 alignment from 69903 , 69897 , 69857.
TJ reverted the X arm green references to what they were yesterday morning, seen snapshot here, and re-ran intial alignment of the X arm. Ryan S and I went to ISCT1 and saw that the beatnotes weren't well aligned, but the whole table alignment was better than what TJ and I found yesterday. We toggled PR3 between 151.6 and 148.5 yaw slider, and saw that the ISCT1 alignment was sort of close to OK for both of these, such that beams were clearing their appertures and making it to the diodes they should have without clipping.
This means that after our table realignment yesterday, the table was in a sort of OK alignment (beams not totally missing their optics) for 4 configurations: both of the green arm alignment set points and both the PR3 pointings. This means that the table alignment that TJ and I did yesterday fixed some gross misalignment that wasn't due to either of those things, (it has been a long time previously since the alignment on table was fixed up).
From the control room TJ, Ryan and I, with PR3 set to 148.5 yaw slider (setting for locks yesterday), adjusted PR2 to lock the X arm in IR, ran the WFS for INPUT align, and did a MICH bright alignment. Ryan and I returned to the table and moved the bottom periscope mirror to improve the alignment of the Y beam onto the trans diode, which allowed TJ to lock the Y arm in green and run inital alignment for that. In the meantime, Ryan and I adjusted the position of the X trans diode, and improved the COMM beatnote using the beatnote BS and the one upstream of it in the X arm path. We eneded up with a COMM beatnote of 6dBm, which drifted down later when TJ re-ran inital alignment for the X arm and is now at -1dBm.
After TJ finished the Y arm green inital alignment, and a MICH bright alignment, Ryan and I realinged the DIFF beatnote, which is now -7dBm. TJ then completed the intial alignment steps we had skipped (PRC and SRC) and is now trying to lock DRMI.
After struggling with poor PRMI buildups, we realized that there was a typo this morning in the camera offsets, so TJ redid the initial alignment with the offsets from yesterday morning. Then Ryan and I went to ISCT1 and realigned the beatnotes, which were already close but not good enough for locking. Now TJ has finished the initial alignment and we are going to attempt locking again.
Both TJ and Corey had difficulty doing INPUT ALIGN, with Corey just now we found that it would lock once I reset the POP 45 I dark offset to -72, which was consistent with last night. I've hard coded that into the guardian, so that it's for now not averaging the dark offset each time the whitening changes but just being set to -72. With this INPUT_ALIGN was able to run and get the X arm transmission to 1. Corey is doing the rest of IR alignment.
In the end this PR3 and ISCT1 alignment was not needed. The problem was that the HAM-ISIs were all drifting off in Yaw (RZ) slowly, but surely, for weeks -- see LHO:69934.
Thu May 25 10:09:35 2023 INFO: Fill completed in 4min 35secs
Fill did not run at 10:00:00 and was rescheduled for 10:05:00 just for today. Gerardo confirmed a good fill curbside.
FAMIS 19977
The trend for PMC Refl has an interesting shape and looks a bit noisier then normal to me.
TITLE: 05/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 11mph Gusts, 8mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY: The IFO is still down since the PR3 and ISCT1 work yesterday. It's sounding like these changes will need to be reverted this morning.
In the end the PR3 and ISCT1 table alignment was a temporary fix for the true problem. The true problem was that the HAM-ISIs were all drifting off in Yaw (RZ) slowly, but surely, for weeks -- see LHO:69934.
TITLE: 05/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY:
Majority of the shift was focused on attempting to get H1 to lock post ISCT1 table work (worked with Sheila a little before she left for home and then with her over teamspeak). Several Full & Manual alignments were attempted with various changes made to PR2 (and a little with PR3). Unfortunately, all the alignments led to poor locking of anything past Green Arms.
The problem was that the HAM-ISIs were all drifting off in Yaw (RZ) slowly, but surely, for weeks -- see LHO:69934.
We are still having difficulties with SRY locking, but I haven't looked into what is going on. If SRY is not locking in an inital alignment step, it is OK to run SR2 ALIGN offload, and move SRM by hand to increase the SRY flashes.
Editing to add:
Input align wasn't working well because of an apparent dark offset in POP A 45I. I adjusted the dark offset from -68.9 to -72 to get the signal without flashes to be near zero. This must be reset by one of the init align guardian states. This problem has been seen before and TJ attempted to fix it here, but something is still not working.
2nd edit: We weren't able to get above 30 counts on POP18 + POP90 in PRMI with PR3 at 151.6 yaw slider, so we moved it to 150.2 with the renormalizations above and re did initial alignment of the IR cavities. Input align wouldn't lock but the flashes were at 1, so we moved on. SRY has worked twice in a row. With this lower beatnote we've had an ALS lockloss, I think that if a few more tries at this alignment aren't fruitful it would make sense to cancel the owl and go to the table in the morning.
In the end the PR3 and ISCT1 table alignment was a temporary fix for the true problem. The true problem was that the HAM-ISIs were all drifting off in Yaw (RZ) slowly, but surely, for weeks -- see LHO:69934.
After the table work, ran an Initial Alignment (it was a bit rough in that INPUT_ALIGN had trouble locking the X-arm. Restrored PR2 alignment to where it was during the last lock, but it was still tough. However, INPUT ALIGN finally did complete and it looked as though the entire alignment completed.
Unfortunately, DRMI did not look good and PRMI also looked misaligned (there were short 1-2 second locks, but they are very low and so far I have not been fast enough to touch up the PRMI alignment to let ASC take over).
0314utc (814pmPT) SEI_ENV in EarthQuake state due to 6.6 from Panama/Colombia. (Picket Fence has been orange and red.)
Will start another Initial Alignment as Sheila looks at trends.
In the end the PR3 and ISCT1 table alignment was a temporary fix to the true problem. The true problem was that the HAM-ISIs were all drifting off in Yaw (RZ) slowly, but surely, for weeks -- see LHO:69934.
Adrian, Betsy, Jim
We noticed correlation between the GW channel and corner station accelerometers earlier today (from about 20:18 to 20:22 UTC, May 24). It was noted that the correlation was coincident with some forklift activities onsite (forklift_oplev_glitching, forklift_oplev_strain) , but didn't know the exact time. To see if forklift activity could be responsible for the noise transients in the accelerometers that were witnessed by the GW channel, Jim and I conducted some tests while we were unlocked in the spirit of last week's site activity injections to try to replicate the phenomenon. We found that stopping the forklift, idling at the point marked with a star in the attached map and then driving away caused a similar time- and frequency-response in the CS accelerometers to the one observed earlier today (forklift_sai_slide).
TJ, Sheila, Betsy, Jim
Since we have had increasing drift in the PR3 yaw alignment needed to get ALS beatnotes good enough to lock, we realigned the beat notes on ISCT1 this afternoon. First we set the green x arm references as we found them yesterday while locked with 2W and the spots centered on the optics (69857). Screenshot attached of accepting these new references in SDF.
TJ then ran an intial alignment through the MICH alignment step, we saw that COMM beatnote was -18dBm, TJ adjusted PR3 yaw to 147.2urad to increase the COMM beatnote to -10dBm; 2 weeks ago PR3 yaw was at 151.6 urad so this is an pparnt drift of 4.4 urad. The osems and optical lever do not indicate that PR3 has had any drift like this.
TJ and I went to ISCT1, looked at the beampaths which were mostly reasonably centered with PR3 at 147.2urad, then asked Jim to move it to 151.6 to restore PR3 to it's alignment of 2 weeks ago. Then the Y arm beam was hitting the X arm side of the prism and being directed into the X arm path. We adjusted mostly yaw and a little pitch on the bottom periscope mirror, (layout here), as well as the yaw of the 1 inch BS that splits the X beam between the trans PD path and the beatnote paths, to get both X and Y beams roughly aligned through the paths to the comm and diff beamsplitters.
From there we manually aligned the COMM beatnote, with an IR card then the scope, and got a beatnote strength of 7-8 dBm, for DIFF we have -8.2dBm. The arm transmission diodes were lower than previously, so Betsy and I went back to the table and checked that the beams were centered on the trans PDs, which didn't increase the power. Corey and I then re-normalized the trans diodes, by adding a gain of 1.16 to the Y channel and 1.35 to the X channel. The changes in screenshots on this alog will need to be accepted in OBSERVE.
In the end the PR3 and ISCT1 table alignment was a temporary fix for the true problem. The true problem was that the HAM-ISIs were all drifting off in Yaw (RZ) slowly, but surely, for weeks -- see LHO:69934.
TITLE: 05/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY: We had a late start to the run but managed to get a 5 hour lock in. We lost lock for reasons unknown so far. Sheila and I went into ISCT1 to fix the beatnotes that we've been struggling with for the last ~9days after reverting PR3 to its old values. We are now about to start an initial alignment with the new table alignment.
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 14:23 | FAC | Betsy | FCES | N | Closeout checks | 14:38 |
| 14:56 | EE | Ken | Carpender shop | N | 17:25 | |
| 15:55 | FAC | Bubba | CS | n | HVAC fan shutdown/startup/adjust | 20:05 |
| 16:20 | PCAL | Dripta | PCAL Lab | local | PCAL meas. | 16:34 |
| 18:52 | PCAL | Dripta | PCAL Lab | local | PCAL meas. | 20:32 |
| 21:16 | VAC | Janos, Gerardo, Jordan | LVEA | n | Take turbo pictures | 21:30 |
| 21:49 | ALS | Sheila, TJ | LVEA - ISCT1 | local | Adjust beatnotes | 23:49 |
TITLE: 05/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 9mph Gusts, 5mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.09 μm/s
QUICK SUMMARY:
Per Robert Schofield, Fans 1 & 2 supplying the LVEA were readjusted to lesser airflows for the Observation Run. The total CFM for AHU 1 was reduced from ~ 26,000 to ~ 11,200 CFM. AHU 2 will require some mechanical adjustment to the linkage in order to achieve the much lower CFM. This will be done at the next opportunity when we are no longer locked or on maintenance tuesday.
Seismon earthquake predictions were gradually slowing down over the last year to the point where no earthquake alerts were issued before the earthquakes reached the site.
The slowdown was related to the size of the earthquake database. I modified the seismon code to make the speed independent of database size, speeding up the prediction calculations.
The attached graph shows the gradual decrease in initial prediction times for earthquakes, and the increase back to normal levels in the last couple of weeks.
The seismon prediction pipeline is now typically 20-25 seconds from when an earthquake event is received from USGS to when a predcition is posted to epics.,
A crew of people moved a ~200lbs empty dewar from the staging building to CP1 by fork lifting to the barricade sign then rolling the dewar on its wheels to CP1. After the fill, they pushed the filled dewar to the forklift and then drove it back to the staging building. This could be seen on the HAM2 STS 10-30Hz blrms channels seen in the attachment.
Note this activity was cleared by Keita.
J. Kissel, J. Rollins, E. Goetz (with ideas, support and inspiration from others) Picking up where I left off on the saga of the IFO's thermalization vs. the DARM sensing function (LHO:69593), the open actions were: (a) more, lower, frequency points (b) synchronizing the thermalization period to a point in time (c) more data sets all to better understand if the sensing function during each thermalization period evolved consistently -- since we saw that the first three examples looked very much not consistent. Namely, we're still craving a single metric of thermalization that we can use to create a look-up table for "when the metric is value Z, apply this XX% / YY deg frequency dependent transfer function amount of extra modeled systematic error to the sensing function's budget of systematic error." I've not yet been able to execute (a), and probably won't before the start of the run (and thus maybe never). So we'll have to model what we can from these four frequency points. This aLOG focuses on (b) and (c). Regarding the number of data sets, (c), where LHO:69593 only characterized 3 data sets, here I've gathered 10, all since May 10th 2023. The times and durations are: UTC Start UTC Stop GPS Start GPS Stop Duration Ref. ID Notes 2023-05-20 22:00 2023-05-21 02:00 1368655218 1368669618 4.00 1 (has systematic error lines amp increase) 2023-05-20 15:40 2023-05-20 19:40 1368632418 1368646818 4.00 2 (has systematic error lines amp increase) 2023-05-20 07:45 2023-05-20 11:45 1368603918 1368618318 4.00 3 (has systematic error lines amp increase) 2023-05-19 19:30 2023-05-19 23:30 1368559818 1368574218 4.00 4 (has systematic error lines amp increase) 2023-05-18 11:00 2023-05-18 15:00 1368442818 1368457218 4.00 5 2023-05-17 18:48 2023-05-17 22:00 1368384498 1368396018 3.20 6 2023-05-16 04:48 2023-05-16 08:47 1368247698 1368262038 3.98 7 (has glitch) 2023-05-15 04:50 2023-05-15 08:50 1368161418 1368175818 4.00 8 2023-05-14 08:15 2023-05-14 11:45 1368087318 1368099918 3.50 9 2023-05-10 11:18 2023-05-10 15:18 1367752698 1367767098 4.00 10 Note, here, unlike the few datasets before, we've (sadly) had enough lock losses that I can select for lock stretches that are 4 hours or longer. This way, if we go with *time* synchronization, we have enough data within each stretch for the data to comfortably be declared "thermalized." Regarding synchronization of data sets, Jamie had the idea of synchronizing the sensing function to IFO Arm Power rather than time, since the varies ways of looking at the data from LHO:69593 indicate that the sensing function evolves on several, mixed, exponential, if not random functions of time (especially with the first 30 minutes of the IFO achieving nominal input power). On the other hand, at least by eye and with what limited study we had, the arm power seemed to be evolving in the same way as the sensing function (and this intuitively made sense given that the optical spring present in the sensing function is some relationship between the carrier power in the arm cavities vs. the power in the SRC). The pdf attachment, sensingFunction_syserror_vs_power.pdf, shows the comparison between these 10 data sets. (i) Page 1: shows all 10 data sets' arm power as a function of time, and then (ii) Pages 2-11: show 3D bode plots, where the frequency, magnitude, and phase of the sensing function systematic error transfer function are show against arm power (on the left) and time (on the right). Note each "arm power" data point over the 4 hour stretch is the median power value over each two minute stride of 16 Hz data, and the data is created by the average of the two arm powers at each 16 Hz data point, from the channels H1:ASC-X_PWR_CIRC_OUT16 H1:ASC-Y_PWR_CIRC_OUT16. Here's what I observe from this collection of data sets: (1) These additional data sets' sensing function is just as inconsistent in the first 30 minutes after achieving the nominal 76W PSL input power as the previous three from LHO:69593. So, that means, if we're going to create a model of this systematic error to add to the collection of time-dependent response function's systematic error like we did in O3, it's going to *have* to have large uncertainty, at least in the first 30 minutes. (2) The arm power vs. time evolution is also inconsistent from lock-stretch to lock strength, both in rate of change as well as final thermalized arm power. (x) During the discussion of the plots in LHO:69593, during last week's LHO commissioning meeting, the suggestion was that the thermalization may be dependent on how cold the optics are at the start of heating them back up, or -- phrased differently -- if it's been a while since the last well-thermalized, high power lock segment, then the heating up may look different. This is consistent with what's seen here -- the orange data set (set 9, starting at 2023-05-14 08:15 UTC) is one of the lower starting powers, and achieves a higher power than the other 9 sets, at a faster rate. Unlike the other 9 data sets, set 9 had 13 hours of "dead time" prior to re-lock-acquisition, vs. most other data sets had "only" 3 hours or less of "dead time," so the supposition is that the optics have not yet full cooled down. (y) Also, the thermalized power is different, but as much as 4-5 kW. This should be less of a big deal, because the SRC detuning will have landed on the SRCL offset value that *retunes* the response, which means the only thing *left* changing with time, or between lock stretch to lock stretch in the sensing function is the optical gain and cavity pole, which are constantly being measured and corrected for. (3) The arm power will be about as good a metric for thermalization as log(time) -- not perfect good, but better than linear time. So, we'll go with the arm power as the metric for thermalization, 'cause ...why not? We're winging this anyways, and we don't have time to research other channels or ideas (or even go back to other older ideas). Now it's on to manipulating this data into arm power (as a proxy for time) slices, and them fitting the *collection* of 10 transfer function values at each arm power, such that we have a representative sensing function systematic error transfer function -- with quantified uncertainty -- at each arm power level. I.e. we'll "stack" these arm-power slices and feed them to the GPR fitting monster and then create a the desired look-up table of fits per arm power.
I've finally wrangled python's handling of 3D data structures enough to re-arrange the above data sets in terms of a "stacked" large meta-collection of arm power slices from all 10 thermalization stretches rather than in terms of segregated the 3D bode plots as a function of time from each thermalization stretch.
Remember the goal is to bin these sensing function systematic error transfer functions synchronized with similar arm power values, such that we can create a representative model what the sensing function systematic error is doing for a given arm power at *any* time.
In other words, in the *response* function systematic error budget, which is computed at an arbitrary time, we trigger that application of "this" systematic error model given the arm cavity power of "that" value at that time.
I've conditioned the data I put into these meta-collections in the following way:
(1) How fine a grain do we need in terms of arm power? 1 kW steps? 2 kW steps?
Given that the data collections contain arm powers that range from ~396 kW to ~435 kW, 1 kW steps means 40 different models.
1 kW steps like a good balance between
- "a huge number of options for the look-up table" and
- "don't bin too many arm power values together, because in the beginning of thermalization, the arm power can span 10 kW in 15 minutes."
This choice of 1 kW is admittedly arbitrary, or at least the metric for choosing the step size is human at the moment. "Perfect is the enemy of good enough."
(2) On the low-end of the arm powers, at the earliest parts of the thermalization, we don't have a lot of data, and the transfer function answer varies wildly.
So, I group all powers below 404 kW into one grouping.
While this "all TFs from the 10 data sets, when the power is below 404 kW" collection spans a large magnitude and phase, a fit to this data set is legit, given
(a) it's an accurate, semi-statistical representation of what's going on, but even if not,
(b) this group will be used quite rarely, given that the start of the 4-hour data sets is always *before* we hit nominal low noise. (The temporary thermalization
lines were loud enough, and the low frequency noise was low enough, that we can start getting information about the detuning before we hit NOMINAL_LOW_NOISE.)
So that knocks the number of power groups to 32.
(3) The final data conditioning is to remove the data set Ref ID 7 (starting on 2023-05-16 04:48 UTC) because the glitch in the IFO near the end of the thermalization
stretch spoils the fidelity of the sensing function systematic error measurement. So, I only use the thermalization from 9 data sets instead of 10. Still leaves
*plenty* of transfer functions at a given power to inform a GPR fit.
OK, so let's show the results of this sorting and data conditioning.
sensingFunction_syserror_vs_power.pdf is an updated version of the same collection of 3D bode plots from the above aLOG (LHO:69796), but with the Ref ID 7 data set removed, and an updated first page which is labeled a bit better.
sensingFunction_syserror_vs_powerslices.pdf is the newly sorted and conditioned data, showing
Page 1: a repeat of the first page to show the power trend vs. time (this really helps be guide the decisions I made regarding power bin step sizes and upper and lower limits of the power bins, so I think it's useful to re-include)
Pages 2 thru 33: the 32 groups of sensing function systematic error transfer functions, grouped by arm power.
As you page through them, you get the gratifying sense that the span of TF magnitude and phase of the data really reflects the range of TF magnitude and phase that had occurred in these nine data sets.
Further, you see, gratifyingly, that as the arm power increases, the sensing function's systematic error transfer function magnitude and phase spread gets smaller, tightening up, and the value of the error sets closer and closer to 1.0 and 0 deg phase, or at a least landing on the thermalized systematic error that we've seen from, e.g. from the sensing_gpr.png from the GPR collection in report 20230510T062635Z which is currently being used in the response function uncertainty budget.
So, we'll use each of these pages of transfer function meta-collections as the input to the GPR fits that will inform the look-up table.