Displaying reports 14561-14580 of 86387.Go to page Start 725 726 727 728 729 730 731 732 733 End
Reports until 16:00, Saturday 04 November 2023
LHO General
austin.jennings@LIGO.ORG - posted 16:00, Saturday 04 November 2023 (73982)
Saturday Shift Summary

TITLE: 11/04 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

- EX saturation @ 17:40

- Robert is currently running PEM measurements until LLO comes back up

Quiet shift otherwise, nothing else to note.
LOG:

No log for this shift.

LHO General
austin.jennings@LIGO.ORG - posted 13:00, Saturday 04 November 2023 (73987)
Mid Shift Report

Quiet day so far, H1 has been locked for 17:19 hours and all systems appear to be stable. Range does appear to be on the lower end of the spectrum, hovering around 150 Mpc.

H1 CDS
david.barker@LIGO.ORG - posted 11:24, Saturday 04 November 2023 (73985)
New NDSCOPE trends web page

I've created a new ndscope trends web service which, for each ndscope yaml defined, generates 1 day, 1 week, 1 month and 3 months trend plots.

The URL is https://lhocds.ligo-wa.caltech.edu/ndshooter.html

Previously the yaml file's title line defined the lookback time, but this lead to many duplicate yamls. The new python code defines the lookbacks in a dictionary, which defaults to 1, 7, 30 and 90 days.

Please contact me if you would like to see more trends added.

Currently the update time is every 15 minutes, which includes a 4 minute acquisition time.

Images attached to this report
LHO VE
david.barker@LIGO.ORG - posted 11:17, Saturday 04 November 2023 (73986)
Sat CP1 Fill

Sat Nov 04 10:09:53 2023 INFO: Fill completed in 9min 50secs

Images attached to this report
H1 CDS
david.barker@LIGO.ORG - posted 09:59, Saturday 04 November 2023 (73984)
New timing master fanout looking good

There have been no gps errors since the timing master was replaced 17 hours ago.

Images attached to this report
LHO General
austin.jennings@LIGO.ORG - posted 08:00, Saturday 04 November 2023 (73981)
Ops Day Shift Start

TITLE: 11/04 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 3mph Gusts, 1mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.18 μm/s 
QUICK SUMMARY:

- Wet and foggy morning, but H1 has been locked for 12.5 hours

- CDS/DMs/SEI ok

LHO General (TCS)
ryan.short@LIGO.ORG - posted 00:03, Saturday 04 November 2023 (73969)
Ops Eve Shift Summary

TITLE: 11/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:

I've engaged damping settings for IY05/06 of FM6, FM8, FM10 and gain of -0.025, which appear to be working well so far.

LOG:

Start Time System Name Location Lazer_Haz Task Time End
22:52 PCAL Tony/Rick PCAL Lab Y (LOCAL) Measurement 23:55
H1 CDS
david.barker@LIGO.ORG - posted 20:48, Friday 03 November 2023 (73980)
h1tcshwssdf was reporting diffs because h1hwsey is not running, fixed by temporarily zeroing OBSERVE.snap settings

Ryan was unable to go into observe because of 14 TCS HWS ETMY SDF diffs. These diffs appeared this evening because during the model restarts when the new timing master was installed this afternoon, the slow controls SDF systems were restarted on h1ecatmon0. Prior to this h1tcshwssdf was keeping the last values from h1hwsey after it died on Tuesday, which kept the diff count to zero. After h1tcshwssdf was restarted, these values went to zero and the 14 diffs appeared.

For a temporary solution, I edited /opt/rtcds/userapps/release/tcs/h1/burtfiles/h1tcshwssdf_OBSERVE.snap and set all the ETMY setpoints to zero (there are 15 of them, but one was already zero, hence the 14 diffs).

Editing the file changed its ownership to me, I changed it back to controls:advligorts but the file permissions red border persists. I'll work on that later, but for now the diff count is zero and H1 is not blocked to go into observe.

-H1:TCS-ETMY_HWS_BEAM_POS_X 1 5.11500000000000000000e+02 1
-H1:TCS-ETMY_HWS_BEAM_POS_Y 1 5.11500000000000000000e+02 1
-H1:TCS-ETMY_HWS_CENTROID_WEIGHT_FACTOR 1 2.00000000000000000000e+00 1
-H1:TCS-ETMY_HWS_CO2_POS_X 1 5.11500000000000000000e+02 1
-H1:TCS-ETMY_HWS_CO2_POS_Y 1 5.11500000000000000000e+02 1
+H1:TCS-ETMY_HWS_BEAM_POS_X 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_BEAM_POS_Y 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_CENTROID_WEIGHT_FACTOR 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_CO2_POS_X 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_CO2_POS_Y 1 0.00000000000000000000e+00 1
 H1:TCS-ETMY_HWS_DEFOCUS_TEMPERATURE_COEFFICIENT 1 0.00000000000000000000e+00 1
-H1:TCS-ETMY_HWS_LEVER_ARM 1 1.00000000000000002082e-02 1
-H1:TCS-ETMY_HWS_MAGNIFICATION 1 2.05000000000000000000e+01 1
-H1:TCS-ETMY_HWS_NO_LIVE_IMAGES_TO_AVERAGE 1 5.00000000000000000000e+00 1
-H1:TCS-ETMY_HWS_NO_REF_IMAGES_TO_AVERAGE 1 1.00000000000000000000e+02 1
-H1:TCS-ETMY_HWS_ORIGIN_X 1 5.11500000000000000000e+02 1
-H1:TCS-ETMY_HWS_ORIGIN_Y 1 5.11500000000000000000e+02 1
-H1:TCS-ETMY_HWS_RH_POS_X 1 5.11500000000000000000e+02 1
-H1:TCS-ETMY_HWS_RH_POS_Y 1 5.11500000000000000000e+02 1
-H1:TCS-ETMY_HWS_SPOT_RADIUS 1 1.00000000000000000000e+01 1
+H1:TCS-ETMY_HWS_LEVER_ARM 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_MAGNIFICATION 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_NO_LIVE_IMAGES_TO_AVERAGE 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_NO_REF_IMAGES_TO_AVERAGE 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_ORIGIN_X 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_ORIGIN_Y 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_RH_POS_X 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_RH_POS_Y 1 0.00000000000000000000e+00 1
+H1:TCS-ETMY_HWS_SPOT_RADIUS 1 0.00000000000000000000e+00 1
 

Images attached to this report
LHO General (CAL, ISC, SEI)
ryan.short@LIGO.ORG - posted 20:46, Friday 03 November 2023 (73978)
Ops Eve Mid Shift Report

State of H1: Observing at 151Mpc

Following an emergency swap of the MSR timing master (see alog 73970), H1 has been recovered, relocked, and just started observing as of 03:37 UTC.

There were several SDF diffs when H1 reached NLN likely caused by models being restarted this evening as part of the timing master swap. The models with SDF diffs were h1hpiham2, h1isietmy, h1isiham7, h1oaf, and h1caley. All of these have been REVERTED as the channels with diffs had been at their previous values for some days prior to this evening's model restarts.

Images attached to this report
H1 ISC (CSWG, PSL, SEI)
jeffrey.kissel@LIGO.ORG - posted 19:16, Friday 03 November 2023 - last comment - 14:23, Tuesday 07 November 2023(73976)
PSL Frequency Noise as measured by IMC down to 0.01 Hz; Two IFO Configs and Compared with ISI GS13 Projection of CARM
J. Kissel (encouraged by S. Dywer, D. Sigg)

Continuing along with the plan of building up a complete picture of the PSL frequency noise as a part of understanding the future *effective* frequency noise delivered via fiber to a seismic platform IFO for longitudinal control (aka SPI L, started elsewhere using the PMC LHO:73905), today I switch tactics and look at the main IFO's frequency stabilization servo. 

Today's primary metric is "IMC_F," i.e. the demodulated output of the PD in reflection of the input of the input mode cleaner, a.k.a the IMC REFL PD, which is a part of the global frequency stabilization servo (FSS) for the detector's pre-stabilized laser (PSL) frequency. For more details on how the global frequency stabilization works, check out the most recent published info on it from Craig Cahillane's paper P2100219 and Chapter 3 section 4 of his thesis P1800022. In short, it's a cascaded set of loops that use 
   - the in-vac, rigid-body ~1 m, reference cavity within the PSL, 
   - the suspended in-vac ~16 m input mode cleaner, then 
   - the common motion of the suspended in-vac 4 km arm cavities themselves 
as frequency reference, with ~300 kHz, ~60 kHz, and ~20 kHz UGF, respectively.

This IMC REFL PD's output is converted into a control signal to be pushed to the PSL laser frequency with the IMC "common mode board" analog servo. The output of that analog control filter is digitized and converted into Hz. Because the FSS + IMC + CARM bandwidths are all above 20 kHz and I'm looking at the PD's output below 1 kHz, we're very much in the "infinite gain" region of the control signal. Thus -- assuming we know the frequency actuator calibration (i.e. the voltage-controlled oscillator drive chain) -- then this control signal can be scaled directly to frequency noise without knowledge of the loop. I didn't do anything here, I'm taking advantage of the filtering that's already in place from the original 2013 characterization / calibration -- see LHO:5945 for details. I'm just reading out the channel H1:IMC-F_OUT_DQ, which is actually calibrated into kHz, and multiplying by 1e3 [Hz/kHz].

I look at this metric in two different states of the detector: 
    2023-11-03 10:53 UTC -- Reference Cavity and IMC are stability locked vs.
    2023-11-03 10:27 UTC -- just prior, when the IFO is in nominal low noise, i.e. FSS and IMC and CARM are locked

As a reference, since folks are not typically used to looking at the frequency noise of the stabilized PSL below 10 Hz, I also show the cartoon trace of the canonical frequency noise of a "free running" NPRO, 100 Hz/rtHz at 100 Hz, and falling proportionally with frequency, i.e.
    df = 100 [Hz/rtHz] * ( 100 [Hz] / freq )

This puts the "free running" PSL NPRO frequency noise, if it weren't stabilized at all, at 1e5 [Hz/rtHz] at 0.1 Hz -- i.e. the microseism.

OK, so, to the attachments. I attach four cascading plots of frequency noise where I'm adding more and more traces to the same plot as the plot number increases. Open them all up in tabs, and then walk through them slowly as follows:
(I) 2023-11-03_H1IMC_F_FrequencyNoise_01.png -- this shows IMC F stabilized with just the RevCav+IMC used as reference in blue. This is our starting point for building up an understanding, and I keep the NPRO free-running frequency noise in dashed sea-green. In this attachment you can already kind of guess what's happening.
    (1) Above 10 Hz, there's some noise that falls as (1/f). Littered throughout that noise, are some acoustic peaks scattered throughout. BUT
    (2) Below 10 Hz -- and particularly around 0.1 Hz we see a quite familiar shape -- that of ground motion around the microseism.

(II) 2023-11-03_H1IMC_F_FrequencyNoise_02.png -- this shows IMC F in nominal low noise, stabilized against with RevCav, IMC, and CARM used as reference. The changes actually help further ellucidate and validate the story:
    (3) Above 10 Hz, the (1/f) noise doesn't change. Fine.
    (4) Below 10 Hz, the part of the spectrum we thought was seismic is indeed changed dramatically:
        (a) The 0.5 to 10 Hz region looks to convert 
             (i) from a noisy-ish displacement noise -- that looks a lot like the shape of 0.5 to 10 Hz noise budgets from HAM2-HAM3 optic motion -- dominated by the HSTS MC1, MC2, and MC3 OSEM sensor noise re-injected through the damping loops
             (ii) to better displacement noise of optics quadruply suspended from a collection of BSC-ISIs
            retaining the the shape of first two quad longitudinal resonances at 0.43 Hz and 1.0 Hz, and then falling into the ever present (1/f) noise -- now "earlier" / lower in frequency at around 2 Hz.
        (b) Below the 0.15 Hz microseism peak, we see kinda also what we expect -- that the BSC-ISIs are re-injecting a lot of local tilt into the local longitudinal motion of each of the test masses, and is creating common longitudinal noise in CARM -- increasing the spectral density by a factor 50x, but the RMS only by a factor of 2x because the 0.15 Hz microseism peak dominates the RMS.

To further re-enforce that this is real seismic motion -- I recall that we've done a lot of work converting all ISI's GS13s -- include those of the BSC-ISIs suspended the QUADs -- into local suspension point motion -- and then even further converting those local suspension point motions into the basis of the IFO cavities.
    H1:OAF-SUSPOINT_CARM_OUT_DQ
is that ISI GS13s for the QUADs converted into local Sus. Point and then to CARM all coherently. So, if we convert this to frequency noise, via 
    >> c = 2.9989e8;
    >> L = 3995;
    >> lambda = 1064e-9;
    >> (c / (lambda*L))
        ans =
            7.0551e+10 [Hz/m]

and because the all GS13s are calibrated into nano-meters, we get the final "frequency noise" calibration of these channels as 
    >> 1e-9 * (c / (lambda*L))
        ans =
            70.551 [Hz/nm]

then we land on the black trace in 
(III) 2023-11-03_H1IMC_F_FrequencyNoise_03.png 
    I'm not gunna lie, I'm *delighted* at how well this matches up with what I expect!
    (a) in the 0.5 to 10 Hz region, comparing black to red, we confirm that the QUAD modes are amplifying the ST2 motion on resonance at 0.43 Hz and 1 Hz --and maybe 2.7 and 5 Hz.
    (b) but look at how identically matched the motion is from 0.1 to 0.2 Hz! WOW Wow wow.
    (c) We can tell though, that the GS13s themselves are not reporting real motion below 0.1 Hz -- they're are either "tilt dominated" or "self noise" dominated or something.

But, for me, this confirms without-a-doubt the thing that Daniel and Sheila "knew all along" that "at the microseism, the laser PSL frequency is just following the ARMs."
Maybe someone has plotted this up before for, but I couldn't find it. Anyways, very convincing, and very enlightening.

Finally, 
(IV) 2023-11-03_H1IMC_F_FrequencyNoise_04.png compares this BLUE and RED answer against our bigger picture question -- how does this relate to what we need for SPI L? It looks like it may meet our "bare minimum" needs, but will become interesting if we're trying to push the performance of the SPI.
     Very interesting indeed....

Anyways, just for another point of reference, I also show the ground motion at all stations in X and Y -- since this is another thing that folks are used to looking at (rather than the SUSPOINT CARM). This gives you the feel that this measurement is during a "medium" microseism day, with little-to-no wind. 
Images attached to this report
Comments related to this report
jeffrey.kissel@LIGO.ORG - 14:23, Tuesday 07 November 2023 (74071)CSWG, ISC, SEI
Attached are the data from this aLOG, exported from the DTT template:
    RED: IMC_F when Full IFO was locked, 10:27 UTC freqnoise_IMCF_FullIFO_kHz_per_rtHz_ASD.txt
    Blue Trace: IMC_F when only IMC and FSS were locked, 10:53 UTC freqnoise_IMCF_IMCFSSOnly_kHz_per_rtHz_ASD.txt
    Black Trace: OAF CARM when Full IFO was locked, 10:27 UTC freqnoise_OAFCARM_FullIFO_m_per_rtHz_ASD_use70p551Hzperm.txt

Note these data are *not* calibrated, or rather, they carry with them the calibration that came with the channel rather than the addition steps it took to get them in the same Hz/rtHz units. Once you load in the data,
- Multiply the IMCF ASDs by 1000 [Hz/kHz]
- Multiple the OAF CARM by 70.551 [Hz/m]

To recreate the DTT plot with matlab and these attached text files:
>> imcf.fullifo.kHz_per_rtHz = load('freqnoise_IMCF_FullIFO_kHz_per_rtHz_ASD.txt');
>> imcf.imcfssonly.kHz_per_rtHz = load('freqnoise_IMCF_IMCFSSOnly_kHz_per_rtHz_ASD.txt');
>> oafcarm.fullifo.m_per_rtHz = load('freqnoise_OAFCARM_FullIFO_m_per_rtHz_ASD_use70p551Hzperm.txt');
>> loglog(imcf.fullifo.kHz_per_rtHz(:,1),imcf.fullifo.kHz_per_rtHz(:,2)*1000,...
          imcf.imcfssonly.kHz_per_rtHz(:,1),imcf.imcfssonly.kHz_per_rtHz(:,2)*1000,...
          oafcarm.fullifo.m_per_rtHz(:,1),oafcarm.fullifo.m_per_rtHz(:,2)*70.551);
>> legend('IMC F, Full IFO 2023-11-03 10:27 UTC',...
          'IMC F, IMC+FSS Only 2023-11-03 10:53 UTC',...
          'OAF CARM, Full IFO 2023-11-03 10:27 UTC')
>> xlabel('Frequency (Hz)')
>> ylabel('Frequency Noise (Hz/rtHz)')

(and you can add the free running NPRO noise model, SPI requirement, and SPI goal traces to the plot by loading in the files as they are, since I generated them by hand and did so in [Hz/rtHz]. These are also attached for your convenience.)

The DTT template and this data are also committed to the SeiSVN here:
    /ligo/svncommon/SeiSVN/seismic/Common/SPI/Results/
Non-image files attached to this comment
H1 CDS
david.barker@LIGO.ORG - posted 17:00, Friday 03 November 2023 - last comment - 20:46, Friday 03 November 2023(73970)
WP11508 Replace MSR Timing Master

H1 was brought into a safe state.

All models are being shutdown in preparation for replacing the MSR timing master. We are keeping h1susauxb123 going for as long as it and the DAQ can to maintain vacuum trends.

Comments related to this report
david.barker@LIGO.ORG - 17:03, Friday 03 November 2023 (73971)
Images attached to this comment
david.barker@LIGO.ORG - 17:32, Friday 03 November 2023 (73972)

The spare timing master has been installed, all SFPs are green. Daniel reports timing system is green.

The DAQ restarted itself, looking ok now.

I hand started the susaux models, all looks good. We restarted the running susauxb123 models once it was no longer the only front end running.

We are now starting all of the models.

filiberto.clara@LIGO.ORG - 17:35, Friday 03 November 2023 (73974)

Old Unit S1103261
New Unit S1103262

david.barker@LIGO.ORG - 17:45, Friday 03 November 2023 (73975)

All models running, SWWD have been untripped. DAQ is good.

Handing over to control room to recover the IFO.

ryan.short@LIGO.ORG - 20:46, Friday 03 November 2023 (73979)

H1 has been recovered and started observing as of 03:37 UTC.

LHO General
ryan.short@LIGO.ORG - posted 16:08, Friday 03 November 2023 (73968)
Ops Eve Shift Start

TITLE: 11/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acqusition
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 4mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.17 μm/s
QUICK SUMMARY: H1 had just relocked to NLN and lost lock while waiting for ADS to converge. H1 is now relocking.

H1 AWC (AWC, ISC)
keita.kawabe@LIGO.ORG - posted 15:26, Friday 03 November 2023 - last comment - 10:19, Tuesday 07 November 2023(73959)
OM2 thermistor injections.

As was briefly mentioned in alogs 73853 and 73917, I injected into thermistor 2 (hot one which is directly mounted on the compression ring of the OM2 mirror) and thermistor 1 (cold one mounted on the back of the T-SAMS structure) independently, and measured the transfer fucntion from the injection to DCPD-SUM (and some other TFs).

All traces in the attached show the transfer function and coherence from OMC-TEST_EXC (calibrated to the voltage across the Thermistor) and DCPD_SUM (in mA) when H1 is in high power. Red is the injection into Thermistor 2 (hot one), green is into Thermistor 1 (cold one), both while the heater was ON. There are many peaks, most notably the one centered at around 277Hz (that's where CW people observed the most problematic comb structure) but there are others like 78, 93, 105, 144, 154, 166, 314 and 356Hz with good coherence.

The coupling is actually very small for broadband noise, it's only problematic for lines like we've experienced. For example, even if we take the peak at 277Hz (-75dB = 1.8E-4 mA/V) for Thermistor 2, in order for the voltage noise to be comparable to e.g. the shot noise for 40mA DC (~1.1E-7 mA/sqrtHz), the voltage noise should be ~600uV/sqrtHz (because 1.1E-7 [mA/sqrtHz] / 1.8E-4 [mA/V]) at 277Hz, which is a big number.

Thermistor 1 TF amplitude is smaller by more than 30dB than Thermistor 2 measured at around the 277Hz peak. This cannot be explained by the current difference due to lower resistance of the hotter thermistor (thermistor 1 is 7.41k, thermistor 2 is 4.08k, measured on the floor). It should be the location of the thermistor. FYI Thermistor 2 is mounted on the compression ring of the T-SAMS and is therefore much closer to the mirror than Thermistor 1 that is mounted on the back.

Within the resolution of this measurement, frequencies of Thermistor 1 peaks that are visible do match with Thermistor 2, but the resolution is not great (~2Hz at around 277Hz) so it's not clear if we're looking at the same resonance driven by different thermistors, or if this is actually two different resonances, each belonging to one thermistor but not the other.

Blue is the injection into Thermistor 2, but this time the measurement was done immediately after the heater was turned OFF. I only measured down to 220Hz or so, but anyway I see no real difference between the blue and the red. This means that the magnetic field formed by the heater loop around the mirror is NOT relevant for the coupling. Since our experience was that the comb was NOT there while the OM2 was cold, maybe the coupling is dependent on the tightness of the compression ring.

Injection amplitude was constant (A=8000 i.e. 16000cts pp sine wave for all frequencies, corresponding to ~4.9Vpp across the thermistor), and I was making huge lines in DARM (2nd attachment).

After the measurement, all breakout boards and temporary cables and handheld voltage reference and things were disconnected from the system (but were left on the work table by HAM6, I'll pick them up during the next maintenance). EXC cable was connected back to the DCPD whitening, and Beckhoff cable was connected back to the OM2 heater driver.

Images attached to this report
Comments related to this report
valery.frolov@LIGO.ORG - 08:51, Saturday 04 November 2023 (73983)

Do you see the responce in the DCPDs without the light i.e. is the coupling purely electric ?

daniel.sigg@LIGO.ORG - 10:19, Tuesday 07 November 2023 (74050)

No, without any light on the DCPD there were no lines on it.

We also tested direct coupling into the OMC PZT by putting the OMC half off the fringe using a direct laser beam. No signal as well.

H1 CDS
david.barker@LIGO.ORG - posted 11:17, Friday 03 November 2023 - last comment - 18:23, Friday 03 November 2023(73956)
Possible timing system cause of Thu night lock-loss + IPC errors + DAQ crashes

Daniel, Erik, Dave, Fil:

Erik discovered that the DTS DAQ also crashed at the same times last night, indicating a site wide timing issue could be the cause.

Daniel just found that the MSR timing server is reporting GPS issues. We are investigating and checking the spares status.

I've opened FRS29567 for this issue

Comments related to this report
david.barker@LIGO.ORG - 13:39, Friday 03 November 2023 (73961)

Daniel, Fil, Marc, Erik, Dave:

Daniel has taken a closer look at the timing master issues and he says it appears more likely that the timing master will need to be replaced rather than an issue with the roof antenna or its cabling.

The attached 24 hour trend shows the timing master's GPS error count (blue) GPS locked status (orange) and number of tracked satellites (green).

The error burst on the left is 19:00 - 22:00 Thu night, which caused the two crashes. The smaller middle burst is 03:00 - 04:00 this morning. The larger right hand burst is 07:00 - 08:00 which was close to causing a crash.

Daniel further points out that the number of satellites remains good through out, and is sometimes too big (>60).

This all points to an internal error in the timing master and not an antenna issue.

We are readying the spare master chassis for a swap out.

Images attached to this comment
austin.jennings@LIGO.ORG - 14:14, Friday 03 November 2023 (73963)

In preparation for another potential timing error this weekend, I have pushed the current pitch/yaw offsets to SDF for the following suspensions: ETMX/Y, TMSX/Y, BS, PRM, SRM, SR2/3, PR2/3, MC1/2/3, IM1/4.

david.barker@LIGO.ORG - 14:49, Friday 03 November 2023 (73964)

Daniel, Erik, Dave:

For a sanity check we verified that the DAQ's GPS time is correct.

Currently this is a hand calculation, Erik is writing code to automate it.

I checked two times:

14:00:00 Fri 03 Nov 2023 PDT (just a few minutes ago)

21:10:00 Thu 02 Nov 2023 PDT (between the first and second crash last night)

Decoding the CNS-II GPS receivers' IRIG-B channels recorded in the DAQ as H1:CAL-PCALX_IRIGB_DQ, H1:CAL-PCALY_IRIGB_DQ I get the UTC times:

21:00:18

04:10:18

PDT/UTC diff is +7hr. The GPS/UTC leap second diff is currently 18 seconds, so the DAQ timing is correct.

jenne.driggers@LIGO.ORG - 18:23, Friday 03 November 2023 (73977)

Austin's having pushed the slider values to the suspensions was extremely helpful in our recovery later in the afternoon.  For the future, I suggest we also (a) Offload the IMC WFS (which I had forgotten to do on Tuesday, and forgetting caused a lot of difficulty) before doing this, and then (b) including *every* suspension just in case, and (c) Jeff reminds me that we should also include the IMC PZT when doing this. But, already having had most of them pushed was already a huge time saver!

H1 CDS
david.barker@LIGO.ORG - posted 12:41, Tuesday 31 October 2023 - last comment - 17:34, Friday 03 November 2023(73871)
h1lsc0 h1asc0 powered down for IO Chassis +24V DC power supply replacement

Fil, Dave, control room:

In preparation for Fil's replacement of the +24V DC Power Supply which is powering h1lsc0 and h1asc0's IO Chassis, I have powered down these two front end computers.

Process was:

IPMI Reports:

Comments related to this report
filiberto.clara@LIGO.ORG - 13:16, Tuesday 31 October 2023 (73874)

WP 11502

Kepco power supply replaced. Both IO chassis and FE computers powered on.

david.barker@LIGO.ORG - 13:45, Tuesday 31 October 2023 (73875)

Tue31Oct2023
LOC TIME HOSTNAME     MODEL/REBOOT
12:59:06 h1lsc0       ***REBOOT***
12:59:07 h1asc0       ***REBOOT***
13:00:42 h1asc0       h1iopasc0   
13:00:42 h1lsc0       h1ioplsc0   
13:00:55 h1asc0       h1asc       
13:00:55 h1lsc0       h1lsc       
13:01:08 h1asc0       h1ascimc    
13:01:08 h1lsc0       h1lscaux    
13:01:21 h1asc0       h1ascsqzifo 
13:01:21 h1lsc0       h1sqz       
13:01:34 h1lsc0       h1ascsqzfc  
 

camilla.compton@LIGO.ORG - 13:50, Tuesday 31 October 2023 (73876)

Attached sqz sdfs that Naoki and I set so that SQZ would come back in a good (with FC DOWN) state. Lots of these will need to be re-accepted later when SQZ is relocked as some of our sfae.snap and observe.snaps are linked files.

Jenne checked sdfs for h1asc, h1ascimc, h1lsc, h1lscaux.

Images attached to this comment
jenne.driggers@LIGO.ORG - 14:58, Tuesday 31 October 2023 (73880)

Attached are SDF tables for h1lsc, h1lscaux, and h1asc.  There were no diffs for h1ascimc. 

The tables have the mask set to ALL, so that it was showing also differences that are not monitored, since *everything* gets reverted to the safe value when we boot a computer.

For h1lsc, I only accepted the TR QPD B offsets, to keep them roughly close to how they are when we've been locking.  These do get reset every lock (which is why they are not monitored), but if we don't accept them then we'll struggle more than usual with finding IR when we're ready to relock.  Everything else is either guardian controlled, or otherwise not so important.

For h1lscaux, I don't really know why those values of 500 were saved to have those oscillators on, but the scope shows that we don't usually actually use them.  This sdf table is not reverted upon each relock (but would have been when we rebooted), so that's likely why we never noticed / cared about these values.  Anyhow, we don't want random oscillators on (even if the output matrix is zeros so those don't go anywhere), so I accepted them to zeros.

For h1asc, I didn't accept any changes, but just in case we have confusion later, I took a screenshot of the diffs that we have.

There were no diffs at all for h1ascimc, so I didn't bother taking a screenshot of an empty SDF table.

Images attached to this comment
jenne.driggers@LIGO.ORG - 15:54, Tuesday 31 October 2023 (73882)

Austin is working on bringing the IMC back online, and the alignment looks quite poor.  In retrospect, I should have (and forgot to do) offloaded the IMC WFS to the suspensions, since the reboot will have lost the integrated alignment offsets that we were sending to the IMC suspensions.  Austin is going to revert the IMC sus to their earlier top mass osem values as a way of hand-offloading the WFS, and hopefully the IMC will lock and behave after that.

marc.pirello@LIGO.ORG - 17:34, Friday 03 November 2023 (73973)

H1-ISC-C1 IO ASC & LSC power supply was replaced on Tuesday.  Supply location H1-VDC-C2 slot 22A (left side).

Outgoing Supply S1201935 Incoming supply (blank)  New supply has ball bearing fan.

Outgoing supply had sleeve bearing fan that failed due to being worn out.  Will replace with ball bearing fan and put back into spares inventory when parts are available.

F. Mera, F. Clara, M. Pirello

 

Displaying reports 14561-14580 of 86387.Go to page Start 725 726 727 728 729 730 731 732 733 End