Displaying reports 2301-2320 of 83228.Go to page Start 112 113 114 115 116 117 118 119 120 End
Reports until 10:49, Wednesday 12 March 2025
H1 SUS (ISC)
thomas.shaffer@LIGO.ORG - posted 10:49, Wednesday 12 March 2025 - last comment - 12:50, Thursday 13 March 2025(83332)
pi31 slow ringup causing range drop

We noticed that the range was dropping even lower than our already low ~145Mpc. There was a lot of low frequency noise that was creeping up. Sheila suggested that we look at the PIs and sure enough, PI31 had started to slowly creep up the same time as the range degredation (see attached). The SUS_PI guardian node will turn on the damping at 3 according to the H1:SUS-PI_PROC_COMPUTE_MODE31_RMSMON channel and turn it back off when it gets below 3. For now we just tried changing that threshold to 0 to all for continuous damping. This let the mode damp down completely and it brought our range back up.

We should think more about damping this down lower and having two thresholds, an on an off.

Images attached to this report
Comments related to this report
thomas.shaffer@LIGO.ORG - 11:06, Wednesday 12 March 2025 (83334)

Changed the threshold to 1. At 0 the node would continuously search for a new phase thinking that it needed to damp it further.

sheila.dwyer@LIGO.ORG - 13:16, Wednesday 12 March 2025 (83335)

The first attached screenshot shows the spectrum around 10.4 kHz (edited to fix typo), the mode which rang up and caused this issue is at 10428.38 Hz, previously identified as being in the y arm (Oli's slides)

The behavoir seemed pretty similar to what we saw a few weeks ago, 82961, with broad nonstationary noise in DARM. 

The downconversion doesn't seem to be caused by the damping, it started to have a noticable impact on the range while the damping was off, and when the damping came on and reduced the amplitude of the mode the range improved. 

In the ndscope screenshot attached you can see the range that we are using on the DCPDs ADC, this is a 20 bit DAC so it would saturate at 524k counts, when this PI was at it's highest today it was using 10% of the range, the DARM offset takes up about 20% of the ADC range.

The third attachment is the DARM spectrum at the time of this issue, as requested by Peter.  As described in 83330 our range is decreasing with thermalization in each lock, the spectrum shows the typical degredation in the spectrum in these last few days since the SQS angle servo has been keeping the squeezing more stable.  The excess noise seen when the PI was rung up has a similar spectrum to the excess noise we get after thermalization.

Also, for some information about these 10.4kHz PI modes, see G1901351

Images attached to this comment
sheila.dwyer@LIGO.ORG - 12:50, Thursday 13 March 2025 (83358)

Jeff pointed me to 82686 with the suggestion to check the channels that have different digital AA. 

The attached plot shows DCPD sum, and two alternatives, in red is H1:OMC-DCPD_16K_SUM1_OUT_DQ which has a high pass to reduce the single precision noise and no digital AA filters (and some extra lines), SUM2 has 1 digital AA filter and the high pass, and DCPD sum is the darm channel and has 2 digital AA filters and no high pass.  The broadband noise is the same for all of these, so digital aliasing doesn't seem to be involved in adding this broadband noise.

Images attached to this comment
H1 SQZ
sheila.dwyer@LIGO.ORG - posted 10:26, Wednesday 12 March 2025 (83330)
range trends with sqz angle servo on

The attached screenshot shows that turning on the SQZ angle servo 83270, which is shown by the time when CLF_RF6 phase starts to move around, the 350Hz squeezing (yellow BLRMS3) has been better, which is what this servo is intended to do. 

You can also see that the higher frequency squeezing has been worse, and now seems correlated with the range dropping.  Early in each lock there is a time when the high freq squeezing reaches 4.7dB, which corresponds with the times when the range is highest around 156 Mpc.  This is during the thermalization, you can also see "f_s" which is the estimate of the SRC spring frequency based on the calibration lines, and kappa_c are also changing rapidly in this time, so the change in range could be explained by these changes more than the impact on squeezing.

TJ has posted a sensitivity comparison with coherence here:  83327 which shows that we have some A2L coherence to fix. 

Images attached to this report
LHO VE
david.barker@LIGO.ORG - posted 10:22, Wednesday 12 March 2025 (83331)
Wed CP1 Fill

Wed Mar 12 10:08:11 2025 INFO: Fill completed in 8min 8secs

Gerardo confirmed a good fill curbside.

Images attached to this report
H1 CDS
david.barker@LIGO.ORG - posted 08:18, Wednesday 12 March 2025 (83328)
VACSTAT restart for BSC3 sensor glitch

Late entry from last night. VACSTAT went into single-alarm-mode with a BSC3 sensor glitch 20:29:44 Tue 11mar2025 PDT. I reset the vacstat_ioc.service on cdsioc0 at 20:36 to clear this.

Images attached to this report
LHO General
thomas.shaffer@LIGO.ORG - posted 07:37, Wednesday 12 March 2025 - last comment - 08:13, Wednesday 12 March 2025(83324)
Ops Day Shift Start

TITLE: 03/12 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 141Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 15mph Gusts, 11mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.31 μm/s
QUICK SUMMARY: Locked for just over and hour, looks like Ryan had to accept some SDFs overnight, and there were some ALS issues yesterday, but we had 3 relocks overnight. No alarms this morning.

 

Comments related to this report
thomas.shaffer@LIGO.ORG - 08:13, Wednesday 12 March 2025 (83327)

Since our range is a bit lower right now I ran the coherence check template. CHARD is definitely showing come coherence so we could probably run A2L, though with our other issues going on perhaps there is something else afoot.

Images attached to this comment
H1 General
ryan.crouch@LIGO.ORG - posted 06:36, Wednesday 12 March 2025 (83323)
OPS OWL assistance

Another ALS_Y sdf diff

13:33 UTC Observing

Images attached to this report
H1 General
ryan.crouch@LIGO.ORG - posted 02:19, Wednesday 12 March 2025 - last comment - 07:39, Wednesday 12 March 2025(83322)
OPS OWL assistance

I had to accept two sets of SDF diffs to get us into observing, for HAM1 and ALS_Y.

09:16 UTC Observing

Images attached to this report
Comments related to this report
thomas.shaffer@LIGO.ORG - 07:39, Wednesday 12 March 2025 (83325)OpsInfo

I added HAM1 ISI model to the DIAG_SDF exclude list yesterday, so it shouldn't have been necessary to accept these ones to get back into observing. They will still show up as differences, but aren't looked at by the DIAG_SDF node.

H1 General (ISC, OpsInfo)
oli.patane@LIGO.ORG - posted 23:32, Tuesday 11 March 2025 - last comment - 08:33, Wednesday 12 March 2025(83321)
Ops Eve Shift End

TITLE: 03/12 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently relocking and in POWER_25W. We've been down for a while trying to figure out a weird issue with the ALSY WFS and we at least have a plan for the night, but it might end up really annoying for Ryan C unfortunately.

For Ryan C (and maybe TJ tomorrow morning): tagging OPSINFO

I started an FRS ticket FRS#33562. Wasn't sure what category to put it under though so I moved it to be under operators, but that's definitely not right.

 

What we saw/what we tried to do to fix it:

After the lockloss earlier, I wanted to run an initial alignment since we had not run one since yesterday. I noticed that once the WFS would turn on for ALSY, the transmission would slowly drift down until it lost lock(ndscope1, ndscope2). I thought the issue might be DOF2 or 3, since both of those seemed to be diverging and DOF3 WFS were much larger than they usually are(ndscope3). I tried turning DOF2, DOF3, and both sets of WFS off, but we would just lose lock. I left IA and tried locking green arms the normal relocking way, which only uses DOF1 and DOF2 aka ETM and TMS, and that worked well, so I knew it had to be an issue with DOF3 specifically. I ended up needing an initial alignment, and either way we didn't want it to need an initial alignment in the middle of the night. Jenne suggested I take the YARM to LOCKED_SLOW_ETM_TMS_WFS and then slowly align ITMY to lower the ITMY camera error. I tried that, but I kept getting to a point where the DOF2 WFS stopped being able to handle it and would just diverge until we lost lock, even if I tried stepping back the other way, all while the camera errors didn't get much better. After trying this multiple times, Jenne came onto TS and we tried this method again with no success. We also tried adjusting ITMY (usually in Pitch) with YARM in LOCKED_SLOW_NO_WFS, again in LOCKED_SLOW_ETM_TMS_WFS but with DOF3 YAW WFS engaged, we tried resetting the slider values for ITMY, ETMY, and TMSY to the end of the last initial alignment, and we tried aligning MICH to get the beamsplitter in a better spot. None of these methods worked.
Eventually we decided that the only thing we could do for tonight was to have an initial alignment run with ALSY aligning only using ETMY and TMSY. Hopefully during the night there isn't a need for another initial alignment, because if there is Ryan C will need to change the states ALSY will need to go through to be ETM_TMS_WFS_OFFLOADED instead of INITIAL_ALIGNMENT_OFFLOADED.

Now, after finishing an initial alignment where we only used the ETMY and TMSY WFS, I've been trying to relock, but it looks like DOF2 diverging is still happening, ALSY goes down, and then ALSY loses lock(ndscope4). This wasn't happening during the relocks earlier in the day right after maintenance. To get around this, once we got to ARMS_OFF_RESONANCE and ALSY transmission started dropping, I turned off the ALSY DOF2 WFS in both pitch and yaw. We were able to get through the rest of the ALS states like this.


LOG:

22:30 Observing

23:45 We dropped out of Observing for a second and then went back in because I accidently clicked something monitored by SDF

01:02 Lockloss                                                                                                                                                                                                                                                                                                                                                                                                                         

Start Time System Name Location Lazer_Haz Task Time End
22:19 ISC Mayank, Siva Opt Lab Local ISS PD array 00:59
23:42 ISC Keita OpticsLab n WFS work 01:09
Images attached to this report
Comments related to this report
thomas.shaffer@LIGO.ORG - 08:33, Wednesday 12 March 2025 (83329)

Our most recent lock acquisition was fully automatic and even went through an initial alignment successfully. It looks like DOF3 did pull the ALSY power down, but not enough to cause it to lose lock.

Images attached to this comment
H1 General (Lockloss)
oli.patane@LIGO.ORG - posted 18:24, Tuesday 11 March 2025 - last comment - 21:09, Tuesday 11 March 2025(83319)
Lockloss

Lockloss @ 03/12 01:02 UTC

Comments related to this report
oli.patane@LIGO.ORG - 21:09, Tuesday 11 March 2025 (83320)

Still unlocked and having trouble with ALSY WFS DOF3, so troubleshooting that with Jenne.

LHO General
thomas.shaffer@LIGO.ORG - posted 16:39, Tuesday 11 March 2025 - last comment - 12:12, Wednesday 12 March 2025(83306)
Ops Day Shift End

TITLE: 03/11 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: List of maintenance activities below, the most impactful to locking being the camera server change that we ended up having to revert to resume locking (alog83308). We will try to implement these cameras that are used in locking a different day when we can dedicate time to setting the IFO to their new offsets.

Locking notes:

LOG:

Start Time System Name Location Lazer_Haz Task Time End
14:36 FAC Nelly Opt Lab n Tech clean 14:48
15:06 SUS Jeff CER n Move PR3 OpLev cable 16:01
15:06 FAC Kim EY n Tech clean 16:53
15:06 FAC Nelly EX n Tech clean 16:05
15:14 CDS Dave, Erik, Jeff CR n Model restarts (PR3, HTTS, IMs, SUSAUX) also SEIH16 15:35
15:17 CDS Fil, Erik, Marc CER n SEI rack work (HAM1) 18:21
15:18 FAC Tyler LVEA, Ends, Mids n 3IFO checks 17:34
15:20 VAC Ken EY MR n Compressor electrics 18:45
15:25 FAC Contractor EY n Well testing - Large van, water truck, SUV 16:22
15:29 IAS Jason, Ryan C LVEA n FARO 18:59
15:35 OPS Richard EY n Check on panels and Ken 16:51
15:35 VAC Janos, Travis, Jordan EY, EX, mids n Compressor cleanup 18:18
15:51 PCAL Rick, Francisco, Sivananda EX not yet PCAL meas 17:35
16:17 VAC Gerardo LVEA n Vac rack UPSs 17:07
16:18 ISC Camilla, job shadow LVEA n Mark table location 17:17
16:29 FAC Nelly FCES n Tech clean 17:14
16:32 VAC Norco CP2 (CS) n LN2 fill 18:19
16:49 CDS Jonathan, Tony MSR n Pulling digivideo0&1 18:03
16:54 FAC Kim, Nelly LVEA n Tech clean 18:16
16:55 SEI Jim, Mitchell Ends n Dropping off wind fence materials 17:58
17:10 VAC Gerardo, Jackie Ends, Mids n Vac rack UPSs 18:36
17:11 SYS Betsy LVEA n Check on HAM1 vent prep 17:11
17:30 VAC Janos, job shadow LVEA n Turbo test 18:45
17:31 SYS Betsy Garbing room n Check on covers and other 18:03
17:35 FAC Tyler, Eric EY n Replacing pump 18:23
17:41 ISC Mayank, Sivananda, Matt Opt Lab Local ISS PD array 20:12
17:46 VAC Travis, Jordan LVEA n HAM1 feedthrough check 18:18
18:25 - Daniel, Fil, Betsy LVEA n HAM1 vent prep 19:04
18:34 - Richard LVEA n Look at cleanrooms 18:42
18:38 VAC Gerardo, Jackie LVEA n Join on turbo testing 18:55
18:41 FAC Eric EY, MY n Glycol pumping 19:59
18:45 OPS Tony LVEA n Sweep 19:09
18:47 VAC Marc, Fil LVEA n BSC1 temperature sensor 18:55
18:58 VAC Janos LVEA n Turbo test end 19:04
19:47 PCAL Dripta PCAL Lab Yes Setting PS5//PS12 Measurement 20:00
20:31 CDS Fil, Jackie, Marc CER n Replacing power supply for susb123 20:33
20:34 FAC Jim, Mitch EndY N Wind fence investigation 21:36
22:19 ISC Mayank, Siva Opt Lab Local ISS PD array ongoing
Comments related to this report
gerardo.moreno@LIGO.ORG - 12:12, Wednesday 12 March 2025 (83336)VE

We had a cold cathode gauge trip during yesterday's maintenance period, H0:VAC-EX_X2_PT524B_PRESS_TORR is the channel from the signal for the cold cathode gauge.  The gauge pair, cold cathode and the pirani gauges, are located next to the gate valve number 20 (GV20), not far from the P-Cal transmitter pier.  These gauges are very sensitive to interference from communication devices, such as two way radios and mobile phones, please try to keep communication devices away from all vacuum gauges, thank you.

Now, we wait for the gauge to comeback on its own.

Images attached to this comment
H1 General
oli.patane@LIGO.ORG - posted 16:27, Tuesday 11 March 2025 (83315)
Ops Eve Shift Start

TITLE: 03/11 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 2mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.28 μm/s
QUICK SUMMARY:

Observing at 152Mpc and have been Locked for 1 hour.

H1 CDS
david.barker@LIGO.ORG - posted 16:20, Tuesday 11 March 2025 - last comment - 07:52, Wednesday 12 March 2025(83312)
CDS Maintenance Summary: Tuesday 11th March 2025

WP12374 WP12370 ASC and SUS model changes.

Daniel, Jeff, Jonathan, Erik, TJ:

New models were installed for h1asc, h1susim, h1sushtts and h1suspr3. A DAQ restart was required.

WP12372 Add ISI HAM1 cards to h1seih16 IO Chassis

Fil, Marc, Erik, Dave, Jim, TJ:

h1seih16 was powered down. In its IO Chassis the bad A2 Adnaco backplane was replaced and the cards which had been moved to A3 were returned. Two additional ADCs and one additional BIO6464 were installed as per drawing (attached) which will be used by the new h1isiham1 model. Currently these cards have no external cables attached.

A new h1iopseih16 model was installed with these cards added. Note an additional 16bitDAC was not required because seismic ham models allocate a DAC per chamber, with the HEPI model using the lower 8 channels and the ISI model using the upper 8 channels. DAC-1 was already installed for h1hpiham1.

Note that this was one of two changes to the h1iopseih16 model (see SWWD section).

DAQ restart was required.

WP12373 Install new h1isiham1 Model

Jonathan, Erik, Jim, Dave:

The first version of the ISI HAM1 model was added to h1seih16. It is currently running with nothing connected to its ADC, DAC and BIO cards.

This is not something we do regularly, so we took the opportunity to document the steps needed. The list we have so far has been included as a spreadsheet.

The model was added to the DAQ. A related change to h1hpiham1 was also made. A DAQ restart was required for both.

Attachment shows the CDS Overvew MEDM h1seih16 section, now with a full complement of models.

WP12382 HAM1 SWWD

Jeff, Dave:

In preparation for HAM1 ISI install, we created a HAM1 Software Watchdog (SWWD) on h1iopsush2b and h1iopseih16. Note this is the second change to h1iopseih16 today.

There are two suspensions on HAM1; RM1 and RM2.

Despite its name, h1iopsush2b also controls ham1 suspensions. A SWWD part on h1iopsush2b reads the UL, LL, UR, LR raw ADC channels and calculates their integrated RMS values. If any value exceeds the trip level, the Dolphin IPC channel sent to h1iopseih16 is set to zero.

The IPC receiver on h1iopseih16, on receipt of a trigger signal, starts a DACKILL count-down timer. If the timer expires, it executes a DACKILL for the second DAC card, which drives HAM1 HEPI+ISI.

New h1iopsush2b and h1iopseih16 models were installed. The h1iopsush2b was done at the same time as the h1susim and h1sushtts models since these are the only models running on this front end. h1iopseih16 restart was done when this front end was powered back up after the IO Chassis work.

The SWWD MEDMs were updated with the new HAM1 system (attached)

Note: one complication is that up until now SUS SWWDs have been for larger suspensions with 6 OSEM channels (F1, F2, F3, LF, RT, SD). RM1,2 only have 4 (UL, LL, UR, LR) and so the two unused OSEM inputs have been grounded in the models. On the MEDM I just show ADC0_30, the channel prior to the duotone which is zero.

WP12358 Move Digital Video Cameras from h1digivideo2 to h1digivideo5

Patrick, Jonathan, Erik, TJ, Sheila, Dave:

We moved the six cameras which were on the last of the old servers (h1digivideo2) onto the new AMD/deb12 h1digivideo5 and powered down the old machine.

All was good until locking started, which is was discovered that for cameras which are copied to front end models the X,Y centroid EPICS channels now had offsets which the models could not handle.

For locking expediency it was decided to move the cameras back to the old server for now until the models could be configured to use to the new centroid values. Of the six models, five were moved back to h1digivideo2 (BS, ITMX_RED, ITMX_GREEN, ITMY_RED andITMY_GREEN). POP-AIR is not copied to FE models and so could stay on h1digivideo5.

At time of writing H1 has locked and been in OBSERVE for 50 minutes.

To "green up" the EDC I am running a dummy IOC on opslogin0 to serve the new channels for these cameras which were lost when we reverted them to the old software.

The camera MEDM shows which ones are still running on VID2 (attached).

DAQ Restarts:

Jonathan, Erik, Dave:

Due to the large number of changes we did two DAQ restarts today. The first was primarily for the ASC and SUS model changes and SWWD. The second was for the addition of the new h1isiham1 model and the camera move.

The EDC restart was due to new:

H1EPICS_DIGVIDEO.ini (at the time, moved 6 cameras to new server)

H1EPICS_DAQ.ini (add h1isiham1)

H1EPICS_FEC.ini (add h1isiham1)

H1EPICS_SDF.ini (add h1isiham1)

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 16:29, Tuesday 11 March 2025 (83316)

Tue11Mar2025
LOC TIME HOSTNAME     MODEL/REBOOT
08:14:26 h1asc0       h1asc       <<< ASC and SUS model changes
08:15:06 h1sush2a     h1suspr3    
08:17:36 h1sush2b     h1iopsush2b 
08:18:12 h1sush2b     h1susim     
08:18:34 h1sush2b     h1sushtts   
08:19:08 h1susauxh2   h1susauxh2  


08:22:02 h1daqdc0     [DAQ] <<< 1st DAQ Restart (no EDC)
08:22:15 h1daqfw0     [DAQ]
08:22:15 h1daqtw0     [DAQ]
08:22:19 h1daqnds0    [DAQ]
08:22:23 h1daqgds0    [DAQ]
08:22:56 h1daqgds0    [DAQ]
08:26:16 h1daqdc1     [DAQ]
08:26:28 h1daqfw1     [DAQ]
08:26:29 h1daqtw1     [DAQ]
08:26:30 h1daqnds1    [DAQ]
08:26:39 h1daqgds1    [DAQ]
08:27:07 h1daqgds1    [DAQ]  <<< GDS1 restart, but too quick
08:27:28 h1daqgds1    [DAQ] <<< seconds GDS1 restart good
08:29:46 h1daqfw0     [DAQ] <<< FW0 spontaneous restart


09:38:05 h1seih16     ***REBOOT*** <<< h1seih16 back from BP fix and card install
09:39:40 h1seih16     h1iopseih16 
09:39:53 h1seih16     h1hpiham1   
09:40:06 h1seih16     h1isiham1   
09:40:19 h1seih16     h1hpiham6   
09:40:32 h1seih16     h1isiham6   
10:04:19 h1seih16     ***REBOOT***  <<< a second IO Chassis incursion to remove 3rd DAC card which is not needed
10:05:54 h1seih16     h1iopseih16 
10:06:07 h1seih16     h1hpiham1   
10:06:20 h1seih16     h1isiham1   
10:06:33 h1seih16     h1hpiham6   
10:06:46 h1seih16     h1isiham6   


10:48:15 h1daqdc0     [DAQ]  <<< second DAQ restart for new h1isiham1 model and various EDC INI files
10:48:28 h1daqfw0     [DAQ]
10:48:28 h1daqtw0     [DAQ]
10:48:32 h1daqnds0    [DAQ]
10:48:37 h1daqgds0    [DAQ]
10:48:48 h1susauxb123 h1edc[DAQ] <<< EDC restart
10:51:22 h1daqdc1     [DAQ]
10:51:34 h1daqfw1     [DAQ]
10:51:34 h1daqtw1     [DAQ]
10:51:35 h1daqnds1    [DAQ]
10:51:43 h1daqgds1    [DAQ]
10:52:18 h1daqgds1    [DAQ] <<< GDS1 restart


11:51:32 h1seih16     h1isiham1   <<< fix ADC card_nums, no DAQ restart needed
 

patrick.thomas@LIGO.ORG - 17:09, Tuesday 11 March 2025 (83317)
The camera server code on both h1digivideo4 and h1digivideo5 was updated from 0.1.16 to 0.1.17 to fix a bug in the periodic archiving of camera images (https://git.ligo.org/cds/software/cameras/pylon-camera-server/-/issues/6).
jonathan.hanks@LIGO.ORG - 07:52, Wednesday 12 March 2025 (83326)
In addition Tony and Jonathan removed h1digivideo0 and h1digivideo1 from the racks as a cleanup job as part of the camera update WP.
LHO VE (VE)
gerardo.moreno@LIGO.ORG - posted 16:02, Tuesday 11 March 2025 (83313)
Replaced batteries on the Vacuum UPS systems

(Jackie F., Gerardo M.)

Replaced the old batteries on the UPS units at both mid stations, both end stations, and two at the corner station.
The UPS units are dedicated supplies for the vacuum racks, the following racks were updated today:

No issues were encountered.  One set of batteries had been already installed previously, see aLOG 83151.

Work done under WP #12377.

Images attached to this report
Displaying reports 2301-2320 of 83228.Go to page Start 112 113 114 115 116 117 118 119 120 End