Displaying reports 12801-12820 of 84292.Go to page Start 637 638 639 640 641 642 643 644 645 End
Reports until 10:40, Saturday 21 October 2023
H1 General
ryan.crouch@LIGO.ORG - posted 10:40, Saturday 21 October 2023 - last comment - 13:16, Saturday 21 October 2023(73637)
OPS Saturday day shift update

Kept losing lock from green arms, the ALS error signals didn't look great, so I started an Initial alignment at 15:12UTC which finished at 15:33UTC. I kept losing it at ALS from XARM_WFS_DOF1_P so I turned it off (I had to do this every lock attempt) and then DIFF_IR kept reported "IR not actually found" so I had to move it around by hand. During PRMI I misaligned PRM to check on the BS and it looked great on AS_AIR, guardian took us to CHECK_MICH while I was in the middle of checking the BS.

Defiantly something is misaligned, I couldn't get PRMI for a while and finally I tried to adjust SR3 in yaw to match its OPLEV from before the glitch and that got PRMI to finally lock, but it was unstable and lost it soon after, so something else is still badly aligned, the PIT signal looked good for SR3 based on the OPLEVS. PR3 OPLEV looks a little different as well so I adjusted it back to where it was.

I have to slowly step up in states to not lose lock going through ALS and IR, even with the DOF dance Xarm just loses it for seemingly no reason. I tried requesting LOCK_SLOW_NO_WFS let it settle in LOCKING_GREEN_ARMS then requested LOCKING_ALS and I did not have to turn off any WFS for Xarm. In CHECK_IR DIFF IR was not really found as it hasn't been every attempt. As I was trying to find DIFF_IR we went to EQ mode and lost lock from the quick ground motion increase (5.0 from Mexico?). I going to go make a cup of coffee while the ground motion calms down.

Comments related to this report
ryan.crouch@LIGO.ORG - 11:07, Saturday 21 October 2023 (73643)

After then getting zero flashes on PRMI after getting back up, I decided to try to restore all the sliders to when we were locking before the previous lock (10/19 18:19UTC) at 18:03UTC. The arms are going through increase flashes now.

Things seemed worse after the IA that I ran? During the IA everything looked good though, the signals were all reasonable and AS_AIR looked correct after each section. If this restore all doesn't help I'm going to restore things to how they were after Camilla finshed her IA earlier in the morning (~14:00UTC)

ryan.crouch@LIGO.ORG - 11:30, Saturday 21 October 2023 (73644)

Going through each state manually and slowly, no Xarm issues this time, IR seems to be looking good too. AS_AIR looks bad at DRMI. Lockloss at CHECK_MICH

ryan.crouch@LIGO.ORG - 12:04, Saturday 21 October 2023 (73646)

After talking with Jenne, I'm runing another intial alignment and I did not restore the optics to after Camillas IA. We're still at the 10/19 18:19UTC alignment that I restored too.

ryan.crouch@LIGO.ORG - 13:16, Saturday 21 October 2023 (73647)

Reaquired NLN at 20:16, waiting for ADS to converge to go into observing.

LHO VE
david.barker@LIGO.ORG - posted 10:34, Saturday 21 October 2023 (73642)
Sat CP1 Fill

Sat Oct 21 10:10:54 2023 INFO: Fill completed in 10min 50secs

Images attached to this report
H1 General
ryan.crouch@LIGO.ORG - posted 08:02, Saturday 21 October 2023 - last comment - 09:17, Saturday 21 October 2023(73636)
OPS Saturday day shift start

TITLE: 10/21 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Camilla (OWL)
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.33 μm/s
QUICK SUMMARY:

Comments related to this report
ryan.crouch@LIGO.ORG - 08:24, Saturday 21 October 2023 (73638)

15:12UTC started an IA after struggling to LOCK_ALS and seeing the ALS error signals weren't great.

ryan.crouch@LIGO.ORG - 09:06, Saturday 21 October 2023 (73639)

IA finished at 15:33UTC, but since then I've been struggling to get DRMI, PRMI locks pretty easily but DRMI struggles even with some decent flashes, POPAIR_B_RF18 can't seem to flash above 120. Lots of ALS locklosses too, sometime from WFS sometimes for no apparent reason.

david.barker@LIGO.ORG - 09:17, Saturday 21 October 2023 (73640)

Follow up on Dolphin crash:

I looked at the logs for the front ends which had an issue at 01:05:25 PDT (lsc0, oaf0, susb123, sush2a, sush34, sush56), nothing was logged at this time.

All the IOP models experienced a very large timing glitch at this time. The size of the glitchs are

iop model cpu max (mS)
oaf0 24.3
lsc0 23.4
susb123 15.7
sush2a 18.7
sush34 18.7
sush56 15.7

 

ryan.crouch@LIGO.ORG - 09:17, Saturday 21 October 2023 (73641)
H1 General (CDS, SEI, SUS)
camilla.compton@LIGO.ORG - posted 02:46, Saturday 21 October 2023 - last comment - 10:35, Monday 23 October 2023(73628)
CDS dolphin crash, causing lockloss and HAM1 watched tripp
Comments related to this report
camilla.compton@LIGO.ORG - 04:47, Saturday 21 October 2023 (73634)
  • Put ISC_LOCK into IDLE to run the baffle align scripts so it wasn't taken to INITAL ALIGNMENT by H1_MANAGER. TMS ones sucessful but the ITM baffle align screens failed and I checked the ITM oplevs moved sliders to get oplevs in same potison as in NLN. This moved ITMX 70urad and ITMX 45urad in pitch. A little light now on green cameras/ AS AIR. Reran ITM baffle align scipts. 
  • 10:20UTC began Inital Alginment. ALSX PLL Unlocked. 
  • 10:50UTC Not much light on X-arm green. Moved BS back to BS oplev postion, 10urad PIT.
  • 11:20UTC Tried moving PR following sliders, greens arms PDs or cameras not better, reverted. 
camilla.compton@LIGO.ORG - 06:51, Saturday 21 October 2023 (73635)

Summary: To lock green arms I put ITMs, BS, PR2 and PR3  back to Oplev positions before trip/last time we were aligning. Also touched ETMs and TMSs as usual in green arms aligning. During intal alignment I touched H1:LSC-XARM_GAIN (and reverted), PR2, PRM, SRM. 

  • Trended the IM's LKIN signals, all seem similar to before trip, IM4 changes when locking usually. 
  • Moved PR2 to oplev values (again).  Starting (P 1566.8, Y 3227.6) Ending (1589.8, 3237.6).
  • Moved PR3 to oplev values. Starting (P-120.2, Y149.2). Ending (-117.2, 156.2). This made the green camera images and ITMY flashes MUCH better.  After ITM, ETM, TMS alignment tweeks, have both green arms locking. 
  • 12:34UTC Started Initial Alignment. A few placed need to restore optics to their oplevs from last initial alignment position.
    • 12:59 UTC stuck at XARM_IR_LOCKED
      • Again adjusted PR2. Starting (1566.8, 3237.6), ending  (1581.8, 3237.6), doubled H1:LSC-XARM_GAIN gain, touch PR2 while watching TRX flashes  (and then restored gain) as Corey describes in 69917 and wiki.
    • Couldn't Acquire PRX. Sliders for Oplevs for PRM restored. Starting (-1377.9, 220.1) Ending: (-573.9, 335.1).
    • SRC_ALIGNMENT offloaded a wring alignment and SRM was saturating with L filters having  really large numbers. Took ALIGN_IFO to DOWN and this cleared. Moved SRM to oplev value. Starting (2231.2, -3158.3) after (2247, -3199) before requesting SRC_ALIGN_OFFLOADED.
  • 13:48 UTC INITAL ALIGNMENT Complete. 
camilla.compton@LIGO.ORG - 10:35, Monday 23 October 2023 (73665)OpsInfo

Documented in FRS 29441. Ryan C got us back to NLN at 20:16UTC by restoring all the sliders to when we were locking before the previous lock and then running another initial alignment alog 73637.  

Restoring sliders with (sitemap > ASC > Init Align Compat > ! Restore) should have been the first thing to do as during the computer restart the sliders are reset to their safe.snap files, which are not (currently) recent values. Tagging OpsInfo and updated Recovery from Dolphin/Computer Crash wiki.

H1 AOS
david.barker@LIGO.ORG - posted 01:39, Saturday 21 October 2023 - last comment - 11:30, Saturday 21 October 2023(73629)
Several corner station front ends down

Investigating a dolphin crash of several corner station front ends.

Images attached to this report
Comments related to this report
camilla.compton@LIGO.ORG - 02:06, Saturday 21 October 2023 (73630)

Opened FRS 29441. This crash appeared to cause locklosss at 8:05UTC.

david.barker@LIGO.ORG - 02:19, Saturday 21 October 2023 (73631)

I first verified for the IOPs with DACKILLs that their IO Chassis could be seen, which was the case.

I restarted all the models (to clear IOP DACKILLS) on h1oaf0, h1lsc0, h1susb123, h1sush2a, h1sush34, h1sush56. I put all the corresponding SEI IOP SWWD into bypass mode before restarting the SUS IOPs.

All the models restarted with no issues, no reboots (with their associated Dolphin fencing) were needed.

In the morning I'll see if I can find which front end caused the Dolphin crash.

david.barker@LIGO.ORG - 02:20, Saturday 21 October 2023 (73632)

The EDC is slow in reconnecting to susmc3 and susprm channels, but it looks like it will get there in time.

david.barker@LIGO.ORG - 02:26, Saturday 21 October 2023 (73633)
Images attached to this comment
david.barker@LIGO.ORG - 11:30, Saturday 21 October 2023 (73645)

Sat21Oct2023
LOC TIME HOSTNAME     MODEL/REBOOT
01:48:20 h1oaf0       h1iopoaf0   
01:48:34 h1oaf0       h1pemcs     
01:48:48 h1oaf0       h1tcscs     
01:49:02 h1oaf0       h1susprocpi 
01:49:16 h1oaf0       h1seiproc   
01:49:30 h1oaf0       h1oaf       
01:49:44 h1oaf0       h1calcs     
01:49:58 h1oaf0       h1susproc   
01:50:12 h1oaf0       h1calinj    
01:50:26 h1oaf0       h1bos       
01:52:02 h1lsc0       h1ioplsc0   
01:52:16 h1lsc0       h1lsc       
01:52:30 h1lsc0       h1lscaux    
01:52:44 h1lsc0       h1sqz       
01:52:58 h1lsc0       h1ascsqzfc  
01:55:25 h1susb123    h1iopsusb123
01:55:39 h1susb123    h1susitmy   
01:55:53 h1susb123    h1susbs     
01:56:07 h1susb123    h1susitmx   
01:56:21 h1susb123    h1susitmpi  
01:58:57 h1sush2a     h1iopsush2a 
01:59:11 h1sush2a     h1susmc1    
01:59:25 h1sush2a     h1susmc3    
01:59:39 h1sush2a     h1susprm    
01:59:53 h1sush2a     h1suspr3    
02:01:57 h1sush34     h1iopsush34 
02:02:11 h1sush34     h1susmc2    
02:02:25 h1sush34     h1suspr2    
02:02:39 h1sush34     h1sussr2    
02:04:13 h1sush56     h1iopsush56 
02:04:27 h1sush56     h1sussrm    
02:04:41 h1sush56     h1sussr3    
02:04:55 h1sush56     h1susifoout 
02:05:09 h1sush56     h1sussqzout 
 
 

Displaying reports 12801-12820 of 84292.Go to page Start 637 638 639 640 641 642 643 644 645 End