Displaying reports 15081-15100 of 88212.Go to page Start 751 752 753 754 755 756 757 758 759 End
Reports until 06:59, Friday 12 January 2024
H1 General (SQZ)
ryan.short@LIGO.ORG - posted 06:59, Friday 12 January 2024 - last comment - 16:06, Friday 12 January 2024(75331)
CER Temperature Increase; Failed Chiller; No SQZ

H1 called for help this morning at 6:15 PST when it could not go to observing because of the SQZ system not functioning properly. Eventually I saw that the Slow Controls section of the CDS overview was red, so I called in Dave for assistance. So far we have not found evidence of an actualm hardware failure, but Richard noted that a chiller in the CER had failed this morning, causing temperatures to rapidly rise there, potentially creating this issue with the SQZ Beckhoff terminals. Richard has turned on a fan and both chillers in the CER to bring the temperatures back down.

Comments related to this report
david.barker@LIGO.ORG - 07:34, Friday 12 January 2024 (75332)

SQZ slow control Beckhoff is reporting errors in most systems: laser, fiber, pmc, shg, opo, clf and adf.

fiber lock has a "temperature feedback limits" error which triggered when the CER went above 26C at 05:44 PST this morning.

Images attached to this comment
ryan.short@LIGO.ORG - 08:03, Friday 12 January 2024 (75333)ISC, OpsInfo, SQZ

As troubleshooting was ongoing, I went through the ObservationWithOrWithoutSqueezing wiki to make the changes necessary for observing without squeezing, since SQZ seems to have been the only system impacted at the moment and our range without squeezing was hovering just above 120Mpc. I did not have the chance to actually start observing as H1 lost lock, but the intent bit was green and I could have begun observing.

To allow the IFO to observe without squeezing, I took the following actions:

  • Changed several Guardians nominal states to DOWN:
    • SQZ_MANAGER
    • SQZ_LO_LR
    • SQZ_FC
    • SQZ_OPO_LR
    • SQZ_CLF_LR
    • SQZ_SHG (currently not in list on wiki page)
    • SQZ_ANG_ADJUST (also not mentioned in wiki, but this is a newer node)
  • Commented out the TTFSS_LOCKED checker in line 97 of the SQZ_LO_LR Guardian
    • This was neccesary because the TTFSS could not lock due to "temperature feedback limits" and the DOWN state would not return True otherwise
  • Accepted several SDFs on the following models (screenshots attached):
    • CS_SQZ
    • SQZ
    • ASCSQZIFO
    • ASCSQZFC
    • SUSFC2

All of this must be reverted once the SQZ Beckhoff issue is resolved to observe with squeezing.

Images attached to this comment
richard.mccarthy@LIGO.ORG - 08:31, Friday 12 January 2024 (75335)

This morning the ciller bank that feeds the CER quit chilling.  Quite cold outside.  Normailly this would not have been a major issue but the secondary bank of chillers was shut down for noise issues in the IFO.  With the second bank of chillers off, the loss of the running bank allowed for the temerature to rise quickly and to a point we had not seen before. (30c)

I reset the failed bank and turned the secondary bank on as well.  A fan was also added to bring the temperature down quickly.  The fan has been shut off.

The power supply that feeds the 5 Beckhoff squeezer chassis tripped off.  I have not found a reason for this trip as this power supply is les than 50% loaded and the fan had already been replaced so temperature no different than the other units around it. 

I will get with Marc P. to have him measure temperatures of this supply.

camilla.compton@LIGO.ORG - 08:59, Friday 12 January 2024 (75337)

Requested SQZ_MANGER to SQZ_READY_IFO, as expected it got suck on TTFSS error. Took H1:SQZ-LASER_HEAD_CRYSTALFREQUENCY from where it had railed at 200 to 40MHz. This took Beatnote close to nominal 160MHz. Still not locking with Temperature Feedback Error. I trended TTFSS screen and after trying H1:SQZ-FIBR_LOCK_TEMPERATURECONTROLS_RESET (no effect) toggled H1:SQZ-FIBR_LOCK_TEMPERATURECONTROLS_ON Off and On (in the Slow Frequency Servo" box. TTFSS then locked fine.

All SQZ are in their nominal SQZ_READY_IFO states and I'll revert Ryan's "Observing with no sqz" changes.

Images attached to this comment
camilla.compton@LIGO.ORG - 09:38, Friday 12 January 2024 (75340)

Reverted all Ryan's changes and added new GRDs to wiki, we'll need to accept sdf's once we get to NLN. I touched pitch sliders in FC1 (mainly) and FC2, following SQZ Wiki: Issues with SQZ FC Guardian to get FC to lock green.

marc.pirello@LIGO.ORG - 09:56, Friday 12 January 2024 (75341)

I checked the supply in question, H1-VDC-C2 U34 RHS (H1-ISC-SQZ SC).  This supply originally drew 24V 5A, but we have added more Slow Controls (SC) chassis to this rail since, so it looks to be drawing 10A.

Temperature at the front of the supply is 95F, rear of the supply is 104F, only 9F delta.  The ambient air is cool and moving around averaging 68F.  Air flow is not restricted on this supply, and it has a new fan.

I do not feel good airflow through this supply like I do others, I would expect a higher temperature delta, most supplies in that area are sitting at 75F front 105F rear, delta of 30F. 

If this supply trips again I recommend replacement.  In either case, it will be replaced during the vent.

camilla.compton@LIGO.ORG - 16:06, Friday 12 January 2024 (75354)

SDF's for Observing with SQZ accepted.

Images attached to this comment
H1 General (ISC)
ryan.short@LIGO.ORG - posted 05:01, Friday 12 January 2024 (75330)
H1 Back to Observing

H1 is back to observing as of 12:58 UTC. I accepted two SDF diffs on EY_ISC (these were accepted by Austin during his shift, so I've essentially undone his accepting).

Images attached to this report
LHO General
ryan.short@LIGO.ORG - posted 00:21, Friday 12 January 2024 (75329)
Ops Owl Shift Start

TITLE: 01/12 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
    SEI_ENV state: EARTH_QUAKE
    Wind: 23mph Gusts, 20mph 5min avg
    Primary useism: 1.06 μm/s
    Secondary useism: 0.90 μm/s
QUICK SUMMARY:

Austin called and reported on a large nearby earthquake unlocking H1 and tripping ISIs for all four QUADs, BS, HAM7, and HAM8. There was also a typo in the SEI_ENV Guardian which caused it to go into error at this time, but this has been fixed.

Since the earthquake is still shaking, I'll wait until it has fully passed and SEI_ENV goes back to CALM to untip the ISIs, then I'll start an initial alignment and have H1 try to lock. The wind is elevated, but not terrible (20-30mph gusts), so there's potential for that to cause issues.

LHO FMCS (ISC)
austin.jennings@LIGO.ORG - posted 00:17, Friday 12 January 2024 (75325)
Thursday Eve Shift Summary

TITLE: 01/12 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

- EX saturation @ 1:08

- Lockloss @ 5:53 - cause unknown

- Airhandler alert on alarm handler for MY REHEAT channel - looks like there has been a sharp spike over the past hour, Tagging FMCS

- Relocking was mostly straightforward, I did give ETMY a little tap during ALS to get it to catch but locking was otherwise automated

- Back to NLN @ 7:17/OBSERVE @ 7:24

- Following SDFs on ALS Y popped up, preventing me from going into observing, accepting for now for them to be looked at later, Tagging ISC

- Lockloss @ 7:49 - caused by 5.9 EQ from Alaska

- Called Ryan S. and let him know of the situation, ground motion is already starting to settle and he intends to untrip the WDs and begin an initial alignment soon after

LOG:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    

Start Time System Name Location Lazer_Haz Task Time End
23:51 PCAL Rick, Tony PCAL lab local Maintenance 01:23
00:19 VAC Janos MX N VAC checks 01:09
Images attached to this report
H1 General (Lockloss)
austin.jennings@LIGO.ORG - posted 23:52, Thursday 11 January 2024 (75328)
Lockloss @ 7:49

Lockloss @ 7:49, caused by a 5.9 EQ from Alaska.

H1 General (Lockloss)
austin.jennings@LIGO.ORG - posted 22:05, Thursday 11 January 2024 (75327)
Lockloss @ 5:53

Lockloss @ 5:53 - cause unknown. Potentialy see a slight rise in oscillations in the DSOFT YAW loops that starts around ~35 seconds before the lockloss. Looking at the LSC loops, it looks like LSC DARM sees the motion first.

Images attached to this report
LHO General
austin.jennings@LIGO.ORG - posted 20:03, Thursday 11 January 2024 (75326)
Mid Shift Eve Report

Tonight has been uneventful this far. H1 is going strong, 28 hours into its current lock. Secondary microseism has stabilized at 0.5 um/s, but the wind has been steadily on the rise, now ~30 mph here on site.

LHO General
austin.jennings@LIGO.ORG - posted 16:18, Thursday 11 January 2024 (75324)
Ops Eve Shift Start

TITLE: 01/12 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 26mph Gusts, 20mph 5min avg
    Primary useism: 0.10 μm/s
    Secondary useism: 0.42 μm/s
QUICK SUMMARY:

- H1 still locked, just hit 24 hours and appears to still be stable

- Microseism continues to trend down

- CDS/DMs ok

LHO General
thomas.shaffer@LIGO.ORG - posted 16:05, Thursday 11 January 2024 (75321)
Ops Day Shift Summary

TITLE: 01/11 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
INCOMING OPERATOR: Austin
SHIFT SUMMARY: Locked for 24 hours, quiet day.
LOG:

Start Time System Name Location Lazer_Haz Task Time End
16:43 FAC Karen Opt lab n Tech clean 17:04
16:51 FAC Kim H2 n Tech clean 17:05
18:56 FAC Randy EY n Wind fence checks 19:21
20:34 - Camilla Yarm n Running 21:04
21:39 VAC Travis FCTE n Looking for parts 21:40
23:51 PCAL Rick, Tony PCAL lab local Maintenance 01:51
H1 CAL
anthony.sanchez@LIGO.ORG - posted 15:45, Thursday 11 January 2024 (75322)
PCAL EY End Station Measurement & Lab measurements

ENDY Station Measurement:
During the Tuesday maintenace, the PCAL team(Rick Savage & Tony Sanchez) went to ENDY with Working Standard Hanford aka WSH(PS4) and took an End station measurement.

The ENDY Station Measurement was carried out according to this Modified version of the procedure outlined in Document LIGO-T1500062-v15,  Pcal End Station Power Sensor Responsivity Ratio Measurements: Procedures and Log, and was completed by 11 am. The modification was just an extra background measurement with the covers on at the very end of the procedure.

First thing we did is take a picture of the beam spot before anything is touched!

Martel:
We started by setting up a Martel Voltage source to apply voltage into the PCAL Chassis's Input 1 channel and we record the times that a -4.000V, -2.000V and a 0.000V signal was sent to the Chassis. Martel_Voltage_Test.png graph. We also did a measurement of the Martel's voltages in the PCAL lab to calculate the ADC conversion factor, which is included on the document.

After the Martel measurement, we make a series of plots while the Working Standard(PS4) is in the Transmitter Module. These plots are shown in WS_at_TX.png.

Next steps include: The WS in the Receiver Module, These plots are shown in WS_at_RX.png.

Followed by TX_RX.png which are plots of the Transmitter module and the receiver module operation without the WS in the beam path at all.

The last picture is of the Beam spot after we had finished the measurement.

Special Notes: In the pcal_params.py
PCALPARAMS['WHG'] = 0.91536 # 12/05/2023
was changed to:
PCALPARAMS['WHG'] = 0.91616 # 01/10/2024
The updated "Local measurement" value for WHG is the average of two lab measurements taken specifically for the End Station measurement. A "FrontBack"(PS4 on the Front track) and a "BackFront" (PS4 on the Back track) of PS4_PS5 Responsivity ratios. They were measured on the day of the end station measurement Jan 9th 2024. This pcal_params.py  For future reference, changing this number multiplys the entire series of measurements for the entire run by the new number.

The above change might have a slight impact on the RxPD calibration reported in LHO_EndY_PD_ReportV2.pdf.

All data and Analysis has been commited to the SVN.
https://svn.ligo.caltech.edu/svn/aligocalibration/trunk/Projects/PhotonCalibrator/measurements/LHO_EndY/tD20240109/

LAB Measurements Notes:
The PCAL Team has made a proceedural change that now requires the lab results to be processed before the End station results to allow for what we are now calling a "Local in time measurement" of the integrating sphere ratios.
The calculation I used to get this "Local measurement" is:
(BackFront_mean + FrontBack_mean)/2 . It's literally just an average of the last two lab measurements in "BF FB pairs".
This is the first time we have tried this change, but it allows us to treat these measurements more like a metrology shop would.
This is the number we measured today, thus we will use this number today baring any data aquisition or quality issues which will be addressed as they arrive.

BackFront configuration PS4/PS5 Responsivity Ratio.
PCAL Lab Responsivity Ratio Measurement:
A WSH/GSHL (PS4/PS5)BF Responsivity Ratio measurement was ran, analyzed, and pushed to the SVN.
The analysis of this measurement produces 4 PDF files which we use to vet the data for problems.
BFraw_voltages.pdf
BFavg_voltages.pdf
BFraw_ratios.pdf
BFavg_ratios.pdf The mean on the bottom of this plot is where I got the BF number to use for the Local measurement.

PCAL Lab Responsivity Ratio Measurement:
A WSH/GSHL (PS4/PS5)FrontBack Responsivity Ratio Measurement was ran, analyzed, and pushed to the SVN.
The analysis of this measurement produces 4 PDF files which we use to vet the data for problems.
raw_voltages.pdf
avg_voltages.pdf
raw_ratios.pdf
avg_ratios.pdf The mean on the bottom of this plot is where I got the FB number to use for the Local measurement.
alphatrends.pdf shows the history and temperature correction of the lab measurements.

All data and Analysis has been commited to the SVN.
https://svn.ligo.caltech.edu/svn/aligocalibration/trunk/Projects/PhotonCalibrator/measurements/LabData/PS4_PS5/

Special Note: I had to retake a lab measurement due to one of the background data files for the FrontBack PS4_PS5 measurement only having one entry of non-zero data, seems like labview failed to properly communicate with the Keithley after the first background voltage value was read. I have since retaken that measurement on the 10th.

This adventure has been brought to you by Rick Savage & Tony Sanchez.

 

Images attached to this report
Non-image files attached to this report
H1 CDS
david.barker@LIGO.ORG - posted 11:04, Thursday 11 January 2024 - last comment - 15:03, Friday 12 January 2024(75318)
Problems with new HWS camera control code

I discovered this morning that my new HWS camera control code, which only talks to the camera when it needs to turn the camera on or off, was locking up in its caget of the IFO lock status. The last change was for the lock status function to return both the caget call status and the ifo_lock_state.

I reverted the code back to the previous version, and it is now running correctly on h1hwsex, h1hwsey and h1hwsmsr1 but for an unknown reason is still locking up on caget on h1hwsmsr. When I run the caget command on h1hwsmsr's command line it works. Investigation is continuing.

Note due to this bug HWS ITMX camera was mistakingly left ON overnight until I manually turned it OFF at 08:33 PST this morning.

Comments related to this report
david.barker@LIGO.ORG - 12:40, Thursday 11 January 2024 (75319)

code is running again on h1hwsmsr.

camilla.compton@LIGO.ORG - 14:55, Thursday 11 January 2024 (75323)DetChar

Tagging DetChar: the ITMX HWS camera was on at 5Hz (comb visible in H1:PEM-CS_MAG_LVEA_OUTPUTOPTICS_Y_DQ) from 2024/01/09 20:20UTC to 2024/01/11 16:33UTC. Thank you Dave for finding this and turning off.

camilla.compton@LIGO.ORG - 15:03, Friday 12 January 2024 (75352)

Today I checked on the status of these cameras. IX and EX were working as expected and Dave's code is working well on all of them.

The HWS code on EY had crashed (or never correctly started) with a segmentation fault  but started again fine.

ITMY was failing even though Dave's code successfully turned the camera on. Couldn't restart code, had error:  "HWS.HS_Camera.AcquisitionFailure: Image acquisition failed: 1 timeouts and 1 overruns occurred." in state 2B. Even though it successfully connected to the camera. I think we see similar  errors when we are trying to stream images and take data at the same time. I tried re-initing the camera but it didn't change. Streaming images gave pixelated noise, no image. This code has properly been working for the last weeks... Unsure of the issue.

H1 CDS
david.barker@LIGO.ORG - posted 10:53, Thursday 11 January 2024 - last comment - 10:57, Thursday 11 January 2024(75316)
Update on flashing blue-screens for cameras running on h1digivideo0

TJ, Jonathan, Erik, Dave:

yesterday the camera client flashing-blue-screen for cameras whose servers are running on h1digivideo1 steadily got worse as time progressed. This was seen before on 21 Nov 2023, at that time reseating the ethernet cable which connects h1digivideo1 to the cameras fixed it.

We tried, with no success, the following:

In all cases the switch port utilization went close to 50%, packets were dropped and camera-clients flashed blue-screen

We then stopped the server processes of three cameras which were not being used often in the control room and were not on any FOM. The port utilization dropped to 36% (still higher than h1digivideo0 (28%) and h1digivideo2 (20%)) but it resolved the flashing.

Current status:

h1digivideo1 eth1 ethernet is temporarily connected to sw-msr-h1aux port 34. Should put this back to port 16.

h1digivideo1 monit is not running the camera service for three cameras:

  1. cam17: SRM
  2. cam18: SQZ SHG TRANS
  3. cam20: SQZ IR TRANS
Comments related to this report
david.barker@LIGO.ORG - 10:57, Thursday 11 January 2024 (75317)

Note that while the problem persisted, making any of the changes described above risked losing a camera completely, at which point the camera needed to be power cycled via POE and its server restarted.

When the ethernet cable was reseated, we lost cam11 (MC1) and cam13 (PRM)

When the ethernet cable was replaced, we lost cam13 (PRM) and cam17 (SRM)

LHO VE
david.barker@LIGO.ORG - posted 10:23, Thursday 11 January 2024 (75315)
Thu CP1 Fill

Thu Jan 11 10:15:36 2024 INFO: Fill completed in 15min 32secs

Images attached to this report
LHO General
thomas.shaffer@LIGO.ORG - posted 08:00, Thursday 11 January 2024 (75313)
Ops Day Shift Start

TITLE: 01/11 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 18mph Gusts, 13mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.40 μm/s
QUICK SUMMARY: Locked for 16 hours. useism still trending down, but the time is picking back up a bit.

LHO General
austin.jennings@LIGO.ORG - posted 00:00, Thursday 11 January 2024 (75311)
Wednesday Eve Shift Summary

TITLE: 01/11 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

- EX saturations @ 1:21/3:32/4:35/4:46

- Quiet night otherwise, H1 has just reached an 8 hour lock

- Secondary microseism continues its steady decline, now just above 0.5 um/s velocity
LOG:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

Start Time System Name Location Lazer_Haz Task Time End
01:19 PCAL Tony PCAL Lab Y (LOCAL) PCAL measurement 03:09
LHO General
austin.jennings@LIGO.ORG - posted 20:02, Wednesday 10 January 2024 (75312)
Mid Shift Eve Report

Uneventful night so far other than a couple glitches. The detector appears stable, microseism is still above the 95th percentile but is beginning to trend down.

H1 SQZ
camilla.compton@LIGO.ORG - posted 12:24, Monday 08 January 2024 - last comment - 08:58, Thursday 11 January 2024(75250)
Attempted to change ADF frequency at 20:05UTC - Unsure of Issue so Reverted

We've been seeing SQZ angle not optimizing correctly (75245, 75151). At 20:05 Ibrahim took us into commissioning and I tried to change ADF frequency H1:SQZ-ADF_VCXO_FREQ_SET from 1300Hz to 200Hz. The ADF line didn't move from the 1300Hz region, just became noisy when I changed it. PLL didn't lock. Unsure why the ADF wouldn't move, I also tried 800Hz to no avail.  ADF frequency hasn't successfully been changed since Daniel adjusted the model in May 69453.

Comments related to this report
camilla.compton@LIGO.ORG - 17:00, Monday 08 January 2024 (75257)

At ~16:15UTC when we got to NLN, I tried and failed at this again.

Vicky showed (image) that H1:SQZ-ADF_VCXO_CONTROLS_SETFREQUENCYOFFSET and H1:SQZ-ADF_VCXO_FREQ_SET need to be changed then the ADF successfully moved. I also turned the size of the line down by turning up H1:SQZ-RLF_INTEGRATION_ADFATTENUATE, but it was still big and probably reduced our range a few MPc. Attached settings changed and then reverted.

It didn't seem to be able to converge on zero, plot attached. After trying twice I  reverted the changes.

Images attached to this comment
camilla.compton@LIGO.ORG - 08:58, Thursday 11 January 2024 (75314)

Dhruva points out to correctly change both of these settings we can use the script in /sqz/h1/scripts/ADF/ 'python setADF.py -f newfrequency'

Displaying reports 15081-15100 of 88212.Go to page Start 751 752 753 754 755 756 757 758 759 End