TITLE: 10/15 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 159Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 18mph Gusts, 13mph 5min avg
Primary useism: 0.06 μm/s
Secondary useism: 0.30 μm/s
QUICK SUMMARY:
Detector Observing and has been Locked for almost 12hours now. Everything is looking good. Wind has picked up a bit in the last few hours.
TITLE: 10/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 165Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Quiet day, EY chiller issue thats seems to have been resolved. The wind has been picking up over the past 3 hours. We've been locked for 11:35 as of 23:00 UTC.
LOG:
No log
We've been locked for 7:55, everythings stable. The EY water temperature has recovered it seems.
I was just going to put in an alarm bypass for the EY chiller supply water temperature and noticed that it has returned to its operational temperature in the past 30 mins. 6 hour trend shown.
Sun Oct 15 10:08:06 2023 INFO: Fill completed in 8min 2secs
TITLE: 10/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 4mph Gusts, 2mph 5min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.32 μm/s
QUICK SUMMARY:
Richard called to let me know there was an EY chiller alarm, the VEA temperatures look stable but the H1:FMC-EY_CY_H20_SUP_DEGF channels temperature is rising (Chiller water temperature).
That temperature channel seems to be stabilizing, although at a higher than nominal temperature.
The temperature has come back down, I'll keep watching it though.
After a 1hr59min lock, H1 had a lockloss and H1 was stuck at FIND_IR with ALS DIFF needing "Search for IR Resonance by hand" at 947utc/247amPT.
Unfortunately, my computer has been monumentally slow so. I have rebooted my computer and it is a little bit easier, but still very hard to work on H1 because of the slow NoMachine interface. I was able to FIND IR (very slowly due to my computer) by moving the DIFF OFFSET only a few clicks.
ISC LOCK continued.
Although I had requested NLN for ISC_LOCK, on my (slow) computer it was "stuck" at the ACQUIRE_DRMI_1F state (it was hard for me to tell why it was stuck here....it should continue on to DRMI_LOCKED_PREP_ASC, etc. etc., but ISC_LOCK was stuck at ACQUIRE_DRMI_1F.
QUES/NOTE: --Could this be due to GRD_IFO being in the Automatic Operation state?
I then requested ISC_LOCK to go to the next step: DRMI_LOCKED_PREP_ASC
It immediately went there with no problem. I then selcted one-step-at-a-time on ISC_LOCK & it proceed fine to each step.
On ISC_LOCK, I then requested a bigger step of going to PREP_ASC_FOR_FULL_IFO, and ISC_LOCK continued on its own to PREP_ASC (as normal!). (around this time, since I had time to NOT babysit, ISC_LOCK, I took GRD_IFO to MANUAL operation...in case that could be an issue.)
While waiting at the ENGAGE ASC state....I requested ISC_LOCK to go to CHECK_VIOLINS_BEFORE_POWERUP.
ADDENDUM: It his hard for me to determine if there was an actual problem here, because my computer has been very slow....so all troubleshooting has been painful to perform. But since ISC_LOCK appears to be returning to normalcy (as seen from my SLOW computer), I am going to take GRD_IFO to AUTOMATIC operation and let H1 continue as it has done (from what I'm told, has been 3+ weeks with no issues...up until this morning's issue....which is either (1) a legit problem with H1, (2) a problem with my extremely slow computer, or (3) a combination of 1 & 2.
It is 1058utc (358amPT), and ISC_LOCK is at Check VIOLINS (just made it to this state), so I then requested NLN on ISC_LOCK. I am hoping ISC_LOCK can continue on its way, as normal.
At 1101utc/401amPT, I am taking GRD_IFO to AUTOMATIC operation....the hope is H1 will be back up to OBSERVING within the next 30-45min.
[Crossing fingers]
After taking GRD_IFO to AUTOMATIC operation (via OPS_Overview medm), I returned to the H1_MANAGER guardian node, because it had been having a notification from the beginning about requesting an INIT to clear it's notification (I remember trying an INIT several times, but this notification would NOT clear.).
However this time (about an hour later), I was able to clear this notification when requesting an INIT.
As of 1107utc/407amPT, ISC_LOCK is at MOVE_SPOTS.
Additional Notes:
NoMachine interface & my computer appear to be much better, although still not great/normal. I do have a new wireless mouse that I'm using and it's operation is dodgy, so I only use it as a last resort.
Another note with my laptop is that in the evening, I log into NoMachine and leave it logged in overnight (so if action is needed, I can jump into my session and start working on H1.) Perhaps this is a bad procedure? I noticed that although my laptop is plugged into power, it's battery was very low when I first logged in to work on H1 this morning. (not sure why this is the case since it is plugged into power) After working on H1 the last 90+min, my laptop battery has increased from a low percentage to 52%. Maybe I should not keep my laptop logged into NoMachine overnight? (I only did this so I could save time and could hit the ground running....but maybe my computer (and I!) can not handle this.)
1114utc/411amPT & ISC_LOCK is at TRANSITION FROM ETMX.
Will NOW let H1 continue on its way.
1126utc: Will continue to baby sit H1 until it is back to OBSERVING. Currently waiting for CAMERA SERVO.
1134utc/434amPT: H1 Back To Observing AUTOMATICALLY
In the hopes I do not have an issue with my computer/NoMachine, I am going to completely log out of NoMachine, close my laptop for the night, and keep it plugged into power tonight/this morning.....just....in ....case.
TITLE: 10/15 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Second half of my shift had a lot of ground motion and a lockloss. We are now in NOMINAL_LOW_NOISE and just waiting for the camera servos before getting into Observing.
LOG:
23:00UTC Detector Observing and has been Locked for 35.5hours
03:54 Earthquake mode activated due to local ground motion
04:14 Back to CALM
04:14 Earthquake mode activated due to incoming earthquake from Afghanistan
04:25 DIAG_MAIN message: SERVO_BOARDS: IMC-REFL_SERVO_SPLITMON > 8V
04:30-04:38 SERVO_BOARDS: IMC-REFL_SERVO_SPLITMON > 8V message occurring more frequently
04:37-04:38 SRM saturations every few seconds
04:38 Lockloss after 41hours (73466)
04:58 Starting INITIAL_ALIGNMENT
05:36 Back to CALM
05:46 Earthquake mode activated due to local earthquake
05:56 Back to CALM
06:01 Lost lock at CHECK_AS_SHUTTERS
06:53 Reached NOMINAL_LOW_NOISE
Lockloss @ 10/15 04:38UTC
We were staying locked through an earthquake but got the DIAG_MAIN message: SERVO_BOARDS: IMC-REFL_SERVO_SPLITMON > 8V at 04:25 and then multiple times and more often in the last few minutes. SRM was constantly saturated in the last minute of the lock. T1 is where DARM_IN1 gave out.
Because ground motion is still very high and we will need to do an initial alignment, I'll be holding us in DOWN for at least ~10mins
04:58 Attempting an Initial Alignment
06:01UTC Lost lock at CHECK_AS_SHUTTERS
06:53 Reached NOMINAL_LOW_NOISE