After a 1hr59min lock, H1 had a lockloss and H1 was stuck at FIND_IR with ALS DIFF needing "Search for IR Resonance by hand" at 947utc/247amPT.
Unfortunately, my computer has been monumentally slow so. I have rebooted my computer and it is a little bit easier, but still very hard to work on H1 because of the slow NoMachine interface. I was able to FIND IR (very slowly due to my computer) by moving the DIFF OFFSET only a few clicks.
ISC LOCK continued.
TITLE: 10/15 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Second half of my shift had a lot of ground motion and a lockloss. We are now in NOMINAL_LOW_NOISE and just waiting for the camera servos before getting into Observing.
LOG:
23:00UTC Detector Observing and has been Locked for 35.5hours
03:54 Earthquake mode activated due to local ground motion
04:14 Back to CALM
04:14 Earthquake mode activated due to incoming earthquake from Afghanistan
04:25 DIAG_MAIN message: SERVO_BOARDS: IMC-REFL_SERVO_SPLITMON > 8V
04:30-04:38 SERVO_BOARDS: IMC-REFL_SERVO_SPLITMON > 8V message occurring more frequently
04:37-04:38 SRM saturations every few seconds
04:38 Lockloss after 41hours (73466)
04:58 Starting INITIAL_ALIGNMENT
05:36 Back to CALM
05:46 Earthquake mode activated due to local earthquake
05:56 Back to CALM
06:01 Lost lock at CHECK_AS_SHUTTERS
06:53 Reached NOMINAL_LOW_NOISE
Lockloss @ 10/15 04:38UTC
We were staying locked through an earthquake but got the DIAG_MAIN message: SERVO_BOARDS: IMC-REFL_SERVO_SPLITMON > 8V at 04:25 and then multiple times and more often in the last few minutes. SRM was constantly saturated in the last minute of the lock. T1 is where DARM_IN1 gave out.
Because ground motion is still very high and we will need to do an initial alignment, I'll be holding us in DOWN for at least ~10mins
04:58 Attempting an Initial Alignment
06:01UTC Lost lock at CHECK_AS_SHUTTERS
06:53 Reached NOMINAL_LOW_NOISE
We've been Locked for 40 hours now and are Observing at 155Mpc
Quiet night so far. useism is trending up but isn't too bad. Wind has gone up to ~15mph in the last half hour.
TITLE: 10/14 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 5mph Gusts, 4mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.29 μm/s
QUICK SUMMARY:
In Observing and Locked for 35.5 hours! Everything looking good!
TITLE: 10/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: The microseism is still rising, winds been very low today, we've been locked for 35:27 as of 23:00UTC.
We dropped observing briefly from me miss clicking something from 18:56:29 - 18:56:47 UTC.
LOG:
No log
We dropped observing briefly from me miss clicking something from 18:56:29 - 18:56:47 UTC.
We've been locked for 31:40, everything looks stable, saturday tours ongoing.
Sat Oct 14 10:05:05 2023 INFO: Fill completed in 5min 1secs
A quick fill, LLCV had just ramped to 100% open.
TITLE: 10/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 163Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.26 μm/s
QUICK SUMMARY:
TITLE: 10/14 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 163Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: We have now been Locked for 19.5hours. PSL ISS % diff is low, around 1.9%. Uneventful evening overall, but there is an earthquake from New Zealand heading in soon. The ring heater changes that Camilla made (73445) are still okay and we will hopefully continue to not have any issues with them. I didn't notice any issues with the ETMY coildriver(73442) pop up either.
LOG:
23:00UTC Detector Observing and has been Locked for 11.5 hours
02:28 Earthquake mode activated without warning due to sudden spike in ground motion
02:38 Back to CALM
Observing at 158Mpc
Calm evening and have now been locked for 15hrs 40mins. We entered Earthquake mode from 10/14 02:28-02:38 UTC due to a small local earthquake, but managed to stay locked.
Closes FAMIS#26257, last checked 73321
Corner Station Fans (attachment1)
All fans are looking normal and within range.
Outbuilding Fans (attachment2)
All fans are looking normal and within range.
[Vicky, Regina, Dorotea, Sheila]
The attached plot shows the ratio of green light transmitted vs reflected through the OPO over the past year to see how green OPO cavity losses have changed.
We last moved the OPO crystal in Nov. 2022 (see relevant alog). Shortly after moving the crystal, the ratio was around 5% and degraded to around 4% within the first week; it has since settled between 2-3% for most of the past year.
We think the discontinuity around 23/04/16 was due to rejecting power away from the refl PD, which was done around that time because the PD was being saturated. If we look at the refl power alone we can see clear saturation occurring on the PD in the time before the power was lowered. We think the loss curve would be fairly continuous if not for the saturation, but can't exactly compensate since we don't know how much power is being lost. The plot shows only times from when the detector was "stable", but stability was calculated fairly naively, leading to some outlier points remaining from when the detector wasn't actually locked; however, the general trend seems pretty consistent.
History of crystal movements before O4, after HAM7 installation. Compared to ~1000+ steps between crystal co-resonances, the opo crystal has not really moved to a different co-resonance position since installation.
Estimating spots between crystal co-resonances: From e.g. Maggie's MIT/Lasti ilog and LLO crystal scans (e.g. LLO:49568 one side to other side of crystal was from (+3,180 and -11,820) counts, for either LLO:53429 10 co-resonance spots, or LLO:60710 13 co-resonance spots), it seems to average about (15000/13) ~ 1150 steps between fixed-temperature crystal co-resonances, save for the fine-tuning needed at each position. Compared to this ~1150 steps between crystal spots, our previous moves have basically been circling one spot.
Both ETM RH turned up from 1.0W to 1.1W/segment at 19:03UTC, plan to increase another 0.1W later this afternoon. Follow on from 73437. Will stay in observing during this test.
Made another step up of +0.1W to 1.2W/Segment on ETMX and ETMY at 21:07UTC.
Plots attached of HOM, DARM and ndscopes. Jenne pointed out we should use GDS-CALIB for DARM as it isn't effected bt the calibration changing with the RH changes. On ndscopes, not that at -4hours theres a step in SQZ that effects the range 73446.
High frequency noise reduce, DARM maybe better in in the bucket but circulating power down 7kW and Kappa_C down 0.8%.
This doesn't seem to be doing anything bad to the range so we can leave the ETM Ring Heaters in this 1.2W setting for the weekend. If Operators have any troubles, they can reduce H1:TCS-{ETMY,ETMY}_RH_SET{UPPER,LOWER}POWER from 1.2W to 1.0W.
Adding plots of 11:00UTC,13h30 after ETM RH change. DARM ~1000Hz looks like it thermalized worse than 2 hours after the RH change. 6kHz DARM continued to reduce, but it started that lock particularly high. Circulating power settled at 367kW, 7kW less than nominal. KAPPA_C dropped nearly 1%.
These RH changes were reverted 16 October 2023 73503.
Although I had requested NLN for ISC_LOCK, on my (slow) computer it was "stuck" at the ACQUIRE_DRMI_1F state (it was hard for me to tell why it was stuck here....it should continue on to DRMI_LOCKED_PREP_ASC, etc. etc., but ISC_LOCK was stuck at ACQUIRE_DRMI_1F.
QUES/NOTE: --Could this be due to GRD_IFO being in the Automatic Operation state?
I then requested ISC_LOCK to go to the next step: DRMI_LOCKED_PREP_ASC
It immediately went there with no problem. I then selcted one-step-at-a-time on ISC_LOCK & it proceed fine to each step.
On ISC_LOCK, I then requested a bigger step of going to PREP_ASC_FOR_FULL_IFO, and ISC_LOCK continued on its own to PREP_ASC (as normal!). (around this time, since I had time to NOT babysit, ISC_LOCK, I took GRD_IFO to MANUAL operation...in case that could be an issue.)
While waiting at the ENGAGE ASC state....I requested ISC_LOCK to go to CHECK_VIOLINS_BEFORE_POWERUP.
ADDENDUM: It his hard for me to determine if there was an actual problem here, because my computer has been very slow....so all troubleshooting has been painful to perform. But since ISC_LOCK appears to be returning to normalcy (as seen from my SLOW computer), I am going to take GRD_IFO to AUTOMATIC operation and let H1 continue as it has done (from what I'm told, has been 3+ weeks with no issues...up until this morning's issue....which is either (1) a legit problem with H1, (2) a problem with my extremely slow computer, or (3) a combination of 1 & 2.
It is 1058utc (358amPT), and ISC_LOCK is at Check VIOLINS (just made it to this state), so I then requested NLN on ISC_LOCK. I am hoping ISC_LOCK can continue on its way, as normal.
At 1101utc/401amPT, I am taking GRD_IFO to AUTOMATIC operation....the hope is H1 will be back up to OBSERVING within the next 30-45min.
[Crossing fingers]
After taking GRD_IFO to AUTOMATIC operation (via OPS_Overview medm), I returned to the H1_MANAGER guardian node, because it had been having a notification from the beginning about requesting an INIT to clear it's notification (I remember trying an INIT several times, but this notification would NOT clear.).
However this time (about an hour later), I was able to clear this notification when requesting an INIT.
As of 1107utc/407amPT, ISC_LOCK is at MOVE_SPOTS.
Additional Notes:
NoMachine interface & my computer appear to be much better, although still not great/normal. I do have a new wireless mouse that I'm using and it's operation is dodgy, so I only use it as a last resort.
Another note with my laptop is that in the evening, I log into NoMachine and leave it logged in overnight (so if action is needed, I can jump into my session and start working on H1.) Perhaps this is a bad procedure? I noticed that although my laptop is plugged into power, it's battery was very low when I first logged in to work on H1 this morning. (not sure why this is the case since it is plugged into power) After working on H1 the last 90+min, my laptop battery has increased from a low percentage to 52%. Maybe I should not keep my laptop logged into NoMachine overnight? (I only did this so I could save time and could hit the ground running....but maybe my computer (and I!) can not handle this.)
1114utc/411amPT & ISC_LOCK is at TRANSITION FROM ETMX.
Will NOW let H1 continue on its way.
1126utc: Will continue to baby sit H1 until it is back to OBSERVING. Currently waiting for CAMERA SERVO.
1134utc/434amPT: H1 Back To Observing AUTOMATICALLY
In the hopes I do not have an issue with my computer/NoMachine, I am going to completely log out of NoMachine, close my laptop for the night, and keep it plugged into power tonight/this morning.....just....in ....case.