Because the fire water tank is in need of water both for it's primary function(s) as well as upcoming demand placed on it by DGR for the storage building work, I have enabled the pump to run beginning at 7:55a PST. The pump will run for a duration of 4 hours. T. Guidry
TITLE: 10/10 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 162Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 4mph Gusts, 2mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.21 μm/s
QUICK SUMMARY:
Observing at 162Mpc and have been Locked for 4 hours. Looks like this last lock has been much smoother than the others during the night were.
H1 requested assistance around 11pm PT due to the SQZ filter cavity not being able to lock. Based on previous alogs, it sounds like there was some SQZ issues earlier today, but there's no mention as to what exactly. Looking at the FC green camera it was clearly trying to lock with a pitch misalingment, so I adjusted FC2 and maximized H1:SQZ-FC_TRANS_C_LF_OUTPUT. This seemed to help, but then after it would find IR it would lose lock. I'm not sure why, but we just lost lock (0720UTC) so I guess I'll see if it happens again when we get back.
TITLE: 10/10 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
IFO is LOCKING at CHECK_AS_SHUTTERS (although squeezing is broken).
Overall terrible shift and locking due to many affected suspension sliders due to a dolphin crash at 22:30 UTC (around 4:30 PM). Not one minute spent in observing due to this crash that somehow changed everything. Here's a rough story:
IR Not Found:
IR was not found. Went into Input_Align in Initial_Alignment, bright eyed and optimistic only to realize that this was also not working. Checked the ALS Overview to find that we were getting a beatnote, and that guardian had indeed swept the full frequency range, without finding any resonances. After some finagling and calling Daniel, he pointed me to check IMs and PRs, and indeed IM4 and PR2 were off in P and Y, so I trended those values and then set them to their original. We were able to Find IR, complete initial alignment and went all the way up to NLN.
Unable to SQZ:
Was happy to finally get to NLN (2:07 UTC) but then the SQZ was unable to lock FC. Breaking News: Lockloss due to 40mph winds! Lockloss alog 80579 After a lot of help from Sheila, we managed to get it working again. Now onto locking!
Lock Attempt 2 (and unable to SQZ again):
Got to NLN after losing lock frequently in ALS and PRMI/DRMI (winds). Wind calmed down and we were able to get all the way up to NLN (4:46 UTC) until yet again, SQZ was broken. Around 2 minutes after alerting Sheila, we lost lock again.. Lockloss alog 80580
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 01:57 | PEM | Dan Morraru | MSR | N | LSB Warehouse Connection | 02:16 |
Another probably wind-related lockloss. While EX X-Axis ground motion was high, overall wind was not usually high enough to cause NLN lockloss. So more of a toss-up - lockloss tool does have a windy tag though...
Winds approaching 40mph so will assume environmental. EX X-axis was experiencing a lot of ground motion and ALSX was struggling to lock for a while.
TITLE: 10/09 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: Currently relocking and at FIND_IR. The camera issue seems to have been fixed for now, but we are still seeing the glitches in the PSL.
LOG:
14:30 Observing and Locked for 7 hours
15:25 Out of Observing due to camera issues (80559)
15:34 Back into Observing (we were good to go back two minutes after we were taken out, but it wasn''t set to auto go back into Observing and I missed it being okay)
16:25 Out of observing from the camera issue
16:26 Back into Observing
16:50 Out of observing due to TCS CO2 laser unlock
16:52 Back into Observing
17:01 I took us out of Observing to try SQZ offloading before taking us down for corrective maintenance
17:03 I unlocked the IFO using the IMC REFL servo toggle, sitting in IDLE
18:49 Started a manual initial alignment
19:17 initial alignment done, relocking
20:08 NOMINAL_LOW_NOISE
20:14 Observing
20:28 Out of Observing so Keita can grab stuff from the LVEA
20:33 Back into Observing
22:05 Superevent S241009em
22:34 Lockloss
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 15:19 | FAC | Karen | OpticsLab/VacPrep | n | Tech clean | 16:17 |
| 16:17 | FAC | Karen | WoodShop | n | Tech clean | 17:02 |
| 17:03 | FAC | Richard, Karen | FCETube | n | 17:09 | |
| 17:10 | FAC | Karen, Kim | FCETube | n | Tech clean | 18:29 |
| 18:06 | PSL | Keita, Sheila, Rick | LVEA, DiodeRoom | n | Swapping PSL controller | 18:44 |
| 20:28 | Keita | LVEA | n | Grabbing stuff | 20:32 |
At 15:34:40 Wed 09oct2024 PDT a corner station dolphin glitch caused DACKILLs for h1sush34, h1lsc0 and h1omc0.
h1sush7 has other Dolphin logs associated with previous glitches. https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=75921 [Wed Feb 21 11:33:40 2024] IXH Adapter 0 : Port 0 is not operational -- UpTime: 0 sec - Event = 0 https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=79561 [Thu Aug 15 16:37:15 2024] IXH Adapter 0 : Port 0 is not operational -- UpTime: 0 sec - Event = 0 And today: [Wed Oct 9 15:34:25 2024] IXH Adapter 0 : Port 0 is not operational -- UpTime: 0 sec - Event = 0 [Wed Oct 9 15:34:25 2024] dis-ix-ntb: dis-ix-ntb0: Detected IXS600 (601) switch, SUB2, port 1, global port 12 [Wed Oct 9 15:34:25 2024] IXH Adapter 0 : Local adapter is connected to IXS600 switch, SUB2, port 1 [Wed Oct 9 15:34:25 2024] IXH Adapter 0 : Link 0 is operational - Link width: x8 - PCIe Gen 2 -- Downtime: 0 sec We might want to check cable when we get a chance, and if that dose not resolve the issue consider replacing the card.
TITLE: 10/09 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 16mph Gusts, 9mph 3min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.22 μm/s
QUICK SUMMARY:
IFO is LOCKING following a Dolphin Crash that's still being investigated. alog 80572.
Lockloss @ 10/09 22:34UTC from PSL :( possibly from a CDS error
FAMIS 28374 Weekly In-Lock SUS Charge Measurements
The nominal set of plots that the Famis task requests are plotted as usual.
ETMX looks fine
ETMY also looks fine
ITMX has a point on the plot that skews the plot so much that we cant really see any recent relative motion of the measurement.
Same thing happened with ITMY.
I'm not sure if this is actually useful.... so here are the same ones but only 4 months of data instead to get those really low ones off the plots.
4 Month ETMX
4 Month ETMY
4 Month ITMX
4 Month ITMY
Maybe this isn't useful to anyone but I was mildly interested.
The filter_bank_changes script, which shows how filters and on/off switches within a filter bank have changed over a period of time (example output, original alog), is now executable within medm screens by doing right click > Execute > filter_bank_changes. It works similar to ndscope and fmscan in that you have to select a channel on the medm screen. The channel can be any of the channels within the filter bank (ex. _SWSTAT, _INMON, _Name00, etc).
If you'd like to run it outside of an MEDM screen, the script can be found in /ligo/gitcommon/filterBankChanges/ and can be run with ./filterbankchanges.sh CHANNEL_NAME , where CHANNEL_NAME should end in _SWSTAT or one of the other aforementioned filter bank channel endings
Rick, Keita, Sheila, Daniel, remote help from Matt Heintze
This morning we swapped the NPRO laser controller S2200009 out for S2200008.
Settings before we started: Laser diode A set temperature 18.3C, Laser diode B set temperature 15.99C, laser diode injection current 2.134 Amps, laser crystal set temperature 26.04 C, laser crystal actual temperature 26.10 C.
We went to the diode room, and followed the notes from the procedure Ryan Short outlined to me to shut down the amplifiers, turning off amp2, then amp1, then closing the shutter, then we went one step beyond his instructions and shut off the NPRO. We swapped the controller with all the cables, including the interlock.
When we powered on S2200008 laser diode A temperature was set to 17.14, B set to 22.96C. We adjusted the pots on the front panel until they matched the values we had written down from the original controller. We turned the knob on the front panel for the injection current to 0. Rick and I went to the laser diode room and turned the NPRO back on, Keita confirmed that this turned on the controller. We noticed that the laser diode 1 and 2 set temps were what we had set them to be by adjusting the pots for A and B, but the act temp readbacks weren't matching, we confirmed with photos that with the original controller the set and actual temps matched. (I will attach a photo of this state). At the rack Ketia turned the injection current up to about 100mA, this didn't change the temperature readbacks.
We had a discussion with Matt Heinze, who agreed this is probably a readback issue and that it was safe to keep going. We think that this is because we haven't adjusted the pots on the LIGO daughter board following T1900456. Keita slowly turned up the injection current knob while Rick and I watched from the diode room, the NPRO power came back to 1.8W which was the same as before. The laser diode act power readbacks for diode 1 says 3.6W, while it was 5W with the original controller, and diode 2 says 2.48W where it also said 5W with the original controller. Since the power output is the same, we think these are also just readback problems due to not following T1900456. We opened the shutter, turned on the amplifiers after a few minutes, and set the power watchdogs back on, as well as the pump diode watchdog.
The PMC and FSS locked without issue, Daniel found that the squeezer laser frequency had to increase by 850MHz to get the squeezer TTFSS locked, so we moved the NPRO temperature up on the FSS by about 0.3 K. After this the squeezer and ALS lasers locked easily, and Oli relocked the IFO.
The first attached photo is the NPRO screen on the beckhoff machine before the controller swap, the second photo is after.
While Oli was relocking we saw that there are still glitches, both of the attached screenshots were taken while the IFO was in the engage ASC state.
Seeing this and based on Camilla's observation that the locklosses started 80561 on the same day that we redistributed gains to increase range in the IMC servo board in 79998, we decided to revert the change, which was intended to help us ride out earth quakes. Daniel points out that this makes some sense, that it might help to ride out the earthquake if we limit the range available to the carm servo. We hope that this will help us to ride out the glitches better without loosing lock.
Rick went back to the diode room and reset the watchdogs after the laser had been running for about 2 hours.
After going down to swap the NPRO controller (80566), we are back to Observing
As part of yesterday's maintenance, the atomic clock has been resynchronized with GPS. The tolerance has been reduced to 1000ns again. Will see how long it lasts this time.
Wed Oct 09 10:07:43 2024 INFO: Fill completed in 7min 40secs
Gerardo confirmed a good fill curbside. Note new fill time of 10am daily.
Sheila, Camilla.
New SQZ ASC using AS42 signals with feedback to ZM4 and ZM6 tested and implemented. We still need to watch that this can keep a good SQZ alignment during thermalization. In O4a we used a SQZ ASC with ZM5/6, we have not had a SQZ ASC for the majority of O4b.
Prep to improve SQZ:
Testing ASC from 80373:
In the first 20 minutes of the lock, the SQZ ASC appears to be working well, plot.
Note to operator team: if the squeezing gets really bad, you should be able to use the SQZ Overview > IFO ASC (black linked button) > "!graceful clear history" script to turn off the SQZ ASC. Then change /opt/rtcds/userapps/release/sqz/h1/guardian/sqzparams.py use_ifo_as42_asc to False and go though NO_SQZEEZING then FREQ_DEP_SQUEEZING in SQZ_MANAGER and accept the sdfs for not using SQZ ASC. If SQZ still looks bad, put ZM4/6 osems (H1:SUS-ZM4/6_M1_DAMP_P/Y_INMON) back to when squeezing was last good and if needed run scan sqz alignment and scan sqz angle with SQZ_MANAGER.
Sheila moved the "0.01:0" integrators from the ASC_POS/ANG_P/Y filters into the ZM4/5/6_M1_LOCK_P/Y filter banks.
This will allow us to more easily adjust the ASC gains and to use the guardian ZM offload states. We turned them on on ZM4/6. Edited OFFLOAD_SQZ_ASC to offload for ZM4,5,6. And tested by putting an offset on ZM4. We put ZM4/6 back to positions they were in in lock via the osesms. SDFs for filters accepted. I removed the "!offload AS42" button from the SQZ > IFO ASC screen (liked to sqz/h1/scripts/ASC/offload_IFO_AS42_ASC.py) as it caused a lockloss yesterday.
Oli tested the SQZ_MANAGER OFFLOAD_SQZ_ASC guardian state today and it worked well. We still need to make the state request-able.
ASC now turns off before SCAN_SQZANG_FDS/FIS and SCAN_ALIGNMENT_FDS/FIS. It wil check if the ASC is on via H1:SQZ-ASC_WFS_SWITCH and turn the asc off before scanning alignment or angle.
We changed the paths so that to get from SCAN_SQZANG_FDS/FIS and SCAN_ALIGNMENT_FDS/FIS back to squeezing, the guardian will go though SQZ_ASC_FDS/FIS to turn back on ASC afterwards.
-Brice, Sheila, Camilla
We are looking to see if there are any aux channels that are affected by certain types of locklosses. Understanding if a threshold is reached in the last few seconds prior to a lockloss can help determine the type of lockloss, which channels are affected more than others, as well as
We have gathered a list of lockloss times (using https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi) with:
(issue: the plots for the first 3 lockloss types wouldn't upload to this aLog. Created a dcc for them: G2401806)
We wrote a python code to pull the data of various auxilliary channels 15 seconds before a lockloss. Graphs for each channel are created, a plot for each lockloss time are stacked on each of the graphs, and the graphs are saved to a png file. All the graphs have been shifted so that the time of lockloss is at t=0.
Histograms for each channel are created that compare the maximum displacement from zero for each lockloss time. There are also a stacked histogram based on 12 quiet microseism times (all taken from between 4.12.24 0900-0930 UTC). The histrograms are created using only the last second of data before lockloss, are normalized by dividing by the numbe rof lockloss times, and saved to a seperate pnd file from the plots.
These channels are provided via a list inside the python file and can be easily adjusted to fit a user's needs. We used the following channels:
After talking with Camilla and Sheila, I adjusted the histogram plots. I excluded the last 0.1 sec before lockloss from the analysis. This is due to (in the original post plots) the H1:ASC-AS_A_NSUM_OUT_DQ channel have most of the last second (blue) histogram at a value of 1.3x10^5. Indicating that the last second of data is capturing the lockloss causing a runawawy in the channels. I also combined the ground motion locklosses (EQ, Windy, and microseism) into one set of plots (45 locklosses) and left the only observe (and Refined) tagged locklosses as another set of plots (15 locklosses). Both groups of plots have 2 stacked histograms for each channel:
Take notice of the histogram for the H1:ASC-DC2_P_IN1_DQ channel for the ground motion locklosses. In the last second before lockloss (blue), we can see a bimodal distribution with the right groupling centered around 0.10. The numbers above the blue bars is the percentage of the counts in that bin: about 33.33% is in the grouping around 0.10. This is in contrast to the distribution for the observe, refined locklosses where the entire (blue) distribution is under 0.02. This could indicate a threshold could be placed on this channel for lockloss tagging. More analysis will be required before that (I am going to next look at times without locklosses for comparison).
I started looking at the DC2 channel and the REFL_B channel, to see if there is a threshold in REFL_B that can be put for a new lockloss tag. I plotted the last eight seconds before lockloss for the various lockloss times. This time I split up the times into different graphs based on if the DC2 max displacement from zero in the last second before lockloss was above 0.06 (based on the histogram in previous comment): Greater = the max displacement is greater than 0.06, Less = the max displacement is less than 0.06. However, I discovered that some of the locklosses that are above 0.06 for the DC2 channel, are failing the logic test in the code: getting considered as having a max displacement less than 0.06 and getting plotted on the lower plots. I wonder if this is also happening in the histograms, but this would only mean that we are underestimating the number of locklosses above the threshold. This could be suppressing possible bimodal distributions for other histograms as well. (Looking into debugging this)
I split the locklosses into 5groups of 8 and 1 group of 5 to make it easier to distinghuish between the lines in the plots.
Based on the plots, I think a threshold for H1:ASC-REFL_B_DC_PIT_OUT_DQ would be 0.06 in the last 3 seconds prior to lockloss
Fixed the logic issue for splitting the plots into pass/fail the threshold of 0.06 as seen in the plot.
The histograms were unaffected by the issue.
Added code to the gitLab
Relocking was automatic but again the SQZ FC couldn't tranistion to IR. Vicky came to rescue and was able to get it working. Something I should have seen earlier is that there was a dolphin crash, so the ZMs were way off from their nominal position. Notes from Vicky:
Accepted SDFs for SQZ ZMs/FC2 and LSC-LOCKIN_OSC_FREQ
Sheila, Camilla
Trending back what happened last night:
To avoid this in future:
Tagging OpsInfo: if you run SCAN_SQZ_ALIGN and don't like it, you'll need to trend back ZM4/6 optical align sliders as they aren't in sdf.
Also you need to turn off SQZ ASC before running scan_sqz_alignment, updated instructions in wiki.<- added this the the guardian.While looking at this we also decided to remove SQZ_FC state LOCKED (nominal is IR_LOCKED we don't see a need for a state that can be GREEN_LOCKED or IR_LOCKED).