TITLE: 02/09 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 119Mpc
INCOMING OPERATOR: Camilla
SHIFT SUMMARY:
LOG:
0:18 Out of observe for Robert at EX
0:34 Back to Observe
6:44 Switch SEI_CONF to Earthquake for Papua New Guinea 6.0 earthquake
7:14 SEI_CONF back to Windy
TITLE: 02/08 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY:
LOG:
Switching off the gains for the following SUS, to see if we are over damping them. The current monitor levels are below 0.5, if it rises above this value then I will turn them back ON.
1. ITMY - MODE 3, 14
2. ITMX - MODE 11
3. ETMY - MODE 15
4. ETMX - MODE 17
TITLE: 02/08 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ed
SHIFT SUMMARY: Locked until 10 minutes before Ed takes over! Microseism and wind and some ground motion maybe were too much for H1. Handing off with green arms in INCREASE_FLASHES.
LOG:
TITLE: 02/08 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Camilla
CURRENT ENVIRONMENT:
SEI_CONF state: EARTH_QUAKE
Wind: 13mph Gusts, 8mph 5min avg
Primary useism: 0.23 μm/s
Secondary useism: 0.65 μm/s
QUICK SUMMARY:
Happened a third time: 2020-03-17 05:04:09 UTC. Still much reduced since the chiller swap.
Looked back at this after TJ suggested we should check that both chillers are keeping constant temperatures and flow rates. Temperatures in ITMX and ITMY cCO2 chillers seems to be stable. See plot over last 2 years here. As the laser hasn't ben on recently we cannot check how stable it is.
Flow rate reported by ITMX chiller seems to regularly drop from the nominal 3.8 to 2 GPM to up to 10 times an hour. However these drops only last one data point which doesn't effect the laser controller safety controls. See zoomed plot here. The chiller has been like for the past 3.5 years (as far back as I can easily see in EPICS).
2021/09/21 Tuesday 17:30UTC Both ITMX and ITMY chiller temperatures decreased from 22 degC to 19.5 degC. The set point is still at 22degC but the channel H1:TCS-{ITMY/ITMX}_CO2_CHILLER_OUT_GAIN_OUT16 dropped to zero Simulink for h1tcscs . Ndscope trace here. This apears to be the day the h1tcscs model was installed on h1oaf0 - see alog 60002. Tagging cds.
ECR for upgrading the CO2 controller is here: https://dcc.ligo.org/E1600312
Latest concept for redesigning CO2 controller is attached. Project is stalled right now due to lack of manpower.
Daniel fixed this setpoint problem by finding the filters had been removed (see alog 60660).
Once the filters were added back in ~19.15UTC 11/16/2021, the filters seems to be maxed out. Daniel turned off the laser power servo loop (sitemap > TCS > CO2{X/Y} > LASER > ON/OFF for PZT_SERVO_GAIN) and they behaved as expected. These values are not recorded in SDF. It doesn't make sence for the chillers to be tied to the laser when the laser is off so will investigate if the matrix changed.
The PZT_SERVO_GAIN OUTPUT we turned OFF 2021/11/16 20UTC must have not being svn saved and turned back on on 2021/11/17 22UTC. See attached trend.
The design is to stabilize the laser PZT error signal by controlling the chiller temperature (E1300233p20). When the laser is off this seems to break down and change the have the setpoint given to the chiller (CHILLER_SETPT1) far from the ~20deg setpoint offset H1:TCS-ITM{}_CO2_CHILLER_SET_POINT_OFFSET (seen in TCS CO2 LASER screen). See table below.
TJ and I are looking into these settings. Maybe the matrix gain values or PZT gains are incorrect or the loop should be switched off when the laser is off.
| Current values | CHILLER_SET_POINT_OFFSET | PZT_SERVO_GAIN | CHILLER_SETPT1 (setpoint sent to chiller) |
| CO2X | 20.6 | -111 | 16.6* |
| CO2Y | 20.5 | 55 | 22.4 |
*Documentation states it's important not to let the chillers be +/- 5deg from 20deg otherwise there may be condensation formed in the laser. The chillers have limits to shut off if they get outside this range but only compared to the CHILLER_SETPT1 value.
The PZT_SERVO_GAIN OUTPUT is turned off in the CO2 Guardian's DOWN state (well done guardian), running DOWN fixed the problem.
Now the lasers think the set point given to the chillers is 20deg but the chillers are receiving set point request of 22degC. So another issue to solve.
| Date | UTC | Time | RMSLP_LOG10 | Drive Output |
| seconds | top of first bounce | +/- envelope | ||
| 7 Feb 2020 | 22:35 UTC | T = 0 | 2 | 0.009 |
| 7 Feb 2020 | 23:17 UTC | T = 2561 | 4.32 | 3.4 |
| 7 Feb 2020 | 23:47 UTC | T = 4336 | 4.92 | 9.8 |
| 7 Feb 2020 | 00:51 UTC | T = 8197 | 5.02 | 13.4 |
| 8 Feb 2020 | 01:13 UTC | T = 9500 | 5.21 | 21 |
TITLE: 02/08 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
INCOMING OPERATOR: Camilla
SHIFT SUMMARY: winds were bad
LOG:
0:00 Niko started initial alignment as I came in, winds also started to pick up, got to NLN around 1:00 utc, but winds promptly hit 60 mph and broke the lock, next couple hours were high winds
5:37 back to NLN and observing
TITLE: 02/08 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 19mph Gusts, 13mph 5min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.43 μm/s
QUICK SUMMARY: Locked 2h30m. Microseism is on the rise but the winds have died down considerably since Jim was locking.
[Sheila, Jenne]
Sheila remembered that there are some ASC offsets that change in ASC engagement, and pointed out that those could be affecting our ability to lock. In the attachment we see that the POP18 drops and POP90 comes up a bit when the AS36Q filter banks change their SWSTAT (which is them turning off the offsets).
We put in offsets in the MICH ASC sensor AS36Q when we have DRMI locked before the CARM offset reduction, in preparation for switching from AS45Q to AS36Q. But, once we're engaging the full IFO ASC, we want to get rid of those offsets. Sheila had moved the offset turn-off from PREP_ASC_FOR_FULL_IFO to the end of ENGAGE_ASC_FOR_FULL_IFO in September of last year, but maybe we should move it even further down the sequence, to after the arm soft loops have converged.
However, I'm not sure that this is the thing that is killing the lock, since it happens ~80 seconds before lockloss.
For now, we've followed the hands-off procedure and since we've had several locklosses in a row, Jim has just completed initial alignment, and we'll see how this acquisition goes.
TITLE: 02/07 Day Shift 16:00 – 00:00 (08:00-16:00), all times posted in UTC
STATE of H1: Locking
INCOMING OPERATOR: Jim
SHIFT SUMMARY: Did some SQZ troubleshooting, followed by commissioning. Lost lock at 22:40, been struggling to relock since then
LOG:
16:32 (08:32) Chandra to MX/MY -- do an inspection
17:15 (09:15) Leaving Observing to investigate SQZ oscillations
17:29 (09:29) Sheila to LVEA -- continue SQZ investigation
17:37 (09:37) Chandra back from MX/MY
17:55 (09:55) Sheila out of LVEA
18:00 (10:00) Starting scheduled commissioning break
19:27 (11:27) Done with commissioning, going into Observing
22:12 (14:12) Gerardo to MX/MY -- pick up equipment, inspect gauges
22:40 (14:40) Fast lockloss
22:45 (14:45) Gerardo back from MX/MY
22:47 (14:47) Chandra to LVEA
23:00 (15:00) Chandra out of LVEA
23:09 (15:09) Travis to Optics Lab
23:12 (15:12) Travis out of Optics Lab
Keita, Sheila
We had another example of the squeezer oscillations that were seen last weekend: 54864 The first screenshot shows the signals related to gree power in the OPO getting very noisy over ~ 6 hours this morning. We went out of observing earlier than our planned commisioning time to try to address this. We turned off the green ISS, which brough all the signals back to normal. Turning that loop back on brought the noise back. We also tried reducing the gain in that loop, which did reduce the peak to peak noise seen in the ndscope. Based on loop measurements taken on Tuesday, 54888, the shape of this loop at its nominal 31dB gain setting should be fine.
Next I went to the floor, while Keita took spectra and changed the gain from the control room. On the floor I measured a spectrum of IN1 (the error point) on the green ISS, and I saw large lines at 15kHz and 30kHz. Keita saw that there is an aliased line at 1249Hz in the spectra in the control room, which would correspond to a line at 15135 Hz. The height of the 15kHz and 30kHz lines didn't change with gain changes, although the overall noise floor did change. Ketia disabled the loop, and I could no longer see the lines on the error point, and the low frequency intensity noise increased (as you would expect).
We then unlocked the squeezer by requesting no squeezing, and relocked it. We saw no evidence of the 15kHz and 30kHz lines on the analyzer, although the red trace in the second attachment shows that the 1kHz line is still present. We reset the ISS gain to 31Hz, which is nomal, and everything seems fine for the time being.
Photos attached (after screenshots):
For operators:
The first attached screenshot is from the ndscope that is accessible from the upper right corner of the SQZ_OVERVIEW screen, called locking signals. If you see the range dropping, open this ndscope and see if there is trend similar to what is shown in the attached screenshot that lines up in time with the range drop. In the left column of the screenshot you can see that the OPO REFL power is changing and the max-min difference gets larger, the max-min difference of OPO_IR_RESONANCE trigger gets larger, and the SQZ_SHG_LAUNCH power max-min difference also gets larger. If this is happening and corresponds to a range drop, please unlock the squeezer by requesting "NO_SQUEEZING" from SQZ_MANAGER, then once it arrives, request SQZ_READY_IFO. Once it arrives there hopefully the signals in the ndscope will have returned to normal, if so you can request "SQUEEZING" again. If not you can try again or call for help.
Tagging OpsInfo to propagate the message
During today's calibration time, this oscillation happened again, and Patrick called me so I could make a few more observations.
Richard, Fil, Rahul, Dave:
The EX hardware watchdog (HWWD) starting having periods of low led-monitor signal warning (STAT=8) starting 12th Nov 2019. The transitions between periods of low signal and normal signal occur mostly on Tuesdays. Attached plot shows a 6 month trend of the EX-HWWD STAT channel. Prior to 12th Nov there was little noise, on that date the signal went noisy. The transition back to no noise was Tue 19th Dec 2019, then noisy Tue 07 Jan 2020, then to a short period of low noise Tue 04 Feb 2020. For these transitions there is evidence that the monthly charge measurement may be the trigger. The only transition which does not fit this theory is on Wed 05Feb when it went noisy again at noon local, at this time H1 was in observe.
We will work with Rahul on the next charge measurement in March to see if the LED-mon voltage is indeed varying.
This is not a problem per-se (for example EY HWWD is much more noisy). However it would be nice to know why the HWWD changed its behaviour around 12th November 2020. A look at the alog says that NCAL and PEM PWR work was ongoing in EX at that time.
[Sheila, Jenne]
We did a few tests of changing the DARM offset, to see if there are any notable differences in the noise. In particular, we're looking to see if there are noise terms like RIN from 9MHz that scale with the DARM offset. Because of this test, the front end calibration is not correct during this time, and so the range reported in the control room is not correct. The offline calibrated data takes into account the calibration lines, and will correct for these changes, so range on the summary pages should be fine to trust.
Attached is a screenshot of an ndscope showing the DARM offset changes, and the resulting online (incorrectly calibrated) range calculation.
Analysis of the data is forthcoming, I just wanted to let people who wander into the control room (or our screenshots) what is going on.
We may need to repeat this measurement with the squeezer off, since changing the DARM offset seems to affect the squeezing that we see.
In the attached screenshot, the top plot is whitened DARM, with the time dependent correction applied, and the bottom is SRCL coherence with DARM. Green is when we had a smaller-than-usual DARM offset, and red is when we had a larger-than-usual DARM offset. I checked that the CFTD time dependence correction is working by checking that the calibration lines are at the same height in both DARM traces (they aren't the same height if you look at the pre-correction CAL-DELTAL that we plot on the front wall). So, the differences in the 2 DARM traces are real.
The high frequency classical noise shouldn't have been affected by changing the DARM offset. Probably the squeezing angle shouldn't have been either, if there were no weird effects. But, perhaps changing the DARM offset is changing the SRCL detuning, which would then change the optimal injected squeezing phase angle.
At low frequency, the extra noise in the green trace is likely mostly due to the excess SRCL control noise in DARM, as seen by the increased coherence. since the SRCL FF is tuned only for our usual DARM offset.
All that said, since squeezing shouldn't be affecting the noise between 10-30 Hz, it is still probably worth while to subtract away that SRCL noise to see if there is any evidence of 9 MHz noise change (or other noise change) when we changed the DARM offset.
Laser Status:
Front End Power is 32.4W (should be around 30 W)
70W Output Power is 70W
Front End Watch is GREEN
70W Watch is GREEN
PMC:
It has been locked 9 days, 22 hr 37 minutes (should be days/weeks)
Reflected power = 11.7Watts
Transmitted power = 51.9Watts
PowerSum = 63.6Watts.
FSS:
It has been locked for 0 days 16 hr and 3 min (should be days/weeks)
TPD[V] = 5.367V (min 0.9V)
ISS:
The diffracted power is around 2.5%
Last saturation event was 0 days 16 hours and 2 minutes ago (should be days/weeks)
Possible Issues:
While watching the lock acquisition, I was surprised at how long we were in the state MOVE_SPOTS before the A2L gains are actually changed. Looking at the code, it is because there are some sleep statements in there for initializing the TMS centering servo that don't need to be in there.
Recall that we use a servo (that Craig implemented over the October break) to move the TMSs such that we are centered on the IR trans QPDs, and maintain that centering as we engage ASC and change the IFO alignment. The first time the servo is used in ENGAGE_SOFT_LOOPS, we are careful to zero the offsets of the TMS TEST filter banks carefully, if they were somehow left on (they get turned off in DOWN). Then a cds.servo is used, and when we leave the state we leave the TMS in its new pointing.
The servos are started up again when we go to MOVE_SPOTS. But, we seem to be resetting the pointing of the TMS before starting the servos up. The way the reset code was written, it waits 30 seconds to ramp off each TMS offset, but it does it in series, for a total of 2 minutes of sleeping. We shouldn't be resetting the offsets here, because that erases the centering that was done earlier, so I commented out this whole section which also removed the sleeps.
It's possible that the IFO liked this 2 min wait to get some time for convergence and thermalization after going up to 10W. If there are problems at this state, please uncomment lines 3611-3621 in ISC_LOCK to put these sleeps back in (if this is needed, I'll fix them on ~Tuesday so that they are timers and not sleeps, since we're not checking for lockloss during each of these 30 sec sleeps).
Separately, we should probably be resetting the A2L gains to their centered locations in PREP_FOR_LOCKING, rather than waiting until PREP_ASC_FOR_FULL_IFO, since we engage DHARD before prep_asc_for_full_IFO. Seems to not be a big deal, but we are jerking that around on poor DHARD when we set them to their centered values just before turning on the dither lines, so we're changing the amount of DHARD noise injected to DARM.
The lines to uncomment are now 3616-3626 if needed, since I added a few lines in PREP_FOR_LOCKING to center the A2L gains (see alog 54950).
We lost lock once in move spots today (after Jim did an initial alignment). I serached on the lockloss page for locklosses at move spots (506). We've had two lcoklosses here since this change was made, and we had 3 locklosses from move spots in January. I rearranged the order of these guardian states on Jan1st, 54219, which is how some of these pauses got in here. Since we've only had two locklosses, and we made it though for now, I'm not going to make a change, but if we keep having locklosses here it might be that we do need to wait for the TMS servo.
The lockloss was very fast with no obvious channel saturations, so maybe it was just a coincidence that it happened around the time of the (small and definitely survivable) EQ.