TITLE: 06/15 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
INCOMING OPERATOR: Travis
SHIFT SUMMARY: locked in Observe entire shift
Locked and almost no glitches in the last 2 hours, winds are below 10mph accross the site.
TITLE: 06/15 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
Wind: 8mph Gusts, 7mph 5min avg
Primary useism: 0.06 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY: locked and in Observe for 6+ hours
TITLE: 06/15 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 51Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
Quiet shift. Nothing eventful. Handing off to Cheryl.
LOG:
This afternoon we spent about 3 hours struggling to lock ALS.
At 20:45 UTC we lost lock while transitioning to Earthquake mode (though the ground motion wasn't really that high)
While reacquiring we kept losing lock while trying to lock ALS. We saw a few different problems:
In conclusion: for some reason we don't know, we couldn't lock ALS for a couple of hours this afternoon. Then for some reason we don't know, we could lock it again.
very similar symptoms now, July 8th 2019 around 23:00 UTC
The problem yesterday evening and probably at the earlier time as well was that the COMM beatnote strength was too low (less than -10dBm). We fixed this by moving a pico in HAM3. For now I've added a check to the ALS_COMM guardian so that it will give a notification and not try to lock if the comm beatnote is less than -9dBm.
During O1+O2 operators used to adjust the alignment of PR3 when the X arm was locked in green to increase the beatnote strength for COMM, we stopped doing this because we thought that PR3 is actually more stable when left alone, and that it might be better to just move the pico when the beatnote gets misaligned, which is what is done at LLO.
Corresponds to FRS Ticket 13231, which has now been closed as per above course action.
While watching operators relock today there were two things that we changed to help make future re-locking more reliable. (This wasn't commissioning, we just addressed some existing problems with the locking while we were relocking).
New state for green arm guardians:
Offsets in AS36 for DRMI ASC:
Relocking after and earthquake last night our range dropped from around 115 to around 107Mpc, this was because a bug in the squeezer LO guardian would under some circumstances not engage the boost in the LO servo .
TJ noted the range drop and Cheryl looked at some alignment changes:
The problem was that there was a check on the power in the 3MHz beatnote that was executed twice, in a way that left us open to unintended configurations. This wrong configuration must have also been accepted in SDF. There was probably confusion because there was also a beckhoff crash from which things were not correctly restored last night.
I've edited the SQZ_LO_LR guardian to avoid this situation in the future. I also simplified the graph for the LO guardian because there were unneeded and potentially problematic connections in it. (screenshot of the changes to the graph attached).
The last attachment shows the change in the DARM spectrum when the LO boosts were off, there is excess noise around 100Hz.
Attached are some SDF diffs we accepted:
I think the PRM coil drivers are in their nominal state so not clear why there is a diff, but also M2 is not used in full lock so it doesnt matter.
The ASC offsets and TRAMPs are from a change Sheila made to the lock acquisition today that she will write an alog about.
(Kyle R, Gerardo M)
The crane was used at Y-Mid to lift the new turbo pump assembly, to aid on drilling 4 holes on the turbo support structure for feet. The crane movement took place during 3 different intervals:
Starting at 20:20 - stopping at 20:45 utc.
Starting at 21:10 - stopping at 21:17 utc
Starting at 21:27 - stopping at 22:15 utc
TITLE: 06/14 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: Travis
CURRENT ENVIRONMENT:
Wind: 16mph Gusts, 10mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.08 μm/s
QUICK SUMMARY:
Commissioners working on ALS
This was a Corrective Maintenance effort. Observatory Mode should have been set as such.
TITLE: 06/14 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Ed
SHIFT SUMMARY: A couple of locklosses early during the first half of the shift. The second half of the shift has been used for Commissioning since ALS has been having issues relocking.
LOG:
19:56 Kyle and Gerardo to MY
22:32 Kyle and Gerardo back
I can see from H1:ODC-OBSERVATORY_MODE that there appears to be about 70 minutes of commissioning today even though in reality there was zero.
Please only use
H1:ODC-OBSERVATORY_MODE = COMMISSIONING [40]
for commissioning.
Don't use it for tasks that are CORRECTIVE MAINTENANCE [53] in nature, e.g. fixing problems when IFO doesn't work, even if the commissioners are working on that problem. That's no different from EE shop trying to identify and eventually replacing broken electronics.
Commissioning is a planned detector improvement project (IFO characterization measurements, noise and/or stability and/or lock sequence improvement etc.). When you find it on a whiteboard for commissioning tasks, that's commissioning.
Lockloss while transitioning to EQ mode. Maybe I waited to long to transition.
Ran through initial alignment after which it relocked without issue.
Some SDF diffs for ASC, PRM, and TCSCS that I accepted. Screenshots attached.
Searches started for before and after lock loss last night Started around 16:32 UTC https://ldas-jobs.ligo.caltech.edu/~ldvw/bruco/patrick.thomas/H1-CAL-DELTAL_EXTERNAL_DQ_2019-06-13-18.30.53-600 https://ldas-jobs.ligo.caltech.edu/~ldvw/bruco/patrick.thomas/H1-CAL-DELTAL_EXTERNAL_DQ_2019-06-14-11.09.47-600
In the second of these reports, the range was low because the squeezer LO servo had no boosts on due to a bug in the guardian code. There is was an extra peak in DARM around 272Hz, but the BRUCO report doesn't show anything very different there between the 2 locks.
Currently talking to Patrick to get it going again.
We restarted the computer twice, but each time PLC3 would not run. All the terminals were in OP and the IOC had started correctly both times. We ended up having to open PLC3, login and start it manually. When TJ did so he said that it reported that the program had changed and asked whether to download the new program. I had him say yes.
h1ecatc1 C1_PLC1_Device3 is under reporting its slave count (actual = 156, configured = 291).
We lost the units around 13:20 UTC (06:20 PDT)
Connection between corner chassis 4 L0 and M0 has failed. The first red block in the topology screenshot is corner chassis 4 M0 (EK1100).
Patrick and Corey are working the problem.
The time for the error Dave mentions above coincides with the H1 Lockloss (and issues with IMC) at 13:21utc (6:21amPDT).
Patrick has tracked this down to the Corner Chasis 4 in the CER and says this may need to get pulled. Fil has been notified and will work with Patrick.
For something like this which might have taken H1 down, should this be a Verbal Alarm? (only way an operator would know about this issue is to happen to catch the red box on the CDS Overview (which we both didn't notice).
[Attached is the RED box on the CDS Overview.]
FRS Ticket submitted: #13049.
https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=13049
So far down time is 3.5hrs, but now I need to try locking.
Side Note: Alog would not let me link the FRS ticket to the text of "13049" above. ![]()
EK1100 EtherCAT Coupler failed and replaced. Unit was scanned and communication errors have cleared. We did noticed a buzzing sound coming from one of the terminals. Tried various combinations of powering off different components to isolate noisy terminal(s), but decided to reinstall unit to get IFO back up. Corner 4 Slow Controls Chassis S1107450 F. Clara, P. Thomas
On the CDS Overview MEDM, if there is a Beckhoff hardware error the slow controls banner will turn RED for better visibility. I simulated a C1PLC1 error to demonstrate (see attachment).
All in all this cost us about 5 hours. See e.g. https://ldas-jobs.ligo.caltech.edu/~detchar/summary/day/20190612/ .
This FRS ticket (13049) is now CLOSED/RESOLVED.
We took a look at several ALS PLLs to check if the ALS DIFF/COMM glitches we saw last night were associated with them. From Plot 1 we see that both the COMM and DIFF PLL control signals glitch strongly at the same time as the ETMY coils. Plot 2: We ran a test where we locked ALS COMM normally and looked for glitches, which we found (red). Then we shuttered the Y arm and the glitches in ALS_COMM went away. Seems that the COMM PLL was partially polluted by Y arm light, which is close in frequency to the X arm light. Solutions being considered are fixing the ISCT1 alignment/changing the polarization of the X vs Y transmission beam, or moving the VCO frequencies. Attachment 3 shows the beat frequencies of the entire ALS system. Seems that the X and Y arm beat frequencies are within 10s of kHz and the X and Y beatnotes could be getting confused on ALS COMM.
Georgia, Keita, Craig Georgia installed a 532 nm half wave plate at ISCT1 in the Y path while Keita and I installed a HWP at End Y to try and reject Y light from the ALS_COMM beatnote. We were unable to eliminate glitches in ALS COMM PLL CTRL.
Note: The half wave plate installed on ISCT1 is on the ALS-Y path, directly before the combining beamsplitter for the ALS-DIFF beatnote, i.e. between ALS-BS6 and ALS-BS7 on this handy layout of ISCT1.
After Keita and Craig installed the half waveplate at end-y, but before I installed the half waveplate on ISCT1, I noted that there was still a substantial beatnote on the DIFF PD. If all had gone to plan the X and Y path on ISCT1 should have been in orthogonal polarisations at this point. With the waveplate installed I could increase the beatnote pk-pk by a factor of 2.
Maybe somewhere along the line (steering optics, vacuum windows?) we are picking up some circular polarisation.
Investigations ongoing.
Just a note that LLO dealt with ALS-DIFF glitches by changing the beam interference on ISCT1, so that instead of interfering the ALS X beam with the ALS Y beam, each ALS arm beam is interfered with the PSL frequency-doubled beam. This is described in LLO log entry 28005.
Associated with FRS Ticket 11113.
Note for future reference:
This problem was caused by electrical cross coupling, not light..
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=43002