CDS Maintenance Summary, Tuesday 10th September 2019

WP8341 Remove old ext_alert service

Jonathan, TJ, Dave:

I removed the old ext_alert service on the ext-alert machine. This was being managed by systemd. The removal process is:

systemctl stop ext_alert.service

systemctl disable ext_alert.service

mv /etc/systemd/system/ext_alert /etc/systemd/removed_systems

systemctl deamon-reload

systemctl reset-failed

The remaining systemd services are ext-alert-credentials.service and lv_alert.service (note we have not been too consistent with underscores and dashes).

WP8347 SDF of ALS shutter settings

Dave:

h1sysecat[x,y]1plc1sdf systems were modified to SDF the end station ALS shutter settings. Please see related alog for details.

These SDF systems were restarted, and the new channels were ACCEPT+MON

BRSX Recentered, still rung up

Fil and I went to EX to look inside the BRSX enclosure in prep for the work next month. While we were there I touched up the centering of the BRS. While we had the box open, we must have disturbed the cable into the interface box inside, because when we got back to the corner station the BRS wasn't damping down. I went back down to look at it and it took a little to realize there is not retention on the ethercat cable into the interface box, so it's really easy to knock loose. It immediately started damping down when I pushed the ethernet cable into the interface chassis.

The BRS is still calming down, so we shouldn't use it for a little while yet. The operator should watch the the BRS health ndscope and wait for the BRSX RY OUT to look more like the BRSY RX OUT signal, i.e. the blue trace on the middle timeseries looks like the yellow(?) trace on that same plot.

Opened FRS Ticket 13570 to track that H1 BRS EX ethernet connector at interface box needs to be repaired.

Shift Summary - Maintenance Day

TITLE: 09/10 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Niko
SHIFT SUMMARY:

Maintenenace went well. Squeezer Table work went a litle long due to a cable issue. Recovery locking has been unsuccessful for the remainder of my shift. Handing of to Niko.
LOG:

15:00 Bubba out to move bees

15:01 Sensor Correction turned off

15:01 Karen out to EY

15:02 Richard and Ken out to LVEA then out to X-Arm to survey for future PEM work

15:03 Jeff B to outbuildings to check Dust monitor pumps, HEPI pumps and TCS

15:08 PCal team to EX

15:25 Vanessa out to LVEA

15:40 Fil out to LVEA

15:45 TJ to LVEA

15:52 TJ out

16:01 Bub, Ty, and Chris out to LVEA ot start craning

16:06 Rich and Ken back from x-arm

16:08 Jeff B back

16:12 Karen headed back

16:!6 H1 unlocked - holding t DOWN for teh duration of maintenance

16:21 Norco on site - dewar 75 Y-Corner

17:00 Jim out to LVEA to deliver parts to HAM6 area

17:16 Hugh and contractor out to EX for Wind Fence

17:16 EX LASER SAFE - PCal team done

17:21 TJ out to LVEA for TCSY investigation

17:41 Richard out to LVEA for physical measurements

17:49 TJ back

18:00 Ethan back from a return trip to EX to disable WAP

18:35 Jeff K out to LVEA to check on maintenance progress

18:37 Bubba is back - Chris and Tyler still out

18:46 Chris and Tyler back

18:49 Jason running rotation stage calibration script - about 5 minutes

19:07 Begin Initial Alignment

19:15 Hugh back

19:09 Sensor correction turned back ON

19:27 Initial Alignment finished - Re- locking attempt 1

19:49 Fil back

20:?? Jim and Fil out to EX during EQ TOO for BRS tuning

21:05 Daniel back

21:35 Initial Alignment

22:20 Kyle to X-mid

 

script for restoring all sliders

Patrick Thomas wrote a script that restores alignment sliders to a time in the past.  Dave showed me this today, when we wanted to restore to before the IMs were moved on Sunday (see Corey's alog and comments 51837), and we tested it.  We added some rounding to avoid having the 10^-15 SDF differences, but the script is working.  I'm just writing it here because it is something that could be useful for operators occasionally.  

/opt/rtcds/userapps/release/sus/h1/scripts$ python restore_suspensions.py 1251771354

It sounds like Dave and Patrick have plans to make this a button that can be on an medm screen and allow the user to type in a gps time. 

Thanks Patrick!

LVEA Swept

The LVEA was swept at 1430 local time. Kyle was still in there but promised to turn the lights off when he left.

And he informed the operator that he did so upon departure. The lights are indeed OFF.

Replacing heliax cable, feethrough and SQZ AOM1

Kara Sheila Fil Daniel

Continuing replacing cabling for the CLF AOM1, old alog 51537.

We replaced the feedthrough and the helix cable to AOM1 of teh CLF. First, we saw the RF power monitor coming back to the old value of 33.5dBm. After a minute or so it went down to 5dBm. Most likely the RF amp blew. Unclear why, but after replacing it we carefully measured the RF power delivered to the AOM. We needed an additional 3dB attenuation to get back to the previous value of 34dBm, see alog 40071. Maybe the cable was defective and we got more power than the AOM could handle? Assuming the AOM potententially has a problem, we swapped it with the spare and reconnected. A lot of alignment tweaking was required to get back to the previous difraction power. All seems to be working again.

We made the following measurements of the power on ISCT6:

After 1st periscope mirror: 3.32mW
Out of vacuum: 3.32mW
After 2nd periscope mirror: 3.25mW
Input to cube: 3.24mW
Output of Cube: 3.16mW
wrong polarization: .09mW
Front of Detector: 3.11mW

We made the follow power measurements on the Squeezer table:

OPO REFL: 3.06mW and 1.28v
Output monitor: 32.6dBm
Drive into AOM: 33.7dBm

3 Lock Losses during CARM Reduction (CARM_ON_TR)

J. Kissel, E. Merilh

Just before a big EQ in Alaska/Canada took us out of maintenance recovery lock acquisition attempts, we had 2 lock losses at / around the start of the CARM reduction sequence. 

Grabbing some plots from the lockloss tool (though the web server appears to be down): it appears as though there's some positive feedback that triggers around this step, ringing up and causing the locklosses. At the moment, I only have POP_A_LF, and the arm powers in red (ASC-[X,Y]_TR_NSUM) showing this symptom, but it's worth looking in to, and I will. Help would be appreciated, though.

Plots are of the three lock losses, attached in chronological order:
    UTC time                GPS Time
    2019-09-10 10:47:42     1252147680
    2019-09-10 20:00:11     1252180829
    2019-09-10 20:26:58     1252182436

Continued preparations in the LVEA for the upcoming Turbo Station installations

For this week's Tuesday, 4-hour maintenance break, I cut-to-length, dressed and terminated the individual wires with the correct crimp sockets on the turbo-end of the signal cable for the Vertex Turbo Station and then installed the CPC connector.  I also cored a hole in the LVEA floor and installed the sole remaining (4th of 4) threaded adhesive anchors for the Vertex Turbo Station.  Prior to the coring operation, I had to use the portable band saw to cut/remove a section of the existing Turbo's support stand weldment so as to provide clearance for the coring drill head, 

PSL Rotation Stage Re-calibrated

Since I was unable to get the PMC transmitted power back to where it was at the start of the run, I re-calibrated the PSL/IO rotation stage.  I followed the Jenne's instructions in alog 30479 and updated the PSL rotation stage Calibrate MEDM screen.  The only changes were the input power went from 49 W to 50.07 W and the minimum power angle changed from -25.2 to -24.78.  I've attached a screenshot of the data fit plotted in Matlab and the new rotation stage calibration screen.  These values are monitored in SDF, so I accepted these changes in the safe.snap SDF file (the OBSERVE.snap file did not register the change, so I'm guessing these channels are unmonitored in OBSERVE?).  No change was seen in the low power state the rotation stage is currently in, so hopefully when we request 37W into the IFO we are closer to 37W than we have been; we'll see how close this actually is as the IFO recovery from today's maintenance plays out.

PSL PMC Tune-Up Work Today & Power Watchdog Reset (FAMIS 10727)

In an effort to get more power out of the PMC, with the ISS OFF I tweaked the beam alignment into the PMC.  Unfortunately I was unable to get any more power out of the PMC.  Since the alignment tweak didn't help, I adjusted the operating currents and temperatures of the pump diodes for the 35W FE and the 70W amp (with the ISS still OFF).  The thinking here is that since the power drop out of the PMC is not due to alignment, it's likely due to changes in mode content caused by the normal deterioration of the pump diodes.  The changes are summarized in the below table.

	35W FE				70W Amp
	Current (A)		Temperature (°C)		Current (A)		Temperature (°C)
	Old	New	Old	New	Old	New	Old	New
D1	47.5	47.7	26.3	26.3	43.2	43.3	28.0	28.0
D2	47.5	47.7	22.3	22.3	43.2	43.3	21.8	22.0
D3	44.5	44.7	26.8	26.5	38.5	38.7	21.8	22.0
D4	44.5	44.7	26.8	26.5	38.5	38.7	21.8	21.7

I've attached screenshots of the relevant PSL Beckhoff screen for future reference.  After the above changes the 35W FE is outputting 32.4 W (was outputting 32.0 W before I started) and the 70W amp is outputting 68.0 W (was outputting 67.1 W before I started).  I should mention that the 70W amp power is as read by the old HPO power monitor PD (H1:PSL-PWR_HPL_DC_LP_OUTPUT).  I'm using this PD as I think something isn't quite right with the 70W amp power monitor PD; it did not register any changes in output power as I adjusted the 70W amp pump diodes (I'll look into this next time I'm able to go into the enclosure).

This adjustment increased both the transmitted and reflected PMC powers, but increased the transmitted more than the reflected (so moving in the right direction).  I re-tweaked the alignment to try to lower the reflected power to no avail.  At this point I think I will need to go into the enclosure to tweak the PMC up in order to impove the reflected power.  All things said and done, with the ISS ON we now have 53.8 W transmitted through the PMC and 10.8 W reflected (when I started it was 52.9 W transmitted and 10.6 W reflected).  So all in all, not too bad.  Managed to get almost 1 W back in transmitted power and only increased the reflected power by 0.2 W.  Because of this change in PMC output power I had to adjust the ISS RefSignal from -2.02 V to -2.04 V, which now has the ISS diffraction % at ~2.1%; this change was accepted in both the safe.snap and OBSERVE.snap SDF files.

In addtion, I reset both PSL power watchdogs at 17:52 UTC (10:52 PDT), thereby completing FAMIS 10727.

Regular Reconciliation of some safe.snaps in SDF System; SUS, ISC, and CAL

J. Kissel

Browsing through SDFs compared against our recent commissioning activity, I've reconciled a few safe.snaps this morning:

H1SUSETMX: 
- accepted L2_DRIVEALIGN_A2L matrices as 1.0 (i.e. permanently turning ON the L2A decoupling filters, recommissioned with new frequency-dependent matrix topology); see LHO aLOGs 51717, 51709, 51604, and 49825.

H1SUSETMY:
- Accepted ETMY PUM A2L spot position gains as 4.3 and 3.6.
H1SUSITMY: 
- Accepted ITMY PUM P2L spot position gain as -3.0
H1SUSITMX:
- Accepted ITMX PUM P2L spot position gain as -3.965
These are the "FULL POWER" positions, according to the current revision of 
${userapps}/release/isc/h1/guardian/lscparams.py
     The "CENTERED" positions, at which we now "start" the ADS loops (see last paragraph of LHO aLOG 51715, which is a shortened version of the dance around the point absorber position, discussed in LHO aLOG 51426), don't get reset (from FULL POWER) until just before the ADS loops are turn on in PREP_ASC_FOR_FULL_IFO, quite a ways along in the lock acquisition sequence. Thus, after a lock loss, and even after the DOWN state is run, the A2L gains will remain at their FULL POWER position, and not be reset. Apparently, some of the FULL POWER A2L gains had been accepted in some safe.snaps, but not consistently in all.
     This has now been remedied. In reality, since the spot position gains *are* set by guardian *eventually* before they're used, this dosn't really matter, but it reduces the number of SDFs which are noise when reconciling in the future.

H1LSC:
- Accepted the filter configuration needed for the improved PRCL FF filter (accepting the configuration shown in the OBSERVE.snap as an attachment to LHO aLOG 51553).
- Also accepted PRCL FF TRAMP to be 3 seconds, as was apparently commissioned on Aug 28 2019, though now appears to be in guardian as well, so again, just reducing noise here.

H1SUSPRM:
- Reverted SUS-PRM_M1_DITHER_P_TRAMP from 10.0 to 0.0. Now inconsequential tramp turned on for a brief test back on Aug 29th. Reverting the tramp to 0.0, since it's otherwise been unused for the entire run. #AlarmNoiseReduction

H1CALCS:
- Accepted values of new Calibration Reference Model Values at Calibration Line Frequencies (aka "EPICS RECORDS", installed and accepted in yesterday's install; see ), but these are different only at the sub-double precision level -- so these diffs will continue re-appear until we round these numbers.
I'll take this as an action to fix the precision at which we write these values.

There are several other unknown differences that may be maintenance / commissioning related activities currently on-going, so we'll take a look after we've re-run the DOWN state at the *end* of maintenance today.

LSC feedforward: rooms for improvement

As the attached shows, there are many peaks/bumps in DARM that are highly coherent with LSC auxiliary control outputs. Low frequency behavior is not good either. I used a glitch-free segment of about 2000 seconds in observing. (It's yet to be seen if the noise transfer function would be stable over longer time scale.)

Anyway, this could be improved. Circled in cyan are the ones that look easier than the others.

BS: 17.8Hz, 32.2Hz and a tiny bump right next to it, ~40Hz.

PRCL: 40.9Hz, a tiny bump around 108.9Hz.

SRCL: ~25Hz

All: Behavior for f<15Hz or so is not good and I believe could be improved without making lower frequency behavior much worse.

Note that 17.8Hz peak which is next to EX L3 17.6Hz calline is BS bounce (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=49643). We should move the band stop from MICH filter to SUS-BS, that way it's easier to finesse the MICH feed forward at this frequency than e.g. using PRCL as a poor man's sensor for feed forward.

I don't have time to work on this but I'll work with Kara/Jenne. This kind of work has been done routinely before, but not in O3, it's just that obtaining long glitch-free observation segment has been difficult in O3.

 

Tagging @DetChar, in hopes that it reaches the CW group.

Bi-Monthly Dust Monitor Vacuum Pump Checks

   Checked all three (End-X, End-Y, & CS) dust monitor vacuum pumps. Both end station pumps are running great. There temps are all in range and there is no accumulation of carbon dust on the mufflers. 

   The CS pump (which was rebuilt on 09/04) new vanes have broken in well and the pump is running with low temps and no carbon dust accumulations. Made an adjustment in the air bypass to bring the vacuum pressure to 18inHg. This adjustment is normal for a newly rebuild pump. 

Back To OBSERVING After Almost 7-hrs

Now that we are back to Observing, figured I'd post locking notes from the night.

Summary:

• DOWN time:  4:39-11:28utc (almost 7hrs)
• Took 8 Lock attempts & 3 Alignments (my long alignment was not needed though!)
• Due to the down time, we should probably still back out the IM alignment change from Sunday night during the next lockloss.

Locking Notes:

• 7:07 Niko's LOCK#3:  Lockloss at INCREASE POWER after the first step in powering up.
• 7:08 - 7:26 My LOCK #1:  Lockloss at ENGAGE SOFT LOOPS
• 7:30-10:08 LOCK#2:
  • After several attempts at locking with the new IM pointing from Sunday night, decided to back IM pointing out.
  • 7:33 Restored IM1, IM2, IM3 to where they were before the last lock.  (settings restored from SDF list here).
  • 7:35-9:37 Starting a 3rd Initial Alignment (because there is a new input pointing)
    • Notes/Issues from Initial Alignment are in this alog.
  • NOTE:  getting "IMC WFS not centered" messages for IMC LOCK.  Observe this in all the space it takes up in the ISC LOCK log.
  • DRMI did not lock.  PRMI locked & then DRMI locked quickly afterward.
  • 10:08 Lockloss at ENGAGE SOFT LOOPS:  this happened fairly quickly and before the "usual" ITMy point absorber manuever which has the "usual" OMC & all-test mass Saturations noted by Verbal.
• 10:09-10:34 LOCK #3:
  • DRMI locks after 7.5minutes w/ no operator assistance
  •  Our new "Usual Point Absorber Manuever":  Verbal talks to me about a saturating OMC & all test masses, and then goes quiet as it moves our beam spot from an Initial Alignment spot to locking spot (Verbal gets vocal as we traverse the point absorber on the ITMy).  This is our new normal.  Oh--you will also see DHARD Pit get rung up during this traverse, but it will come down, too.  And I believe we also get the guardian note of "OMC not ready" during this maneuver as well.
  • Lockloss at INCREASE POWER (this is after the first "step" for increase power, sort of what Niko and I saw for his last 
• 10:35-10:47 LOCK#4:
  • DRMI locks in under 5min
  • Lockloss at CARM TO TR
• 10:49 LOCK #5:
  • DRMI locks in about 9min
  • Went directly to NOMINAL LOW NOISE
  • Had a long list of Diffs for CALCS (accepted all of them & snapshot is attached)
• 11:28 Back to OBSERVING!

J. Kissel

Regarding Corey's comment:
    NOTE:  getting "IMC WFS not centered" messages for IMC LOCK.  Observe this in all the space it takes up in the ISC LOCK log.

This is a known issue/problem that is not resolvable without a hardware change: check out my aLOG from tail end of July: LHO aLOG 50920, and Daniel's subsequent reference to 5109.

To quote my aLOG: "This message appears, typically, after PSL incursions (rare), site power outages (rare), or computer failures (rare) when the IMC suspensions's or PSL Periscope PT alignments get lost. Because these events are rare, instituional memory loss causes confusion for folks when they see that error message and wonder if action is needed. However, once these alignments are roughly recovered (via reseting sliders on suspensions), the WFS, again typically, eventually recover their centering all on their own as the RF loops converge [there are no "DC centering" loops on IMC WFS, as there are for many of the the WFS systems]."

However, last night, none of these things happen, and the IM alignment is all down stream of the IMC WFS. So I'm equally confused as to why the WFS got outside of their range. Worth trending.

But in short -- while these warnings are annoying -- they are reflecting of a real issue, so don't ignore them.

CAL CS differences are due to poor rounding of the installation of Calibration Model Reference Values art Calibration Line frequencies. Will work on resolving this later today.

OPS Eve Summary

TITLE: 09/09 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: dropped lock during EQ, relocked with small changes to IMs, in Observe
LOG:

• 04:20:17 - lockloss, SEI was in EQ, DCPD and EX verbal alarms
• plot attached shows that bounce mode for ETMX increased right before lockloss
  • IX and EX bounce were hovering arounf 1.0, and about 8 seconds before lockloss EX bounce shot up to 1.7
• first attempt at relock ended before DRMI
• second attempt I made some small changes to IM1, IM2, IM3
  • maximized to a smooth signal POPAIR_B_RF18_I_NORM_MON, minimized POPAIR_B_RF18_I_NORM_MON, maximized LSC_REFL_A_LF_OUT16
• third attempt didn't survive ENGAGE SOFT LOOPS
• fourth attempt went to NLN
  • range is around 119Mpc

These IM alignment slider positions have been reverted -- see LHO aLOG 51858.