Good observing for first half of the shift. No issues or problems to report. There is fog moving in across the site. Not too thick, yet.
The script to generate an alog template for the shift summary is giving errors.
Dec. 3 2019
State: Observing at ~115 MPc.
Summary: One lock loss, cause not immediately apparent. Struggled relocking until I commented out rsp.rs_bootstrap_power('PSL', power) in the LASER_PWR guardian and reloaded it. I can't say for certain if this was what fixed it. We are currently only at 36.7 W, likely because of this change. The INCREASE_FLASHES state seemed to work. I had to manually request it, and I think it might have left the SDF differences that I attached in the previous log.
Log:
01:10 UTC Vlad to optics lab
01:33 UTC Vlad done
03:28 UTC ETMY violin mode 2 has climbed to around 3. Clicked out of observing. Reloaded VIOLIN_DAMPING guardian as per Cheryl's request. Set VIOLIN_DAMPING guardian to DAMPING_ON_SIMPLE. Set ETMY mode 2 gain to -2. Set VIOLIN_DAMPING guardian to IDLE and then to DAMPING_ON_DC.
03:32 UTC Clicked back to observing
04:43 UTC Lock loss
04:50 UTC Green arms are not locking. Trying INCREASE_FLASHES on both arms.
04:58 UTC INCREASE_FLASHES seems to have worked. ISC_LOCK is moving to LOCKING_ALS.
05:06 UTC ISC_LOCK moved to ACQUIRE_PRMI automatically
05:06 UTC PRMI_LOCKED
05:18 UTC ENGAGE_ASC_FOR_FULL_IFO, OMC saturated many times
05:28 UTC Lock loss from INCREASE_POWER
ISC_LOCK had to go to ACQUIRE_PRMI again
06:05 UTC Lock loss from LOWNOISE_ASC
06:10 UTC IR not found. Adjusted COMM and DIFF offsets by hand
DRMI locked without going to PRMI
06:42 UTC Lock loss from LOWNOISE_ASC
06:48 UTC Commented out rsp.rs_bootstrap_power('PSL', power) in the LASER_PWR guardian and reloaded it
07:19 UTC NLN
07:21 UTC Observing
07:21 UTC
I was getting suspicious that the adjustment of the rotation stage at the end of each step in power might be leading to lock losses, so I commented out rsp.rs_bootstrap_power('PSL', power) in the LASER_PWR guardian and reloaded it. The next attempt at locking succeeded, although we are only at 36.6 W. I hope this doesn't mess with the calibration?
I reverted the SDF differences in the attached screenshot. I suspect they are from the INCREASE_FLASHES state?
I'd agree the SDF diffs are from INCREASE_FLASHES. The state just saves the TEST offset values and then puts them back in when it's done, so I'm surprised that the values it put back in seem rounded from the originals. I'll look into it.
What was happening that made you think that the bootstrapping was the issue?
Went out of observing briefly to address violin modes. Reloaded the VIOLIN_DAMPING guardian during this. No other issues.
Since the NCal electronics have remained on since GPS=1258930068 (Wed Nov 27 22:47:30 UTC 2019), see LHO aLOG 53524, I have taken the weekly Fscan averages from the summary pages to look for any additional noise from spectral artifacts. Granted, this is not a conclusive study to confirm or vindicate any couplings caused by NCal, but it is a first look at long duration averages and suggests we should investigate this further. I would also suggest that it's worth turning off the Ncal as long as it does not remain an essential tool for IFO operations. Past experiences indicate that all non-essential electronics should be disconnected during observing runs. This suggestion is, at the moment, due to an abundance of caution regarding new, non-essential IFO operations electronics connected during an observing run. Below I attach three ratios of the weekly Fscan averages of H1:GDS-CALIB_STRAIN in the 10-100 Hz band, a common region for noise lines from electronics. Figure 1: ratio of Fscan weekly average Nov. 3-10 / Nov. 17-25 Figure 2: ratio of Fscan weekly average Nov. 10-17 / Nov. 17-25 Figure 3: ratio of Fscan weekly average Nov. 25-30 / Nov. 17-25 <-- NCal electronics remain on from Nov. 27 In the two weeks preceding the Ncal being left on, the Fscan ratios show some variations, as indicated by the ratio deviating from a nominal (unchanged) value of 1.0. These variations are below a maximum ratio value of 20 (one line only). See figures 1 and 2. However, the final figure shows the change from the week the NCal electronics remain on and many lines show large ratio values up to maximum values greater than 50. Further study is needed to determine if this really is caused by the NCal electronics remaining on since Nov. 27. At the moment, I'd suggest turning the electronics off and possibly disconnecting as well to be fully cautious.
I think that a reasonable check of the NCal as a source of lines is to look for cohernece between DARM and the temporary magnetometer, H1:PEM-EX_ADC_0_11_OUT_DQ, that we left on the sattelite box during this period (and it is still there). I would expect that any coupling through power supplies or grounds would be associated with periodic currents in the NCal system that would be detected by this magnetometer more strongly than the VEA magnetometer. If there are no lines on this magneometer that are coherent with DARM, and greater in amplitude in this magnetometer than the VEA magnetometer, then I think the NCal is clear.
First round last week can be seen in alog53499
Today I tried the Increase_Flashes state on the Y arm. I pushed it out a few urads in both pitch and yaw until we were only getting flashes to around 0.3, and then waited to see if the state could recover an alignment that was good enough for the WFS to catch it. After it had adjusted both P & Y, it was able to get to an alignment that was giving flashes >0.9. This is the threshold set for this arm, so once it found flashes that were this high, it stopped trying to find a better alignment. I changed it so ISC_LOCK will ask it to run a second time if it cannot lock for some time after the first try. For this case, a second try was quickly helpful.
I didn't have time to add the TMS into the code. This is for anther time.
Operators: The ops intervention sheet was been temporarily updated, asking to allow this code to try for 2 attempts. If there are any problems, you can maneuver around this state and align by hand, or give me a call.
Is this supposed to run automatically at LOCKING_ARMS_GREEN? I had to manually request it.
After a waiting period it will run automatically. I think its 5min? Perhaps too long, but I was keeping consistent with the intervention sheet.
Today, I performed the 1st stage of modification to the instrument air supply setup for the Corner Station which, when completed, will reduce the compressor's runtime (duty-cycle) by ~65%. Step 1 - added double check-valves to the output of the drying towers. Also, added isolation valves and Tees w/swagelok ports between the reciever tank and the pressure switches (both LEAD and LAG).
This is an improvement in reducing energy and increasing equipment life. Additionally, we plan to replace this reciprocating piston style compressor unit with a scroll style similar to original.
TITLE: 12/03 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.09 μm/s
Secondary useism: 0.32 μm/s
QUICK SUMMARY: No issues.
TITLE: 12/03 Day Shift 16:00 – 00:00 (00:08-16:00), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Not too invasive maintenance day, followed by MICH_BRIGHT_ALIGN initial alignment issues. Went all the way up to NLN after lockloss due to known (and now fixed) issue. Locked and Observing for 2 hours.
LOG:
16:00 (08:00) Scott, Tyler to EX -- bale tumbleweeds
16:00 (08:00) TJ taking IFO to down for GRD tests
16:00 (08:00) Karen to EY
16:10 (08:10) Kyle to LVEA -- compressor work
16:18 (08:18) Chris, pest control to LVEA
16:31 (08:31) Fil to CER
16:36 (08:36) Vanessa to EX
16:50 (08:50) Chris, pest control out of LVEA
16:50 (08:50) Chris, pest control to EY, EX
16:54 (08:54) Hugh to HAM7
17:00 (09:00) Jason starting FSS tune-up
17:01 (09:01) Karen leaving EY
17:32 (09:32) Cheryl to LVEA -- prep camera work
17:33 (09:33) Jim to EY, EX -- plug in BRS temperature sensors
17:34 (09:34) FSS tune-up complete
17:47 (09:47) Christina to LVEA -- inventory
17:48 (09:48) Cheryl to LVEA -- start HAM2 camera work
17:51 (09:51) Dave to MY/EX
17:51 (09:51) Vanessa leaving EX, going to LVEA
18:00 (10:00) Vlad to Optics Lab
18:16 (10:16) Pest control finished on site
18:21 (10:21) Norco onsite to fill CP6
18:30 (10:30) Chandra to LVEA -- grab RGA electronics
18:40 (10:40) Christina out of LVEA, going to MX -- continue inventory
18:42 (10:42) Gerardo to LVEA -- find Karen
18:43 (10:43) Chandra out of LVEA
18:54 (10:54) Beginning initial alignment
19:03 (11:03) Carlos, Richard to EX, EY -- deploy wireless access points
19:11 (11:11) Chandra to LVEA -- return RGA electronics
19:17 (11:17) Karen to EY
19:32 (11:32) Chris to LVEA -- unplug forklift
19:37 (11:37) Chris out of LVEA
19:39 (11:39) Gerardo to LVEA -- grab equipment
19:47 (11:47) Carlos, Richard to CER/mid-stations -- deploy wireless access points
19:52 (11:52) Gerardo out of LVEA
19:54 (11:54) Carlos, Richard out of LVEA
19:59 (11:59) Fil out of LVEA
20:01 (12:01) Tyler, Scott out of LVEA
20:04 (12:04) Cheryl out of LVEA
20:09 (12:09) Carlos to CER -- assist Dick
20:17 (12:17) Corey, Camilla to LVEA -- sweep
20:39 (12:39) Corey, Camilla out of LVEA
20:53 (12:53) Initial alignment complete, starting re-lock
21:22 (13:22) Dick and Marc finished in CER
22:12 (14:12) Vlad to Optics Lab
23:07 (15:07) Taking IFO to EQ mode for 5.6 EQ near Papua New Guinea
23:26 (15:26) Transitioning back to WINDY
Commissioners started work to improve MICH_BRIGHT acquisition during initial alignment, but it has not been completed. Operators having issues should skip MICH_BRIGHT during initial alignment or do the old MICH_DARK_LOCKED hand alignment
A follow up to the measurement of the "mystery" UIM dynamics:
I have fitted the measured transfer functions measured in the above link:
Given that the UIM keeps rolling off after this, I don't think the higher frequency parts of the recorded data will show up in the above plot.
continuing from 5 Nov, alog 53026:
Jack & Hugh WP 8384. Brought in the north crane and disconnected the HAM7 N. Crossbeam from the HEPI housings. Pulled the crossbeam and got the radius plates connected to the U-Chan and Chamber. This is not completely secure yet. We ran out of maintenance time and need to shim. Attached photo shows the progress
Initial Alignment:
Difficulty with MICH_BRIGHT. Jenne, Jeff, and TJ worked on it and we got through
Attempt #1:
Greens arms ran automatically
Find IR ran automatically
DRMI ran automatically
Lockloss from CARM_OFFSET_REDUCTION (Jenne recognized and fixed problem)
Attempt #2:
Green arms ran automatically
Find IR ran automatically
DRMI ran automatically
CARM reduction states ran automatically
ENGAGE_ASC_FOR_FULL_IFO/ENGAGE_SOFT_LOOPS ran automatically
NLN
End station front end IPMI
Erik, Dave:
we visited EX and EY to confirm the ethernet cabling of the managment ports to the SEI and SUSAUX front end computers.
FOM nuc24
Dave:
I updated nuc24 FOM:
TEST OFFSETs on all optics are typically a fixed value, and not active during NLN/Observe. Currentl there are two situations that need correction/understanding, listed below:
Plots of ETMX TEST YAW and TMS TEST OFFSETs attached.
Current list of TEST OFFSET values:
Looks like the only work in the alog on Nov. 19th, that mentions the TMSs, is Craig's QPD servo work (alog53362). The script he mentions doesn't seem to be checked into the SVN, so I can't see it at the moment to see if it uses the TEST bank.
Looking at the tms_qpd_centering_servos.py script, this is where the TMS change on the 19th came from.
All mysteries solved.
Some of the SUS TEST OFFSETS are used to park the ETMs in specific no-light places during initial alignment. So, an offset remains there and gets toggled on and off as alignment scripts need them. TJ's new ALS alignment code (implemented ~last Wed) will also use these offsets but should be restoring them to the nominal misalignment pointing offsets. However, killing the script (as may have been done once or twice over the early part of the weekend) likely did not put the misalignment value back, and Patrick ended up accepting an incorrect misalignment offset.
SO, the next time we are out of OBSERVE, please enter and SDF accept the following:
H1:SUS-ETMX_M0_TEST_Y_OFFSET -8.45
The attached plot is of the last few months of these settings.
This has been changed and accepted into SDF
[J.Kissel, T.Mistry]
After the calibration measurement were complete, I went to the X end and powered on and off the NCAL optical encoder system in 20 intervals. The interferometer was in Nominal Low Noise (NLN) at the after the calibration measurements. The motivation of this was to investigate whether the encoder would cause glitches and having known on and off times within the same lock stretch may indicate if this is the case. The first figure attached shows the encoder signal from the channel H1:CAL-NCALX_ENCODER_VELOCITY_OUT_DQ. When the encoder system is powered off, the channel measures 80 +/-1 DAC counts. When the encoder system is turned on, the channels measures 6932 +/-1 (approx 4V). A follow up invesitgation, in addition with investigation from LHO alogs 53503,53441 and 53396. A time log of the activities as taken by J.Kissel are as follows:
1258926919 NLN (Timesh arriving at end station, Jenne feedforward off time stops, Robert still somewhere in/around the YVEA with Wifi and phones ON doing camera recordings of test mass glint, wind fence crews are driving around at EY, audible within the YVEA).
21:55:01 UTC
1258927033 Timesh calls for first go in
21:56:55 UTC
1258927169 +/-15V power supply to mini-field rack Flipped ON. Begin first 20 minutes of ON data.
21:58:59 UTC
(XVEA Lights were left ON)
1258927372 lights OFF now
22:02:34 UTC
1258928389 Timesh begins to head in to turn off NCAL power supply (with lights remaining off)
22:19:31 UTC
1258928508 power supply flipped OFF
22:21:30 UTC
1258928589 Timesh calls back to say he's done turning OFF NCAL.
1258929539 Timesh heads back in to turn NCAL ON
1258929649 NCAL powered ON
22:40:31 UTC
1258930068
22:47:30 UTC
Keita OKs leaving the NCAL ON, as long as we are committed to showing that it doesn't matter.
Timesh goes in to garb room to clean up.
1258930168 Timesh leaves EX
1258930425 IFO handed over to Robert for his tests.
In case it wasn't clear from Timesh's entry: We have left the NCAL system *powered ON,* but not spinning. The data was supremely glitchy today, in general, because if the current wind storm. Thus, it was difficult to make any sound conclusions about whether we were able to reproduce the elevated glitch rate that was reported previously when the NCAL system was similarly powered on, but not spinning (see LHO aLOG 53396). So, we're leaving it powered on but not spinning for a few days to (a) hopefully get some data while the environment is more typical (i.e. quiet) (b) gather days-long stretches of data to be sure leaving it powered on doesn't adversely affect the CW search group. If exonerated, then we don't have to drive to the end station every time we want to use the thing. Stay tuned!
See LHO aLOG 53666 for a first look at this data using Fscans. Further study is needed, but the first look indicates some additional lines or increased line artifacts around this time.
