Accepted the following SDF differences after recovering lock
At mid shift recovering from LSC Servo Board Split Voltage is High lockloss. Had to tweak X-Arm ALS several times to get green to lock. Had to tweak BS and PRM to get DRMI_1F to acquire. There was one locking break due to another LSC Servo Board Split Voltage is High event. Lock acquisition is going better but not all the way to NLN yet.
08:41 (01:41) dropped out of observing to manually damp violin mode-9. Applied the -3 gain from the Lscparams file, to suppress the ring up. Went back into observing at 08:58 (01:58) after turning back on Guardian damping and accepting the SDF Diffs in the snapshots for ITMY & ETMY. Could not find the snapshot for ETMX. There are additional "mode is growing" notifications for ETMX Mode-7 and ETMY Mode-14. Both are bouncing around, but neither is growing. Will continue to monitor them.
TITLE: 03/26 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC STATE of H1: Observing at 100Mpc INCOMING OPERATOR: Jeff SHIFT SUMMARY: Lock loss during calibration measurements. Jenne and Sheila helped recover. Had to do some tweaking of the BS / PRM / SRM. Did not need an initial alignment. Keita had me reload the INJ_TRANS guardian and request INJECT_SUCCESS. Adam had me reload the INJ_TRANS guardian a couple of times later. I turned off the lights in the LVEA before going to observing. There was a random drop out of observing due to the laser power guardian (see log). LOG: 23:29 UTC NLN_CAL_MEAS -> NLN. 23:41 UTC NLN -> NLN_CAL_MEAS. 00:09 UTC NLN_CAL_MEAS -> NLN. 00:25 UTC NLN -> NLN_CAL_MEAS. 00:49 UTC Lock loss. LSC Servo Board Split Voltage is High. 01:01 UTC Per Keita: Load INJ_TRANS. Requested INJECT_SUCCESS. 01:37 UTC Per Adam: Load INJ_TRANS. 02:03 UTC Per Adam: Load INJ_TRANS. 02:17 UTC Back to NLN 02:20 UTC Craig running calibration measurement. Set observatory mode to calibration. 02:57 UTC Turned off lights in the LVEA 03:29 UTC Observing 05:07 UTC Dropped out of observing. Laser power guardian went into error. Craig stopped and exec the laser power guardian to restart it. It is continuously writing to H1:PSL-POWER_SCALE_OFFSET and momentarily failed to connect to the channel. Craig put back to observing.
Finished calibration measurements for the night. Just went into observing. Georgia will make ring heater measurements.
Forgot to post some SDF diffs from last night (2019-03-26) that I accepted to get to observing. These are about the Cal line magnitude for EX (first screenshot), and some ALS_Y differences (second screenshot).
Reed Essick, TJ Massinger, Adam,
Some detchar safety injections have been scheduled to go into the IFO at 5:00:00 PDT (12:00:00 UTC) tomorrow morning. They'll take about 91 secs. They'll only go in if the IFO is in observation mode, but the operator should know that it's not critical for these injections to happen, so they don't have to stress if the IFO isn't in observation mode.
If for some reason, these injections need to be stopped, the operator can request INJECT_KILL in the INJ_TRANS guardian (found on the guardian overview screen).
These injections were not successful. The interferometer was not in observing intent at the correct time.
[Daniel, Jenne]
At 17:06 local, so 00:06 on 27 Mar 2019 UTC, we turned off by hand the corner station ALS VCOs (COMM VCO, COMM FDD, DIFF VCO, DIFF FDD), using the new relays mentioned in alogs 47894 and 47893.
I have also added turning them on and off to the main ISC_LOCK guardian. When we get to PARK_ALS_VCOs (no longer using ALS for any lock) the VCOs and FDDs will now be turned off. The PREP_FOR_LOCKING state will turn them back on. Note that the channels are "disable" switches rather than our more usual "enable" switches, so to turn off the VCO, you must set the bit to 1. The MEDM screens just say on/off, so it's clear, but the guardian uses the numerical values, which is why I make note of it.
DetChar: It would be helpful if you could have a look at tonight's lock and see if the whistle situation is improved. Hopefully it will be!
Laura, Andy There are still many whistles in the long lock at 4 UTC (Mar 27). First we confirmed that the ALS comm and diff frequencies are zero, so those VCOs are turned off in the lock. The summary pages show a high rate of high frequency glitches. We can find the IMC VCO frequencies at which these whistles occur, by plotting the VCO frequency of the lock, with all glitch times marked as points - see attachment. They make a clear line at 79.169 MHz. The ALS VCOs all park below 79 MHz, so even if they were on they're not really close enough to the IMC VCO to explain the whistles. The code for the plot is on git.ligo in this Jupyter notebook.
Thanks for looking at this. We'll do a slow sweep of the IMC VCO frequency over a longer range to hopefully find a good place.
While looking at the history of the situation, I see that the reboots due to the new on/off switches that happened on Tuesday seem to have removed the offset of the setpoint that we'd been using since Daniel set it on March 12th (alog 47470).
Some of the liquid nitrogen dewar Try-Cock exhaust-to-atmosphere tubes were blocked or, partially blocked, with mud dauber wasp nests (a.k.a "mud') -> So, today I cleaned out each one via opening the Try-Cock valve and then feeding a wire up through the exhaust tube so as to dislodge the obstruction. Once dislodged, the ~10 psig GN2 vapor blew the debris out onto the ground. .
I finished the changes that were discussed in a SQZ meeting on March 12 to make the squeezing automation more common among the two sites, and to set up a contingency plan to getting into observing if we cannot squeeze. Namely:
I have written instructions in a wiki on how to get into observing without squeezing for the operators to use, but this procedure has not been tested yet. (Link here) If someone has to test it before I am around, please call/text me with with any issue you encounter.
While I was changing these around, I also changed the management of the SQZ_MANAGER in ALIGN_IFO so that we would not have the stolen node notifications anymore.
NikoL, RickS
The Working Standard responsivity ratios have changed slightly as noted in LHO aLOG 47858.
The updated values were loaded into the EPICS records used the FrontEnd code calculations and updated files with these new values were added to the SVN:
ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/CALCS_FE/
lho_xend_pcal_epics_created-20190326.txt and
lho_yend_pcal_epics_created-20190326.txt
The change in H1 overall calibraiton is about 0.21%
The changes were accepted in the SDF.
TJ, Jonathan, Dave:
Late last week Jonathan installed his system monitor EPICS-IOC on h1guardian1 and we striptool'ed it over the past 4 days. The trends have revealed a memory leak, with the available memory dropping from 80% to 40% and occasionally resetting back (see attached striptool plot).
The times the memory is recouped is coincident with seismic blend switching, which is usually associated with operator earthquake switching. TJ is taking the investigation further.
I forgot to mention the system load fluctuates at these times as well.
After Dave found that the SEI_CONF node was changed to EARTHQUAKE the time that the memory % freed up a bit, I tried each of the 3 types of nodes changed in that state: HPI sensor correction, ISI sensor correction, and ISI blends. The first didn't change anything, but changing the ISI_BS_ST1_BLND node made an immediate change.
These nodes will sit in its idle state after making the blend transition, and the only thing running is a decorator checking the configuration. This decorator is creating a new LIGOBlendManager class reference every cycle. I thought that the end of each new cycle would release this from memory, but perhaps I was wrong. I will test this in the coming days, when given the opportunity.
In the mean time, if memory becomes an issue, we can always re-request the same blend. It will blend to the same one, and no harm will be done to the IFO.
Potential offender here:
def check_blend_config(config):
"""
dof_dict - Dictionary with the dofs as the keys and
what should be the correct fm#s as their values
"""
config_name, dof_dict = config
for dof, fm in dof_dict.items():
if fm == 0:
continue
cur_fm = LIGOBlendManager([dof], ezca=ezca).get_cur_blend_name(dof)[0]
if cur_fm != fm:
return False
return True
I ran a test today of commenting out that decorator, reloading all of the blend nodes, and then watching the available memory rate and CPU load. This definitely seems to be the problem.
In the attached trend the large drop in the CPU load (blue) was me reloading the blend nodes without the decorator. The available memory (yellow) from before this point was steadily declining, but after was basically stable. I did not get the chance to change blend states, so it makes sense that we did not free up more memory like we had seen previously.
TITLE: 03/26 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Calibration
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
Wind: 8mph Gusts, 6mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.16 μm/s
QUICK SUMMARY:
Locked at NLN_CAL_MEAS.
23:20 UTC Changed observatory mode from commissioning to calibration.
NikoL, SudarshanK, RickS
We've analyzed the measurements made to date with our updated Pcal laser power calibration standards (see LIGO-T1800359-v2 and LIGO-G1900393-v4).
The method used to arrive at the O3 starting values is described in LIGO-T1900134-v2.
The WS/GS responsivitiy ratios ("Final values" in table) at in the attached table.
TITLE: 03/26 Day Shift: 15:00 - 23:00 UTC (8:00 - 16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Maintenance from 8am -12:30 pm. Initial alignment went smoothly, then an LSC servo board splitmon-induced lockloss gave us trouble getting past ACQUIRE_DRMI_1F. Hit NLN around 3:30pm, leaving still locked with Cal measurements ongoing.
LOG:
15:00 (8:00) Start of shift
15:08 (8:08) Vanessa to EX
15:11 (8:11) Hugh to LVEA -- look at seismic shorts
15:13 (8:13) Richard to LVEA -- add remote controlled relay to VCO power
15:18 (8:18) Bubba to LVEA -- look at things for O3
15:19 (8:19) Phillipe to LVEA -- grab equipment
15:28 (8:28) Bubba out of LVEA
15:50 (8:50) Hugh out of LVEA
15:53 (8:53) TJ to LVEA -- HWS temperature sensor install
15:53 (8:53) Nutsinee to LVEA -- replace ISCT6 diode
15:58 (8:58) Hugh to EX, EY -- HEPI fluid check, look for seismic shorts
16:23 (9:23) Dave to EY -- network switch work
16:31 (9:31) Karen leaving EY
16:39 (9:39) Hugh done with end station work
16:59 (9:59) Vanessa, Karen to LVEA
NOTE: Karen said her key fob wouldn’t work while trying to get into EY
17:02 (10:02) Chandra to LVEA -- take photos of HAM6
17:09 (10:09) Ed to EY, Fil to EX -- grab VCO’s
17:10 (10:10) Betsy, Jim to LVEA -- fit checking C3 cover
17:14 (10:14) Carlos restarting H1 video cameras
17:18 (10:18) TJ out of LVEA
17:25 (10:25) Christina to EX, EY -- collect Norco reciepts
17:33 (10:33) Chris to EX, EY
17:35 (10:35) Dick to LVEA -- trace noise
17:38 (10:38) Betsy, Jim out of LVEA
17:39 (10:39) Chandra out of LVEA (mentioned she crawled on HAM6)
17:42 (10:42) Corey to EX, EY -- ALS table pictures
17:58 (10:58) Corey transitioning EY to LASER HAZARD
17:58 (10:58) Paradise water at gate
18:01 (11:01) Phillipe, Sharan to EY -- grab speaker
18:25 (11:25) Chris back from End Stations
18:42 (11:42) Betsy, Rahul to LVEA
18:55 (11:55) Betsy, Rahul out of LVEA
19:02 (12:02) Nutsinee out of LVEA
19:10 (12:10) Dick out of LVEA
19:10 (12:10) Ed back from EY
19:12 (12:12) Fil, Corey back from EX
19:15 (12:15) Richard to CER -- “look at some stuff”
19:43 (12:43) Starting initial alignment
20:32 (13:32) Kyle to MY
20:34 (13:34) Nutsinee to ISCT6
20:39 (13:39) Carlos to CER
20:47 (13:47) Carlos out of CER
20:53 (13:53) Lockloss from DRMI_TO_POP - LSC Servo Board Splitmon Voltage High
21:22 (14:22) Christina to MX -- grabbing Norco receipt
21:43 (14:43) Lockloss from ACQUIRE_DRMI_1F - IMC Servo Board Splitmon Voltage High
21:54 (14:54) Kyle back from MY
23:00 (15:00) End of shift
Pep Covas, T. Shaffer
Started the HIGH_FREQ_LINES Guardian node to sweep H1:CAL-PCALX_PACALOSC1_OSC_FREQ from starting frequency to 2000Hz by 500Hz steps. The steps will happen every 12 hours, but it will wait until we are not locked to change the frequency, so it could be longer than the 12.
Line amp = 35000
Starting freq = 5950.0Hz
Step size = 500Hz
We unmonitored H1:CAL-PCALX_PCALOSC1_OSC_FREQ in SDF.
J. Kissel, P. Covas
We found that the PCALX saturated, so we turned off the line at 5950.0 Hz, then turned off the optical follower servo and then reenabled it. Then we turned up the line around 0:18 UTC. We saw the non-linearity (down-conversion) seen previously in O2 is still there, which produces a line at 86 Hz, as can be seen in the attached plot. At 0:24:45 the frequency of the line was changed to 3 kHz.
We have seen that the line had been reverted back to ~6kHz. At 18:47 UTC we have changed it to 3001.3 Hz
Following on from Jenne's common ring heater change (lowering power), we are taking a differential ring heater step tonight. We will reduce ITMY to the same power as Jenne did, and take a smaller step on ITMX. If there are problems with ASC while reacquiring lock, these changes might need to be reverted.
| ITMX RH Filter input [W] | ITMY RH filter input [W] | |
| Nominal | 0.5 | 1.4 |
| Common step today | 0 | 0.9 |
| Differential step tonight | 0.25 | 0.9 |
( - I'm quoting filter input requested power here, there's an offset from the actual ring heater set power I'm not sure about.)
Craig has coded up a live monitor of the frequency and intensity noise lines which I'm running on ZOTWS22. The realtime monitors are found here:
/ligo/home/craig.cahillane/monitors/AWG_Line_Monitors.py
To run them you might need to use craig's environment, by running:
source /opt/rtcds/userapps/release/cds/h1/scripts/setup_anaconda
source activate cragenv
(edit: original title to alog quoted wrong start time timezone for measurement)
After ASC lockloss reset the Ring Heaters back to original values. RH Filter Input for ITM-X is now 0.5w, RH Filter Input for ITM-Y is now 1.4w.
Just to clarify: the nominal ring heater values I quoted in this alog are 0.1 W higher than the actual nominal ring heater vales: I got confused by the slow response of the filtered output.
Before yesterday, ITMX_RH_INVERSE_FILTER_IN was 0.4 W, and ITMY_RH_INVERSE_FILTER_IN was 1.3 W.
