This morning I tweaked the beam alignment into the FSS RefCav. Before the tweak the RefCav TPD was reading ~3.8 V, after the tweak it is reading ~4.2 V.
TITLE: 05/07 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 108Mpc
INCOMING OPERATOR: Jeff
SHIFT SUMMARY:
Nice shift with about 4hrs of triple coincidence.
Summary Pages look to be 5hrs old for L1 & H1.
LOG:
Triple coincidence going on now (and Virgo just completed their Maintenance Day). H1 has a 6.5+ hour lock with range between 105-110 Mpc. Mostly quiet with winds just starting to pick up.
Summary: Operators, until Commissioners say otherwise, please do following:
Niko tried a variety of things to help get H1 back, but he mentioned there was not a smoking gun other than zipping through the ISC_LOCK steps for this current lock. A possible source of grief could be the phase for AS_A_36. Since Thurs night, Jenne has requested operators to address the phase for AS_A_36 (as noted above & in her alog). We've only had a few locklosses since Thurs, so not had lots of practice remembering to do this, so just posting another reminder to be mindful of doing this after locklosses. I reckon we want to keep on ACCEPTING the SDF diffs we get for these channels. (Attached is one of these phase channels and H1 range for the last few days, and you can see we forgot to transition phases after today's earthquake lockloss.)
I have put up a sticky note reminder for this on the Operator computer. We will do this until commissioners let us know otherwise.
Bigger Picture Question:
Do we know how long we need to do this? If it's short term we can remember to do this, but if it's longer, can we have this in the DOWN state?
TITLE: 05/07 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 109Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
Wind: 5mph Gusts, 4mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.11 μm/s
QUICK SUMMARY:
TITLE: 05/06 Eve Shift 23:00 – 07:00 (16:00-00:00), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Waited a little under 2 hours for the earthquake near Papua New Guinea (and subsequent earthquake off the coast of Alaska) to ring down. Then started initial alignment, but had ALS fiber polarization issues. Eventually started locking, and lost lock several times from ENGAGE_DC_VIOLINS and POWER_30W. I initially assumed it was the SQZ TTFSS and turned on the noise eater. When it happened again, I tried to damp high violin modes (though none of them seemed to be increasing). The final time, I just moved out of POWER_30W as quickly as I could and the IFO quieted down. Been in Observing for 1.5 hours.
LOG:
23:00 (16:00) Start of shift
23:44 (16:44) Dave performing DAQ restart while we wait for the EQ to ring down
01:05 (18:05) Starting initial alignment
01:30 (18:30) Adjusting fiber polarization in both arms with Keita. X won’t go below 19%
02:15 (19:15) Starting locking sequence
03:40 (20:40) Lost lock from POWER_30W, I believe due to TTFSS being noisy. Turning on SQZ noise eater (temporarily)
04:11 (21:11) Lost lock from POWER_30W. Unsuccessful at damping violin modes
04:45 (21:45) Lost lock from ENGAGE_DC_VIOLINS
05:36 (22:36) Reached NLN, accepting SDF’s and checking violin modes
05:42 (22:42) In Observing mode
05:45 (22:45) Out of Observing briefly to turn off SQZ noise eater, back to Observing within the minute
06:28 (23:28) GRB (E331637)
06:49 (23:49) Going to EQ seismic configuration as a precaution for a 5.2 NE of Japan
07:00 (00:00) End of shift
Accepted the attached sdf change to go into Observing
Sheila, Daniel, Nutsinee
Last week we measured a set of sqz/asqz vs. CLF angle measurement. Using equation 2.71, 2.71, and 4.5 from Sheila's thesis and some adjustment to CLF angle read back we were able to make a reasonable fit to the measurement.
There are 5 variables, loss, phase noise, initial offset, CLF angle offset, and sqz angle offset. The uncertainty band is the rms error value between the fit and the measured data. To get the uncertainty on each parameter I tweaked each one to the point the rmse is increased by 10%, which at that point the fit looked very unlikely to be a good fit, giving us the upper bound of the uncertainty of each variable. Suggestion by David McManus.
Loss = 51.86% +/- 5%
Phase noise (rad) = 0.1577 +0.066/-0.1177
Initial offset (rad) CLF angle offset (rad) = 3.789 +/- 0.023
CLF angle offset (rad) Ellipse Axes ratio (r as Daniel described below) = 1.7252 +0.1693/-0.1327
SQZ angle offset (rad) = -1.0304 +0.0654/-0.0151
CLF angle offset is the discrepancy between the slide bar read back and the real angle offset. This was due to unequal side bands.
The relation between demodulation angle and squeezer angle can be written as
with:
ϕdemod : demodulation angle
ϕ0 : offset to the demodulation angle
ϕSQZ : squeeze angle
ϕOfs : offset to the squeeze angle
r : ratio between major and minor axes of the demodulation ellipse in the IQ plane
The ratio r is related to the non-linear gain of the CLF field.
The above fit uses ϕOfs as the squeeze angle offset and ϕ0/2 as the demodulation (CLF) offset.
If I were to tweak one parameter and let the function re-fit the rest of the parameters (as all of these variables are dependent of each other), using the same criteria as before (10% increase in RMSE) here's what I came up with in terms of uncertainty:
Loss (%) = 51.86 +14.64/-8.56
Phase noise (rad) = 0.1577 +0.1223/-0.1577
CLF angle offset (rad) = 3.789 +0.096/-0.151
r = 1.7252 +0.3298/-0.2182
SQZ angle offset (rad) = -1.0304 +-0.4
Note that most of the error bars have become larger to get to the point where RMSE increase by 10% as other parameters are free to move during each tweak.
Daniel also suggested another method to estimate uncertainties is to use covariance matrix. I have been having issue getting the function to spit out jacobian. This will take a little longer.
Here's another goodness of fit estimate using Chi^2, suggestion by Daniel:
Chi^2 = SUM( ((Y2-Y1 / dY))^2 ) ; Y2 is the fit, Y1 is measured data, dY is the uncertainty in the measured data.
I tweaked dY so that Chi^2 = number of data - number of parameter, this chi^2 should be the lowest.
Once I have dY, I tweaked each parameter and let other refit until I get Chi^2+1.
Below are each parameter with 1sigma uncertainty:
Loss (%) = 51.86 +3.84/-1.89
Phase noise (rad) = 0.1577 +0.0447/-0.0337
CLF angle offset (rad) = 3.789 +0.0235/-0.0399
r = 1.7252 +0.0913/-0.04
SQZ angle offset = -1.0304 +0.1142/-0.0674
The uncertainty has become much smaller with this method. Sadly the phase noise is still within 100mrad range even with the lower bound uncertainty.
Lost lock a few times while trying to get to NLN. Most recent was from Power 30W, likely due to TTFSS being noisy. Following 48996 and turning the SQZ noise eater on until we can get locked.
On Saturday 04 May 2019 there was an alignment problem for Y arm starting at Corey's owl shift (alog 48973) throughout Ed's shift alog 48979) and into Patrick's evening shift (alog 48982).
Seems like a part of the problem was a big temperature swing over multiple days, a problem first reported by Bubba (alog 48936).
In the first attachment that is about 20 hours' worth of trend, t=0 is when the lock was lost during Corey's shift.
ETMY M0 LOCK_P was absolutely huge just before the lock loss, -400ct ~ -20urad. As a result, after the lock was lost, ETMY moved by about +17 urad in PIT (my scribbling in yellow). There were also some non-negligible jumps in ITMY and TMSY, but nothing comparable to ETMY.
After an hour and a half or so Corey managed to lock ALSY with OK alignment twice (cyan scribble), and then couldn't lock for a while and again manages to lock twice (red scribble), but nothing came back to where they should be.
Second attachment is the same as the first one except for the time axis which is from about 10 days ago until now. t=0 is still Corey's lock loss.
The most revealing is the correlation between reaction chain top mass P and the temperature, because reaction chain M0 is not affected by ASC at this time scale as ASC feedback to ETM goes to test mass chain M0 at DC via an integrator. I didn't show it here but the temperature change of the EY VEA (as opposed to PEM BSC10 temperature sensor) was almost 2F peak to peak.
The drifts are huge, more than 20urad for ERM top mass, TMSY is a bit harder to tell due to initial alignments but seems to be ~20urad too. As a result, during the long lock ASC had to push ETMY really hard to keep it aligned, but when the lock was lost the feedback is zeroed and the ETMY experienced a huge kick.
During recovery, the suspensions were still moving fairly rapidly. L2 witness sensors would have been useless for long term trend because ERM has drifted and was drifting, which means that we didn't have a reasonable long term reference without knowing that. The drift only slowed down after the first NLN lock was acquired by Patrick (red vertical line that I've overlayed on the plots).
In conclusion, it seems extremely likely that ETMY and TMSY drifted by a huge amount due to the temperature swing in EY VEA, causing a sudden huge kick when the lock was lost. Recovery was difficult because things were still moving as we were working, and our usual long term references, i.e. L2 witness sensors, were unreliable due to ERM chain drift.
Even when temperature is doing OK, whenever NLN lock is lost there are non-negligible kicks, and for any given DOF the sign is always the same and the amplitude is very roughly the same. For example for ITMY PIT, the kicks related to lock losses are always positive and from 2 to 5 urad. This is to be expected given that we intentionally offset the beam position and therefore we have a constant DC torque due to radiation pressure.
New h1ascimc model.
Daniel, Dave:
Daniel installed a new h1ascimc model (IOT1_SPAREIN1 FM removed, REFL_TRIG_LF FM added). Model was restarted at 16:45 PDT, DAQ was restarted at 16:47.
Took advantage of the earthquake break and set up the software for the IMC REFL PD shutter. This required a configuration change in the corner PLC1. I also added a command in the IMC_LOCK guardian to open the shutter during acquire. The medm screens are updated as well. However, no hardware shutter/trigger is installed as of yet.
WP8197 h1edc DAQ-CRC mitigation
Jonathan, Dave:
During this afternoon's earthquake downtime, I made the h1edc code change to stop it checking H1EDC.ini file's checksum. There is clear evidence that the /opt/rtcds file system slows between 4am and 5am local, and for the past two mornings h1edc's DAQ-CRC counter has incremented around these times.
Remember that h1edc is unique in that it builds out of RCG-3.5.0 area. I changed the file
/opt/rtcds/rtscore/advLigoRTS-3.5.0/src/epics/seq/edcu.c and commented out the code block:
// Check file CRCs every 5 seconds.
// DAQ and COEFF file checking was moved from skeleton.st to here RCG V2.9.
/* %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Do not check H1EDC.ini checksums. This may be causing DAQ-CRC errors at LHO during O3
D.Barker LHO 06may2019
if(!fivesectimer) {
status = checkFileCrc(daqFile);
if(status != daqFileCrc) {
daqFileCrc = status;
status = dbPutField(&daqmsgaddr,DBR_STRING,modfilemsg,1);
logFileEntry("Detected Change to DAQ Config file.");
}
}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% */
On h1susauxh34, in the /opt/rtcds/lho/h1/rtbuild/rt-3.5.0 build area I did the compile sequence
Note that the install overwrites the H1EDC.ini with a 'fec version' of this file. This generated a 0x2000 DAQ error, though the actual data was unchanged. I cleared this error by regenerating H1EDC.ini using my script.
I restarted h1edc at 15:35 PDT, and then cleared the DAQ-CRC counters.
To test that h1edc is no longer checking H1EDC.ini, I edited this file (added a trailing blank line) and no 0x2000 error ensued.
We have seen occasional CRC errors on l1edc, but not at this level. Perhaps we will trend to see if a pattern persists At LLO, a separate SSD-based file server pair (1lfs1/l1fs2) handles the front-end file system, with a larger spinning-media server pair (cdsfs1/cdsfs2) for workstation-specific file systems
Dave, Jonathan, During earthquake down time we reworked systemd config controlling the CW injection. We make sure that startup/shutdown of the injection enforces a 10s ramp time. The system will try to auto start up to 4 time in 1/2 hour, with 5 minutes between attempts. We changed the startup script to: source /ligo/cdscfg/stdenv.sh caput H1:CAL-INJ_CW_TRAMP 10.0 caput H1:CAL-INJ_CW_GAIN 0.0 sleep 11 /home/hinj/Details/bin/x_start_psinject monit start caput H1:CAL-INJ_CW_GAIN 1.0 and the stop script to: source /ligo/cdscfg/stdenv.sh caput H1:CAL-INJ_CW_TRAMP 10.0 caput H1:CAL-INJ_CW_GAIN 0.0 sleep 11 /home/hinj/Details/bin/x_stop_psinject monit stop As a side effect of how systemd monitors it, when the process crashes, the stop script gets run which ensure that on a failure the gain is forced down to 0 with a ramp. So whatever was left in the awg buffers gets ramped down.
Ops Shift Transition: 05/06/2019, Day Shift 23:00 – 7:00 (16:00 -00:00) - UTC (PT)
State of H1: Down
Intent Bit: Commissioning
Weather: Low wind, sunny
Primary 0.03 – 0.1Hz: ~1 um/s, rung up from a 7.2 magnitude earthquake near Papua New Guinea
Secondary 0.1 – 0.3Hz: 0.1 um/s
Outgoing Operator: Ed
Quick Summary: Waiting for earthquake to ring down before we re-lock
