During this morning's PSL ISS code change, I tested if restarting h1pslpmc model with the FLOW IIR filter replaced with an EPICS part still closes the shutter, and indeed it did.
Looking at full data DAQ trends at the time of model startup shows what is going on. The FLOW ADC data is downsampled from the 64kHz IOP model to the 32kHz h1psliss model. The downsampling filter starts the data at zero and takes several cycles to settle to the digitized value. As an example the plot below shows an LSC channel at the time we restarted h1lsc yesterday. In the PSL PMC case, since any instantaneous FLOW value outside of the acceptible range causes a shutter close trigger, this will always close the shutter on model startup.
One solution we can think of is to add code in the FLOW_ERR C-file which triggers FLOW_OK to become False only if the flow values are out of range for several consecutive cycles. For a decision on whether the issue is of enough concern to warrent this code change we'll defer to the PSL team.
This morning the RMS watchdogs for all of the test masses have been tripping, usually during locklosses at higher ISC_LOCK states. Resetting these requires opening the tripped Quad overview, and setting H1:SUS-*TM*_BIO_L2_UL_RMSRESET to zero, then back to 1. Since I've had all 4 trip at once, and Verbal_Alarms yells each one at you multiple times, I made an alias in my .bashrc profile to do this. Resetting all of the RMS WDs doesn't seem to effect anything, or disrupt lock, so this alias can be used any time one of these WDs trips. Entering rms_wd in a terminal now sets the WD bit to zero, sleeps for a second, then sets it to 1.
Operators can easily add this to their .bashrc file from a terminal by copy/pasting :
echo 'alias rms_wd="caput H1:SUS-ITMX_BIO_L2_UL_RMSRESET 0 && caput H1:SUS-ITMY_BIO_L2_UL_RMSRESET 0 && caput H1:SUS-ETMY_BIO_L2_UL_RMSRESET 0 && caput H1:SUS-ETMX_BIO_L2_UL_RMSRESET 0 && sleep 1 && caput H1:SUS-ITMX_BIO_L2_UL_RMSRESET 1 && caput H1:SUS-ITMY_BIO_L2_UL_RMSRESET 1 && caput H1:SUS-ETMY_BIO_L2_UL_RMSRESET 1 && caput H1:SUS-ETMX_BIO_L2_UL_RMSRESET 1"' >> .bashrc
You then need to open a new terminal to get the new alias.
Terra, Sheila
After recovery tonight we did a few things:
We have noticed that since we are going to higher powers, things are misalined after a lockloss and become realigned as they cool off. We might want to account for this someday to save time.
Log was meant to be posted ~3:30 this morning.
WP 6122 h1psliss model change (and subsequent DAQ restart)
Keita, Peter, Patrick, Dave:
We installed Keita's PSL ISS model change on h1psl0. I also restarted the PMC model to see if it closed the shutter (it did) so now I have DAQ data on this problem.
We restarted the DAQ for the new ISS INI file. This also applied the FMCS changes (addition of Relative Humidity channels).
This will generate an alarm on pump level. Filling at 45% open on LLCV, past 100% full until I see a change in flow meter curve or until ~20 minutes go by (takes around 30 min. for LN2 to spew out the exhaust).
J. Kissel, J. Warner As Jim brought the instrument up this morning, we were alarmed to see a forest of lines around 20 to 25 Hz. As they did not disappear or change height as we went through the lock acquisition sequence, as ASC loops turned on, noticed them in MICH and PRCL, found a whole bunch of ASC-ADS SDF differences, and I heard Shiela suggest that she might try an LLO scheme for dithering the alignment, I began to suspect that these lines were intentional. Indeed, after poking around, I found the dither alignment overview screen and found several oscillators pushing out excitations at 19.1, 19.7. 20.1. and 20.8 Hz in Pitch and 21.3. 21.9, 22.3, and 23.0 Hz in yaw, being sent to PR2, PR3, and the BS. I confirmed that these lines actually make it to the SUS by gathering an ASD of the input control request at the bottom of PR2, PR3 and the BS indeed I see the same features in PR3 and BS (why not PR2?). *phew!* We'll leave these on, in case the plan was to just gather data with these lines present. See attached ASD of input request to the SUS, a screen cap of the Dither Overview screen, and the corresponding DARM ASD.
For the record and/or future study, this lock stretch -- chilling in DC_READOUT at 2W -- lasted ~1.5 hours, from 2016-08-26 16:10:26 UTC to 2016-08-26 17:47:?? UTC. This would be an excellent lock stretch to, for example, pull out the 3501.3 Hz PCALX to DARM transfer function. From DTT alone, I was able to estimate the TF to be: mag: 2.531e-6 [m/ct] pha: -40.885 [deg] re: 1.852e-6 [m/ct] im: -1.682e-6 [m/ct] coh: 0.607 BW: 0.0025 ENBW: 0.00292 nAvgs: 35 unc: sqrt((1-coh)/(2*nAvgs*coh)) = 0.096 = 9.6%
Late entry from yesterday. The AC power for the Kepco high voltage supplies (provide ±430V to the LN ITM ESD Driver) were connected through the safety interface chassis D1400047. Interlock enables/disables the high voltage power supplies. The safety relay is controlled by the output of the Inficon guage installed on BSC7. The positive high voltage supply was also replaced. Unit was found tripped and even after power cycing, would trip again every 10 minutes or so. One thing we noticed was the screen would lock up and then go blank.
Started Famis Task to check all the CPSs weekly. This noise has not been seen since June3, aLog 27549. Will move FRS 5615 to pending and recommend closing. Maybe power outage 6 June helped.
100mL of water was added to the Xtal chiller to bring the sight-tube level to MAX. Xtal flow is 20.1 l/min. Diode chiller was not in alarm and flow rate showed 31.5 l/min. FIlters showed no discoloration to note.
The front end pump diode temperatures were adjusted to increase the laser power.
- D1 was 26.6 degC is now 26.4 degC
- D2 was 24.6 degC is now 24.4 degC
- D3 was 27.8 degC is now 27.5 degC
- D4 unchanged
The temperature changes increased the output power by ~300 mW.
Increased the diode currents for the front end laser to bring the power back to where it should be.
- D1/D2 was 45.5 A is now 46.0 A
- D3/D4 was 47.5 A is now 49.5 A
The net result is that the front end laser power increased by 1 W. That helps to preserve the
injection locking bandwidth a trifle.
The temperatures of the high power oscillator pump diodes was also adjusted. A 4 W increase
was obtained. The diode currents were not touched.
Observatory Mode taken from Unknown to Commissioning
Some recovery from Maintenance Thurs (Roll Modes and checks of violin modes). Rest of the evening consisted of Commissioning work (Jeff
Worked on damping out Roll Modes of ITMx (with Sheila's tutorial). Spent about ~30min (started at 4:58pm)
Received some Test Mass Coil WD Verbal Alarms (i.e. ETMX RMS ['LR'] WD has tripped). And this was a first for me. Kissel pointed me toward how to untrip these (Jeff said this WD is for the Test Mass penultimate masses (which is L2?). For L2, the WD Trips are located on the right side of the SUS screen (RMS WD RESET). To clear, put in a 1.0 for the UL & then take it back to 0.0. This seems to take care of it for all the coils! Kissel was concerned that these sort of WD trips could be a sign for violin modes ringing up.
SIDE NOTE: The operator electric table apparently couldn't take my weight & failed. Please do not use this table until it is repaired.
Please do not sit on the electric table at the operator's station. It is not intended to support your weight.
Bubba repaired the table this morning.
We had a couple locklosses when the ISS saturated, both with the third loop on and off. We just increased the ref signal to -1.52 V to move the diffracted power to around 4%. Was some setting incorect after a reboot this morning? SDF indicated that everythind is OK.
The 1st loop's reference offset voltage (H1:PSL-ISS_REFSIGNAL) is not monitored in the SDF system. During O1, because of the alignment drift in the PSL and or interactions with the 2nd loop, the operators would often adjust the diffracted power with this offset. I've now re-monitored the channel, for now, because we don't have any restriction of SDF difference to do science. Keita restarted the PSL ISS model again this morning, and we had begun relocking too quickly (!!) after that reboot, so we had to adjust the offset on the fly, when it was have too late. This resulted in a close, but not the same, value of -1.602. Also had to restore the "AOM Offset" H1:PSL-ISS_CTRL_OFFSET, which should be 3.85 (it had come up as 2.5). I've accepted both the -1.602 offset voltage and 3.85 AOM offset in the SDF's safe.snap and down.snap for the PSLISS front-end.
Note that the H1:PSL-ISS_CTRL_OFFSET used to be 2.5, but yesterday Keita and I changed it to 3.85. Thank you Jason & Jeff for pointing out that I forgot to write an alog.
We made this adjustment with the ISS first loop off, and set the offset such that the diffracted power was a little over 3%. Before doing this, the diffracted power was 1.7% with the first loop off, and we kept hitting the bottom rail, which would send the first loop into oscillation and cause it to unlock. We were able to close the 1st loop, and everything looked okay. So, 3.85 is the new best value.
With all that said, we've had a few times over the last 3 days of hitting the bottom rail of the ISS when we've got the IFO locked, and it changes the power injected into the vacuum and the IFO oscillates a bit although usually retains lock. We may need to consider increasing the first loop offset even more, so that we're a teeny bit farther away from that rail.
Running a Worden experiment: -CP4 at 88% full -Set LLCV to nominal in manual mode (41% open) -Increase LLCV by 5% (43% open) and leave it at this value -Monitor increase in fill -Allow fill to exceed 100% and monitor exhaust flow meter (expect a sharp increase in flow when liquid starts to hit warm exhaust) -After this sharp increase, reduce LLCV by 10% to below nominal (<41%) -Hope that no LN2 reaches the flow meter this time (when flow approaches 100% I will monitor while sitting next to exhaust to be ready to open bypass exhaust valve and remove flow meter with LN2 gloves on) **THIS WILL GENERATE CP4 ALARM ON PUMP LEVEL**
Note: this is done with Dewar at 39.6% full
I increased LLCV to 45% (10% increase from nominal) to speed things up. 5% LLCV increase is about 1% fill increase every 40 min.
Taking too long to reach critical point (>100% full), so I'm terminating this experiment before alarms start (>98% full), and will restart early tomorrow at 90% full set point.
the Verbal Alarms code was logging to the ops home directory. Prior to the move of this home directory (WP5658) I have modified the code to log to a new directory: /ligo/logs/VerbalAlarms We restarted the program at 14:04 and verified the log files are logging correctly.
These verbal log files actually live one level deeper, in /ligo/logs/VerbalAlarms/Verbal_logs/ For the current month, the log files live in that folder. However, at the end of every month, they're moved into the dated subfolders, e.g. /ligo/logs/VerbalAlarms/Verbal_logs/2016/7/ The text files themselves are named "verbal_m_dd_yyyy.txt". Unfortunately, these are not committed to repo where these logs might be viewed off site. Maybe we;ll work on that. Happy hunting!
The Verbal logs are now copied over to the web-exported directory via a cronjob. Here, they live in /VerbalAlarms_logs/$(year)/$(month)/
The logs in /ligo/logs/VerbalAlarms/Verbal_logs/ will now always be in their month, even the curent ones.