Will comment this entry when leaving
We are still losing lock more than 50% of the time once we start to increase the power into the interferometer. I slowed down the rotation stage to 10% of its normal speed. In this way I was able to make it to 10 W once. On the second try, the interferometer lost lock right after a gain step in the summing node board. The behavior of the CARM error signal does not seem any different from past successful power increases, so it is not clear to me what could be wrong with the summing node board.
In order to make it easier to engage the soft loops, I made the 20 dB boosts come on only when the soft loop error signals are less than 0.01 ct.
I also turned down all the hard loop gains by 6 dB and re-engaged the bounce mode notches in the suspension PUM drives, because the bounce modes were becoming unmanageable.
Today I redistributed gain in the CARM and IMC boards so that we can use the CARM gain slider to power up, rather than the SNB gain slider. I powered up to 10 W three times and didn't have any locklosses
I reoptimized the green QPD offsets for the Y arm in full lock.
TITLE: 06/11 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: None
SHIFT SUMMARY: We have had good locking all night, getting to INCREASE_POWER for the first time in a long time. It has been a bit windy, but just under the threshold for locking.
LOG:
Locking, commissioning, locking, commissioning . . . Mostly keeping Evan company.
As noted in alog 27683, turning on the 10dB gains in the M0 Lock filter banks for increased offloading speed was killing the lock. We saw that again today. I turned on all the filters by hand, and it seems that it's the ETMY Y that is the problem - all others are fine. Sheila pointed out that ETMY has filters for Y2Y and P2P in the M0 drivealign banks, but no other quads do. We turned off the invY2Y filter, and we were able to run the EngageSoftLoops state. We didn't turn off the invP2P, but we probably should so that all the quads are the same.
Title: 06/10/2016, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT) State of H1: IFO unlocked. ITMX ISI Stage 2 WD tripped. ETMX RMS WD tripped. Commissioning: Outgoing Operator: None Activity Log: All Times in UTC (PT) 14:48 (07:48) Reset ITMX ST-2 WD trip. 15:00 (08:00) ETMX RMS WD tripped. 16:16 (09:16) Kyle – Going to Mid-Y for CP3 fill 16:21 (09:21) Richard – Going to End-X to work on RMS WD 16:30 (09:30) Kyle – Back from Mid-Y 16:36 (09:36) Ed – Going to Mid-Y to look for test board 17:04 (10:04) Ed – Back from Mid-Y 17:05 (10:05) Richard – Back from End-X – The RMS WD has been reset 17:35 (10:35) Karen – Cleaning the H2 building 17:37 (10:37) Richard – Going to End-Y CER 17:48 (10:48) Richard – Finished at End-Y 19:15 (12:15) Sheila & Ed – Going into squeezer bay looking for electronics 19:34 (12:34) Sheila & Ed – Out of LVEA 20:39 (13:39) Ed – Going to Mid-Y 23:00 (16:00) Turn over to Travis End of Shift Summary: Title: 06/10/2016, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT) Support: Jenne, Sheila, Jim, TJ Incoming Operator: Travis Shift Detail Summary: Reset ITMX ISI St-2 WD. ETMX RMS WD is tripped. Per aLOG entries tried toggling the Reset from 1 to 0 back to 1. No change. Cannot lock X-Arm until this is cleared. RMS WD is reset. Did an initial alignment, and started trying to lock. Commissioners continue working through various locking problems
Some time ago one of the water manifolds for the laser failed. In dismantling the manifold, traces of PTFE tape are evident in the bores of the flow meter and pressure regulator. There are no signs of the anodising from the homogeniser or fibre bundle holders in the fittings.
I had a look at lock loss times and their durations during O1 (only lock losses from nominal low noise). I've had a brief look at a couple of the physical environment channels (H1:PEM-EY_WIND_ROOF_WEATHER_MPH for wind and H1:ISI-GND_STS_ITMY_Z_BLRMS_100M_300M for seismic) to see how they are correlated to these lock losses. There are a few plots I've attached here below. All of the data for these plots are mean minute trends
Plots:
Duty_cycle_noncumulative.pdf - This is a 3D plot showing the duty cycle for different wind and seismic bins (percentiles)
Duty_cycle_cumulative.pdf - This is a 3D plot showing the duty cycle when the wind or seismic behaviour is greater than or equal to the given percentiles
Downtime_wind_seis.pdf - This is a 3D column plot showing the integrated 'downtime' that results from a lock loss in the given wind and seismic bins (percentiles)
lock_losses.pdf - This plot shows the number of lock losses per percentile bins, surf (matlab) makes it look a bit weird.
I've also attached lock_losses.txt which is a list of the times in GPS when the lock losses occur (first column) and their duration in seconds (second column). The durations and loss times are only to the nearest minute unfortunately since I used minute trend data.
Percentiles.txt contains the wind (first row) and seismic (second row) channel values that correspond to 5% intervals starting from the 0th percentile (0,5,10...). These are for the mean minute trends though which are much lower than the maximum minute trends.
The scripts used to generate these plots are located at: https://dcc.ligo.org/T1600211
I have built a 24-hour version of SenseMonitor on the redundant machine (h1dmt2) and it is apparently running now. I also modified it to initialize the dmtviewer time series from the minute trends (this should get rid of the errors that showed up every time the dmt was shut down without stutting down SenseMon cleanly in advance). This version is available right now as the SenseMonitor_rhoft_H1 monitor. If it looks like it is working well, I will make this version the default. A few notes: a) as of now, the the rhoft version will not come back without a little fidgeting. Let me know it stops working. b) because of (a) it would be good to get a definitive statement about whether this is what is desired so I can install the modified code. c) Jim requested that the _CAL_ version be modified but I changed the _hoft_ version. This was purely for ease of testing. Once I install the new executable, all versions will be 24hours. If you want some to be 24 and some to be 12 hours, I can fix that by juggling configurations. d) Yes, Chris and Vern, This means that I can now start up a BBH SenseMon.
Attached are two illustrations: The first is a notional schematic of the pum rms "watch dog" showing the interconnections between the rms circuit, the "trigger logic", and the driver amplifier. The second drawing shows the signal levels through the circuitry. The information for these illustrations came from drawings D070483, PUM Coil Driver Sheets 1 & 2, and D0900975, Monitor Board. (The D0900975 is an annotated update of the old D070480 Monitor board.)
The resetting of the rms-wd is accomplished by either powering off, waiting a minute, and powering back on, which allows the reset circuit (R17, R18, C18, and U15B) to clear the D-flip-flops or a High-to-Low voltage transition on P18 of connector J7 on the PUM Coil Driver card. There is a binary I/O and software (MEDM button) available to do this remotely. Richard M. and Fil C. are checking the integrity of this. The state of the trigger D-flip-flops (tripped or not-tripped) is available on the MEDM screen for the PUM drivers.
0915 - 0930 hrs. local -> To and from Y-mid Opened exhaust check valve bypass valve. Opened LLCV bypass valve 1/2 turn -> LN2 at exhaust after 28 seconds -> Returned valves to as found state. Next overfill to be Monday, June 13th.
SEI: BRS upgrade at End-X underway and going well. SUS: RMS WD at End-X is tripped and cannot be reset. Being worked on. VAC: Installing RGAs at both mid stations. All other groups report no outstanding issues.
Sheila, Evan, Carl Terra
At the end of the night we had several locklosses when trying to engage the soft loops. It seems like the problem could be turning on the 10 dB more offloading gain in the top mass, which we have been doing without a problem for several weeks now. In the last lockloss like this, L2 rms watchdogs tripped on all 4 test masses (not all coils). Evan manually reset the Y arm optics, but we couldn't reset the X arm optics. For ETMX, Terra and I went to the end station and tried power cycling the coil driver, but this didn't allow us to reset the watchdogs. Carl and Evan power cycled the ITMY coil driver, and were able to reset the RMS WDs but the software WD cannot be reset.
Also, we have had several EQs tonight (both large and small) and tried out turning sensor correction on and off.
Here is a time series of L2 master outs durring the lockloss where the RMS WDs tripped. This lockloss happened durring a 15 second sleep in the guardian, which is why down was not run for so long after the lock was lost. We've eliminated this particular long sleep but there are other places where we have similar sleeps that need to be fixed. We went through the guardian states up to DC_READOUT this afternoon, rewritting in places that have long sleeps. We've tested this up to ENGAGE_SOFT_LOOPS.
The version of ISC_LOCK committed just before starting these edits is 13570, the version after is 13576.
TITLE: 06/10 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: None
SHIFT SUMMARY: H1 was locking fairly well until the commissioners left for their dinner break. Then we were slammed by a series of EQs (6.1 in Nicaragua and 3 aftershocks and 5.9 in Solomon Islands). Since then, things have not been locking so well.
LOG:
3:46 Sensor Correction turned off in anticipation of incoming EQ.
3:57 ITMx and ITMy ISI WD tripped.
4:35 Attempted to start locking again, but could not get past DRMI. Realized this was due to another incoming EQ. Set IFO to DOWN.
5:58 Started Initial Alignment
6:35 Finished IA, back to locking
Bounce damping setting was changed for IY and EX so they damp at all. Don't know why the old settings don't work.
For EX I just flipped the sign.
For IY, I rotated the phase by 210 degrees in effect and increased the gain significantly. Gain increase might be unnecessary but I leave it like that because it seems to work faster.
| Old | NEW | |
| EX |
Gain= -0.03 (negative), FM1 (+60dg), FM4 (bp) |
Gain=+0.03, FM1 (+60dg), FM4 (bp) |
| IY |
Gain= -0.03 (negative), FM1 (+60dg), FM3 (BP), FM6 (+30dg) negative sign + 60 deg + 30deg = -90 deg total |
Gain= -0.15 (negative), FM2 (-60dg), FM3 (BP) negative sign -60 deg = +120 deg total |
ISC_Lock was changed but not loaded.
Travis and I added a filter with negative gain to IY bounce (and EX) damping, so the gains for all bounce and roll damping filters will be positive from now on. The guardian is updated for this.
We found that in the state DARM offset, we got good damping for ETMX with a positive gain and -60 degrees of phase. This worked better than the settings that Keita described in the DARM offset state, but we also used Keita's settings sucsesfully in DHARD WFS.
TJ and I saw that with the Keita settings at DARM_OFFSET we were ringing up the bounce mode for ETMX. So, now in the Resonance state the bounce damping gets set to the Sheila settings. Unclear why this one needs a 120 deg phase change but no others do. Perhaps related to why the settings needed changing in the first place? When the bounce damping is first turned on in DHARD_WFS, it goes back to the Keita settings.
We have been using a bandpass filter for ETMX bounce mode damping that is designed for the ETMY bounce frequency. For the other optics we use a broad bandpass, so I've now set the gaurdian to do that for ETMX as well. The new settings are:
FM3 (broad BP), and FM8 (gain -1). These are in the guardian for now, to be used in all states, so I've commented out the filter change in RESONANCE.
When we have the notches in the ASC off, we can damp the bounce modes as seen in the DARM spectrum. However, even when they appear to be reduced in the DARM spectrum they seem to be increasing in the MICH ASC signals.
I copied quad bounce and roll notches to the BS ASC loops as well.
I looked at a 1 mHz spectrum of the bounce modes from last nights lock, and the frequencies are consistent with what we reported one year ago:
| ETMY | 9.731 Hz |
| ETMX | 9.776 Hz |
| ITMY | 9.831 Hz |
| ITMX | not rung up last night but should be 9.847 Hz |
The confusion about the ITMY mode frequency (corrected by Evan in the comment) was unfortunately propagated to our monitoring filters. I've corrected the monitor bandpass for ITMY and made all of the bandpasses 10 mHz wide (before they were all 30 mHz wide, meaning that the monitors could not distinguish between the ITMs).
Sheila and I rewrote the analog CARM transition so that we now hop directly from the digital normalized in-air REFL signal to the analog in-vacuum REFL signal (rather than the analog in-air signal). This has worked twice in a row, so maybe this will help us avoid some of the CARM switching instabilities.
While I was checking through the list of running nodes, I noticed that there were a total of 5 TCS nodes. It seems that the old TCS_ITMX diagnostic node was still running, but no one knew since it was not on the overview or anywhere else. Since the newer TCS_ITMX_CO2 node has replaced this one, I destroyed the node after confirming with Nutsinee.
DAQ EDCU was briefly GREEN until this node was removed, now back to PURPLE.
ETMX Mode3: FM1, FM2, FM4, +10 gain (BP, -60deg, 100dB) seems to have worked quite well. No need to turn on a damping for 505.707 (which should be checked again with 0.0001 BW to see if it's really exists). 505.707 should be watched while damping 505.710 just in case.
The new violin mode table is being uplated slowly. The modes with Damp Setting column filled out are the one that's been sucessfully damped by the filter settings indicated in the table.
Jenne, Sheila
With the long lock tonight we got to do some more violin damping. I haven't updated the wiki or guardian.
At first we thought the largest mode was the 505.71 mode that Nutsinee logged about, and tried to damp it without sucsess using the settings she described. After getting a 0.002 Hz spectrum we could see that it was really 505.706Hz. (on the wiki there is a note asking if this mode really exists, looks like it does) I tried to damp this with ETMX MODE 2, which has a narrower filter, and in the last 5 minutes of the lock it looked like a phase of -60 and gain of 30 was working, with the 505.71 notch engaged, but we unlocked before we could be really sure it was working.
We damped the mode at 506.922 (ETMX mode5) using a phase of 0 degrees and a gain of 30. FM1 and FM4. (This is different from what is on the old wiki) the new wiki has this mode not damped.
We also damped a 500.062Hz mode (ITMX mode 1, 0 degrees gain +30, also different from the wiki) With higher gain and the same filter settings we rang the mode up.
The top panel shows the fundamental violin mode forest at the begining of the lock in blue, and the end in red. (There is a tiny improvement in 505.707Hz) The bottom panel shows the 0.002 Hz spectrum with cursors at 505.71 and 505.707Hz.
A negative gain for the 505.707 mode worked for me (although it's not clear whether it damped 505.707 or 505.710).
The setting I had seems to work on both 505.707 and 505.71.
