After seeing that stepping the ITM ring heaters up in common didn't make improvements to the IFO (alog 47851), I have made a step down in common. This means that we have to be watchful for any ITM violin modes to go bonkers, but so far (3000 sec after the step) things still seem okay for now.
Just before doing anything, Patrick took the IFO out of Observing for me, so that we can easily flag that this is when the change happened.
I followed the instructions in Danny's alog 47770, although the ITM ring heater guardians were already in the FILTER_RH_INPUT state, presumably because they needed to be there for a few days after the Saturday night test. I changed H1:TCS-ITMY_RH_INVERSE_FILTER_IN from 1.3 W to 1.1 W, and I changed H1:TCS-ITMX_RH_INVERSE_FILTER_IN from 0.4 W to 0.2 W. I think I'm a little confused as to what those really mean, since the requested and measured power for each segment on ITMY is going to about 1.2 W, not 1.1 W, and the requested and measured power for each segment on ITMX is going to about 0.3 W not 0.2 W. But, it's still a step down in power, roughly in common, so the step will still show us if this is the direction we want to go.
As soon as I had made the ring heater change, I remembered to turn on the AWG_LINES guardian injections for intensity, frequency, and 9 MHz RFAM. So, I won't have peak heights to compare to from before any changes happened, but the lines are at least going now, and we can see how they change from the start of the step to the end of the step, and back to nominal (if I go back to nominal). I requested the AWG_LINES guardian by hand to go to INJECTING, but later realized that I could have just done it with the ISC_LOCK guardian.
To then get the IFO back to Observing, I accepted the SDF diffs for LSC, PSLISS and one of the Beckhoff modules to let the excitations for my injections go through. I also had to uncomment the line in ISC_LOCK, so that it would think that the AWG_LINES guardian should be in the injecting state, not the idle state, so that the ISC_LOCK node would be happy with its subordinate. I also (while in the IDLE state of AWG_LINES, after having manual-ed there from INJECTING) changed the nominal of that guardian to be INJECTING, so that it would report OK, so that ISC_LOCK would report OK, so that we could go to Observe.
After these minor guardian shenanigans, and the SDF accepts, I put the IFO back into Observing. (Sorry Patrick - I should have let you know I was done and let you do any checks necesssary)
Betsy, Rahul
If any of the Violin modes are rung-up high (say peak excitation above 4.5 or 5 or ASD 10^-16 m/sqrt Hz level, please follow the instruction given below (or see the attached pdf file).
The up to date instructions for manually damping the run-up violin modes is now available on the wiki page. You can access it through Justin's LHO Launch Pad (Click on LHO Operator Wiki, then click on Violin_Mode_Table_v2).
https://cdswiki.ligo-wa.caltech.edu/wiki/Violin_Mode_Table_v2
The other way to reach this link is through VIOLIN_ALL_MONITORS.adl medm screen. Click on the 'Open Violin Mode wiki button'.
TITLE: 03/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 107Mpc
OUTGOING OPERATOR: Travis (for Cheryl)
CURRENT ENVIRONMENT:
Wind: 21mph Gusts, 17mph 5min avg
Primary useism: 0.07 μm/s
Secondary useism: 0.26 μm/s
QUICK SUMMARY:
Jenne has made a change to the ITM ring heater settings.
Coherence plots from the current lock, BW = 0.01, Averages = 1000, 0 to 60Hz, show peaks below 2Hz.
Alogging for reference.
TITLE: 03/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 109Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: H1 has been locked for ~20 hours and in Observing for most of that time.
LOG: Covering for Cheryl for last 1.5 hours of day shift.
22:43 Nutsinee to optics lab.
NikoL, JasonO, RickS
We recently noticed that the output power from our spare Pcal Tx module that is used for Working Standard / Gold Standard responsivity measurements had significantly degraded output power (~0.5 W, down from 2 W) and very poor beam quality.
Today, we adjusted the pump laser temperature (potentiometer W1) and were able to recover most of the power. The pump laser current (potentiometer W3) seems to be at about the maximum level already.
In the end, we have the following power levels:
Thus the diffraction efficiency is a bit better than 70%.
The beam quality looks reasonably good.
The maximum OFS PD output is about 7.6 V, so we set the operating setpoint at about 3.5 V.
At this offset, the power downstream of the wedged beamsplitter is about 0,6 W, so we expect about 0.3 W in each beam for the power standard ratio measurements.
I had a brief look at some data from a short ring heater test that was done Saturday night. It's not alogged when the change was made, but Ed alogged when he did the step back (alog 47797). It looks like both ITM ring heaters were increased by 0.2 W total (0.1 W upper, 0.1 W lower) for one hour, and then lowered back to their nominal values.
Attached is a screenshot showing the step times, some of the circulating powers and power recycling gain, as well as spectra taken at 2 times. The spectra are from the times of the 2 vertical cursors on the time series plots. The step up in ring heater power was done about an hour into the lock, so there are still some thermalization effects going on. But the step down an hour later helps make things more clear. In the spectra, red is the hotter ring heater state, and blue is the nominal ring heater state. Recall for the spectra that we'd like all of the peaks to be smaller, indicating a lower coupling of that noise to DARM. For POP90 we'd like it low, and for PRG and POP18 we'd like them high.
So, overall, I conclude that the nominal common ITM ring heater values are better than the hotter ring heater values. This is opposite of what one might expect with the increase in total circulating power in the IFO, if we assume that we had fine tuned the TCS for our previous 30W settings. But, we hadn't actually done the fine tuning for TCS at 30W.
WP8139 Remove inverse-filter feedback
Danni, TJ., Dave:
Attached are images of a ring-heater inverse filter section before and after the changes applied 18:00 fri22mar2019.
The feedback from the INVERSE_INVERSE_FILTER_OUTVAL back to the EpicsInCtrl input part was removed. The EpicsInCtrl part was replaced with a standard EpicsIn. This meant that the mask record controlling the EpicsIn part (INVERSE_FILTER_MASK) was no longer needed. Given that this change was being made on a Friday night, I decided we will keep this part in the model, which meant that a DAQ restart was not needed when the model was loaded.
At this point in time, I will recommend we continue to keep the non-functional INVERSE_FILTER_MASK parts in the model, since removing them will cause MEDM and or Guardian problems. I am annotating the h1tcscs simulink model with a full description and a reference to this alog.
TITLE: 03/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 106Mpc
INCOMING OPERATOR: Travis, covering
SHIFT SUMMARY: Morning Meeting to plan Maintenance, Commissioning Meeting to plan the week. H1 locked all day
LOG:
Commissioning Plan for the week: Calibration is top priority next few days, commissioning as available/needed, later in the week ramp up for running over the weekend.
Dave loaded some filters in software being loaded tomorrow, so there is one red dot on the CDS overview, on the line H1EDC, which does not effect H1, and will green up tomorrow.
Inadvertently tested my new CRC bit for the external EDCU (h1edc) this afternoon. I was prepping a new H1EPICS_CDSMON.ini file for tomorrow's maintenance, and forgot my new code which generates the H1EDC.ini file every minute so h1edc can flag the CFC bit in its STATE_WORD. This has no impact on H1 until we install this system during tomorrow's maintenance, it is just cosmetic and ruins our perfectly green CDS overview.
Morning Meeting - plan for Maintenance:
Jenne, Nutsinee
Just in case we need one in the future, SQZ_MANAGER guardian can just look at this value and then decide which path to take.
Based on measurements Niko and I made last week (see Log 47669 and Log 47668) the ranges, limited by AOM diffraction saturation, for the Pcals are:
Xarm 3.7 V x 2^18 counts/20 V = 48,500 counts
Yarm 2.725 V x 2^18 counts/20 V = 35,700 counts
These are a bit lower than during O2, especially at Yend. We plan to adjust AOM alignment at Yend when time allows. That would hopefully give us more range at Yend.
Damping of rung up violin - dropped OBS MODE for ~20 mins:
Sheila pointed us to set the Violin Mode Guardian state to DAMPING_ON_SIMPLE which then allows for manual tuning of the gains of any mode. (At the moment, the guardian looks to see if the mode is ringing up, then turns off the gain and hollers at the operator. It does not do any auto-intervention.) The manual intervention is:
- Go to violin Guardian DAMPING_ON_SIMPLE
- Adjust gain of problem violin mode (in our case turned back on the -3 gain for ETMX mode9)
- Watch on ndscope for mode to turn over, also watch others since now guardian isn't looking at any mode
When mode looks like it has turned over and is ringing down
- Set guardian to IDLE (to recollect the current status)
- Set guardian back to DAMP_ON_DC (guardian back on watch duty now)
- Go back to OBS MODE
I have updated the CDSWiki page which shows the step by step process for damping the violin modes. The link to this wiki can be found on the VIOLIN_ALL_MONITORS.adl medm screen (click - Open Violin Mode Wiki). I will update the link on the LHO Operator wiki page (since this the troubleshooting page for the Operators present during the shifts).
The old sets of instructions (valid for O2) is now obsolete (old version - ViolinModeDamping101).
TITLE: 03/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 107Mpc
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
Wind: 8mph Gusts, 5mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.26 μm/s
QUICK SUMMARY: Great long lock - a violin mode ringing up at 500Hz
Ops Shift Log: 03/25/2019, Owl Shift 07:00 – 15:00 (00:00 - 08:00) Time - UTC (PT)
Violin mode-9 (ETM-X)is ringing up, and am unable to damp it. Guardian message is to turn off gain, however the gain on the filter bank is 0.000. When I try to change the gain, Guardian (I assume) is overriding my changes and putting it back to zero.
Looking at the Guardian VIOLIN_MODE.STATE screen, there is a state to turn off the DAMPING for a suspension, but not the gain.
Could find no information in the WIKIs or in the aLOG that addresses this situation. Leaving the IFO in Observation mode and hoping a commissioner comes in before it breaks lock.
IFO has been locked in Observing for the past 7.75 hours. The input power is 35.2w. The IFO has been a bit glitchy all night, but has remained locked. The RF9 and RF45 plots are bouncing around, which may be influencing the glitches. Environmental conditions are good. No outstanding concerns or issues at this time.
attached are photos of the Guardian Nodes which I took action on.
Does ISC_LOCK not take SQZ_MANAGER to its nominal state anymore? Or did ISC_LOCK requested SQZ_MANAGER nominal state and failed? What's in Inject_squeezing state in the ISC_LOCK?
If ISC_LOCK requested SQZ_MANAGER to go to nominal state but failed for got stuck, can I have more details of how it fails next time this happens? Including guardian log, screenshots, would be helpful for improving squeezer automation.
ISC_LOCK takes the SQZ _Manager to its nominal state during normal locking, but with what happened tonight (would we call this the "Squeezer losing lock" or something else?), the ISC_LOCK & ALIGN_IFO were transitioned to "NOT OK" states, when the Squeezer did what it did.
Luckily I was able to get the SQZ Guardian nodes mentioned above to their nominal state. Otherwise, ISC_LOCK & ALIGN_IFO would have remained "Not OK" and would prevent OBSERVING. Alternatively, if one is comfortable with changing the ISC_LOCK & ALIGN_IFO scripts, I reckon there must be a way to break this remaining connection they have to the squeezer.
Anyway, when I looked at SQZ_MANAGER and SQZ_LO_LR, they were not actively doing anything according to their logs. SQZ_LO_LR was down and staying there. After I (1) tried getting it to its nominal state, and then (2) getting SQZ_MANAGER to its nominal state, then ISC_LOCK & ALIGN_IFO were back to being OK.
Since this was easy to recover from, maybe it's good to have it go down in this way? Mainly because recovery was fairly straightforward? And we also get a heads up that Squeezer's state has changed.
Oh, and good idea about posting more info. Attached is screenshot showing all the logs of pertinent guardian nodes from this event. Now that I look at them, it looks like ISC_LOCK & ALIGN_IFO probably did not take us out...but they point the way to SQZ since they give the messages when they were listed as "not ok" (but their logs don't show anything until the very end when I go back to Observing).
I reckon it's the SDF diffs from the squeezer which took us out of OBSERVING.
Hope this helps!
Oh, and the other thing from the screenshots is that you can see ISC_LOCK definitely does not do anything to the SQZ_MANAGER when this happened. (but ISC_LOCK & align_ifo were listed as NOT OK....and this would not have let us return to Observing until we fixed their situation with the SQZ or edited their scripts).
After ~75 minutes, the circulating powers are increasing (a very small amount), but the frequency and intensity noise coupling at 75 Hz are getting worse. In the spectra, red is the original hotter, and blue is the current cooler state.
Violin modes all seem okay for now.
Just made another step, setting ITMX filter input to 0 W, and ITMY filter input to 0.9 W. Both of these are -0.4 W from where they started earlier today.
The IFO is pretty much thermalized from the second ring heater step I took today. The arm circulating powers are going up, and they are getting closer to eachother - the Xarm power is increasing slightly faster than the Yarm power.
The frequency and intensity noise couplings in the bucket are certainly worse - we likely need to do some differential tuning next. PRCL coherence is much higher now than it was earlier, suggesting (as was a suspicion) that it is mostly just reporting frequency noise, not direct PRCL noise. The MICH and SRCL feedforward look like they could use some tuning, but that's not really going to be a good use of time until we've done a bit of differential tuning.
Georgia is going to keep stepping TCS gently around tonight, and is getting the script ready to look at the injected lines.