At the request of Adam Mullavey (via a phone call from Rick), I have turned off the 5001.3Hz PCal line at EndX for some injection work. I did this by zeroing the amplitude of the line, leaving all other parameters unchanged. See attached screenshot for setting before I changed them.
In a later email forwarded by Rick, Shivaraj mentions having to change Guardian code to keep these lines off at LLO. I'm unaware of any such PCal Guardian code at LHO (my guess is that this is due to the fact that LHO doesn't use PCal for locking like LLO does), so made no changes to Guardian. If someone reading this knows otherwise, please contact myself and the current operator.
We are just starting to do initial alignment after the Beckhoff issue was resolved by Patrick and Richard. So far, so good.
~0945 - 0955 hrs local -> Kyle R. and Travis S. in and out of LVEA Suspected that the backing pump cart's pirani gauge might have increased from its steady, bottomed-out, value of 1.7 x 10-3 torr at the start of the bake to, a now steady, 1.9 x 10-3 torr value -> Closed the O-ring valve at the "hung" turbo's exhaust for several tens of seconds while Travis monitored the pirani reading -> Displayed value remained unchanged while valve closed and after valve opened (should have taken notes - initial value was probably "1.9" all along). Temperature survey a little high after yesterday's modest variac increase -> As found today, more like 95C +/- 10C -> Decreased variacs slightly. The plan now is to start the temperature step down beginning tomorrow morning such that pump hardware can be removed and PT180 be re-exposed to the site vacuum volume (i.e. return to nominal state) on Tuesday
It appears the problem is either with ISC Common L20, ISC Common M0, or the link between (see attached screenshots). I will go out to the CER and take a look.
I tried slowly power cycling the ISC Common chassis in the CER twice. Each time it came back in the same state. Since the problem is reported as being between two terminals internal to the chassis, I'm not sure what else can be done without pulling the chassis out of the rack.
I've left a voice message with Richard. I want to touch base with him before I go further.
Richard came in and we opened the top of the chassis in situ. We swapped the internal cables around and determined that the problem was with the ISC Common M0 module. This is the EK1100 EtherCAT module in the middle row of the ISC Common chassis. We replaced that module and it is back up and running. The only cable removed during this was one of the Ethernet cables in the back of the chassis. This cable should be replaced with a longer one. The serial number of the module that was replaced is 04110621.
Everything looks fine to me, but I'm no expert. See attached screenshots.
TITLE: 11/26 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Down due to Beckhoff issues
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
Wind: 2mph Gusts, 1mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.49 μm/s
QUICK SUMMARY: Down due to Beckhoff issues per Nutsinee's aLog. Have contacted Keita and Patrick. Keita is looking at things remotely and Patrick is on his way to the site.
TITLE: 11/26 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Unknown
INCOMING OPERATOR: Travis
SHIFT SUMMARY: H1 stayed locked most of the shift. Now I'm running into an issue I've never dealt with before. See alog31847 for details. This alog has disappeared magically. See alog31850 (comment below) for details.
LOG:
9:05 Noticed range slowly dropping, out of Observe to adjust SRM P
9:08 Back to Observe
9:12 Noticed that I made it worse, back to commissioning to move SRM P the other way round.
9:16 back to Observe.
13:39 Lockloss
14:24 THE confusing lockloss. Below I copied some of the Verbal Alarm message that might be useful in giving people insight of what's going on. It would be useful to know how long does the code take to loop though all the status of things.
14:23:16 VIOLIN_MODE_DAMPING2
14:23:28 TCS CO2 lasers tripped
14:23:24 ISI HAM6 WD tripped
14:23:48 ETMX ISI ST1 WD tripped
14:23:48 ETMX ISI ST2 WD tripped.
14:24:01 DOWN
My alog describing Beckhoff issues seems to have dissapeared (HOW IS THIS EVEN POSSIBLE?) so I'm copying the content here.
----------------------------------------------------------------------------------------------------
After 7 hours H1 lost lock for not so obvious reason (as always, I just haven't had a chance to investigate). This confusing lockloss occured as I tried to recover from the 7-hour lockloss. I stepped out after got through DC READOUT and came back to find that the lock broke at Violin Mode Damping 2 according to the Verbal alarm.
Couple of strange things happened (and still happening)
1) PSL power request suddenly went all the way up to 156W but the actual output got stuck at 30W. I couldn't use LASER_PWR guardian to move it back to 2W. The log repeatedly output the message:
2016-11-26T14:47:54.58953 LASER_PWR [GOTO_POWER_2W_FAST.run] ezca: H1:PSL-POWER_SCALE_OFFSET => 28.95
Seems like LASER_PWR guardian keeps outputing the offset to the channel. I can't manually put it in. I can't caput it. This offset is probably based on the input power to IMC so it make sense that it agrees with the input power. I also couldn't turn the rotation state manually on the rotation state screen.
2) Both TCSX and Y mysteriously tripped during prior to this lockloss. The medm status is READY so I went to check the interlock box. GATE and ENABLE lights are on but Remote Enable light is off. I've never seen this before. CDS overview doesn't have anything red on it. If the IR sensors saw a flash of carrier light as the lock broke, the GATE light wouldn't have been on and it shouldn't do anything to the Remote Enable light.
I'm really really confused right now. Still investigating and hopefully fixing seomthing. If somebody wake up and see this log and have some idea of what might be going on, please call the control room. Otherwise I (and Day shift op) will be calling people after 8am.
----------------------------------------------------------------------------------------------------
**UPDATE**
The rotation state medm screen seems to be stuck. It won't calculate angle requested when you put in new power, and true the other way round. The state is stuck at "busy"
According to ECAT1PLC3 SDF differences. It seems to me that TCS (and likely PSL) rotation states got stuck some how. How do I verify if Beckhoff computer controling RS is working properly?
**UPDATE 15:30**
Found CDS overview medm shows ERROR status for ECAT C1PLC1 and 2
**UPDATE 16:08**
Beckhoff error may have caused the most recent lockloss. C1PLC1 error flag came up prior to ISC_LOCK guardian went down. This is also true for the 7-hour lockloss. I don't know what's going on with C1PLC2.
The screen in alog 30842 can be used to diagnose hardware errors. It can be found from the menu SYS under EtherCAT. Each computer has one somewhere under PLC1.
Slavecountactual: The number of EtherCAT modules seen by the software
Slaveccountconfig: The number of configured modules
These two numbers should agree, indicated by slavecounterror.
Lostframes: Number of lost Ethernet frames since last reset. Any large number, or one that increases steadily is a problem.
Same goes for Lostqueuedframes.
The subscreen under workingcounters keeps track of the different types of EtherCAT frames, but is maybe more for the experts.
TITLE: 11/26 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 70.3054Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT: Wind: 4mph Gusts, 3mph 5min avg Primary useism: 0.04 μm/s Secondary useism: 0.62 μm/s
QUICK SUMMARY: TJ had it locked for me.
TITLE: 11/26 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY: PEM work for the whole shift, broke lock a few times, but recovery was pretty usual. POP90 jumped on me once, but I missed it and didn't notice till Anamaria pointed out the loss of range. I turned off the SCR1 loops and thenmoved SRM back to where it was.
The PEM crew is running the lines from alog31845
LOG:
Today we shook HEPIs, all chambers, and shook the input and output arm beam tubes at the corner station. A few of the lines in DARM were excited by the BS injections. We left low frequency lines In the cal line of ETMX L1 (0.3 Hz lenght) and ITMX L1 (0.2 Hz yaw) to produce side bands on certain lines. These lines are considerably smaller than the microseismic peak, so we didnt want them to keep us from going to observation mode.
TITLE: 11/26 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Calibration
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.46 μm/s
QUICK SUMMARY: Robert and Anamaria are doing their injections, we are currently locked at 72Mpc.
20:40 LockLoss OMC PD saturation. Robert is possibly the cause, doing low frequency acoustic injections, and suggests that maybe the ITMY oplev is sensitive to the freq range he's using. Maybe we'll try turning the OpLev damping off for next lock.
21:14 Lockloss at SWITCH_TO_QPDS
21:19 Wind has been picking up a little and blowing moderatly from the NE. X arm has been shaky. I switched ISI config to WINDY_MOD_WIND_BRSX and that seems to be holding rock steady for now, once all the nodes arrived. TRX and TRY don't look 'raggedy' any more.
21:30 Lockloss at DRMI_ON_POP
22:02 NLN. Beam Diverters closed 68Mpc. OpLev damping off experiment proved to be not good to try.
22:08 Robert out to LVEA to do PEM injections
22:07 PI mode 28 ringing up. Changed from -60ø to -100ø
22:17 PI mode 27 ringing up. Changed to 120ø
22:21 GRB alert. PEM injections happening at LHO. Not in Observing .
22:39 Lockloss. HAM6 ISI WD trip. May have been my fault playing with bp filters for roll mode while trying to damp.
23:19 back to NLN. Diverters closed 75Mpc
23:31 Manual overfill of CP3
25:51 Lockloss. HAM6 ISI WD trip
23:50 Tour(s) in e control room
00:35 Last tour out of control room. Handing off to TJ.
23:31 Took about 17 sec. to overfill CP3 at 50% open on LLCV.
This is a follow up of 31765.
Sheila reported a very interesting behaviour in this past Wednesday (31765) and also the last Friday (31559) in which lowering the power recycling gain improved the sensitivity for some reason. This reminds us about the behaviour we encountered at 50 W in the past -- we spent a good amount of time trying to increase the power recycling gain, however, ending up with worse noise in the end. So I wanted to see if the behaviour Sheila saw was a same kind or not with a hope to get a hint for future improvement. I started looking at various power signals and etc, but no conclusion yet.
[Some observations]
See the plot below.
This is a collection of some relevant signals for a lock stretch during of which Sheila was moving the PR3 spot position. In the middle plot in the 2nd row, POP_B_PIT shows the movement, changing its value from -0.2 to 0.2 counts. The upper left panel shows the improved BNS range coincided wit the PR3 move.
At the same time as the PR3 move, many power monitors dropped by almost 5% including, TR signals, POP powers. For the RF build ups, the 45 MHz sidebands seem to have dropped by the same amount of 5% by the PR3 movement, but the 9 MHz was affected a bit less. Another interesting thing is REFL LF, which decreased as the power recycling gain dropped, indicating that the interferometer approached closer to the critical coupling (assuming the mode matching stayed the same), or probably even passed the critical coupling as we have been having difficulty engaging the REFL WFSs (for example 31557).
Also, at the same time, the kappa_c value dropped by roughly 3% which is consistent with the decreased power recycling gain because the optical gain scales with the square root of the arm circulating power (when the DCPD's DC current was kept same). The cavity pole increased by 5Hz or so although I am not sure what this indicates in this case.
[Comparison with the 50 W case]
Looking at Stefan's plot in alog 28337, I see the typical behaviour that the 9 MHz sidebands drops a lot quicker than the recycling gains of the carrier and 45 MHz sidebands. This is due to the fact that the effective finesse for the 9 MHz is higher that the others, making the 9 MHz more vulnerable to losses. However, looking at the data from this past Wednesday, I found that the 9 MHz droped less steeper than the other two, see the lower right panel in the above plot. I don't know how this could happen in a smiple loss-increasing picture. Also, besides, the 9 MHz's initial drop (for 15 minutes or so at the very beginning) is smaller than the 50 W case presumably due to the less thermal lensing in the interferometer. It used to drop all the way to 40% of its maximum build up, while the recent drop goes to 75% of the maximum only.