Commissioners and the PEM crew are working when we have the IFO up, but we have had a few unknown issues causing locklosses in some of the states after DRMI making it take a bit longer to get there. Currently working to get back up to resume PEM work.
Sheila, Kiwamu
Since last wensday, we have had several sudden changes in SRC alingment that started in the middle of the week and have continued (31865 31849 31844 31823) We don't know what causes this, but it only shows up in the alignment of SRM and SR2, (as well as the OMs and OMC) and it seems to be a pitch problem.
One likely suspect is the AS36I sensing, allthough we don't have any reason to suspect it changed mid week. We decided to try running a dither loop on SRM pitch for a few days as a test. We changed the demodulated signal to SRCL control, which has a higher SNR than any of the build ups. We reduce the amplitude of the dither line from 600 counts on the clock gain to 100 counts, use a gain of -100 and no 60dB filter (FM1). We set the lock point of this loop by adjusting the P2L gain in SRM M3, which we set to -1.3 for now to get the best build ups.
We are hoping to see if problems continue in the next few days with this on, or if there are jumps in the AS36 I signals. We have added it to guardian but this is not tested yet.
CDS: The python nds2 client that the script uses could not get data today and yesterday.
FAMIS#:4703
Since the script was not working, I had to do this the old fashioned way. I'm glad this was made into a script!
ITMX P, ETMY P are getting close to out of bounds, but other than that it all looks good.
Sheila, Kiwamu,
This is a follow up of the latest AS90 jump incident reported in 31844.
We think this anomalous behaviour started in this past Wednesday (31804). This time, we looked into some other signals and found elevated noise in SRCL, although we are still unable to localize the issue.
[Time line]
[Some observations]
The symptom is very similar to what was reported in 31804. The POP 90 increased from 13 cnts to 16 cnts. Since the AS diverter was closed, it is hard to say whether there is a jump in AS90 or not. The jump also seems to have caused misalignment in SRC1 as was the case in 31804. The first attachment is trend of some relevant signals. The leftmost vertical line in each column represents the time when the POP90 jumped. The second vertical line represents the time when TJ opened the SRC1 loop and did manual alignment. Similarly to 31804, the POP90 became noisier after the jump. The LO signals in AS36 WFSs didn't show any obvious glitch or abrupt change.
This time we looked at the spectra of the AS A 36 signals and the SRCL error signal. See the second attachment. In each panel, the blue curve is taken before the jump or at evaluation point #1 (see the above list). The green curve is taken right after the jump or at evaluation point #2. Finally the red curve is taken after TJ's fix or at evaluation point #3. I was hoping to see an obvious change in the spectra that might indicate some electronics issue, but the AS 36 signals didn't really show an obvious change as far as the spectra are concerned. However, on the other hand, we found a difference in the SRCL length error signal although this is not easy to interpret. Right after the jump the SRCL noise increased in 3 - 30 Hz by roughly a factor of two. This cannot be explained by the change in the optical gain beucase the increased POP90 should mean a lower optical gain for seinsing SRCL. Another explanation might be a worse A2L coupling contaminating SRCL due to the different spot positions on the SR mirrors.
I was curious to see how the BRSX was doing, so attached is a quick 5 day trend. It seems that it may be slowly ringing down after Jim vistied on Thursday.
Even better today. Might be time to turn the damper back on soon.
TITLE: 11/27 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Travis
CURRENT ENVIRONMENT:
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.36 μm/s
QUICK SUMMARY: On our way back up after some issues described in the log below mine.
TITLE: 11/26 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
LOG:
17:50 Kyle on site checking pumps.
18:01 Patrick to CER
18:05 Robert to EY
19:50 Robert and AnnaMaria to put beam dump in MC Refl, Patrick to CER taking top off of Beckhoff module.
20:30 ECat chassis is fixed. Start relocking.
23:50 PCal X 5001.3 Hz line turned off.
At the request of Adam Mullavey (via a phone call from Rick), I have turned off the 5001.3Hz PCal line at EndX for some injection work. I did this by zeroing the amplitude of the line, leaving all other parameters unchanged. See attached screenshot for setting before I changed them.
In a later email forwarded by Rick, Shivaraj mentions having to change Guardian code to keep these lines off at LLO. I'm unaware of any such PCal Guardian code at LHO (my guess is that this is due to the fact that LHO doesn't use PCal for locking like LLO does), so made no changes to Guardian. If someone reading this knows otherwise, please contact myself and the current operator.
We are just starting to do initial alignment after the Beckhoff issue was resolved by Patrick and Richard. So far, so good.
~0945 - 0955 hrs local -> Kyle R. and Travis S. in and out of LVEA Suspected that the backing pump cart's pirani gauge might have increased from its steady, bottomed-out, value of 1.7 x 10-3 torr at the start of the bake to, a now steady, 1.9 x 10-3 torr value -> Closed the O-ring valve at the "hung" turbo's exhaust for several tens of seconds while Travis monitored the pirani reading -> Displayed value remained unchanged while valve closed and after valve opened (should have taken notes - initial value was probably "1.9" all along). Temperature survey a little high after yesterday's modest variac increase -> As found today, more like 95C +/- 10C -> Decreased variacs slightly. The plan now is to start the temperature step down beginning tomorrow morning such that pump hardware can be removed and PT180 be re-exposed to the site vacuum volume (i.e. return to nominal state) on Tuesday
It appears the problem is either with ISC Common L20, ISC Common M0, or the link between (see attached screenshots). I will go out to the CER and take a look.
I tried slowly power cycling the ISC Common chassis in the CER twice. Each time it came back in the same state. Since the problem is reported as being between two terminals internal to the chassis, I'm not sure what else can be done without pulling the chassis out of the rack.
I've left a voice message with Richard. I want to touch base with him before I go further.
Richard came in and we opened the top of the chassis in situ. We swapped the internal cables around and determined that the problem was with the ISC Common M0 module. This is the EK1100 EtherCAT module in the middle row of the ISC Common chassis. We replaced that module and it is back up and running. The only cable removed during this was one of the Ethernet cables in the back of the chassis. This cable should be replaced with a longer one. The serial number of the module that was replaced is 04110621.
Everything looks fine to me, but I'm no expert. See attached screenshots.
TITLE: 11/26 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Down due to Beckhoff issues
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
Wind: 2mph Gusts, 1mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.49 μm/s
QUICK SUMMARY: Down due to Beckhoff issues per Nutsinee's aLog. Have contacted Keita and Patrick. Keita is looking at things remotely and Patrick is on his way to the site.
TITLE: 11/26 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Unknown
INCOMING OPERATOR: Travis
SHIFT SUMMARY: H1 stayed locked most of the shift. Now I'm running into an issue I've never dealt with before. See alog31847 for details. This alog has disappeared magically. See alog31850 (comment below) for details.
LOG:
9:05 Noticed range slowly dropping, out of Observe to adjust SRM P
9:08 Back to Observe
9:12 Noticed that I made it worse, back to commissioning to move SRM P the other way round.
9:16 back to Observe.
13:39 Lockloss
14:24 THE confusing lockloss. Below I copied some of the Verbal Alarm message that might be useful in giving people insight of what's going on. It would be useful to know how long does the code take to loop though all the status of things.
14:23:16 VIOLIN_MODE_DAMPING2
14:23:28 TCS CO2 lasers tripped
14:23:24 ISI HAM6 WD tripped
14:23:48 ETMX ISI ST1 WD tripped
14:23:48 ETMX ISI ST2 WD tripped.
14:24:01 DOWN
My alog describing Beckhoff issues seems to have dissapeared (HOW IS THIS EVEN POSSIBLE?) so I'm copying the content here.
----------------------------------------------------------------------------------------------------
After 7 hours H1 lost lock for not so obvious reason (as always, I just haven't had a chance to investigate). This confusing lockloss occured as I tried to recover from the 7-hour lockloss. I stepped out after got through DC READOUT and came back to find that the lock broke at Violin Mode Damping 2 according to the Verbal alarm.
Couple of strange things happened (and still happening)
1) PSL power request suddenly went all the way up to 156W but the actual output got stuck at 30W. I couldn't use LASER_PWR guardian to move it back to 2W. The log repeatedly output the message:
2016-11-26T14:47:54.58953 LASER_PWR [GOTO_POWER_2W_FAST.run] ezca: H1:PSL-POWER_SCALE_OFFSET => 28.95
Seems like LASER_PWR guardian keeps outputing the offset to the channel. I can't manually put it in. I can't caput it. This offset is probably based on the input power to IMC so it make sense that it agrees with the input power. I also couldn't turn the rotation state manually on the rotation state screen.
2) Both TCSX and Y mysteriously tripped during prior to this lockloss. The medm status is READY so I went to check the interlock box. GATE and ENABLE lights are on but Remote Enable light is off. I've never seen this before. CDS overview doesn't have anything red on it. If the IR sensors saw a flash of carrier light as the lock broke, the GATE light wouldn't have been on and it shouldn't do anything to the Remote Enable light.
I'm really really confused right now. Still investigating and hopefully fixing seomthing. If somebody wake up and see this log and have some idea of what might be going on, please call the control room. Otherwise I (and Day shift op) will be calling people after 8am.
----------------------------------------------------------------------------------------------------
**UPDATE**
The rotation state medm screen seems to be stuck. It won't calculate angle requested when you put in new power, and true the other way round. The state is stuck at "busy"
According to ECAT1PLC3 SDF differences. It seems to me that TCS (and likely PSL) rotation states got stuck some how. How do I verify if Beckhoff computer controling RS is working properly?
**UPDATE 15:30**
Found CDS overview medm shows ERROR status for ECAT C1PLC1 and 2
**UPDATE 16:08**
Beckhoff error may have caused the most recent lockloss. C1PLC1 error flag came up prior to ISC_LOCK guardian went down. This is also true for the 7-hour lockloss. I don't know what's going on with C1PLC2.
The screen in alog 30842 can be used to diagnose hardware errors. It can be found from the menu SYS under EtherCAT. Each computer has one somewhere under PLC1.
Slavecountactual: The number of EtherCAT modules seen by the software
Slaveccountconfig: The number of configured modules
These two numbers should agree, indicated by slavecounterror.
Lostframes: Number of lost Ethernet frames since last reset. Any large number, or one that increases steadily is a problem.
Same goes for Lostqueuedframes.
The subscreen under workingcounters keeps track of the different types of EtherCAT frames, but is maybe more for the experts.
Is it possible that the excess motion in ITMY pitch, that was fixed last week, improved AS36I sensing in the same way that the pitch dither on SRM ASC is this week?
Last week the excess motion in ITMY pitch was identified, alog 31675 , and the filter causing it was turned off, alog 31701.
In searching the alog, I happened to find my own alog from June 2015, alog 18930, where I noticed the ASC ITMY pitch signal was about 3 times larger than ITMX's, at 500pp and 150pp (counts?) respectively. This may have changed since that alog, but if that was still true before the ITMY fix last week, it gives a scale of how much bigger ITMY was vs ITMX, which might be helpful in comparing to what the SRM ASC needs now.