we had some system problems over the weekend, here is a summary of the time-line. (These would appear to be unrelated events)
The last issue caused Guardian problems with the ALS_XARM node. We did the following to try to fix it:
The power up of h1guardian sans epics-gateway gave even more CA connection errors, with HWS IOC and h1seib3 FEC. These were very confusing, and seemed to go away when the h1slow-h1fe epics gateway was restarted which added to the confusion. We need to reproduce this error.
After Patrick restarted the h1ecatx1 IOC the guardian errors went away.
See attached screenshot.
Restarted the IOC.
Forced PT100 Cold Cathode on in Beckhoff on h0vacly per Chandra's request (see attached). It is now reading ~1.01e-07.
Attached are HEPI Pump Trends for the past 45 days. To my untrained eye, I don't see any egregious excursions in pump pressures. SEI folks should review.
This completes FAMIS Request 4520.
Labeled HAM11 annulus IP (physically on HAM12) fell out of scale at around noon yesterday. Labeled HAM12 annulus IP (physically on HAM11) needs to be replaced. i.e. HAM11 annulus IP is working hard https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=26804
9am local 1/2 turn on LLCV bypass --> took 22 seconds to overfill. Lowered LLCV back to 20% (from 21%). Hot weather last week likely cause of long overfill times last Wed. and Fri. *watch exhaust pressure after tomorrow's Dewar fill
SEI - No major maintenance plans scheduled. Ongoing tweaking with BRS.
SUS - model changes for HSTS scheduled for tomorrow.
VAC - GV measurements and manual CP3 overfill scheduled for tomorrow
CDS - Cables to be pulled for ITM ESD tomorrow. Auto restart of workstations to take place tomorrow morning.
PSL - the team sees no pressing reason to go into the enclosure at this time other than to do some DBB aligning tomorow, possibly.
*CP3 Dewar fill from Norco truck tomorrow
The laser was off this morning. The chiller indicated that there was a "Flow sensor 1" error.
Looking at the data from the various flow sensors, maybe, just maybe, the problem is with the flow sensor attached to the auxiliary circuit (which monitors flows to the power meters ...). The attached plot seems to imply that the flow to the power meters dropped before the crystal chiller flow declined. Would need to check that the power meters are really attached to this part of the cooling circuit because the diode chiller was running this morning. For reference the cooling system information is under https://dcc.ligo.org/DocDB/0067/T1100373/002/LIGO-T1100373-v2%20Coolant%20system%20operating%20and%20maintenance%20manual.pdf
While the laser was down I mounted a B&K voice coil shaker on the +X, +Y corner of the table to study the PSL jitter contribution to DARM noise in the 80-200 Hz region. I doubt it will affect alighnment but be aware since it is quite heavy.
BP
Following Friday night's lock, I looked at the spectrum and saw some regular structure, looking like a 2Hz comb at odd frequencies.This looks like a new comb. [figs 1&2]
As followe up, I ran coherence with all of the EBAY magnetometers and saw strong coherence is some places with this 2Hz comb, as well as the persisting 0.5Hz comb (see plots below)
0.5Hz comb: https://ldas-jobs.ligo.caltech.edu/~brynley.pearlstone/comb_investigations/May_2016_comb/H1:PEM-EX_MAG_EBAY_SEIRACK_Z_DQ_25_40Hz.png
0.5Hz comb + 2Hz comb: https://ldas-jobs.ligo.caltech.edu/~brynley.pearlstone/comb_investigations/May_2016_comb/H1:PEM-EX_MAG_EBAY_SEIRACK_X_DQ_25_40Hz.png
Note: These two are the same magnetometer (MAG_EBAY_SEIRACK), looking at 2 different axes. Both combs were also seen in other magnetometers.
Full list of plots: https://ldas-jobs.ligo.caltech.edu/~brynley.pearlstone/comb_investigations/May_2016_comb/
Previous efforts to mitigate the 0.5Hz comb was focussed on powering the timing card independently in the CS EBAY LSC-C! I/O chassis which handles DARM. This has not worked to eliminate the comb. I can't report any reduction yet, as Friday's lock was not sensitive at low (<100Hz) frequencies.
This 2Hz comb on 1Hz offset is the transform of a 1Hz square wave. The 2Hz comb might have to do with the 0.5s and 1s structure seen it Keith's data folding studies here: https://alog.ligo-wa.caltech.edu/aLOG/index.php
Evan, Sheila
Today we blended CHARD P, with the same blend filters Craig and I used last night for yaw. This went smoothly and the attached screenshot shows the settings, which are accepted in SDF and shouldn't need to be in the guardian since they don't need to change. WE haven't made it to low noise to see what kind of improvement this gives us.
We also had a look at dithering SRM and demodulating DARM control to get an alignment signal for SRM. The SNR was not good for a 4Hz or 9 Hz dither, but for a 15 Hz dither there is clearly a signal, although the lock point was not good. I've added filters we could try to supress the length coupling in MICH2, SRCL2, PRCL2 filter banks, but we didn't get a chance to try this again.
Evan noticed that there is an instability in CHARD P at 0.2Hz when we tried to power up past 20 Watts or so last night, so we redisgned the boost that comes on at 17Watts to be gentler and have higher frequency zeros, (the new filter is MSBoost2 and in FM1). With this we seems to be stable at 30Watts, at least for 5 minutes (we broke lock by trying to go to 35W).
We had a bit of trouble with the laser today, nothing that can't be fixed easily ( noise eater and ISS oscialltions, difficulty locking the FSS). Evan changed the temp search ramp parameters for the FSS.
Also, we had to reduce the analog gain of the TMS QPDs even further in order to avoid saturation on some of the segments when going above 25 W. We used to set the gain to 9 dB once we achieved resonance; now we reduce it to 3 dB.
This is a problem that is better solved by picomotoring the TMS beams. The worst offender seems to be the Y-arm B diode; it has a factor of something like 40 in power between segments 2 and 4.
Evan, Kiwamu,
In this afternoon, we closed the ISS third loop which are meant to damp the arm power fluctuation.
We confirmed that it suppressed the common arm power fluctuation at around 0.5 Hz by 10 dB nominally. We tested this loop at various PSL power from 2 to 25 W. Further sophistication of the loop shape can be done if necessary.
[nominal settings]
[injection point changed]
The original plan was to inject the third loop signal to either TRANSFER1 or 2 input. However it turned out that they have seem to have a factor of 10 attenuation, we ran into a range issue where the DAC output and SR560 (which is meant as a dewthitening filter) saturated before getting to the nominal servo gain. We instead moved the injection point to the one for the second loop by disconnecting the second loop and plugging the third loop cable.
Also, there is a low-pass-ish response somewhere in the second loop injection path which let us put a lead filter in FM3.
[Instability at 25 W]
With a 25 W PSL, we see an instability that occured at 0.24 Hz. Similarly to the usual arm power instability at 0.4 - 0.5 Hz, it modulated the arm power, but the frequency seems to be a bit low and it coupled to the HARD mode. This instability seems to happen regardless of whether the third loop is closed or not. A possible source might be an unstable ASC loop. Currently under investigation.
The third loop signal now goes through an SR560 with an inversion, so the gain should now be +0.04 in the filter module.
1:30pm local 1/2 turn open on LLCV bypass --> took 33:17 min. to overfill CP3. Raised LLCV from 20% to 21%.
It looks like we need to do something about the triple coil drivers that we switch in lock, especially PRM M3. We have lost lock a good fraction of the times that we have tried to switch in the last month or so. Screenshot is attached, I also filed an FRS ticket hoping that someone might make an attempt to tune the delays while people are in the PSL tomorrow morning. FRS ticket #5489
Is the DAC spectrum post-switch using up a large fraction of the range? If the noise in the PRC loop has change a little bit it could make this transition less risky.
Here's the PRM drive from last night's lock, in which we just skipped the PRM M3 BIO switching leaving the low pass off (BIO state 1). It seems like we should have plenty of room to turn the low pass on without saturating the DAQ.
FRS Ticket #5489 closed in favor of a long-term fix Integration Issue #5506 (which is now also in the FRS system) and for an eventual ECR. Work permit #5880 indicates we'll install the fix tomorrow. See LHO aLOG 27223.
Rebooting the entire guardian machine just because one node was having a problem seems like extreme overkill to me. I would not recommend that as a solution, since obviously it kills all other guardian processes, causing them to loose their state and current channel connections. I don't see any reason to disturb the other nodes because one is having trouble. Any problem that would supposedly be fixed by rebooting the machine should also be fixed by just kill and restarting the affected node process.
The actual problem with the node is not specified, but the only issue I know of that would cause a node to become unresponsive and immune to a simple "guardctrl restart" is the EPICS mutex thread lock issue, which has been reported both at LLO and LHO, and both with solutions that don't require rebooting the entire machine. Presumably the issue being reported here is somehow different? It would be good to have a better description of what exactly the problem was.