Jeff B. warned me that we would likely be getting a lot of dust monitor alarms after the PEM system was reset this afternoon. He advised that we can set arbitrary alarm values in the meantime and he will remedy them in the morning. I have been setting the thresholds to 1000 minor and 10000 major as the alarms occur.
CP4 LTY250 pump level went into high alarm 20 mins ago. Cell phones have received alarms. Waiting to talk to Chandra or someone VE...
The current alarm is not the normal "glitchiness" that has been ongoing for the last few days - 20 mins ago this signal jumped from ~95 to 100 and is riding up there. See attached.
Got a hold of Kyle. He is looking into it from home and will call back to advise if any action is required.
This is mostly a nuisance that we don't yet understand. Chandra reduced the "smoothing factor" (I love it!) from ~99.9 to 0.00 a day or so ago at John's suggestion but this doesn't seemed to have changed the behavior. We are mostly concerned with low pump levels as opposed to high pump levels these days. We have opened the exhaust check-valve bypasses on all 80K pumps on site. This eliminates any over "pressure" situations that could have been a threat in the past during rapid pump fillings or too high LN2 pump levels. For tonight, OPERATORS please post 4 hour trends (thanks Betsy) twice per shift. The Vacuum members will monitor from home.
It's on the bounce again...
I instructed Travis to put CP4 in manual mode with the LLCV at 35% open overnight. The software limit of 25% is great when the pump level is too high but nothing prevents it from going too open if the PID gets a pump level signal of too low. At least in Manual mode the crazy swings will stop. We can deal with it in the morning.
A couple hours in now on MANUAL at 35% open. Not alarming high anymore.
Seems to still be stable.
Level has been slowly dropping for the last 3 hours. If this continues, I may have to give VE and early wake up call.
Reloading some Guardian code knocked us out of Observe. I took this opportunity to run a2l. PI modes 27 and 28 started ringing up during a2l, so Terra and I tweaked some setting and beat them back down.
Back to Observe 2:01 UTC.
J. Kissel, J. Driggers, S. Dwyer, J. Bartlett, D. Barker Today's (not that big a deal) rabbit hole: while looking for what was preventing the IFO from being OBSERVATION_READY, we found the IFO master node complaining that the IMC_LOCK gaurdian was not ready (i.e. the H1:GRD-IMC_LOCK_OK == False), even though the node was in its nominal state, managed correctly, and in EXEC. The key was that the IMC input power was just ever so slightly out-of-range of a recently added conditional statement that prevented the run portion of the ISS_DC_COUPLED state from ever completing: # Hack, since power_adjust() won't change the gain when it's less than 1dB different, 30Nov2016 JCD if self.counter == 0: if ezca['IMC-PWR_IN_OUTMON'] > 28 and ezca['IMC-PWR_IN_OUTMON'] < 32: ezca['IMC-REFL_SERVO_IN1GAIN'] = -8 self.counter += 1 -- and today, for the first time, we hit an a low of 27.9 [W], the self.counter never incremented, and the code never advanced because all future conditionals relied on that self.counter to have incremented. Jenne recalls that this conditional statement was added to account for the IMC-REFL_SERVO_GAIN adjustment that happens as the ISS figures out what power it wants to send into the interferometer today (see LHO aLOG 32267 for side conversation about that), and prevent SDF from complaining that this gain was set to -7 dB instead of -8 dB. The solution: bring the incrementing of the self.counter out of the power check: # Hack, since power_adjust() won't change the gain when it's less than 1dB different, 30Nov2016 JCD if self.counter == 0: if ezca['IMC-PWR_IN_OUTMON'] > 28 and ezca['IMC-PWR_IN_OUTMON'] < 32: ezca['IMC-REFL_SERVO_IN1GAIN'] = -8 self.counter += 1 This bug fix has been loaded, and committed to the userapps repo. Once the bug fix was installed, the code was able to successfully run, the IMC_LOCK status cleared, and the IFO happily went into OBSERVATION_READY.
TITLE: 12/07 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
Wind: 13mph Gusts, 12mph 5min avg
Primary useism: 0.28 μm/s
Secondary useism: 0.31 μm/s
QUICK SUMMARY: Ran through IA at the beginning of the shift. We are back at Observing at 1:28 UTC.
As I was working on a different script I found that both X and Y camera can be talked to simultaneously. So I'm leaving both codes running again. If DIAG_MAIN guardian complains about HWS code stops running please do not comment out the HWS diagnostic code just to get rid of the message. I need to know if they stop running so we can try to troubleshoot it some more.
All of the dust monitor EPICS IOC's, the FMCS EPICS IOC, and the dewpoint EPICS IOC was restarted due to a reboot of the computer on which they were running.
J. Kissel (with information from S. Dwyer) I have been blaming the rotation stage for the 5 [W] range of IFO input power from lock stretch to lock stretch, but it looks like I've been wrong this whole time. I attach a 25 day trend of the the power into the IMC, compared to - the power transmitted from the PMC, - the power reflected off the input of the PMC. - the relative power diffracted by the AOM (servoed by 1st loop) - the 2nd loop Reference Voltage (servoed by 2nd Loop) One can see - the range of power into the IMC is 5 [W], with an average of 30 [W], which means a relative power range of 16%. - the range of transmitted power from the PMC shows a similar range: 10 [W] or out an average of 60 [W]; a relative range of 16%. - the power changes are anti-correlated with the PSL diffracted power, as expected, but the channel appears to be poorly calibrated since it ranges between 2 and 6%. This is likely the cause of the power variations, and they are under reported. - the power reflected by the PMC shows some long-term trend over the 25 days, but no correlation with the lock-to-lock difference. - the second loop reference voltage seems to track the long term trend of the PMC REFL power. - though there may be some correlated in isolated events, the human-adjusted (and NOT every lock stretch) 1st Loop voltage reference is not correllated with the lock-to-lock stretch changes in power. The fundamental problem -- the DC voltage reference for the ISS' 1st loop is not stable. Currently, we have some alarms on the diffracted power, but they're not very tight, and only designed to prevent saturation of the servo. The power fluctuations *could* be remedied with some hacky slow digital servo, or a tightening of the threshold on the diffracted power, but it seems like we should be able to get a more stable voltage reference.
the virtual machine which serves matlab licences is being rebooted. This also serves FMCS epics data, so RO alarms are being generated.
Ops Shift Log: 12/06/2016, Day Shift 16:00 – 00:00 (08:00 - 16:00) Time - UTC (PT) State of H1: IFO is down due to two earthquakes within about an hour Intent Bit: Commissioning Support: Jeff K., Keita, Jenne, Terra Incoming Operator: Travis Shift Summary: After the scaled down maintenance window (due to maintenance work done on Monday), tried relocking. Found green power in low in both arms. Ran an initial alignment and relocked on the first attempt. Sat at NOMINAL_LOW_NOISE in Commissioning while Evan did some commissioning work on the ITM Reaction Chain alignment. Went to commissioning when Evan finished. Rode through a 5.8Mag EQ centered around Trinidad. Lost lock due to 6.4Mag EQ in Indonesia. Put IFO into DOWN until microseism drops below 1.0um/s. Activity Log: Time - UTC (PT) 07:00 (00:00) Take over from TJ 16:19 (08:19) Chris – Escorting pest control through LVEA 16:20 (08:20) MY_CP4_LT250_PUMP_LEVEL_PCT alarm – Cleared on own 16:34 (08:34) Take IFO to Commissioning for start of maintenance 16:45 (08:45) Richard – Moving fiber bundles in MSR - (WP #6292) 16:45 (08:45) Chris – Finished at CS, Escorting pest control to Mid-Y and End-Y 16:46 (08:46) Christina – Going to End-X for cleaning 16:48 (08:48) Karen – Going into the LVEA to clean around Ham4, 5, 6, and high-bay 16:50 (08:50) Jim – Going to End-X to unplug the WAP 16:50 (08:50) Carlos – Going to End-Y to unplug the WAP 17:00 (09:00) Richard – Taking Scott M (LLO) into LVEA to look at vacuum racks 17:00 (09:00) Joe – Going into LVEA to check eyewash stations 17:05 (09:05) Cintas on site to service matts 17:10 (09:10) Kyle & Gerardo – Going into LVEA to work on PT180 - (WP #6377) 17:11 (09:11) Norco on site to deliver N2 to CP2 17:18 (09:18) Jim – Back from End-X 17:19 (09:19) Richard & Scott – Out of LVEA 17:31 (09:31) Lockloss – Due to maintenance activities 17:37 (09:37) Carlos – Back from End-Y – Reports the parking lot is very icy 17:45 (09:45) Christina – Leaving End-X 17:46 (09:46) Joe – Out of the LVEA 17:46 (09:46) Chris – Pest control finished at ends and corner – Working down the arms 17:50 (09:50) Karen – Out of the LVEA 18:08 (10:08) Alfredo & Elizabeth – Going to Mid-Y to do inventory 18:30 (10:30) Dave – Going to Mid-X and Mid-Y to unplug WAP 18:44 (10:44) Joe – Going into LVEA to service first aid stations 18:47 (10:47) Norco leaving site 18:52 (10:52) Rana – Escorting tour in the LVEA (WP #6387) 18:57 (10:57) Richard & Evan – Focusing camera at SRM (WP #6371) 19:06 (11:06) Dave – Back from Mid-Stations 19:20 (11:20) Richard & Evan – Out of the LVEA 19:27 (11:27) Alfredo & Elizabeth – Back from Mid-Y 19:36 (11:36) Paradise Water on site to deliver water 19:40 (11:40) Dave – Power recycle LIGO file system server (WP #6388) 19:44 (11:44) Bubba – Inspection tour of LVEA 20:05 (12:05) Patrick – Doing sweep of the LVEA 20:10 (12:10) Reset Fiber Polarization on X Arm 20:12 (12:12) Kyle & Gerardo – Out of the LVEA 20:15 (12:15) Start to relock 21:31 (13:31) After Initial alignment – Relocked at NOMINAL_LOW_NOISE, 27.9W, 66.7MPc 21:31 (13:31) In Commissioning mode - Evan & Rana work on ITM reaction chain alignment 22:00 (14:00) Damp PI modes 9, 26, 27, 28 22:14 (14:14) Rode through a Mag 5.8 EQ near Trinidad 22:19 (14:19) Set mode to Observing 23:08 (15:08) Lost lock due to 6.4Mag EQ in Indonesia. Primary microseism up to 7.0um/s 23:09 (15:09) Put IFO into down until microseism settles down 00:00 (16:00) Turn over to Travis
(see WP #6377) Having not asked for permission, I am now asking for forgiveness! It was my intention to shut down the temporary pump setup @ BSC8 at the end of maintenance today. Instead, I have left the setup running. Obviously, we will shut it down if/when directed to do so. As is, we have just this morning entered in to the "pressure region of interest" for our long-term gauge drift data collection. The nature of the problem doesn't lend itself to 4 hours per week of data collecting. Recall that ON/OFF tests of this setup while in a locked "Low Noise" IFO state produced nothing of interest to the PEM folks just prior to the start of O2.
~1630 - 1635 hrs. local -> Kyle in and out of LVEA At the request of others, I shut down the setup at PT180/BSC8. This included isolating PT180, each of the three vacuum pumps and de-energizing all of the related components. The ladder which had been left up against BSC8 was also removed. NOTE: PT180 is left isolated from the site vacuum volume and all pumping sources. As such, PT180's indicated pressure value does not represent the pressure at BSC8 etc. and will be rising until further notice.
Not sure of the cause yet, everything seemed all good and normal. Running lockloss plots now though and I'll update if I find anything.
But don't worry, we are back to Observing at 10:36 UTC.
I haven't seen anything of note for the lockloss. I checked the usual templates, with some screenshots of them attached.
This seems like another example of the SR3 problem. (alog 32220 FRS 6852)
If you want to check for this kind of lockloss, zoom the time axis right around the lockloss time to see if the SR3 sensors change fractions of a second before the lockloss.
J. Kissel, B. Weaver, T. Sadecki Just for reference, I include a lockloss that was definitely *not* caused by the SR3 glitching for future comparison and distinction of whether this SR3 glitch has happened or not. Also, remember, although the OPS wiki's instructions suggest that one must and can only use lockloss2, not everyone has the alias yet for this more advanced version. You can make the plots, and do everything you need with the more basic version: lockloss -c /ligo/home/ops/Templates/Locklosses/channels_to_look_at_SR3.txt select It would also be great to get the support of @DetChar on this one. The fear is that these glitches begin infrequently, but get successively more frequent. Once they do, we should consider replacing electronics. The fishy thing, though, is that LF and RT are on separate electronics chains, given the cable layout of HAM5 (see D1101917). Maybe these glitches are physical motion? Maybe with statistics of two, it's unclear whether it's that LF and RT just *appear* to be the culprit whether it may be a random set of OSEMs glitching.
See my note in alog 32220, namely that Sheila and I looked again and we see that the glitch is on the T1 and LF coils, which share a line of electronics. The second lockloss TJ started with in this log (12/06) are somewhat unconclusively linked to SR3 - no "glitches" like the first one 12/05, but instead all 6 top mass SR3 OSEMs show motion before lockloss.
Sheila, Betsy
Attached is a ~5 day trend of the SR3 top stage OSEMs. T1 and LF do have an overall step in the min/max of their signals which happened at the time of that lockloss which showed the SR3 glitch (12/05 16:02 UTC)...
I used the lockloss2 script that automatically checks for sus saturations and plots them using the lockloss tool, and saw that one of the three locklosses (2016-12-05 16:02:42 UTC) in the last day or so was probably caused by a glitch on SR3. The attached screenshot shows the timeline, there is clearly a glitch on the top mass of SR3 about 0.2 seconds before the lockloss.
The dither outputs (which we use for the cage servo) don't show anything unual until after the lockloss, which means that this is not a cage servo problem. Looking at top mass OSEMINF LF and RT are the two that seem to have a glitch first, at about the same time.
I've added a lockloss template /ligo/home/ops/Templates/Locklosses/channels_to_look_at_SR3.txt for any operators who have an unexplained lockloss and want to check if it is simlar to this one.
Sheila and I looked again at this particular lockloss (2016-12-06 10:05:39 UTC) and agree that the glitch that likely caused the lockloss are actually on the T1 and LF top stage OSEMs. These are indeed on the same set cabling, satellite amp, and driver run. See attached for updated lockloss plot this time with the OSEMINF channels. We'll keep watching locklosses to see if this happens more.
Not sure why the PSL dust alarm has been going off tonight. Nothing looks out of normal on the camera views to inside the enclosure. Lots of signal on longer trends of the dust monitors - to be filed as "look into further"...