kyle.ryan@LIGO.ORG - posted 16:13, Tuesday 08 April 2014 (11229)
Inadvertently de-energized 12V power supply beneath BSC6 -> Plugged back in
(Rich M, Sheila, Alexa)
HEPI and ISI ETMX tripped because we were playing with the suspension watchdogs.
Matlab has been changed to a 2011a version with limited licenses available for Linux control room workstations. Avoid using matlab for model changes. Version 2012b will be reinstalled when licenses become available. No matlab licenses are available for OS X at this time.
LVEA Laser Hazard Nitrogen delivery to Mid-X Ed Watt – Insulation on X-Arm 08:45 Jodi – Working at Mid-Y 08:45 Ken – Replacing flow bench fan motors in Optics Lab 08:57 Filiberto – Camera work at End-Y 09:38 Sheila & Alexa – Alignment work at End-X 09:30 Corey – Survey of ITMX spool cameras 09:42 Schneider Electrical contractor on site working on GC UPS system with Richard 09:50 Jonathan – Restarting the aLOG 09:50 Aaron – Cabling work in LVEA 10:00 Rick, Justin, Peter & Cris – Cleaning up in the H1-PSL enclosure 10:04 BSC2 – HEPI & ISI watchdogs tripped 10:06 BSC1 – SUS, HEPI, & ISI watchdogs tripped 10:15 Schneider Electrical contractor on site working on CDS UPS system with Richard 10:30 Praxair – Nitrogen delivery to Mid-X 11:25 Cyrus – Outside internet & phones are down due to fiber & copper cut at 400 area 12:30 Betsy & Travis – Working in the LVEA SUS Test Stand area 12:30 BSC9 – SUS, HEPI, and ISI watchdogs tripped 12:50 Richard – Changing dust monitor #5 (Beer Garden) to fixed vacuum connection 12:54 Corey – Going to End-X TMS lab 13:15 M itch & Scott – Taking forklift to Mid-X to get BSC plates 13:20 Corey – Going to End-Y 13:30 Michael – Survey tour in LVEA 13:35 Hugh & Greg – Replacing parker valve at HMA4 13:40 Cyrus – Going to End-Y 13:45 Richard – Working on dust monitor #5 tubing 14:06 Alastair & Dave – Turning on CO2 TCS-X laser 14:20 BSC2 – HEPI, ISI watchdogs tripped – Ken working on cabling near BSC2 14:44 Kyle – going to End-Y working on BSC6
@ 14:20 the BSC2 HEPI and ISI watchdogs tripped. Shortly after the HEIP watchdog on HAM3 tripped. There was work on the cable trays above these chambers at this time. Both chambers have been recovered.
For a while the BS chamber's HEPI has been an outlier because I accidentally loaded theposition filters to the level 2 slot, where all the other chambers use the level 1 slot. I used foton to copy everything over this afternoon so now the controllers occupy both slots. This means that now the BS-HEPI can be isolated just like the other chambers by pushing the isolate level 1 button on the Commands2 screen. The level 2 button also still works, I didn't change anything there.
I discovered why the reboot of h1seih45 yesterday did not get logged the same way in which a model restart does. When the OS is rebooted, the startup procedure for the front end models is slightly different and does not created the target/<name>/logs/reboot.log file. On h1boot I modified the /diskless/root/etc/rc.local file to call a new script /diskless/root/etc/log_fe_startup.sh which creates the reboot.log file for each model running on the computer (including the iop model).
While Greg & Jim replaced the Parker Valve here just last week, the replacement had a performance of 133% of nominal.. L124 needs calibration. We'll leave this in bleed mode for at least a couple hours.
While a contractor was on site commissioning our New GC UPS I asked him to look at our CDS UPS as we have been having communication problems with it. He quickly identified the problem we put the unit into bypass and swapped a lead/lag module around and restored the system. We are on the alternate module and a replacement will be ordered.
A Backhoe working in the 400 area for DOE struck our fiber connection to the Hanford fiber loop. Lockheed Martin is in the process evaluating the damage and will let us know how bad it is. Best case it is after our feed and on the Hanford loop so we could reroute. Worst case it is our fiber run and will have to be repaired.
Jeff B
@ 10:00 BSC2 HEPI & ISI Tripped. Reset watchdogs and both have recovered.
Jeff B
@ 10:02 the BCS1 SUS, HEPI, and ISI Watchdogs tripped. Reser watchdogs and all systems have recovered.
I closed these late yesterday but then turned them off again over night. I've collected overnight spectra now and have re-closed the loops. I'll make comparisons with the HEPI L4Cs and the ISI T240s for evidence of minimal harm from the loops. Attached is the comparison--I'll examine it closer too but I think there are elevated areas with the loops on between 0.7 and 10hz. Maybe we can live with this or tune this out.
Adjusted fan 1 output to 12000 SCFM, this also corresponds to the fan output at X end. This request was to see if we could reduce a noise peak that showed up in Jim Warner's test spectra.
model restarts logged for Mon 07/Apr/2014
none reported by the software, but...
looks like I've got a bug in the reporting software, we rebooted the h1seih45 computer and the models started up automatically. Somehow this method of startup was not logged, here are the restarts entered manually:
Mon Apr 7 11:51:26 2014
h1iopseih45
h1hpiham4
h1hpiham5
h1isiham4
h1isiham5
Here is the list of commissioning task for the next 7-14 days:
Green team:
Red team:
Blue team (ALS WFS):
Blue team (ISCTEY):
SEI/SUS team:
Since Saturday the ODC for the PSL and (obviously) IMC have looked a little odd. They both keeping reporting very short, quick times when both the systems are not 'good'. These happen throughout the day as you can see by the two attached plots which show yesterday's activities (you can see more of these plots on the summary pages - https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/). Just wanted to check you knew about this and if so is this nothing to worry about? Thanks!
This was another iteration of the ISS actuation running into its range limit.
Sheila, Chris, Alexa, Stefan
This morning we looked for coherences with many signals with our COMM noise. There weren't any surprises, we see the same coherence with the end station PDH control signal around 1kHz that we are used to, ISCT1 accelerometers around 70-100Hz, and a little coherence (-0.7-0.9) with the oplevs at low frequencies (pitch at 0.5Hz, yaw around 0.1). We don't see much coherence at all with the TMS QPDs, just a small bit at low frequencies. We lose coherence with the in loop sensor below around 100Hz.
Chris and I went out to End X to look into the beam quality problems. We didn't find much that was promising. We checked rather carefully for clipping on the way into the chamber again. We also set up the nanoscan again, and we see that there really isn't anything coming back to the WFS when the ITM and the ETM are both misaligned (so it isn't a second beam that is directly reflecting a lens or some optic on transmon.)
We also tried Daniel's suggestion of closing down an iris in the path. For WFS B this only made things worse. WFS A scans will be attached to this alog soon, but we didn't get any improvement there either.
We came back and tried out some of Rich's blends. We were able to get relatively low noise once without using WFS, 30z down to 0.02 Hz. The attached plot shows the COMM noise (measured by the IR) with different blends on. We are curently seeing a lot of noise on ETMX T240s at around 3Hz, (100 times more) Rich is looking into this.
Optic is Newport 20B20NC.1 which has a 30min wedge. Since this is in a double path configuration it was not clear which orientation is the best, but I put it in such a way that the beam and the ghost beam going to the chamber separate immediately. OTOH this means that, for the return beam in HWS, PDH and WFS path, the main beam and the ghost crosses at a point 7" after the BS, and then separate. This is because it's easier to block the ghost beam(s) in HWS, PDH and WFS paths than at the back of the periscope.
After the installation of the BS, I used two irises downstream of the 2" PZT mirror to move the 2" PZT mirror position as well as the angle to recover the in-vac QPDs. I further adjusted the 2" PZT mirror and the corner steering mirror between the 1" lens and the 1" PZT mirror to center the beam horizontally on the 2" lens without losing the beam on QPDs.
The optic and the mirror holder were supplied by AOS, had to use lab stock for the post and the base plate.
We had a hard time recovering from this trip which caused several more trips, the wd plotting script is not working this afternoon, possibly beause of our network problems so we don't have any plots. Once HEPI was isolated, we tried letting guardian isolate the ISI. It tripped immediately on CPSs (stage1) after we cleared this and tried again it tripped on GS13 (stage2).
We then paused the guardian, moved all the blends to start, and gave the gaurdian another try. (note, the ground motion is fairly quiet today). This tripped again, GS13s.
We then puased the guardian, and tried the command script, stage 1 level 3, tripped on the actuator limit. I then stopped taking notes.
The rough procedure that Rich used to bring it back was damping both stages, turning on 1 DOF at a time RX, RY, Z, RZ, X, Y (stage 1 first, then stage 2). Then Alexa moved the blends over to Tbetter without anything tripping.
We don't know why the things that used to work don't anymore, but now ETMX ISI is harder to untrip and neither the guardian nor the command script seem to be able to do it anymore.
Rich may not have mentioned in the alog, but after our trip the other day while trying to switch blends, he changed the filters from zero crossing to ramp, which should make it easier to switch blends without tripping the ISIs.
One thing about guardian, its not clear to me how we should unpause if we need to pause it for some reason. TO pause it we paused all three, stage 1, stage2 and the manager. Wen unpausing the subpodinates they don't come back in managed mode. I was able to get them back to managed mode by going to init on all three, but in the meantime they were doing their own thing, but happily didn't trip the ISI. What is the best way to do this? Also, maybe the right way to pause and unpause a manager with subordinates is something that needs to be made more obvious to a user.
Alexa also tried pausing, she only paused the manager, bringing that back was not a problem but it doesn't pause the subordinates.
quoth Sheila:
But you did mention one likely important thing that changed: new blend filters (Tbetter?). What happens if we go back to Tcrappy? Do we have the same problems?
also quoth Sheila:
This is definitely sounds like something that could be improved. It is true that all nodes, manager and subordinates, need to independenty be paused and unpaused. I'll try to think about how to make that smoother. In the mean time, here's the procedure that should be used:
The manager INIT state resets all the works, restarts them, and puts them into managed mode. That should be the most straightforward way to do things for the moment.
Thanks Jamie.
We did try both the command script and guardian with all the blends on start, which used to work. I don't think we tried turning sensor correction off, which is another change.