Changes attached.

Updated userapps: isi/common/guardian/isiguardianlib/watchdog:

hugh.radkins@opsws1:watchdog 0$ svn up
U    states.py
Updated to revision 12546.
 

guardctrl restart ISI_ETMX_ST2:  this caused the guardian to deisolate stage2.  Not what I experienced Tuesday when I did similar..  When I restarted stage1 guardian, it did not touch the platform, as I expected.

Tripped the platform and now as expected, the damping loops were turned off when a state4 trip occurs.  Okay, so update fixed issue.

At ETMY, restart Stage1 first, then Stage2 and Guardian did nothing to the platforms...

Restarted HAM3, no problem.  Tested (tripping PR2 in the process) and the DAMPing path was turned off.  So good.

When restarting ISI_HAM6, this ISI guardian switched its request state to NONE but did not do anything, chamber manager was throwing messages.  Toggled ISI to HIGH_ISOLATED and everyone became happy.

TCS crew is working on SRC so I'll not touch them.

Guardian restart & test ETMX, HAM3.

Guardian restart ETMY, HAM2, HAM6.

I'll complete the remainder of the restarts and maybe test a couple more when TCS is clear.

Turns out that HWS at the end stations were temporarily turned on. When HWS is on and not on a separate power supply, it injects 57Hz noise into the ESD, which is picked up by the monitor. Another mystery solved.

On the attached top left, red is with the EX HWS off, blue is on, IFO unlocked, ETMs uncontrolled. EX ESD UL digital output was turned off. The 57Hz line and its harmonics are gone with the HWS.

On the bottom left are the same thing but with EY HWS off/on. No 57Hz in either of the two traces. That's because EY HWS is on a separate temporary power unit (alog 18355). The reason why we also saw 57Hz in EY LVESDAMON in lock (alog 25310) is not clear, maybe EX ESD was not turned off (though all outputs were zeroed) and shook the ETMX, and the line might have propagated to EY through DARM loop (we can see the line in DARM).

I don't know if X-Y difference (bottom right), apart from the callines in Y, is for real or just the sensing noise.

WP#5722

While we have high useism I wanted to revisit how this node is doing since I haven't touched it since before ER8. I created, started the node, and then had it switch blends with no problem. I then tried to mess it up to see how it would react. This is where I left off last time.

Some of my tests:

• If I manually change the blend to one that was not set by the GRD node, then it will complain that it is not in the right configuration and then try to change it immediately. At this point the FE has control of the filter banks so the node cannot actually change the value of the banks. I will need to change the code to wait before trying to change it back, or maybe just jump to DOWN and notify that it isn't in the preferred configuration.
• The DOWN state works as it should. It will jump to it from any of the requestable states and then just sit there and not do anything. If a blend is manually changed while the node is in DOWN, it will simply just ignore it. The node is, well, down.
• INIT doesn't jump to the state that it should, I'm not 100% sure why yet, I have some guesses but need to investigate some more first. If INIT is selected, even if it is in a state that it just set, it will go to DOWN and then work its way throught the graph to the requested state.
• Stopping a node in the middle of the switch still allows the FE to finish the "blending", but the NXT banks do not get turned off at the end. I need to think of a way around this, though this situation shouldn't ever happen...(I'm sure someone will find a way)
• Chaning the name of the filter in the const.py, but with an incorrect filter bank number will cause it to change blends to the FB number but as soon as it is done it will complain that it is not in the right configuration and then try to get to the state again. I should add a quick check somewhere that will check that the name and FB number match.

Overall it went well and I have a few things to work on. When I get another chance to test this out I will try it on ETMX with the BRS SC as well, the BLENSCOR node as I have been calling it.

I flushed the chiller water for both X and Y arms today, with advice from Jeff on what to do.  The water in the LHO chillers has not been a problem so far, and shows no evidence of particulate or discoloration.  However the chiller manual suggests that replacing the water every 6months would be a reasonable maintenance schedule.

Given that there was no problem with the water we could have got away with just draining and refilling the chillers a couple of times to replace the water.  However at LLO we will want to actively flush the system, so for that reason I also flushed the chillers here.

First I turned off the lasers, then their power supplies and the power supply to the AOM driver.  The laser, laser driver, AOM head and AOM driver are all water cooled.  For each chiller in turn I performed the flush as follows:

*Tools required - flat blade screwdriver for hose clamp

1) Turn off the chiller then disconnect the chiller using the quick connects, then drain the chiller using the drain plug at the bottom on the back.***Suggest making sure there are no power cables around on the ground at this point becase water may get on ground***

2) Close drain plug, refill chiller with distilled water ("laboratory grade" = reverse osmosis filtered then distilled).  Reconnect cables and the "Process Outlet" pipe to the top of back of chiller.

3) Leave the process inlet pipe disconnected and remove the quick disconnect attachment from the pipe.  Run this pipe into a water collection basin (I used a mop bucket which has wheels and is a good size for this)

4) Note that when filling the water collection basin, make sure that it doesn't get so full that you can't move it back down the stairs.  5 gallons is about as much as you would want to carry.

5) Repeat the following steps until all water is replaced

     a. Turn on chiller while holding pipe into basin

     b. Wait for approx 10 seconds of water to come out of chiller, then turn chiller off using the switch on the back (not the one on the front which takes too long to turn chiller off)

     c. Top chiller back up to full with distilled water

6) Once the water is replaced reconnect the process inlet pipe and hose clamp

7) Turn on the chiller and run for several minutes.  Check water level and top up as necessary.

I have measured the HARD arm loops at several different input powers, so that we can see how the peaks move around with radiation pressure. Some of the measurements could be a bit better (eg. CHARD YAW 10W to lower freq), but I think it's enough that we get the idea of what's going on, and can confirm with the ASC model. 

For each plot, red is 20W, orange is 10W and blue is 2W.  Shaded regions indicate the ~1 sigma error bars from the coherence of the measurement.  The gains and control filters are different for each of these measurements, but I know what the state was, and the model script takes those differences into account.  I will also divide out the rest of the loop contributions, so that we can see how the suspension changes by itself.

For all of the loops except for DHARD Pitch, we have basically no gain in the frequency region that I had the patience to measure. We should look forward to using the confirmed ASC models to design sensible loops that have some actual gain.

The DHARD Yaw comparison with the model was reported in alog 25054, but I have not yet made the comparison plots for the other degrees of freedom.  The oplev damping is still being added to the quad suspension model, so it may be another little while before the pitch measurements get a model to compare with.

Attached as a tarball is a folder including the DTT templates for these measurements, the transfer functions and coherence text files exported from DTT, the matlab script to create the figures, and all of the figures saved in several different formats.  Note that the coherences that are exported into the text files are just the IN2/IN1 coherence, although others are available in the DTT templates.

In bring the diagnostic breadboard back online, which required powering up the electronics,
the flow sensor sensor tripped causing the shutter to close.  It was reset from the Control
Room.

Back to normal.

The reduction in gain on chiller loop looks to have got rid of the oscillation that we were seeing.  Attached is the over night data, which shows the same small power output range and pzt range, with a smoother chiller response.  Y-arm laser wasn't on last night.

   Transition Summary:
02/04/2016, Day Shift 16:00 – 00:00 (08:00 – 16:00) All times in UTC (PT)
	
State of H1: IFO unlocked. The wind is a light breeze (< 7mph). Seismic and microseism are still as elevated as they were last night. Locking may be somewhat difficult at this time.      

It has been a fight to get DRMI to stay locked for more than a few minutes due to high ground motion.  Commissioners suggest calling it a night as microseism continues to trend upward.

The ground motion is just too high, so Evan, Travis and I agree that we should call it a night.  We can't hold DRMI for more than 2 minutes or so.  Earlier today, I flipped the phase of the ASAIR 90 PD, so that we get roughly the same number of counts when DRMI is locked as we used to.  We only use ASAIR90 for thresholding engagement of the DRMI ASC, so it's not a super critical PD phasing-wise.  I'll come back to it tomorrow.

This is another instance of getting the '75% notification', followed shortly by a WD trip on HAM6.  All I did was to reset the WD.

Vern, Jim, Dave:

Several times in the past few weeks we have seen an invocation of tconvert produce the output:

tconvert NOTICE: Leap-second info in /ligo/apps/linux-x86_64/ligotools/config/public/tcleaps.txt is no longer certain to be valid; We got valid information from the web, but were unable to update the local cache file: No permission to write in directory /ligo/apps/linux-x86_64/ligotools/config/public
 

This afternoon we spent some time tracking this down. tconvert uses a web server in france to access the IERS bulletin-C information page which alerts us to upcoming leap seconds. It appears that this server was pure HTTP until January 11th this year. It is now an HTTP server which redirects the request to a HTTPS page. The TCL code inside of tconvert interprets the redirection return message as an error. It then marks the expiration of the leap second information for 24 hours in the future.

The procedure we used to maintain future leap second notification was to twice a year run tconvert as user controls, which updated a leap seoncds database file (owned by controls) which was good for the next six months. With the recent HTTP issues, controls now produces a file which is good for only a day.

This afternoon we ran a modified tconvert which access a different leap seconds HTTP page and the LHO CDS database file is now good through 8th August. In the mean time we will raise an FRS and work with Peter S for a long term solution.

Late entry from Tuesday Maintenance

Attach test conlog machine to IRIG-B fanout [WP5712]

Fill, Patrick, Jim, Dave:

We ran a 50 Ohm coax from a spare port of the MSR IRIG-B fanout to the test conlog machine, Patrick was able to get all the timing information he needed except for the year.

Model Rate and DAQ change for BSC SUS-AUX models [WP5714]

Jeff K, Betsy, Jim, Dave:

For the h1susauxb123, h1suseuxex and h1susauxey models three changes: change rate of models from 2048Hz to 16kHz, add filters, add one channel to the commissioning frame at 16kHz and remove some obsolete 1kHz channels.

New asc model

Jenne, Sheila, Dave

new h1asc model with additional filters and new excitation injection points.

Add FEC SDF buttons to conlog

Patrick, Dave

We changed the python script which generates the conlog fe_include.txt file to include SDF buttons. Conlog was restarted.

HWS Beckhoff changes

Daniel, Aidan, Alastair, Patrick, Dave:

h1ecatc1plc3 was modified to add HWS channels. Autoburt.req and DAQ INI files were updated.

BSC ISI T240 channel problem identified and fixed

Jim W, Jim B, Dave

Jim found that the T240 X,Y,Z channels were not correct. We identified the problem in the isi2stagemaster common model. Jim fixed the problem and we rebuilt and restarted h1isibs, h1isiitmx, h1isiitmy, h1isietmx and h1isietmy

New PSL ISS model

Kiwamu, Jim, Dave:

Kiwamu installed a new h1psliss model. This required several restarts. There was no impact on the PSL output. We burt restored to 12:10 PST Tuesday after each start.

DAQ Restarts

Jim, Dave:

Due to possible file system slowdowns, we changed the monit configuration on the DAQ systems to 60 seconds for DC, and 30 seconds for all others (was 10 seconds before). We reatarted the DAQ four times during the course of maintenance to support the above changes and did not see any repetition of previous DAQ startup problems.

I started doing a power budget for the table, but got side tracked investigating the in-loop diode that doesn't have any response.  I tried realigning onto it but got a strange response that included negative voltages.  As a result the lens isn't in place in front of the diode now, so I've blocked this path using a 10W power meter head (this path gets ~200mW so this head can easily cope).  The laser is turned off just now at the controller.

There is already a fairly large power reduction through the AOM, from 63W down to 56.6W, then down to 56.1 after the polarizer.  The PDs are getting 450mW, so even after these stages there is more than 50W going into the rest of the table.  It is pretty tricky getting power meters in becasue there isn't a lot of room, so we may not get a complete picture of the power losses.  I'll work more on this tomorrow.

Hugh asked why there were 'dot nfs' files in his directory and why he could not remove them, but later when he exited matlab they were removed. Here is a part of his directory listing when the files were there

-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840bb80000002f
-rw-rw-r-- 1 hugh.radkins controls 76376 Feb 18  2015 .nfs000000000d840bb70000002e
-rw-rw-r-- 1 hugh.radkins controls  5974 Feb 18  2015 .nfs000000000d840bb60000003e
-rw-rw-r-- 1 hugh.radkins controls  5974 Feb 18  2015 .nfs000000000d840b9a0000002d
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840b2c0000002b
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840b210000002a
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840b1200000029
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d84096200000028
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d84095700000027
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d84094c00000026
-rw-rw-r-- 1 hugh.radkins controls 76376 Feb 18  2015 .nfs000000000d84093400000025

-rw-rw-r-- 1 hugh.radkins controls 89335 Feb  3 16:15 .nfs000000000d840bc600000046
-rw-rw-r-- 1 hugh.radkins controls 89335 Feb  3 16:15 .nfs000000000d840bbe0000003c
-rw-rw-r-- 1 hugh.radkins controls 77540 Feb  3 16:15 .nfs000000000d840b4c00000035
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840b9900000036
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840b9c00000037
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba200000038
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba300000039
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba40000003a
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba50000003b
-rw-rw-r-- 1 hugh.radkins controls 77540 Feb  3 16:19 BSC9_H1_Valve_Check.xml
...
 

In Linux is is permitted for a program to open a file and then to unlink it. On a local file system this creates a file with no name (wont show up in 'ls' for example) but the inodes still exist for the file and the program can still read and write to the phantom file. Only when the program stops running does the OS clear the inodes for reuse. This is great for temporary files which should disappear when the program stops, even if it crashes and does not cleanly close the files.

The situation Hugh encountered is when the file is on an NFS mounted file system. In this case when the file is unlinked, the NFS client renames the file to a dot nfs file with a random name. If a user on the same NFS client machine tries to delete the file, they get a "cannot remove, Devicee or resource busy" error and the file is not deleted.

Interestingly we found that on a different NFS client it is possible to delete the file since the controlling process is not on that machine.

So we have potentially two problems. One is that dot nfs files get stuck open (like the ones from last Feb 18th in Hugh's listing, we presume the NFS client was abruptly shutdown). Second is that someone deletes a file on one NFS client which is actually being used by another NSF client.'s program.

Activity Log: All Times in UTC (PT)

15:43 (07:43) Reset tripped ISI WD on HAM6
15:53 (07:53) Peter – Going into the H2 enclosure to look for electronics equipment
16:00 (08:00) Peter – Added 350ml water to the Diode chiller
16:16 (08:16) Peter – Out of the H2 enclosure
16:45 (08:45) Richard & Filiberto – At HAM6 for Triplexer installation
16:53 (08:53) Joe – Going to X-Arm for beam tube sealing
17:15 (09:15) Carpet installers on site to finish OSB installation
17:35 (09:35) Completed initial alignment
17:36 (09:36) IFO in DOWN for Kiwamu, who is working on the ISS Second Loop 
18:05 (10:05) Richard & Filiberto – Out of the LVEA
18:50 (10:50) Christina & Karen – Forklifting boxes from LSB to VPW and High Bay
18:53 (10:53) Bubba – in LVEA near HAM4/5 working on snorkel lift
18:53 (10:53) Chris – Bean tube sealing on the X-Arm
19:30 (11:30) Bubba – Out of the LVEA
19:56 (11:56) Joe & Chris – Back from X-Arm
20:24 (12:24) Completed operator assigned FAMIS ticket tasks
21:15 (13:15) Joe – Beam tube sealing on X-Arm
21:27 (13:27) Kyle – Going to Y-End compressor room
21:50 (13:50) Jenne & Cao – Going to check the Triplexer installed at HAM6
22:16 (14:16) Aidan & Nutsinee – Going to X-Arm HWS table
22:57 (14:57) Jenne & Cao – Out of the LVEA
23:15 (15:15) Joe – Back from X-Arm
23:45 (15:45) Kyle – Back from End-Y
00:00 (16:00) Turn over to Travis


End of Shift Summary:

Title:  02/03/2015, Day Shift 16:00 – 00:00 (08:00 – 16:00) All times in UTC (PT)

Support: Jenne, Kiwamu,   
 
Incoming Operator: Travis

Shift Detail Summary:  Ongoing commissioning work during the 

There seem to be two new rf oddities that appeared after maintenance today:

• AS90 seems to jump back and forth between two different offsets, maybe about once every hour or so.
• POP90 and AS90 both show irregular low-frequency motion, similar to what we see when an in-loop OSEM (or its sensing chain) goes bad. Ease of locking doesn't really seem to be affected, unlike the previous experience with bad OSEM electronics.

Attached are spectra comparing the SUS ETM L3 LV ESD channels from lock segments before (Jan 24, 2016 03:35:00 UTC) and after (Feb 1, 2016 01:50:00 UTC) the ESD Driver update last Tuesday alog 25175 (and the subsequent ESD fixes on Wed 25204).  Before the install, the channels looked like ADC noise, while they look live now.  The ETMx plot is included, but of course the ETMx ESD is not ON during either lock stretch, so all the plot really says is that something happened to the channels.  Whether or not the ETMy channels look like they actually should, according to various ECRs, is to be determined.