In bring the diagnostic breadboard back online, which required powering up the electronics,
the flow sensor sensor tripped causing the shutter to close.  It was reset from the Control
Room.

Back to normal.

The reduction in gain on chiller loop looks to have got rid of the oscillation that we were seeing.  Attached is the over night data, which shows the same small power output range and pzt range, with a smoother chiller response.  Y-arm laser wasn't on last night.

   Transition Summary:
02/04/2016, Day Shift 16:00 – 00:00 (08:00 – 16:00) All times in UTC (PT)
	
State of H1: IFO unlocked. The wind is a light breeze (< 7mph). Seismic and microseism are still as elevated as they were last night. Locking may be somewhat difficult at this time.      

It has been a fight to get DRMI to stay locked for more than a few minutes due to high ground motion.  Commissioners suggest calling it a night as microseism continues to trend upward.

The ground motion is just too high, so Evan, Travis and I agree that we should call it a night.  We can't hold DRMI for more than 2 minutes or so.  Earlier today, I flipped the phase of the ASAIR 90 PD, so that we get roughly the same number of counts when DRMI is locked as we used to.  We only use ASAIR90 for thresholding engagement of the DRMI ASC, so it's not a super critical PD phasing-wise.  I'll come back to it tomorrow.

This is another instance of getting the '75% notification', followed shortly by a WD trip on HAM6.  All I did was to reset the WD.

Vern, Jim, Dave:

Several times in the past few weeks we have seen an invocation of tconvert produce the output:

tconvert NOTICE: Leap-second info in /ligo/apps/linux-x86_64/ligotools/config/public/tcleaps.txt is no longer certain to be valid; We got valid information from the web, but were unable to update the local cache file: No permission to write in directory /ligo/apps/linux-x86_64/ligotools/config/public
 

This afternoon we spent some time tracking this down. tconvert uses a web server in france to access the IERS bulletin-C information page which alerts us to upcoming leap seconds. It appears that this server was pure HTTP until January 11th this year. It is now an HTTP server which redirects the request to a HTTPS page. The TCL code inside of tconvert interprets the redirection return message as an error. It then marks the expiration of the leap second information for 24 hours in the future.

The procedure we used to maintain future leap second notification was to twice a year run tconvert as user controls, which updated a leap seoncds database file (owned by controls) which was good for the next six months. With the recent HTTP issues, controls now produces a file which is good for only a day.

This afternoon we ran a modified tconvert which access a different leap seconds HTTP page and the LHO CDS database file is now good through 8th August. In the mean time we will raise an FRS and work with Peter S for a long term solution.

Late entry from Tuesday Maintenance

Attach test conlog machine to IRIG-B fanout [WP5712]

Fill, Patrick, Jim, Dave:

We ran a 50 Ohm coax from a spare port of the MSR IRIG-B fanout to the test conlog machine, Patrick was able to get all the timing information he needed except for the year.

Model Rate and DAQ change for BSC SUS-AUX models [WP5714]

Jeff K, Betsy, Jim, Dave:

For the h1susauxb123, h1suseuxex and h1susauxey models three changes: change rate of models from 2048Hz to 16kHz, add filters, add one channel to the commissioning frame at 16kHz and remove some obsolete 1kHz channels.

New asc model

Jenne, Sheila, Dave

new h1asc model with additional filters and new excitation injection points.

Add FEC SDF buttons to conlog

Patrick, Dave

We changed the python script which generates the conlog fe_include.txt file to include SDF buttons. Conlog was restarted.

HWS Beckhoff changes

Daniel, Aidan, Alastair, Patrick, Dave:

h1ecatc1plc3 was modified to add HWS channels. Autoburt.req and DAQ INI files were updated.

BSC ISI T240 channel problem identified and fixed

Jim W, Jim B, Dave

Jim found that the T240 X,Y,Z channels were not correct. We identified the problem in the isi2stagemaster common model. Jim fixed the problem and we rebuilt and restarted h1isibs, h1isiitmx, h1isiitmy, h1isietmx and h1isietmy

New PSL ISS model

Kiwamu, Jim, Dave:

Kiwamu installed a new h1psliss model. This required several restarts. There was no impact on the PSL output. We burt restored to 12:10 PST Tuesday after each start.

DAQ Restarts

Jim, Dave:

Due to possible file system slowdowns, we changed the monit configuration on the DAQ systems to 60 seconds for DC, and 30 seconds for all others (was 10 seconds before). We reatarted the DAQ four times during the course of maintenance to support the above changes and did not see any repetition of previous DAQ startup problems.

I started doing a power budget for the table, but got side tracked investigating the in-loop diode that doesn't have any response.  I tried realigning onto it but got a strange response that included negative voltages.  As a result the lens isn't in place in front of the diode now, so I've blocked this path using a 10W power meter head (this path gets ~200mW so this head can easily cope).  The laser is turned off just now at the controller.

There is already a fairly large power reduction through the AOM, from 63W down to 56.6W, then down to 56.1 after the polarizer.  The PDs are getting 450mW, so even after these stages there is more than 50W going into the rest of the table.  It is pretty tricky getting power meters in becasue there isn't a lot of room, so we may not get a complete picture of the power losses.  I'll work more on this tomorrow.

Hugh asked why there were 'dot nfs' files in his directory and why he could not remove them, but later when he exited matlab they were removed. Here is a part of his directory listing when the files were there

-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840bb80000002f
-rw-rw-r-- 1 hugh.radkins controls 76376 Feb 18  2015 .nfs000000000d840bb70000002e
-rw-rw-r-- 1 hugh.radkins controls  5974 Feb 18  2015 .nfs000000000d840bb60000003e
-rw-rw-r-- 1 hugh.radkins controls  5974 Feb 18  2015 .nfs000000000d840b9a0000002d
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840b2c0000002b
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840b210000002a
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840b1200000029
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d84096200000028
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d84095700000027
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d84094c00000026
-rw-rw-r-- 1 hugh.radkins controls 76376 Feb 18  2015 .nfs000000000d84093400000025

-rw-rw-r-- 1 hugh.radkins controls 89335 Feb  3 16:15 .nfs000000000d840bc600000046
-rw-rw-r-- 1 hugh.radkins controls 89335 Feb  3 16:15 .nfs000000000d840bbe0000003c
-rw-rw-r-- 1 hugh.radkins controls 77540 Feb  3 16:15 .nfs000000000d840b4c00000035
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840b9900000036
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840b9c00000037
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba200000038
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba300000039
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba40000003a
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba50000003b
-rw-rw-r-- 1 hugh.radkins controls 77540 Feb  3 16:19 BSC9_H1_Valve_Check.xml
...
 

In Linux is is permitted for a program to open a file and then to unlink it. On a local file system this creates a file with no name (wont show up in 'ls' for example) but the inodes still exist for the file and the program can still read and write to the phantom file. Only when the program stops running does the OS clear the inodes for reuse. This is great for temporary files which should disappear when the program stops, even if it crashes and does not cleanly close the files.

The situation Hugh encountered is when the file is on an NFS mounted file system. In this case when the file is unlinked, the NFS client renames the file to a dot nfs file with a random name. If a user on the same NFS client machine tries to delete the file, they get a "cannot remove, Devicee or resource busy" error and the file is not deleted.

Interestingly we found that on a different NFS client it is possible to delete the file since the controlling process is not on that machine.

So we have potentially two problems. One is that dot nfs files get stuck open (like the ones from last Feb 18th in Hugh's listing, we presume the NFS client was abruptly shutdown). Second is that someone deletes a file on one NFS client which is actually being used by another NSF client.'s program.

Activity Log: All Times in UTC (PT)

15:43 (07:43) Reset tripped ISI WD on HAM6
15:53 (07:53) Peter – Going into the H2 enclosure to look for electronics equipment
16:00 (08:00) Peter – Added 350ml water to the Diode chiller
16:16 (08:16) Peter – Out of the H2 enclosure
16:45 (08:45) Richard & Filiberto – At HAM6 for Triplexer installation
16:53 (08:53) Joe – Going to X-Arm for beam tube sealing
17:15 (09:15) Carpet installers on site to finish OSB installation
17:35 (09:35) Completed initial alignment
17:36 (09:36) IFO in DOWN for Kiwamu, who is working on the ISS Second Loop 
18:05 (10:05) Richard & Filiberto – Out of the LVEA
18:50 (10:50) Christina & Karen – Forklifting boxes from LSB to VPW and High Bay
18:53 (10:53) Bubba – in LVEA near HAM4/5 working on snorkel lift
18:53 (10:53) Chris – Bean tube sealing on the X-Arm
19:30 (11:30) Bubba – Out of the LVEA
19:56 (11:56) Joe & Chris – Back from X-Arm
20:24 (12:24) Completed operator assigned FAMIS ticket tasks
21:15 (13:15) Joe – Beam tube sealing on X-Arm
21:27 (13:27) Kyle – Going to Y-End compressor room
21:50 (13:50) Jenne & Cao – Going to check the Triplexer installed at HAM6
22:16 (14:16) Aidan & Nutsinee – Going to X-Arm HWS table
22:57 (14:57) Jenne & Cao – Out of the LVEA
23:15 (15:15) Joe – Back from X-Arm
23:45 (15:45) Kyle – Back from End-Y
00:00 (16:00) Turn over to Travis


End of Shift Summary:

Title:  02/03/2015, Day Shift 16:00 – 00:00 (08:00 – 16:00) All times in UTC (PT)

Support: Jenne, Kiwamu,   
 
Incoming Operator: Travis

Shift Detail Summary:  Ongoing commissioning work during the 

[Jenne, Cao]

What was once the 90 MHz local oscillator for just the ASAIR 90 WFS now goes to a distribution amplifier box.  3 of the outputs of that board now go to the local oscillator inputs for each of ASAIR_90, AS_A_90 and AS_B_90. The local oscillator inputs want 10dBm each, so we measured the output of the distribution box, and each output was 15dBm.  So, we put 5dB RF attenuators on each spigot.  Next up, we'll lock DRMI and look at phasing.

Yesterday I put both X and Y lasers into the DOWN guardian state, left them for a while to run unlocked.  I then set both guardian states to LASER_UP and left them to lock themselves and run over night.  Attached is the chart (guardian_locking_X_and_Y_lasers.png ) showing the output power, which is +/-0.015W for the Y arm laser and +/-0.008W for the X arm laser.

The PZT movement is +/-5V on both lasers, with the full range available being +/-35V, so this seems good.  The chiller movement is approximately 0.05C on both chillers, which again is very small compared to the chiller range.

There is a glitch seen periodically in the Y-arm laser power output with a 15minute interval.  We believe that we have tracked this down to being the interval that the FLIR camera uses to re-calibrate (this involves some internal mechanical motion) and looking at the attached chart (Y_Laser_FLIR_GLITCH.png) it seems that we have got rid of it by remotely turning off the FLIR camera from the MEDM screen.

The chiller setpoint for both lasers looks to have some small amount of oscillation.  The period is 31mins and it is seen at about the same level in both lasers.  Looking at the LVEA temperature data (temperature_fluctuations.png ) there is no correlation to actual lab temperature changes.  Also we can see that the oscillation appears in the PZT output channel, showing that the faster PZT actuator is subtracting out the length change caused by the oscillation in the chiller.  The chiller uses this voltage as the setpoint for its actuation and it looks like the chiller lags the PZT by 180 degrees.  In order to reduce the unit gain point of the servo I have reduced the gain by a factor of 3 on both X and Y chiller servo loops.  I then put the lasers into DOWN guardian state and then back into LASER_UP.  I will leave them to run and we can look to see if this fixes the problem.

Evan told me that there seems to be an analog pole of 40Hz or so that is unaccounted for if he makes the transfer function from H1:SUS-ETMY_L3_LOCK_L_OUT to the LVESD monitor whitening outputs ( L3_LVESDAMON_LL_OUT and such) after removing all known analog and digital poles/zeros.

I measured the transfer function for EX, not from LOCK_L but from ESDOUTF_LL_IN2 to LVESDAMON_LL_OUT and confirmed that the problem is also in EX driver.

In the attached, blue is the measured transfer function divided by the transfer function of the whitening that is supposed to be (z, p)=([2;2;19.3k;19.3k], [40;40;965;965]) Hz (https://dcc.ligo.org/LIGO-D1500389).

I also multiplied the bule with 42Hz zero and got the green one, it seems like there's really 42Hz-ish analog pole that is not accounted for. (There are also high frequency things I'm not worried at the moment.)

Eventually I remembered that, in the old version of LVESD that we are using, there is an LPF for monitor output which had a 42Hz pole though it was supposed to be 1kHz (the LPF in question is on page 11 of D1500016). Rich Abbott fixed all spares, but not the ones we're using. One mystery solved.

I talked with Richard and Filiberto, and since they have some PI-related task scheduled on next Tue., I hope this can be fixed at the same time by either modifying or swapping.

As part of my larger investigation into IM alignment jumps, I drove IM1, IM2, and IM3, in pitch and yaw by -300 "slider urad" to measure the change in the OSEMs.

The first chart shows the response to the change in alignment drive of -300 "slider urad", change in "OSEM urad", the change on IM4 Trans QPD, and the expected change to the alignment slider drive to change the OSEM urad by 5 urad.

In pitch, the alignment slider changes are about 40 times the OSEM value changes, which is due to the magnetic damping in pitch.

In yaw, the alignment slider changes are about 10 times the OSEM value changes, so closer, but not all that close.

This alignment slider calibration may be something that should be updated.

The first plot also shows what alignment slider change is necessary to move the optic 5urad.  I used these numbers to produce the second table.

The second table shows the drive change calculated in table one did produce a 5urad OSEM change.  It also showed that in the axis that wasn't being drive, there was up to a 1urad shift in alignment, which happened in both pitch and yaw.

Diagonalization of the IMs may be something that should be updated as well.

I was trying to get the EY St2 guardian to turn on RX/RY isolation filters and was not having much luck. I would make the necessary changes in the ISI_ETMY_ST2.py file, save and hit reload on all of the chamber's SEI nodes, then tried cycling the ISI down to damped and back up. When I couldn't get the guardian to turn on the loops I wanted, I tried going the other way and leaving loops off. When I left Z out of the DOF and ALL_DOF lists, the ISI would turn ON the ST2 Z loop, then not turn it off. When I checked the logs, guardian was still turning ON the Z loop, then turning OFF (the still not engaged) RX and RY loops. When I talked to TJ about this, he said that he had seen that it was necessary to sometimes restart Seismic nodes to get them  to fully digest changes. When I did guardctrl restart ISI_ETMY_ST2, then tried another cycle of ST2 to damped then isolated, this time the node turned on the requested loops.

Odd.

I've updated the record of calibration references for the TCS channels.

It is attached here:

Masayuki, Kiwamu,

With the two functions that we implemented yesterday (alog 25316), today we tested the automation to see how they improved the engagement of the second loop.

It locks without a fail out of more than 20 trials within approximately several seconds every time. Very good.

On the other hand, I have noticed that I was missing some signal conditioning filters in order to handle multiple error signals in a user-friendly way. So we need another round of modification on the front end model which should finializes this implementation.

I reverted the settings back to nominal. So the guardian still handles the engagement as usual.

[Advantage over guardian in this case]

The front end code allows for a fast and more complicated servo filter. This is the biggest advantage and therefore the key point of the succeess in the test today.

The reference signal or offset of the second loop has been adjusted by the IMC  guardian which employed a simple PID for servoing the reference signal so as to minimize the "acquisition kick" during the engagement. Looking at the behavior of the suppressed and unsuppressed signals, I found the UGF of this loop to be lower than 0.1 Hz. At one point in the past, this servo became insufficient due to excess intensity fluctuation imposed by the IMC. The IMC seemingly adds extra intensity noise below 1 Hz presumably via misalignment. As a consequence, the guardian servo became unable to keep up with such a large and fast fluctuation. We started assisting the guardian by the manual engagement (alog 22449) where the operator activates the second loop when the second loop error signal momentarily crosses zero.

The new implementation tested today was able to achieve a UGF of 0.8-ish Hz which sufficiently suppressed the fluctuation and therefore a smooth engagement of the 2nd loop with a help from the trigger. In the digital filter, I had to install 3 pairs of poles and zeros ( i.e. zpk([0.03;0.03;0.05], [2;3;3], 1) ) in addition to an integrator in order to compensate for a 0.1 Hz second order analog low-pass. Currently the UGF is limited by the DAC range. The rms fluctuation monitored by SECONDLOOP_SIGNAL reduced by roughly a factor of 5. The dominant component was from around 0.1 Hz and can now be well suppressed with the fast servo.

Svn up'd LHO's common guardian masterswitch and watchdog folders:

hugh.radkins@opsws1:masterswitch 0$ svn up
U    states.py
Updated to revision 12509.
hugh.radkins@opsws1:masterswitch 127$ pwd
/opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/masterswitch
hugh.radkins@opsws1:masterswitch 0$
hugh.radkins@opsws1:masterswitch 0$ cd ../watchdog/
hugh.radkins@opsws1:watchdog 0$ svn st
hugh.radkins@opsws1:watchdog 0$ svn up
U    states.py
Updated to revision 12509.

I restarted HAM6 and tested by disabling an output leg.  The trip appeared to execute the functions as expected-I did not notice any problem.  Restarted HAM5 and tested, it too turned off the FF and reduced the GS13 gains as expected, no problem noticed.  Restarted HAM4 but did not test (by tripping the platform.)

Restarted HAM3 after enabling the GS13 Switching feature as I wanted to test the problem of the guardian being unable to turn the GS13 gain up without tripping the platform.  This is where I noticed the problem.

When guardian turned up the GS13 gain, the HAM3 tripped and it successfully turned off the FF and lowered the GS13 gains but it left the DAMPing loops engaged.  I thought the restart of the guardian may have been responsible for this behaviour.  I cleared the trip but did not turn off the DAMPing path first.  The ISI did not trip until the GS13s were again toggled to high gain but this time, the DAMPing path was turned off as I would expect.  Okay, maybe first time around problem.  Cleared the trip and again the platform tripped when the GS13 gain changed and again the DAMPing path was left on.  I repeatd and again the DAMPing path was left on.  I disabled the GS13 Gain Switching feature and we made it to isolated and I set the GS13 gains to high with the Command Script.

I've repeated the test on HAM5 and there too, DAMPing path remained on.

Meanwhile ITMX tripped due to LVEA activity, DAMPing path not turned off and this guardian has not been restarted with the new update.  Repeated this on ITMY and it too left the DAMPing path enabled.  Okay, it looks like this DAMPing problem is not related to the current code upgrade.

I will continue to restart the guardian with this current upgrade though as the turning off of the GS13s when tripped is a good thing and generally, the platform can deal with untripping the watchdog with stuff coming out of the DAMPing path, as long as the GS13 are in low gain.  And since HAM2 & 3 can't handle guardian gain switching, they must have the gains toggled manually.

[Alastair, Aidan]

Attached are some plots of RIN for the X and Y arm lasers.  There are now channels in the front end for RIN for in loop and out of loop photodiodes on each table.  Here we show the RIN and dark noise for both lasers.  The in loop photodiode for the Y-arm table is giving zero response so either there is a problem with it or the beam is mis-aligned.  I've only attached the data for the other 3 diodes.

The dark noise is consitent between all three diodes.  The RIN of the X-arm laser is approx 5x10^-7 at 20Hz.  The Y-arm laser is higher at 2x10^-7.  The Y-arm result is similar to the best we measured at Caltech (also attached).  It may be that the X arm is lower than we have previously measured due to the low noise enviroment.  The spectrums show no excess at low frequency such as would be attributed to air motion on the table, which is good news.  Also the in loop and out of loop diodes on the X-table show very similar spectrums which is good for any future intensity noise cancellation.