Today VerbalAlarms crashed due to a ValueError, but luckily we didn't need Verbs today because of the environment.

The problem came from code I added on December 9 that would stop the tempurature alarms from repeating too often. I it would look through a date string for the 9th-11th elements, which should be the minutes, and then see if it was five minutes since the last alarm. This was good and all until the date changed from a single digit to two digits. That was something I should have seen, my fault. The reason it did not show up until now is because it was under a few conditionals that have not been met since the 9th.

The fix was two part. First I made a regular expression to look through the date string and find the minutes. This worked well, but increased the loop time a tiny bit more. So I then went for the much simpler solution and made a global variable 'minute' from when I make the date string. Added only one line of code. This worked even better.

This configuration has been tested and I have reloaded Verbs with the new code.

Locking seems to be impossible, I can't keep ALS locked. I've tried many different blend configurations but there is always a ton of movement. It seems to be well aligned but then will just drift back to 0 power or it will immediately lose lock. I think the enviroment has got us for now.

useism ~1.0um/s. Wind <35mph.

Ryan, Jim, Dave

we built a new NFS server called x1fs0 today. It has a ZFS file system installed on two 1TB HDD. Ryan introduced us to the wonders of ZFS on linux. This machine is a new build Ubuntu 14:04LTS. We are currently rsyncing /opt from x1boot to x1fs0 (should complete this evening). Tomorrow we will test if we can move the NFS function off of x1boot over to x1fs0 in preparation for doing the same on H1 post-O1.

Ryan, Jim, Carlos, Dave:

Following the failure of the Tandberg tape backup, as a short term fix we built a 2TB file system on cdsbackup so we could perform a disk-to-disk-to-disk backup of /opt/rtcds and /ligo (replacing the current disk-to-disk-to-tape). With Ryan's help we built a 2TB ZFS file system on cdsbackup (a ZFS mirror on two 2TB disks). I am currently rsyncing /opt/rtcds from h1boot-backup to cdsbackup over the private backup network. This will take many hours to complete. I have verified (with Observium) that this traffic is going over the private backup network and not over the 10.101 or 10.20 VLANS, so it should not impact on the front ends or control room.

I have managed to lock PRMI two times, but could not advance from there. Since my two locks, it has been harder to keep ALS locked. Not going so well.

useism still ~1um/s, wind <30mph.

Today Ken (S&K Electric) and I replaced 5 heating elements in Zone 2A and 2 in Zone 1B.
There were several elements burned out in Zone 3A so we relocated the remaining elements in order to bring these up sequentially as they were designed to do. These elements are a different style and we did not have replacements. I will order more after the first of the year. 

With the exception of 1A, which has some issues that will be addressed after the break, we are now able to turn on heat in all of the zones. 

Manually filled CP3 at 22:43 utc, opened the fill valve by one full turn, time to see LN2 out of the exhaust 4:00 minutes.

At Nutsinee's request, I have re-enabled violin mode damping in full lock.

I turned off the extra 500 Hz stopbands in the PUM pitch/yaw filter modules, and I have commented out the code in the COIL_DRIVERS state which disables the damping.

We had 36 hours of undisturbed data from roughly 2015-12-19 07:00:00 Z to 2015-12-20 19:00:00 Z. This should be enough to analyze the ringdowns. At a glance, all but one of the first harmonics in the spectrum seem to ring down.

Ops Day Summary: 16:00-23:59UTC, 08:00-16:00PT

State of H1: down since 18:15UTC, high useism, winds increasing, not able to get to DRMI

Outgoing Operator: Jeff

Incoming Operator: TJ

Shift Details:

• H1:
• lost lock at 18:15UTC, coincident with a spike in the 0.003-0.1um/s seismic
• needed an Initail Alignment
• woth little hope of locking, we took some time for commissioning
• Jim took some measurements with alternative SEI filters in the Find IR state
• MICH Dark did not work today
• Jim needed PRMI or DRMI for his next measurements, and neither locked
• alignment has been very dynamic, meaning every time H1 drops from trying to lock DRMI, the arms in green need to have the ETMs adjusted to get back to a 0-0 mode, but no WFS troubles, I think just a lot of ground motion
• VerbalAlarms are broken - TJ is here and looking into it

Site:

• Gerardo to MY to manually fill CP3, will be ongoing
• CP3 on VAC overview is reading zero, but it is being watched closely
• deliveries at OSB, LSB, and Staging Building
• useism is about 1um/s in the CS
• winds have increased and are now c-25mph across the site
• HVAC work today increased the elements for heating

• TITLE: 12/23 Eve Shift: 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC"

• STATE Of H1: Down, environment

• OUTGOING OPERATOR: Cheryl

• QUICK SUMMARY: The IFO is down due to highish winds (<25mph) and high useism (1.0um/s). I will continue to try and hope for the best.

Jim had an alternative SC filter that gained a bit at the useism at the expense of lower frequency noise.  With the high useism and lock loss it seemed an opportunity to see if it would improve relocking.  Alas, no independent data captured with the IFO unable to grab and hold.  Another time.

Attached are the two filters.  The green is the new filter.  This filters the ground seismo before it is added to the CPS giving it an inertial flavor before going into the super sensor (Blend.)

Jim's saved the foton files in the svn and we've reverted the ISI back to the nominal configuration--no change in SDF.

We've continued to investigate activity around 74 Hz in order to explain the coherence line between H1 and L1 generated by stochmon. 

Attached is a comprehensive set of results from the coherence tool (the full set of results can be found here). The coherence tool plots coherence with the local (H1 or L1) h(t) channel, averaged over a week with 1mHz resolution. The slides attached show the channels showing coherence structure around 74 Hz over the course of O1. You can find relevant plots via a download link in the attached slides.

As reported in ealier alogs, the EPICS freeze ups are caused by the rsync backups of h1boot by both its clone machine (h1boot-backup) and the tape archive machine (cdsfs1). The file system on h1boot has degraded such that a full rsync which used to complete within an hour is now taking 10 hours. This caused many of the backup jobs to pile up, which in turn caused the front end freeze ups.

I have been re-writing the rsync scripts to spread out the rsync of h1boot by h1boot-backup over many jobs. I am also not rsyncing directories which contain many thousands of files as this always causes a freeze-up. Only h1boot-backup is rsyncing h1boot at deterministic times, I will then rsync h1boot-backup to a third machine to provide a disk-to-disk-to-disk backup.

For reference, here is an email I sent today to lho-all reporting recent CDS backup systems problems, short term and long term solutions:

Over the past few weeks we have developed problems with the CDS backup systems, namely:

Yesterday, the masses on the STS2s were centered but I should have monitored the mass positions more in real time as the centering doesn't always take the first or second time around.  While we did improve things maybe, there are still problems: six of the nine masses (3 masses 3 sts2s) were out of spec but after centering, there are still 3 of the 9 out of spec.

At EndX, the STS2 had two masses out of spec by about 50% (7500 cts vs 5000);  after centering one of those came into spec but the other overshot and is now out by a factor of 5!  Attached is ASDs of before and after.  The refs are from 2am Tuesday and the currents are from 2am this morning.  The useism is likely a bit higher early this morning (not as high as it is right now though.)  There is not anything glaringly wrong to me here.  In general the current traces are higher than the references but again that could just be the actual ground motion.  I thought such a large mass position problem would be obvious but it is not to me.  I'll do a comparison of LVEA instruments at the same time.

I increased the power on the BS and SR3 oplev lasers this morning by ~5%.  I used the voltage read out of the "Current Monitor" port on the back of the laser (this port monitors the laser diode current, outputs a voltage).  The old and new values are listed below, as well as the new SUM counts.  Since I had to open the coolers the lasers are housed in they will need ~4-6 hours to return to thermal equilibium; at that point I can assess whether or not the glitching has improved.

• BS OpLev
  • Old voltage:  0.987 V
  • New voltage:  1.036 V
  • New SUM:  ~33500 counts
• SR3 OpLev
  • Old voltage:  0.889 V
  • New voltage:  0.933 V
  • New SUM:  ~40900 counts