We are having trouble keeping the Y arm locked in green right now, and according to the optical lever ITMY is having large sudden glitches.  Our difficulty locking might be unrelated to the apparently higher motion of this ISI, which has been moving more that ITMX for 20 days according to the BLRMS. 

The colored blocks on the ISI overview screen seem to indicate that there is a problem with the ISI, and Corey noted that the CPS spectra looked noisy earlier today.  (44061)  

I think that Jim looked into that and I think he and Betsy told me that the problem came and went, indeed.  The first attached screenshot shows a comparison of the 4 BSC ISI BLRMS medms. (ETMX has a known problem, ITMY has only starting having this problem as far as I know.)  Also attached is a time series of some of these BLRMS with a couple of the ITMX channels for reference.  There does seem to be some kind of a problem that might be coming and going on ITMY, this has been the case for at least 20 days (3rd screenshot). 

TITLE: 09/19 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: None
SHIFT SUMMARY:

Commissioners working on locking most of the day.  Had issue with ITMx BIO (FIXED), noisy ITMy CPS (maybe fixed?), noisy ITMy oplev (not sure on status), noisy MC1 (not sure on status).
LOG:

• 14:55 Commissioners already working on locking.
• 15:20 Noticed Binary IO issue for ITMx (suspecting software)
• 15:36-15:45 Going on floor to check for mechanical shorts for ITMy (Ed)
• 15:54 End Station cleaning Karen (requested to be mindful around electronics racks & BRS)
  • 15:54-16:35 @EX
  • 16:40-17:06 @EY*---No glitches!  :)
  • *EY is where we have had ISC glitches which have destroyed ALS alignment.  Which sounds like it's relegated to the CER and when an invasive activity is occurring (i.e. accidental unplugging of rack power cable & maneuvering Genie lift).
• 16:23-17:45 ISCT6/rack work (Nutsinee)
• 17:25 Topped off PSL Chiller (Corey)
• 17:32 HVAC welding at VPW (Mark)
• 17:33-done Checking ITMy oplev again (Jason)
• 17:46-done Access Control work (Richard at EY & APS at MY)
• 18:06-done Clean-up & ethernet cable install at EY & then EX (Dave)
• 19:59 SUSB123 reboot to address ITMx binary I/O (CDS)
  • Rebooting hardware (binary IO chassis) seems to have fixed this.
• 20:03-20:51 Slow Controls SDF down, cont from yesterday (Jonathan)
• 20:36-20:47 Fire dept MY to address non-working alarm bell (Ed)
• 20:36-21:00 FMCS work at EY air compressor room (apollo)
• 21:06-21:30 Access Control near PSL (Richard)
• 21:35 Access Control out to MX (APS)
• 22:00-23:02 Site Weekly

As part of the upgrades in CDS we rebuilt all the slow-controls/CA SDF models against RCG 3.4.3.  We also moved the models to run on a new machine h1ecatmon0 which is a diskless computer.  The SDF models auto start on system boot.

The old host for these SDF models (h1build) has been retired, Dave will provide a new h1build(x) with a current OS/build environment later.

• Filled Crystal chiller with 200mL (Diode had no alarm & not filled).
• Crystal chiller filter is a little more yellowish in color compared to the Diode chiller filter.

BSC ITMy CPS corner #2 (mostly H2, a little V2) look noisier than normal.  First attachment is noisier ITMy BSC CPS (compare to last week).  Notifying Hugh & Jim to see if this is a concern or normal.

All other BSC & HAM CPS's look nominal (plots attached) & closing FAMIS.

TITLE: 09/19 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    Wind: 3mph Gusts, 2mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

• Jenne & Gabriele already working on locking (PRMI can lock, working on DRMI).
• Noticed Timing Errors for h1ioplsc0 section of CDS Overview, but it has already gone away.
• A few FOMs have issues with tools not running (i.e. nuc3:  H1_DARM_FOM.xml, video1:  BRS_FOM.xml)

This morning I noticed that an LSC glitch was perhaps was not getting cleared. Yesterday I had modified the auto_diag_reset script to no longer clear the end station SUS glitches (so if we have any, they will latch) but to continue to clear LSC glitches. I'll check today to ensure the script is running.

We have had no end station SUS glitches overnight since yesterday's upgrade of these machines from V2 to V4.

Danny, Georgia, Sheila, Hang, TVo

After fixing the mask today, the HWS ITMY shows that the clipping issue we've had before is now fixed and the heating profile is more circularly symmetric.

We still got a contrast defect of about 0.4%, but we didn't optimize the differential CO2 power because we saw an obvious pringle mode (HG20-HG02) that stems from astigmatism,  this shows up pretty obviously when looking at the single bounce beams where ITMX looks good, but ITMY looks bad.  

Sheila suggested that we should be able to scan the PR3 around the ITMY surface and see the astigmatism in the single bounce ITMY beam vary as we start to sweep over the center of the CO2 lensing.  This worked a bit but we started clipping after moving the spot position on ITMY by about 5 mm so it gave use a direction to try to aim for when changing the alignment of the CO2 beam relative to the test mass. 

Although this had confusing results because we once tried pico-ing to where we thought it should get better by doing a modulation of the laser power and seeing the deflection at the anti-symmetric port with the ASAIR camera, the astigmatism didn't seem to get better.  Changing the picoing in the y-direction showed that the deflection went up whether or not we pico-ed up or down in counts so we thought we might be at a minimum, however, If we went through a minimum then we should see a phase change between the heating cycle and the deflection slope but we weren't able to observe this which makes it even more confusing.

In the end, there was still astigmatism after the pico-ing but Sheila suggested that the morning commissioning crew try to lock and see how these particular lenses change our locking scheme/setpoints

WP7815

Rolf, Keith, Dave, Jonathan, Jamie, Richard, Fil, Ed 

Today we did a major upgrade to the CDS system:

• Upgraded all models to RCG-3.4.3 (from 3.2.0/3.2.3)
• Upgraded all front end OS to Gentoo-3.0.8 (from 2.6.34)
• Upgraded Dolphin hardware from DX to IX series
• Replaced GeFanuc long-range reflective memory IPC with long-range Dolphin
• Upgraded some V1 and V2 front end computers with V4

After the IFO was placed in a safe state, we stopped all models and powered down all computers with Dolphin cards. We stopped DHCP running on the old boot server (h1boot) and started it on the new boot server (h1boot1). We rebooted all the non-Dolphin systems (SUS-AUX) to gt3.0.8/rcg3.4.2 and restarted the DAQ.

Rolf, Jonathan, Fil and Ed upgraded the MSR, Keith upgraded the software systems, Dave and Jamie upgraded the end stations. At each location:

• GeFanuc switches and fibers were removed from rack
• DX-Dolphin switches and cables were removed from rack
• Computers with Dolphin cards were retracted from rack and their DX-Dolphin cards were replace with IXH611 cards
• IXS600 Dolphin switches were installed in the racks (three in MSR, one in each end station)
• IX Dolphin cables ran from switch(es) to cards
• IX Dolphin switch(es) attached to H1FE-LAN (required port activation at end stations)
• At end stations SUS and ISC computers were upgraded to 8-core E5-1660-v4 computers*

The long-range Dolphin server (h1cdsrfm) had been installed on Monday, today its Dolphin cards (both local and at the end stations) were connected to their respective Dolphin switches.

The h1ioplsc0 model was modified to run as a timing server for h1oaf1.

We discovered timing issues with the mid station PEM systems. The remoteGPS function was not obtaining the correct time from h1dc0 via EPICS channel access. When this was turned off, the GPS time was one second too slow. A modified RCG-3.4.3 was installed to advance the GPS time by one second and h1ioppem[mx,my] were built against this. The Mid Station PEM systems are now running correctly.

*V4 computer upgrade:

Initially we were to upgrade 10 front end computers to V4. However we found that the V4 computers could not detect the cards in the IO Chassis if the OneStop fiber transceivers were from 2011 or earlier. Testing on h1iscex and h1susex showed that 2012 and 2015 fibers worked. Because running new OneStop fiber at the end stations is relatively easy (adjacent rooms, open access cable trays) , Fil and Ed ran two 2012 fibers at each end station for the SUS and ISC computer upgrade. Running new fibers between the MSR and CER in the corner station is more problematical, so the two computers which had been upgraded to V4 at that point were returned to their original hardware for now.

The upgrade was completed and the system handed over to the control room at 16:59 PDT.

Fixed:

• Gentoo2.6.34 208-days bug
• Time glitching of the end station SUS systems

Still To Do:

Install new h1oaf1 front end computer, this is a Dolphin only machine (no IRIG-B timing or IO Chassis) which will get its timing from h1ioplsc0.

the ITMY optical lever is noisy (screenshot attached).  It would be a good idea to check for mechanical interferences when there is an oppurtunity. 

There seems to be a problem with the ITMX BIO, screenshot attached.  It has been this way (H1:SUS-ITMX_BIO_M0_MON = 1535) since about 1:22 UTC, we don't know of any activity that would have caused a problem at that time. 

setting H1:SUS-ITMX_BIO_M0_CTENABLE to 0 and back to 1 again doesn't fix the problem. 

J. Kissel, D. Barker

We've been periodically finding front-end modules that show differences when no one reports making changes. Upon looking at the diff provided by the RCG system (an awesome newish feature) between the loaded file, and the *to be* loaded file, there are TONs of differences. Upon close inspection, however, those changes are entirely superficial -- the header of the file where all filter banks in the model are recorded (in "some" order) has been flipped or re-ordered, and/or (sometimes both, sometimes one or the other) the coefficient notation has switched from engineering/decimal notation to scientific notation. 

I'll open an FRS ticket about this, but at this point we don't have a smoking gun as to who what where when or why this happens, we (Dave and I) just have seen it enough that it has become a "usual" problem in our parlance. Usually because it relies on Dave dilligently (but still only occasionally) browsing through the CDS overview screen before a pending computer change, and comes to me asking me to figure it out and resolve it. So it can be up to "many days" from when someone may have altered the filter file with some program, and when we notice it.