Spun up vertex turbo right before lunch at ~0.6 Torr. Still need to close purge/vent isolation valves (OMC and IMC) and IP3 GV and little right angle valve.

The DMT production computers in the MSR, h1dmt0, h1dmt1, and h1dmt2, have been patched and rebooted. WP 7466 is done.

At ~2256utc on 3 April (or precisely 1206831398gps), all the CPS sensor readouts on both stages jumped to a different value. Eight of the twelve spiked to a 33kish number before going somewhere else.  A few on Stage2 are not crazy numbers but still shifted.  Most on Stage1 are extreme ~positive 8k.  See attached

Another item to be investigated is the Coil Driver Thermal trip hit at 1557utc on the fifth...

I confirmed with Patrick yesterday that the RS232 flow rate for both chillers shows that the chillers are on, and the flow rates are around 3gpm.

The flow rates on the medm are negative, both around -2.5gmp.

Greg's most recent alog 41314 from April 5th says the chiller is running in a small closed loop, and it appears that this is for both TCSX and TCSY chillers.

Kyle, Gerardo, Chandra 

Today we pumped IP11 using a local turbo backed by an aux. cart.  IP11 is isolated from the Y-end via its closed GV.  As I squirted IPA around each CFF joint and vacuum welds of the Chevron Baffle nipple, Gerardo M. monitored the pressure gauge on the aux. cart to see if there was a response.  Chandra R. had already established this to be a very large leak (0.1 torr*L/sec).  We tried "listening" for the audio leak using a device intended for that purpose -> Something seems to be originating from the Chevron Baffle but difficult to eliminate background noise in the room -> We shut down the Clean Room (1417 hrs. local) and that helped a little.  Next, we modified the as-found bag such that each CFF joint was exposed but keeping the nipple's tube vacuum welds within the bagged volume.  We then purged the bag with bottled N2 while spraying helium around each CFF joint -> no significant response.  Finally, we reversed the bagging, now purging the CFF joint(s) while spraying helium around the welds of the nipple tube -> found very large leak coming from the bottom tube-to-flange vacuum weld (nipple s/n 01).  

Having found one leaking Chevron Baffle, Chandra R. examined a different one stored in the VPW.  She found suspicious looking welds.  

Also, we prepared IP11+chevron baffle nipple for tomorrow's removal by venting it with bottled UHP N2 and removing all but (3) ea. of the nipple-to-GV nuts and studs.  Note that with the local turbo spun down and disconnected from the aux. cart, I applied the N2 to the Tee at the inlet of the turbo letting it flow trough the turbo and into the room via the open-ended turbo exhaust line.  This setup would prevent the possibility of over-pressurizing IP11 as it remained open to the room.  I then opened the 1 1/2" vent/pump valve to vent IP11.  Gerardo M. noticed that room air was then entering through the exhaust.  Thus, for a short period, I had been back streaming through the local turbo and into IP11.  I then closed the 1 1/2" valve and removed the turbo. This is a "dirty" pump in that it is not a maglev.  I'll consult Chandra R. as to what, if any action is needed (FTIR?).   

Isolated QDP80 at vertex with pressure at 1.3 Torr. Will resume tomorrow morning.

Chandra has requested that the high alarm level for H0:VAC-MY_CP3_TE202A_DISCHARGE_TEMP_DEGC be raised from 130.0 degC to 140.0 degC (this signal is currently operating at 131.1 degC). This was done and the alarm system was restarted.

day started with JeffB's alog: 41345

All Tmes UTC:

• 09-04-18 17:14 Mark and Tyler craning over X beam manifold
• 09-04-18 17:15 Jason, Peter, Ed, to PSL
• 09-04-18 17:15 Fil to EY to test ALS laser
• 09-04-18 17:25 Hugh to EY to work on HEPI
• 09-04-18 17:39 JeffB to LVEA to reset dust monitors 5 and 6
• 09-04-18 17:48 JeffB done in LVEA
• 09-04-18 17:49 JeffB to MX to remove the old floor vacuum system
• 09-04-18 18:37 Hugh done at EY, going to MX and MY
• 09-04-18 18:38 Travis and Betsy back from EX
• 09-04-18 18:43 Mark and Tyler done in LVEA, heading to EY to pick up un-needed items
• 09-04-18 19:29 Ed out of the PSL
• 09-04-18 19:29 Peter and Jason out of the PSL
• 09-04-18 19:30 Hugh back from MX and MY
• 09-04-18 19:31 Fil to EY
• 09-04-18 19:53 JeffB done at MX
• 09-04-18 20:40 Travis and Betsy to EX
• 09-04-18 20:40 Kyle to LVEA
• 09-04-18 20:59 Rich Abbot and guest out of the LVEA
• 09-04-18 20:59 Jason and Ed to the PSL
• 09-04-18 21:37 Mark to EY to reinstall timing box
• 09-04-18 22:01 Travis and Betsy back from EX
• 09-04-18 23:23 PeterK out of the PSL
• 09-04-18 23:23 Jason and Ed out of the PSL

Today, Travis and I finished assembling the ETMX lower reaction chain.  We assembled the PenRe mass in the structure, attaching the balance of parts to make the weight from last week including AOSEMs, cables, and cable routing brackets.  We then installed the new annular AERM07 mass below the PenRe in the lower structure and adjusted it in it's 6 DOF's by eye before suspending it.  We then reclamped everything and rolled the main lower chain up to it.  A quick lift via the Genie duct jack and we set the main lower structure and newly fiber'ed masses into the trolley with the reaction set.  We added the UIM and PUM magnets/flags (set with polarities per the QUAD controls poster), and then shoved the 2 structures together and fastened them.

We wrapped the now complete lower unit and rolled it to the chamber side.  After attaching some of the LSAT lifting blocks, we staged the Genie duct jack so that we are ready to install it on the arm and into the chamber tomorrow.

Vent:
Plan to put HAM6 doors on mid next week

PSL: 
Tweaks and alignment continue. 
Will be adding the shutter shortly. 
Commissioning wants the beam on Wednesday 

VAC:
NO ACCESS TO HAM6 until Wednesday
Leak checking at various points across the site 
Leak at End-Y
CS vertex is pumping down
Viewport work at HAM6
MY CP4 bake out continues

CDS/EE:
Working on End-Y timing system
Working on End-Y AOS laser interlock

PEM:
Moving a weather station to a Beckhoff system

70W install is ongoing.

I have moved a subset of guardian nodes to the new configuration on h1guardian1.  This is to try to catch more of the segfaults we were seeing during the last upgrade attempt, that we have not been able to reproduce in testing.

The nodes should function normally on the new system, but given  what we saw before we expect to see segfaults with a mean time to failure of about 100 hours.  I will be baby sitting the nodes on the new setup, and will restart them as soon as they crash.

The nodes that have been moved to the new system are all the SUS and SEI nodes in the input chambers, BS, and the arms.  No nodes from HAM4, HAM5, or HAM6 were moved.  Full list of nodes now running on h1guardian1:

jameson.rollins@opsws12:~ 0$ ssh guardian@h1guardian1 list
HPI_BS
HPI_ETMX
HPI_ETMY
HPI_HAM1
HPI_HAM2
HPI_HAM3
HPI_ITMX
HPI_ITMY
ISI_BS_ST1
ISI_BS_ST1_BLND
ISI_BS_ST1_SC
ISI_BS_ST2
ISI_BS_ST2_BLND
ISI_BS_ST2_SC
ISI_ETMX_ST1
ISI_ETMX_ST1_BLND
ISI_ETMX_ST1_SC
ISI_ETMX_ST2
ISI_ETMX_ST2_BLND
ISI_ETMX_ST2_SC
ISI_ETMY_ST1
ISI_ETMY_ST1_BLND
ISI_ETMY_ST1_SC
ISI_ETMY_ST2
ISI_ETMY_ST2_BLND
ISI_ETMY_ST2_SC
ISI_HAM2
ISI_HAM2_SC
ISI_HAM3
ISI_HAM3_SC
ISI_ITMX_ST1
ISI_ITMX_ST1_BLND
ISI_ITMX_ST1_SC
ISI_ITMX_ST2
ISI_ITMX_ST2_BLND
ISI_ITMX_ST2_SC
ISI_ITMY_ST1
ISI_ITMY_ST1_BLND
ISI_ITMY_ST1_SC
ISI_ITMY_ST2
ISI_ITMY_ST2_BLND
ISI_ITMY_ST2_SC
SEI_BS
SEI_ETMX
SEI_ETMY
SEI_HAM2
SEI_HAM3
SEI_ITMX
SEI_ITMY
SUS_BS
SUS_ETMX
SUS_ETMY
SUS_IM1
SUS_IM2
SUS_IM3
SUS_IM4
SUS_ITMX
SUS_ITMY
SUS_MC1
SUS_MC2
SUS_MC3
SUS_PR2
SUS_PR3
SUS_PRM
SUS_RM1
SUS_RM2
SUS_TMSX
SUS_TMSY
jameson.rollins@opsws12:~ 0$

guardctrl for nodes on h1guardian1

NOTE: Until the new system has been put fully into production, "guardctrl" interaction with these nodes on h1guardian1 is a bit different.  To start/stop the nodes, or get status or view the logs, you will need to send the appropriate guardctrl command to guardian@h1guardian1 over ssh, e.g.:

jameson.rollins@opsws12:~ 0$ ssh guardian@h1guardian1 status SUS_BS
● guardian@SUS_BS.service - Advanced LIGO Guardian service: SUS_BS
   Loaded: loaded (/usr/lib/systemd/user/guardian@.service; enabled; vendor preset: enabled)
  Drop-In: /home/guardian/.config/systemd/user/guardian@.service.d
           └─timeout.conf
   Active: active (running) since Sun 2018-04-08 14:48:47 PDT; 1h 53min ago
 Main PID: 24724 (guardian SUS_BS)
   CGroup: /user.slice/user-1010.slice/user@1010.service/guardian.slice/guardian@SUS_BS.service
           ├─24724 guardian SUS_BS /opt/rtcds/userapps/release/sus/common/guardian/SUS_BS.py
           └─24745 guardian-worker SUS_BS /opt/rtcds/userapps/release/sus/common/guardian/SUS_BS.py

Apr 08 14:48:50 h1guardian1 guardian[24724]: SUS_BS executing state: ALIGNED (100)
Apr 08 14:48:50 h1guardian1 guardian[24724]: SUS_BS [ALIGNED.enter]
Apr 08 16:01:45 h1guardian1 guardian[24724]: SUS_BS REQUEST: ALIGNED
Apr 08 16:01:45 h1guardian1 guardian[24724]: SUS_BS calculating path: ALIGNED->ALIGNED
Apr 08 16:01:45 h1guardian1 guardian[24724]: SUS_BS same state request redirect
Apr 08 16:01:45 h1guardian1 guardian[24724]: SUS_BS REDIRECT requested, timeout in 1.000 seconds
Apr 08 16:01:45 h1guardian1 guardian[24724]: SUS_BS REDIRECT caught
Apr 08 16:01:45 h1guardian1 guardian[24724]: SUS_BS [ALIGNED.redirect]
Apr 08 16:01:45 h1guardian1 guardian[24724]: SUS_BS executing state: ALIGNED (100)
Apr 08 16:01:45 h1guardian1 guardian[24724]: SUS_BS [ALIGNED.enter]
jameson.rollins@opsws12:~ 0$

Problems encountered during move

A couple of the SEI systems did not come back up to the same states they were in before the move.  This caused a trip on ETMY HPI, and ETMX ISI_ST1.  I eventually recovered everything back to the states they were in at the beginning of the day.

The main problem I've been having is with the ISI_*_SC nodes.  They all are supposed to be in the SC_OFF state, but a couple of the nodes are cycling between TURNING_OFF_SC and SC_OFF.   For instance, ISI_ITMY_ST2_SC is showing the following:

2018-04-09_00:00:39.728236Z ISI_ITMY_ST2_SC new target: SC_OFF
2018-04-09_00:00:39.729272Z ISI_ITMY_ST2_SC executing state: TURNING_OFF_SC (-14)
2018-04-09_00:00:39.729667Z ISI_ITMY_ST2_SC [TURNING_OFF_SC.enter]
2018-04-09_00:00:39.730468Z ISI_ITMY_ST2_SC [TURNING_OFF_SC.main] timer['ramping gains'] = 5
2018-04-09_00:00:39.790070Z ISI_ITMY_ST2_SC [TURNING_OFF_SC.run] USERMSG 0: Waiting for gains to ramp
2018-04-09_00:00:44.730863Z ISI_ITMY_ST2_SC [TURNING_OFF_SC.run] timer['ramping gains'] done
2018-04-09_00:00:44.863962Z ISI_ITMY_ST2_SC EDGE: TURNING_OFF_SC->SC_OFF
2018-04-09_00:00:44.864457Z ISI_ITMY_ST2_SC calculating path: SC_OFF->SC_OFF
2018-04-09_00:00:44.865347Z ISI_ITMY_ST2_SC executing state: SC_OFF (10)
2018-04-09_00:00:44.865730Z ISI_ITMY_ST2_SC [SC_OFF.enter]
2018-04-09_00:00:44.866689Z ISI_ITMY_ST2_SC [SC_OFF.main] SENSCOR_Y_IIRHP FMs:[4] is not in the correct configuration
2018-04-09_00:00:44.866988Z ISI_ITMY_ST2_SC [SC_OFF.main] USERMSG 0: SENSCOR_Y_IIRHP FMs:[4] is not in the correct configuration
2018-04-09_00:00:44.927099Z ISI_ITMY_ST2_SC JUMP target: TURNING_OFF_SC
2018-04-09_00:00:44.927619Z ISI_ITMY_ST2_SC [SC_OFF.exit]
2018-04-09_00:00:44.989053Z ISI_ITMY_ST2_SC JUMP: SC_OFF->TURNING_OFF_SC
2018-04-09_00:00:44.989577Z ISI_ITMY_ST2_SC calculating path: TURNING_OFF_SC->SC_OFF
2018-04-09_00:00:44.989968Z ISI_ITMY_ST2_SC new target: SC_OFF
2018-04-09_00:00:44.991117Z ISI_ITMY_ST2_SC executing state: TURNING_OFF_SC (-14)
2018-04-09_00:00:44.991513Z ISI_ITMY_ST2_SC [TURNING_OFF_SC.enter]
2018-04-09_00:00:44.993546Z ISI_ITMY_ST2_SC [TURNING_OFF_SC.main] timer['ramping gains'] = 5
2018-04-09_00:00:45.053773Z ISI_ITMY_ST2_SC [TURNING_OFF_SC.run] USERMSG 0: Waiting for gains to ramp

Note that the problem seems to be that it's failing a check for the SENSCOR filter banks being in the correct state once SC_OFF has been achieved.  Here are the nodes that are having problems, and the messages they're throwing:

ISI_HAM2_SC     [SC_OFF.main] SENSCOR_GND_STS_Y_FIR FMs:[1] is not in the correct configuration
ISI_HAM3_SC     [SC_OFF.main] SENSCOR_GND_STS_Y_FIR FMs:[1] is not in the correct configuration
ISI_BS_ST2_SC   [SC_OFF.main] SENSCOR_Y_IIRHP FMs:[4] is not in the correct configuration
ISI_BS_ST1_SC   [SC_OFF.main] SENSCOR_GND_STS_Y_WNR FMs:[6] is not in the correct configuration
ISI_ITMX_ST2_SC [SC_OFF.main] SENSCOR_Y_IIRHP FMs:[4] is not in the correct configuration
ISI_ITMY_ST2_SC [SC_OFF.main] SENSCOR_Y_IIRHP FMs:[4] is not in the correct configuration
ISI_ETMY_ST1_SC [SC_OFF.main] SENSCOR_GND_STS_Y_WNR FMs:[6] is not in the correct configuration
ISI_ETMY_ST1_SC [SC_OFF.main] SENSCOR_GND_STS_Y_WNR FMs:[6] is not in the correct configuration

I've tried to track down where exactly the problem is coming from, but haven't been able to figure it out yet.  It looks like the expected configuration just does not match with how they're currently set.  I will need to consult with the SEI folks tomorrow to sort this out.  In the mean time, I'm leaving all of the above nodes paused.

The "sei_config" guardian nodes (ISI_*_{BLND,SC}) were showing import errors having to do with not finding the module "SEI_CONFIG".  It looks like this module was renamed to "sei_config" on March 26, but the nodes that import it were not updated to import the module under the new name.  I've updated them appropriately, and committed to the SVN:

jameson.rollins@opsws12:/opt/rtcds/userapps/release/isi/h1/guardian 0$ svn status
M       ISI_BS_ST1_BLND.py
M       ISI_BS_ST1_SC.py
M       ISI_BS_ST2_BLND.py
M       ISI_BS_ST2_SC.py
M       ISI_ETMX_ST1_BLND.py
M       ISI_ETMX_ST1_SC.py
M       ISI_ETMX_ST2_BLND.py
M       ISI_ETMX_ST2_SC.py
M       ISI_ETMY_ST1_BLND.py
M       ISI_ETMY_ST1_SC.py
M       ISI_ETMY_ST2_BLND.py
M       ISI_ETMY_ST2_SC.py
M       ISI_HAM2_SC.py
M       ISI_HAM3_SC.py
M       ISI_HAM4_SC.py
M       ISI_HAM5_SC.py
M       ISI_HAM6_SC.py
M       ISI_ITMX_ST1_BLND.py
M       ISI_ITMX_ST1_SC.py
M       ISI_ITMX_ST2_BLND.py
M       ISI_ITMX_ST2_SC.py
M       ISI_ITMY_ST1_BLND.py
M       ISI_ITMY_ST1_SC.py
M       ISI_ITMY_ST2_BLND.py
M       ISI_ITMY_ST2_SC.py
jameson.rollins@opsws12:/opt/rtcds/userapps/release/isi/h1/guardian 0$ svn commit
Sending        ISI_BS_ST1_BLND.py
Sending        ISI_BS_ST1_SC.py
Sending        ISI_BS_ST2_BLND.py
Sending        ISI_BS_ST2_SC.py
Sending        ISI_ETMX_ST1_BLND.py
Sending        ISI_ETMX_ST1_SC.py
Sending        ISI_ETMX_ST2_BLND.py
Sending        ISI_ETMX_ST2_SC.py
Sending        ISI_ETMY_ST1_BLND.py
Sending        ISI_ETMY_ST1_SC.py
Sending        ISI_ETMY_ST2_BLND.py
Sending        ISI_ETMY_ST2_SC.py
Sending        ISI_HAM2_SC.py
Sending        ISI_HAM3_SC.py
Sending        ISI_HAM4_SC.py
Sending        ISI_HAM5_SC.py
Sending        ISI_HAM6_SC.py
Sending        ISI_ITMX_ST1_BLND.py
Sending        ISI_ITMX_ST1_SC.py
Sending        ISI_ITMX_ST2_BLND.py
Sending        ISI_ITMX_ST2_SC.py
Sending        ISI_ITMY_ST1_BLND.py
Sending        ISI_ITMY_ST1_SC.py
Sending        ISI_ITMY_ST2_BLND.py
Sending        ISI_ITMY_ST2_SC.py
Transmitting file data .........................
Committed revision 17121.
jameson.rollins@opsws12:/opt/rtcds/userapps/release/isi/h1/guardian 0$