Summary of CDS maintenance model and DAQ restarts. Upgraded DAQ to RCG2.9.4. SEI new models. Upgraded Mid Station PEM to RGG2.9.4. DAQ restart to support upgrades.

* = unexpected restart

model restarts logged for Tue 07/Jul/2015
2015_07_07 00:30 h1fw1*
2015_07_07 09:49 h1nds0
2015_07_07 10:37 h1hpiham1
2015_07_07 10:42 h1hpiham2
2015_07_07 10:42 h1isiham2
2015_07_07 10:48 h1hpiham3
2015_07_07 10:48 h1isiham3
2015_07_07 10:56 h1hpiham4
2015_07_07 10:57 h1isiham4
2015_07_07 11:08 h1hpiham5
2015_07_07 11:08 h1isiham5
2015_07_07 11:13 h1isiham6
2015_07_07 11:15 h1hpiham6
2015_07_07 11:25 h1hpiitmy
2015_07_07 11:27 h1hpibs
2015_07_07 11:27 h1isiitmy
2015_07_07 11:29 h1isibs
2015_07_07 11:33 h1hpiitmx
2015_07_07 11:33 h1isiitmx
2015_07_07 11:38 h1hpietmx
2015_07_07 11:39 h1isietmx
2015_07_07 11:43 h1hpietmy
2015_07_07 11:43 h1isietmy
2015_07_07 11:52 h1broadcast0
2015_07_07 11:52 h1dc0
2015_07_07 11:52 h1fw0
2015_07_07 11:52 h1fw1
2015_07_07 11:52 h1nds0
2015_07_07 11:52 h1nds1
2015_07_07 12:11 h1fw0*
2015_07_07 12:31 h1fw0*
2015_07_07 13:06 h1ioppemmy
2015_07_07 13:06 h1pemmy
2015_07_07 13:07 h1ioppemmy
2015_07_07 13:08 h1ioppemmy
2015_07_07 13:09 h1ioppemmy
2015_07_07 13:10 h1ioppemmy
2015_07_07 13:11 h1ioppemmy
2015_07_07 13:12 h1pemmy
2015_07_07 15:54 h1ioppemmx
2015_07_07 15:54 h1pemmx

[Stefan, Jenne]

Since the Yarm gate valves were opened this afternoon (but the Xarm is still closed), we aligned the Yarm to the green beam, and then aligned the input IR pointing to the Yarm.  (We assumed that the BS pointing was fine from yesterday.)  

* We used the ITMY baffle PDs to set the TMS pointing for the green input, then used the green WFS to align the Yarm.  We touched PR3 a bit to keep the green transmission at the corner on its camera.  

* We used PR2 and IM4 to align the IR beam to the arm.

* We then aligned PRX, followed by SRY.  

After that, we started on the SRY work that I've already reported on.

[Stefan, Jenne]

We were informed that the SRY lock acquisition step of initial alignment was flaky, and could get into a state of ringing up the SR optics.  

The solution was two-fold:
(1) We increase the input power to 10W to improve the SNR on ASAIR_A_RF45_Q.  (After the ASC offload step, the power is returned to the original 2W).
(2) We utilize the front end fast triggering, for both the output and the FM4 integrator.  

For item (1), we did a little bit of finagling in the guardian script to ensure that the value of H1:PSL-POWER_SCALE_OFFSET matches H1:IMC-PWR_IN_OUTMON.  This is required so that we don't lose lock (due to de-triggering) when we are changing the power levels.  Since we're using H1:LSC-ASAIR_LF_NORM_OUT for triggering in item 2, if the value in H1:PSL-POWER_SCALE_OFFSET (which does the "norm" in H1:LSC-ASAIR_LF_NORM) does not match reality (as approximated by H1:IMC-PWR_IN_OUTMON), the loop will often de-trigger even if it shouldn't have, which is why we need those values to match.

For item (2), we changed the SET_SRY guardian to leave the SRCL input always on, and use H1:LSC-ASAIR_LF_NORM for triggering.  We found that values of 50 cts up, and 30 cnts down were good thresholds.  Since only the output of the LSC filters are triggered, this means that we cannot leave the FM4 integrator on unless the loop is engaged.   We use the filter module triggering (which was already triggering FM1, a +6dB gain) to also trigger the FM4 integrator.  Since the integrator's oomph is needed to acquire the lock, we use a "delay" in the filter module of 0 seconds (it used to be 0.2 seconds).  

Nic, Evan

Summary

The frequency noise in CARM is limited in part by the dark noise of REFLAIR9I between 30 Hz and 1 kHz.

Details

First, an explanation of the plot traces:

1. The first trace gives the out-of-loop CARM noise in full lock as sensed by REFL9I, taken on 2015-06-14 06:16:10.
2. The second trace is the dark noise of REFL9I, taken a few days later when the IMC was down.
3. The third trace shows the dark noise of the in-loop sensor (REFLAIR9I), taken today (out of lock, REFL shutter closed). Notes on this:
   • We had to temporarily increase the whitening gain by 30 dB in order to overcome the ADC noise.
   • We have also scaled this noise by a constant factor in order to give the equivalent amount of noise that would be seen on the out-of-loop sensor. [The total scale factor is 10^(-30/20) / 0.0525 ≈ 0.60­, where 0.0525 ct/ct was the previously measured conversion factor from REFL9I to REFLAIR9I in full lock on 2015-06-14.]
   • The detailed shape of the CARM loop is not taken into account (see explanation below about our assumptions for CARM). So above a few kilohertz, where the CARM gain is only a factor of a few, this in-loop dark noise is probably not impressed very strongly onto the OOL sensor.

In the case that the CARM loop is sensing-noise limited, the dark noise (and the in-lock shot noise) of REFLAIR9I should appear on REFL9I. The CARM ugf is about 15 to 20 kHz, and is >30 dB below 1 kHz. Additionally, the white(ish) noise in REFL9I from 30 Hz to 1 kHz strongly suggests that we are seeing some kind of sensing noise. So we believe that the sensing noise from REFLAIR9I is indeed being impressed onto the CARM loop, and is the dominant contributor to the OOL noise between 30 Hz and 1 kHz.

We would like to additionally measure the shot noise as seen in REFLAIR9I (with no lock), to see how large it is relative to the REFLAIR9I dark noise. Either way, it is probably advantageous to switch control of CARM to REFL9I.

around 15:37PDT I noticed my CDS overview MEDM was not connecting to the models running on h1seih23. The control room TV MEDM screen was not showing this problem, but new MEDM connections were not being established. I was also unable to ping or ssh onto h1seih23.

h1seih23's console is reporting many errors (several per second) of the type:

nf_conntrack: table full, dropping packet.

and sometimes

net_ratelimit: 66 callbacks suppressed

a quick goggle search shows that the internal IP connection table has reached its limit.

Over time new MEDM connections are actually being established. Guardian is not reporting any issues with its connections.

So we are thinking that new channel access connections are flakey, but established ones (like guardian) will continue to work. The IFO team would like to delay a restart of h1seih23 until tomorrow unless it becomes a major issue to IFO locking.

J. Kissel, L. Prokhorov, T. Shaffer

Another episode in the saga of the High Voltage ESD Drivers. Though we've not been able to identify why, the high-voltage for both end station's ESD High Voltage Drivers were found OFF. We've queried the usual suspects, and those who were around, and no one has any idea why they might have been *turned* off, so we suspect that they had tripped off from all of the vacuum and electronics work at the end-stations today (the best aLOGs for which are the sparse ops log, LHO aLOG 19462, and VE logs 19472, and 19471).

The ETMY Driver remains railed negative. We've tried all of the possible button mashing and cable plugging and unplugging we've either done before or heard has been tried before, with no success. We request the assistance of the CDS team in the morning.

---------

As you know, we're still in the odd state that the ETMY high-voltage driver is controlled by the new(ish) low-voltage driver, and the ETMX high-voltage driver is controlled via a temporary Beckhoff setup. Therefore, the start-up procedure for both is different.

ETMX
- Flip ON the high-voltage supplies' power switch in the racks near the entrance of the building (you'll hear a ~few second loud continuous beep, wait for the initialization procedure to complete)
- Hit black "V SET" button, followed by entering in 430 V (i.e. hit 4, 3, 0, then enter)
- Hit black "I SET" button, followed by entering in 80 mA (i.e. hit 8, 0, then enter)
- Hit red "OUTPUT ON/OFF" button
You should see the volts go to ~430 V, and the current go to ~3 mA

Now, head out to the driver in the XVEA. You should see both the 430 V and the (low voltage, I don't remember what it is 12 or 15 or something) lights on the front panel showing bright green. You can either turn on the driver manually by hitting the red "start" button, or using the temporary remote system. You'll know the driver is ON when the light above that button goes from OFF to bright RED.
To use the temporary remote system, via a terminal on a work station, toggle one of the remote switches:
caput H1:ISC-EXTRA_X_BO_4 0
caput H1:ISC-EXTRA_X_BO_4 1
then turn on the second,
caput H1:ISC-EXTRA_X_BO_3 1
Once you see all of the lights on on the front panel of the high voltage driver, make sure to confirm that you see a change in the readbacks when turning ON and OFF requested signals (i.e. turn the BIAS "DC" ON and OFF, and turn an OFFSET in each of the quadrants UL, LL, UR, and LR ON and OFF).

ETMY
(This is the same as ETMX)
- Flip ON the high-voltage supplies' power switch in the racks near the entrance of the building (you'll hear a ~few second loud continuous beep, wait for the initialization procedure to complete)
- Hit black "V SET" button, followed by entering in 430 V (i.e. hit 4, 3, 0, then enter)
- Hit black "I SET" button, followed by entering in 80 mA (i.e. hit 8, 0, then enter)
- Hit red "OUTPUT ON/OFF" button
You should see the volts go to ~430 V, and the current go to ~3 mA

(This is different than ETMX)
- Head out to the YVEA (so you can see the front panel of the ESD driver, because these first few steps rarely work), and open the BIO screen of the ETMY SUS.
- Hit the big beige RESET button on the bottom right corner. This *may* toggle the High Voltage Driver on and off, and you *may* regain functionality but likely not.
- Several times now, the driver has been found to be "railed negative" in which he primary symptom is that the readbacks for every channel show about -15700 [ct], and the channels are unresponsive to change in requested drive signal.
Here's what has been tried in the past (ranging from "that's probably it" to "there's no reason hat would have worked anyways, but we're grasping at straws"), and so what I've tried this evening:
- Manually turning ON and OFF the driver with the START button on the front, with the REMOTE RESET cable and INPUT cable still plugged in.
- Unplugging the REMOTE RESET cable and manually turning ON and OFF the driver with the START button (INPUT cable still plugged in)
- Plugging the REMOTE RESET cable back in, and trying both to remotely and manually turn ON and OFF the driver
- Unplugging REMOTE RESET and INPUT cables, manually turning ON and OFF the driver
- Toggling all of the low-voltage driver binary I/O switches (i.e. the HI/LO VOLTAGE and HI VOLT DISCONNECT switches)
None of which were successful this time around.

Elli, Nutsinee

Today we moved theITMY HWS periscope to fix the return beam clipping issue (it was clipped at the lens between the two periscope mirros). I lost the green beam during the alignment (ITMY misaligned) so I left it there for now. The return beam seems to be clipping at the edge of the first iris (the green came right through the center though). *Should* be a quick fix next time (don't I always hope that?).

Scott L. Ed P. Rodney H.

7/6/15
Beginning this week we increased our crew size by one man. Rodney Haux previously worked for LIGO and has graciously agreed to return and assist with the beam tube cleaning.

Monday was spent relocating all equipment to Y-End where we will begin cleaning and moving toward the corner station. The lights were suspended and fans located for air movement. The crew started vacuuming support tubes and capping them as they were cleaned. 

The 1 ton truck was running extremely rough and it was discovered that the air box was 95% blocked with debris from mice. We were able to remove enough of the material to get the truck down there however, we may need a new air box as this one is very damaged with rodent excrement. I plan on picking up a new air filter today.

7/7/15
Cleaned 59 meters of tube including the bellows in those sections ending 10 meters east of HSW-2-096.

I recently purchased some cooling vest for the crew which seem to be working relatively well. The ice packs only last 3-4 hours at most and then need to be replaced with another set. The triple digit temperatures are taking a toll.    

Chris S. (Joe D.50%)

500 meters of the X-Arm beam tube enclosure have metal strips installed on the upper portions of each joint. We started at the bridge and are working northward.

This has been a slow process in part due to the triple digit heat we have been experiencing.  

While at EX with Jeff K. and Leonid for an ESD issue (alog to come), I restarted the code for the BRS since it crashed during ER7. As expected it was very rung up, so I turned the damper off to let it ring down naturally over the week.

Valved out Turbo pump from HAM6 at 11:45 am, ion pump is mantaining pressure, currently at 4.5 x 10^-6 torr.

Turbo pump + cart still running.

The 18 bit DAC card has been removed from the h1pemmx I/O chassis to update it's firmware.  The h1ioppemmx and h1pemmx models were modified to not include the 18 bit DAC.

All Times in UTC:

14:45 Jeff B out to LVEA to move cleanrooms

15:15 Andres out to LVEA to join Jeff

15:26 Karn and Christina to EY

15:36 Joe into LVEA to check on orklift batteries, Fire Extinguishers, eye-wash stations...etc

15:45 Hugh into the Bier Garden

15:47 Kyle to EY

15:52 Bubba goin to MX and EX

15:56 Karen and Christina leaving EY

16:07 Hugh out of LVEA and heading to EXto replace T240 with STS2.  Back at 17:59.

16:16 Fil out to Ends to do P-Cal power (both) and magnetometer power for Jordan at EY

16:26 Betsy and Travis out to the LVEA to check for post-vent fodder.

16:39 Andres out

16:46 Jeff B out

17:02  Leaving EX cristina & karen 

17:25  Restarting all SEI Models (Kissel)

17:30  Praxair called in to say a truck will be on-site in 15min (for "379" which is MX's CP6) & he mentioned that another truck will be heading to "380" (which is EX's CP8).

17:40  Noticed Beckhoff at EX went down, need to figure out how to remote login and restart.

17:53  pemmx rcg upgrade & pulling a card as well (Dave)

18:05  Continuing Vent clean up in the Cleaning Room (Betsy)

18:06  Heading into LVEA  for hose (Joe)

18:10  TCS Chassis Install at EX & then to EY (Sudarshan, Vinny, & Jordan)

18:20  Gerardo said to keep an eye on EY pressure---any alarm for pressure increase 

18:22  Gerardo on top of HAM6 (for how long?)

18:27  Kyle opening GV5 & GV7

18:38 Sudarshan turning on P-Cal LASER at EX

18:48 Fill reported back from Ends

18:49 Joe into LVEA 

18:50 DAQ restart

19:13 Sheila heading to EX to restart slow controls computer

19:28 Kyle out of LVEA. GV 5,7 are now opened.

19:35 Sheila back from EX

19:48 Fill called from EX. Continuing work for PEM

19:51 Sudarshan back from EX

20:03 Dave re-boot  somthing at EY

20:30 Kyle, John & Gerardo to EY to valve in NEG and valve out Turbo

21:15 John called from EY. Arm is open and Gauges are reading inconsistently. (2orders of magnitude)

21:19 Jordan and Vinny to EY WP 5336

21:20 Katie to EY, Mangetomoeter calibration.

21:21 Fil called from EY. While doing cabling, accidentally disconnected  12V oscillator source.

22:11 Nutsinee out to HAM4 to adjust alignment on HWS table

22:44 Dave and Jim at MX. Took PEM down to remove 18bit DAC.

22:47 Ellie out o HWS table to assist Nutsi

22:48 Jeff K, TJ and Leo to EX to debug ESD issue.

Must have inadvertently unplugged it yesterday while working around this rack

Kyle, Gerardo, John

The NEG pump has been valved into the END Y volume and all appears well.

The plot shows 2 hours of the BSC chamber pressure. Initially the main turbo was pumping the station along with the LN2 pump and open to the tube. The LN2 pump had been opened to the BSC earlier in the day and we let the pressure settle over lunch.

With the turbo still connected you can see the pressure fall as we open the NEG to the volume. Next the turbo was valved out and the pressure returns to near the starting value.

So far so good!

J. Kissel
WP: 5335
ECRs: E1500216, E1500245
II: 1046, 1059
Design aLOG: SEI aLOG 721

Through various levels of changes depending on the whether it was the BSC-ISI, HAM-ISI, or HEPI main library part (see SEI aLOG 721), there are now the following channels in the commissioning frames stored at 256 [Hz]:
HEPIs:
${IFO}:HPI-${CHAMBER}_SCSUM_${SENSOR}_IN_${DOF}_DQ
with ${SENSOR} = either IPS or STS

BSCs:
${IFO}:ISI-${CHAMBER}_ST1_SCSUM_${SENSOR}_${DOF}_IN_DQ
with ${SENSOR} = either L4C, CPS, or STS

HAMs:
${IFO}:ISI-${CHAMBER}_SCSUM_${SENSOR}_${DOF}_IN_DQ
with ${SENSOR} = either CPS or STS
sadly, I only realize after documenting what I've done that the order of "IN" and "${DOF}" is different for HEPI than it is for the ISIs. At this point it's not worth the IFO down time to make the bug fix. Sorry :-(.

This closes out ECR E1500216 and Integration Issue 1046 for Hanford, so I'll reassign it to Monsieur Pele. For Livingston to receive this upgrade, one must only update the following library parts, recompile, install, restart, and restore: 
${userapps}/release/isi/common/models/isihammaster.mdl
${userapps}/release/isi/common/models/isi2stagemaster.mdl
${userapps}/release/hpi/common/models/hepitemplate.mdl

I attach screen shots of the latest version of the relevant blocks in the library parts that I've modified.

Also a bonus change that had been pending until today when I re-found it: the HAM-ISI models had not had their SENSCOR BLRMS channels removed, which I thought have been done when Arnaud was here, as per ECR E1500245 and Integration Issue 1059, but it turns out he only modified the BSC-ISIs (see LHO aLOG 19187). HEPI models never had this calculation, so no need to change there.

For the record, I made sure 
- all SDF differences were understood and captured on all SEI systems before restarting
- all ISC guardians were in their DOWN state, and the IMC_LOCK guardian was OFFLINE before restarting.
- all chambers were brought to OFFLINE via their chamber GUARDIANs
- all chambers were able to be re-isolated after the restart of their front-end process via there chamber GUARDIANs
- add SDF systems for the new platforms have no channels "NOT FOUND" or "NOT INIT" -- because I removed the BLRMS of the STS channels from the BSCs, they required the re-saving and re-loading the SDF table, i.e. an OVERWRITE of the EPICS DB TO FILE, where the file is the safe.snap.
- The IMC recovered after the restart of the HAM2 and HAM3 chambers

STS2-B (ITMY) was sn109827.  STS2-A (HAM2-actually located in BierGarten on temp cable) was sn89950.  109827 has moved to endX to replace the T240 that has been there for some time.  This T240 will relocate to the ETF lab.  There is no instrument connected to the STS2-A chassis and sn89950 is now connected to the STS2-B chassis.

All temporary cable to accommodate the T240 as well as the T240 chassis remain in place.  The T240 chassis is powered off.