Marie, Sheila, Kiwamu, Lisa, Stefan

We tried a few things to improve the recycling gain tonight. We  believe that the problem is with something in our PRC.  We can see that the green power transmitted through the X arm stays stable as we power up, we think this means the optics are stable.  When we move the soft offsets to improve the recycling gain we change the X arm alignment to follow something in the vertex.

• We started with dithering IM1+IM3.  The IM1 dither was meant to test is we were clipping into the Faraday, we don't think that is happeneing because we could clearly see our dither in IM4 trans PIT and Yaw, but not in the sum. 
• We tried moving IM3 and PR2 offsets.  We saw taht IM4 trans pit moves as we power up, so we started by trying to correct this by moving IM3. We were repeatably able to improve the reycyling gain from 28 to 29 by moving IM3 by 3000 in pitch counts (we think this is calibrated in urad! Our inital starting point was 58495 on the IM3 pit slider, our best recycling gain was at 55515 urad).  This seems like too big of a move to be the real drift that we are trying to correct, and is much larger than the move needed to correct the position on IM4 trans. 
• We tried moving loop offsets in PRC2 (POPX-> PR3 loop) In pit we went +- 400 counts without seeing any real change, in yaw we saw a repeatable imprvement with a offset of +400 counts. 
• We switched off the loop, moved optics, and broke the lock. 

As TJ noted, since the computer problems earlier we are having trouble relocking the PSL after locklosses, with noise eater, PMC and FSS and ISS problems.  It seems like trying to lock the FSS causes the noise eater to oscillate again after it has been reset. 

TITLE: 06/29 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: None
SHIFT SUMMARY: Commissions working hard. Had one hicup (alog28044) with EY SUS computer crashing, and then the PMC and FSS gve us trouble. After another lockloss the PMC and FSS continued to fight us. The only way we have found to make the PMC lock is to click the autolocker ON and OFF (while waiting between) until it locks on a good mode. Then the FSS may still be oscillating and the PMC HV drifting,  but do the same trick with the FSS autolocker and after fifteen minutes or so you'll have it! But keep an eye on the noise eater.
LOG:

• 00:04 - Jonathan, Carlos done with Wiki/WP done
• 01:18 - Sheila, Hoacun, Marie, Kiwamu to LVEA to realign POPX
• 01:34 - Out.
• 03:23 - Sheila, Lisa to LVEA to realign POPX
• 03:35 - Out

Sheila, Stefan, Lisa, Nutsinee

Had a lockloss that seemed to have been caused by an H1IOPSUSEY computer crash. After we figured that this was the issue and started to start the recovery process by restarting the computer, then clearing all of the watchdogs, and bring alignment sliders back to where they should be. No issues recovering there.

Then we noticed that the front StripTool looked odd and the FSS, ISS, and PMC seemed to be having issues. We called Dave to have him look at all this before he went to bed.

The PMC Locked Light showed that it was locked (green) but the HV was basically zero and the PMC Trans camera image was caught on some terrrible mode.  We managed to get the PMC locked again by turning the autolocking on and off and on and off till eventually it locked on a good mode. BUT after we would think that we had it locked nicely, we would watch as the HV would drift away and lose the lock or lose it immediately as we went to engage the FSS. During all of these locks and unlocks the noise eater would start to oscillate and we would end up having to reset it 3 times before we managed to get everything stable. Then we had to fight the ISS from oscillating as well. In the end we won the battle, but feel very battered and confused.

We seemed to have recoved everything except TrendWriter0 which went down in the chaos sometime after the EY crash.

Kiwamu, Alastair (via email/phone), Nutsinee

WP#5967

Quick Conclusion: No more nasty particulates going into the chiller system. Both AOM and AOM driver were drained but the hoses were left undrained.

Preparation

Since we're done using AOM to align the CO2 beam to the carrier light, it's time to remove them out of the water system so the nasty particulates don't travel into the water system any further. First we went out on the mezzanine to turn off the RF oscillator, then we headed to the RF distribution amplifier. We couldn't find any on/off switch so Richard suggested we just disconnect the cables and replaced it with 50 Ohms terminators. We disconnects TCSX and Y RF driver cables (#3 and #8 -- see first attachment). Then we headed back to the mezzanine and switched off the AOM power supply (which hasn't been tagged out. DO NOT TURN THIS UNIT ON.). The CO2 laser was turned off from the control room at this point. PSL light pipes were shuttered. TCS chillers were left running.

Disconnecting AOMs from water supply

Starting with TCSY table, First I removed the quick connects. Then I removed the hoses from the AOM and the AOM driver. Seeing that the water isn't going to drain itself out, I blew out water using compressed air duster. I didn't have time to drain water out of the hoses to I wrapped them with clean room cloth and left them there (for now, at least until next Tuesday). I also left some dry cloths below the AOM and the driver just in case any more water drip out. I did the same at the quick connects. I repeated the same thing with TCSX table. See 2nd and 3rd attachment for images. There were no more water dripping or seeping out when I closed the tables. I turned the oscillator back on, opened the PSL light pipes, and restored both CO2 lasers at ~ 14:13 PT.

I attached a picture (final attachment) of some nasty particulates I blew off one of the units for your amusement.

earlier today we had a raid issue with h1tw0 which cleared on reboot. This evening it failed again and h1tw0 was continuously restarting its daqd. I have shut this down.

+750ml to TCSY chiller, +100mL to TCSX chiller

Cleaned TCSY chiller filter

We lost lock when the sus EY computer went down, there was a red light by DK.  We followed the instructions here  to restart it, and now the ISI is re isolating. 

Jeff K and myself

Was: SAFE

Now: INIT

A few weeks back we changed the initial request state for the SUS Guardians from ALIGNED to SAFE (alog27627) with the thought that it was too aggressive of a move immediately after a reboot. Today the Guardian machine was rebooted again and the SUSs all went to SAFE, but this had apparently messed up the PCAL team. We decided to just have the nodes run INIT and go back to where we had them previously. Hopefully it actually goes that way.

All of the SUS nodes have been reloaded with the new code that is also committed to the SVN.

Manually set CP5 LLCV to 10% open to lower LN2 level, then adjusted % full set point to 98% and switched to PID mode to maintain overnight.

Cheryl recovered the optic positions and locked the arms on green. I completed an initial alignment and brought the IFO to DC_READOUT_TRANSITION.

Bubba, property inventory, LVEA
Travis, Darkhan and Evan to PCAL X

14:12 UTC Jeff B. and Peter in PSL
14:40 UTC Chis S. to end X to retrieve equipment from receiving area
15:08 UTC Coca Cola delivery truck through gate
15:23 UTC Gerardo to LVEA to retrieve wire
15:23 UTC Jim B. restarting guardian machine
15:23 UTC Karen and Christina to LVEA to clean
15:26 UTC Jeff B. and Peter out of PSL, Peter taking equipment back to anteroom
15:34 UTC John changed setpoint on CP7
Guardian reboot took everything to safe, interfered with PCAL work
15:40 UTC Jeff B. taking ETMX SUS back to aligned
15:48 UTC Bubba and Nicole to end Y and then end X for property management
15:52 UTC Peter done with chiller block investigation and DC blocker install
15:58 UTC Richard swapping IM2 satellite amp, power cycling IM3 and MC1 satellite amps
16:07 UTC Cheryl installing beam blocks on IOT2L
16:07 UTC Joe D. to LVEA to gather flex tubing
16:07 UTC Vinny to LVEA to work on infrasonic microphones
16:09 UTC LN2 delivery truck through gate
16:11 UTC mat delivery truck through gate
16:15 UTC Jeff B. to optics lab to store equipment
16:19 UTC Bubba lifting Styrofoam cooler off tilt meter at end Y to look at serial number
16:23 UTC Jeff B. done
16:27 UTC Cheryl done
16:40 UTC Vinny to EY VEA to work on infrasonic microphones
16:43 UTC Evan and crew done at end X
16:46 UTC Evan and crew to end Y
16:48 UTC retrieval of SR785 from beer garden
16:51 UTC Vern to SR2 by HAM4
16:58 UTC tractor trailer truck through gate to pickup forklifts
17:02 UTC Nutsinee to LVEA
17:05 UTC Richard to LVEA to replace DAC cards in SUSB123 and SUSH2A IO chassis
17:05 UTC Gerardo to LVEA to replace wire for HAM1 vacuum gauge
17:07 UTC Dave B. powering down SUSB123 and SUSH2A frontend computers so Richard can replace DAC cards
17:11 UTC SUSB123 and SUSH2A powered down
17:15 UTC Nutsinee done
17:19 UTC Vinny back from end Y, going to end X
17:22 UTC Dave M. swapping cables on Newtonian Noise seismometers in LVEA
17:26 UTC Richard done replacing DAC cards, Jim B. powering on SUSB123 and SUSH2A frontend computers
17:40 UTC Dave B. restarting ISI ITMY model
17:40 UTC Jim B. restarting models on SUSB123 and SUSH2A frontend computers
17:51 UTC Dave M. done
Satellite amp modifications are done
17:59 UTC Richard to LVEA to reinstall modified whitening chassis
18:00 UTC Vinny back from end X
18:11 UTC Dave M. to LVEA to make measurements of Newtonian Noise array
18:12 UTC Christina and Karen done cleaning at end X, coming back to OSB
18:12 UTC BSC HEPI and IOP models need restarting, not just ISI
18:16 UTC BS seismic HEPI and IOP model restart
18:19 UTC Vinny to LVEA to calibrate microphones
18:19 UTC Nutsinee removing TCSY AOM from water supply
18:25 UTC Kiwamu restarting OMC PI model
18:25 UTC Dale leading tour in CR
18:25 UTC LDAS work starting on nds0
18:28 UTC Travis done
18:28 UTC Richard reinstalling modified AA chassis
18:34 UTC Daniel done Beckhoff work
18:36 UTC Bubba to end X for property management
18:39 UTC Richard done installing modified AA chassis
18:39 UTC Dave restarting OMC PI model
18:42 UTC Vinny done calibrating microphones
18:46 UTC Gerardo done at HAM1
18:49 UTC Filiberto power cycling CPS fanout
18:51 UTC Dave B. restarting PI model
18:52 UTC SEI ITMx (HEPI & ISI) (Dave)
18:52 UTC Kyle to end X and end Y VEAs to take photographs and serial numbers of VAC equipment
18:57 UTC Volker to optics lab to retrieve optics posts
19:02 UTC Dave B. restarting end station HEPI and ISI models
19:27 UTC SUS ETMY PI model
19:31 UTC Jeff K. and Jim W. starting recovery of optics in HAM2
19:38 UTC DAQ restart
19:44 UTC Kyle back, reports end X is laser hazard and end Y is laser safe
19:55 UTC Christina opening OSB receiving door
20:09 UTC Bubba and Ernest to LVEA for property audit
20:15 UTC Filiberto to LVEA to reset cable on HAM4 SR2
20:39 UTC Kiwamu to LVEA to check with Nutsinee
20:57 UTC Nutsinee done, closed tables, reopened light pipe
20:57 UTC Dave B. taking nds1 down
21:01 UTC Starting IFO locking recovery
21:06 UTC Kiwamu to LVEA to realign REFL cam
21:09 UTC Bubba to end stations for property audit
21:15 UTC Nutsinee to TCS racks for CO2 X
21:16 UTC Starting initial alignment
21:19 UTC Carl to end Y to take PI measurements
21:19 UTC Nutsinee back
21:27 UTC Kiwamu done
21:28 UTC Kiwmu to HAM6 to measure DC QPD whitening
21:36 UTC DAQ work complete
21:49 UTC Bubba done property management in VEAs
22:08 UTC Travis to end Y to check illuminator
22:09 UTC Fred taking UW REU students into LVEA
22:55 UTC Initial alignment complete
22:57 UTC Carl done
23:04 UTC Fred and UW REU students out of LVEA

Work done by Carlos and Jonathan

The CDS wiki/web server was upgraded to a new OS, apache release, and newer hardware.  As it was being rebuilt we built the configuration into the CDS configuration management system so that the current state is easily repeatable.

Everything thing appears to have been migrated successfully.  Users should not expect to see changes due to this migration.  However it allows us to upgrade security settings and provides a needed upgrade that will support some experimental remote data access software.

We finally accumulated a lock stretch long enough, with good sensitivity to analyze the interferometer response with the Pcal above 1 kHz. Last night was a lock stretch at 40 Mpc, 40 W input power. For the lock stretch, a good time to start the high frequency analysis is at 05:31:00 06-28-2016 UTC, and end time of 06:32:00 06-28-2016 UTC. The x-end Pcal was running at 1000.3 Hz (different than the previous time we made the injection)

Now that there is enough data accumulated, I moved the Pcal injection to 1501.3 Hz and will wait for the next lock stretch with good enough sensitivity, high power, and long duration.

For reference, the end of O1 also made high frequency, long duration injections. We are repeating these measurements. See alog 24843, and the plan from the end of O1 alog 24802.

Today during Maintenance the HAM2 ISI was not fully isolated but in DAMPED to allow for work in the LVEA.  The IM1-4 optics were in SAFE, so no alignment bias and no damping.  This configuration closely mimicks an ISI / optic trip.

When it was time to restore the HAM2 ISI, and then redamp the IM1-4 optics, I observed the drift in pitch and yaw that I've observed after an ISI trip.

The difference between a typical ISI-IM-IMC recovery, and todays recovery is that we restored the ISI and IMs while the PSL beam was blocked at the shutter between the PSL and HAM1.

The drift in IMs today was no different than the drifts I've observed before.

There was a suggestion that heating from the IMC being restored, when the HAM2 and HAM3 ISIs and optics are restored, might be the source of the IM drift, however since I tracked the drift today and there was no IMC beam, I have to conclude the restoration of the IMC is not the source of the IM drift.

While tracking the drift, I exercised the pitch and yaw alignment biases for each IM by +/-100 counts to see if that would effect the drift.  I was testing the theory that the cause of the alignment shifts that I've tracked since last year might also be causing this recovery drift, but I cannot see that being true, since moving the alignment biases had no effect on the drift.

Here are the amounts each IM changed after being restored:

black: IM DOFs that did not show any significant drift (drift that is less than 1urad)

red: IM DOFs that did show significat drift (drift equal to or greater than 1urad)

Attached plots:

• 25 minutes of drift
• close up of the pitch and yaw alignment biases being exercised, showing the drift of was unchanged

omc_pi model change. WP5957

Kiwamu, Carl, Dave:

h1omcpi was modifed to read in the two new ADC channels. Two new 64kHz channels were added to the commissioning frame, 8 existing 2kHz channels were added to the science frame for permanent archival (H1:OMC-PI_DOWNCONV[1-4]_DEMOD_[I,Q]_OUT_DQ)

h1 SUS PI model changes

Ross, Carl, Dave:

new h1susitmpi,h1susetm[x,y]pi models were installed.

DAC card replacement in SUS B123 and H2A WP5954, WP5953

Richard, Fil, Dave, Jim

h1susb123 and h1sush2a were powered down. The first DAC card in each IO Chassis were replaced. The new DAC cards both pass the autocal. The ADCs were power cycled at the same time.

BSC ISI code change WP5961

Brian, Jim W, Jim, Dave

a new isi2stagemaster.mdl file were installed. All BSC ISI models were rebuilt and restarted. (h1isi[bs, itmx, itmy, etmx, etmy]). Part of this change was to remove the user model's DACKILL part. Since this part registers itelf with the IOP model on startup, the code change required a restart of all the models on h1seib[1,2,3,ex,ey] to resync the IOP with the DAC clients.

We also found that the HEPI watchdog reset button does not work on newer Ubuntu 14 machines. This was tracked to the perl script wd_dackill_reset.pl script requires the CAP5 module, not loaded on newer machines. Perhaps these old PERL scripts should be replaced with newer PYTHON scripts.

DAQ upgrades and new QFS server WP5964

Dan, Carlos  Jim, Dave:

Dan upgraded the QFS servers h1ldasgw0, h1ldasgw1 and h1dmtqfs0 Solaris machines. These were fully patched, and the QFS file sysem they control was file-system-checked.

Dan installed a new QFS server called h1ldasgw2. This is a Sun X2200, with redundent fiber channel ports which directly connect to the two Qlogic switches.

In the new configuration, h1ldasgw0 only exports /ldas-h1-frames to h1fw0 on the private LAN (no longer exports to h1nds0). h1ldasgw1 only exports /cds-h1-frames to h1fw1 on the private LAN (no longer exports to h1nds1). h1ldasgw2 NFS exports both /ldas-h1-frames and /cds-h1-frames to both nds machines in read-only mode.

The hope is that the frame writers will be more stable if their NFS/QFS server is not also serving NDS requests.

h1nds1's RAM was doubled from 24GB to 48GB by the insertion of 3*8GB SIMM cards harvested from x1work.

ASC new code

Jenne, Jim, Dave

new h1asc model was installed. We were surprised that an excitation was started immediately once the model got running. This looks like a script constantly running the excitation of H1:ASC-POP_X_PZT_YAW_EXC

Does anyone know what is doing this?

DAQ Restart

Daniel, Dave:

The DAQ was restarted to support all of the above model changes. Latest Beckhoff INI files were installed.

Tesing Scientific Linux 7.2 on LHO DMT WP5969

Dan:

Dan is upgrading h1dmt2 from SL6.5 to SL7.2 as a test bed for the new OS in preparation for a pre-ER9 upgrade.

cdswiki upgrade WP5956

Jonathan, Carlos:

work is underway to upgrade the cdswiki machine

Terminated and landed a new cable for PT100-A, old cable was 22 AWG, new one is 18 AWG.

As a note, we used a pigtail cable with a HD DB15 connector.

(Richard M, Fil C, Ed M, Daniel S)

ECR E1600192.

Split Whitening Chassis S/N S1101627:

• On the interface chassis remove R7, R8, R9 and R10, R17 and R18 (disconnects in-vac cables and Z-switch).
• On the interface chassis jumper 220 pF capacitors on J3 between pins 1-3, 9-11, 2-4 and 10-12 (adds one pole of high pass at 7.2 kHz).
• On the 2nd whitening board remove C37, C38 and C39 (removes both poles of low pass).

AA Chassis S/N S1102788 & S1202201:

• On channels 15 and 16 remove C1, C6 and C9 (two poles of 10 kHz low pass).
• On channels 15 and 16 remove C7 (one pole of 10 kHz low pass).
• The twin-T notch was left in. It is fairly wide and attenuates ~17 dB at 50 kHz.

The attached plots show the transfer functions of

1. whitening chassis: OMC DCPD_A: same as before
2. whitening chassis: OMC PI DCPD_A: modified
3. whitening chassis: OMC DCPD_B: same as before
4. whitening chassis: OMC PI DCPD_B: modified
5. AA chassis: channel 15
6. AA chassis: channel 16

Nutsinee, Jim, Dave:

The HWS code crashed at 07:50 PDT this morning, Nutsinee tried to restart at 11:46 PDT, but it failed. We found that the 1TB raid is 100% full (has data back to December 2014). We are contacting Aidan to see how to proceed.

BTW: the /data file system on h1hwsmsr is NFS mounted at the end stations, so no HWS camera information is being recorded at the moment.