1040 hrs. local 2.7 x 10-6 torr @ turbo inlet -> Foreline @ 4.1 x 10-3 torr Domestic water storage tank @ 43", raw water storage tank > 60" BT roads still not passable -> tumbleweed crew working towards Y-mid 1100 hrs. local -> Leaving site now
We have completed the removal of the two infrastructure racks in the H2 Building, with their contents now relocated in the MSR. All items went into the right side two MSR racks with the exception of:
the Qlogic switch which was moved to the DTS DAQ rack (previously the H2 DAQ rack)
the GC switch which was relocated into the DTS computer rack
the conlog machine which was relocated to the H1 DMT rack, this will become an H1 FE computer
The cdswiki machine was finally made to boot via a manual grub startup. This machine looks like it lost one of its LVM2 drives. Since it will be replaced with the new Debian box next week, we will hope it holds together until then.
The backup file server died again, we suspect a bad boot drive. It is scheduled for a reinstall on the second boot drive soon.
Work continues on the rack relocation from the H2 Building to the MSR. Today we are fixing the cdswiki problems, moving the conlog machine into the H1 DMT rack, getting all services restarted and gutting the old H2 racks. Access to the DTS is down while we reconfigure the GC network in the H2 Building.
Attached is a picture of the two new MSR racks. They are the computer racks next the the two UPS racks.
WP 3615, Dave B and Jim B
We shutdown all the servers in the H2 Building, starting at 9am today. All machines were re-racked in the MSR. The original network switches were moved along with the computers. All were connected to the Raritan Paragon KVM system.
We powered the machines on in order of importance: file and boot servers, admin servers, ifo servers, web and login servers.
All came back except for cdswiki, which appears to have a disk error. We will continue to work on this tomorrow, but for tonight there is no CDS web services: medm snapshots, web pages, work permit, cds wiki. If we cannot get cdswiki working tomorrow, we will put a temporary web server in place to serve the medm screen shots.
Staff with login account should be able to use their accounts now.
We check that the DAQ is acquiring the Vacuum controls and FMCS signals. All front ends were unaffected by this move.
All Ubuntu workstations in the control room needed a power cycle to get operational, we suspect DHCP issues.
I'll post a more detailed alog later.
1300 hrs. local 3.3 x 10-6 torr @ turbo inlet -> foreline @ 4.4 x 10-3 torr
Tumbleweed crew are working their way to Y-mid
The h1boot computer had a kernel panic Dec. 25 in the morning. Rebooted at 11:19 PST Dec. 26.
logs suggest that h1boot became unresponsive at 01:55am Christmas Morning.
HAM3 pressures as of Dec 24, 11:00 am
Cold Cathode - 5.6 e-6
Pirani - 5.6 e-3
Not a typo - both gauges read 5.6 but 3 decades apart.
Tumbleweed baling continues.
The buildings are relatively clear now. One lane has been opened to XMID, the crew expects this lane to be done to XEND at the end of today.
Wednesday they will start on the YARM - we need access for LN2 and work planned for the YEND station in early January. They expect to work
through the holidays except for Dec 25.
1345 hrs. local -> Leaving site now
Expect to be here 30 mins.
1030 hrs. local -> HAM2-HAM3 @ 1.2 x 10-1 torr -> Relatively fast rough-down (scroll pump) due to 2" hose connection to 2.5" Vent/Purge valve as opposed to pumping through deenergized turbo pump (observed ultimate pressure of this particular scroll pump is 3 x 10-3 torr). Tumbleweed crew bailing near Y-arm penetration. 1115 hrs local -> Leaving site now.
The CDS backup disk server cdsfs1 crashed late last night and would not reboot today. We could not recover its boot disk, so we reinstalled Ubuntu 10.04LTS. The backup raid contents were preserved, so no need to rsync everything from /ligo and h1boot again. We are now re-installing the backup scripts and linking the tape backup system.
cdsfs1 had become very flakey lately, hopefully the rebuild will fix this.
Models running on h1seih16 would not run more than a few minutes. The IOP model would die with complaints of ADC channel hopping, the GDS TP screen showed ADC1 as the culprit. In an effort to find the problem, Richard McCarthy 1.) reseated all the cards in the I/O chassis, then 2.) moved the ADC card for ADC1 to a different slot so it would no longer be ADC1, 3.) Completely rearranged the cards to match the h1seih23 system (minus 2 ADC cards). In all cases, the models would stop after a period of time (up to 25 minutes) with the same complaint of ADC channel hopping and the indication that ADC1 was the problem. David Barker modified the IOP model to only use 1 ADC card and the model was restarted. So far after many hours, the IOP model is still running. All cards were left in the I/O chassis for this test. We need to troubleshoot this to find out if any 2 ADC cards cause channel hopping, or if there is a specific ADC card that is causing the issue, or some other cause.
A quiet day in the CR. The biggest news was the startup of pumping of the volume between HAM's 2 and 3. See Kyle's entry below. Kyle has his eye on the LN2 tank for CP4 which reads 31.3 today. Tumbleweeds nixed last week's scheduled fill of this tank. Dave Barker, Jim and Richard swapped ADC cards for HAM1/HAM6 SEI in an attempt to provide readouts for seismic. See Kiwamu's entry below regarding the move of IOT1. The tumbleweed crew has been working all day. Jeff Barlett, Mark Barton, Kyle and Dave Barker have placed work permits for work that will occur over the holiday break.
Will be in occasionally to check on status, record pressure and to switch over to turbo etc. -> Will make aLOG stating my intended duration when I arrive and an entry when I leave site -> Am continuing to pump down Vertex + YBM over the break for the data.
The IOT1 table [1] was transported from the squeezer bay to the place next to HAM2 where the table is supposed to be. The disk-shaped spacers were installed between the enclosure and legs after the table moved in there.
Some pictures will be uploaded shortely.
[1] LHO alog #4985