WP 12998 cont Installing the new VM cluster

Test stand log book entries that document the setup and testing prior to production

• Setting up Proxmox 9 servers for testing
• Reworking the Proxmox Setup
• Failure testing the new proxmox
• Testing migrating a VM between proxmox clusters

Physical layout

The following components are racked together in rack 12.

1. sw-msr-pve0 - switch that connects the nodes to each other and to the core.  It runs the internal cluster traffic, and contains the links to the outside world.
   1. Connected to the core via a lag.  I have spare ports in both the core and the switch so that I can increase the bandwidth if needed
2. pve-node[0,1,2] - the nodes

Basic setup

When I updated the ipmi address in the bios I also did the install of the proxmox nodes.  I follow the basic routine in reworking the proxmox setup for pve-node0.

• Created a zfs raid z1 using /dev/sda & /dev/sdb
• Gave the nics descriptive names:
  • onboard 10g copper port 0 - nic1 - 10.20.0.50 (the web interface)
  • onboard 10g copper port 1 - nic2 - not used
  • solar flare 10g nic port 0 - nicdata0 - will be used as a bonded interface for VM traffic
  • solar flare 10g nic port 1 - nicdata1 - will be used as a bonded interface for VM traffic
  • intel e810 port 0 - niccluster0 - for the cluster corosync network
  • intel e810 port 1 - nicceph0 - for the ceph networks

Repeat this for pve-node1 & 2, incrementing the ip addresses.

Cluster network configuration creation

Configured the niccluster0 interface on each node

• Set to 10.200.0.50
• Set mtu to 900
• Set to autostart
• Added comment "PVE Corosync Network"

Did the same on each node, and a ping test.

Cluster creation

On pve-node0 go to the datacenter / cluster and click on create cluster

• Pick a name, "CDS-VM"
• Select the 10.200.0.50 interface

Then on the other nodes to to datacenter / cluster and click join cluster

• Get the join information from pve-node0 datacenter / cluster / join information, copy it to the clipboard
• Paste the join information into the join window
• Put in pve-node0's root password in the password box
• Select the 10.200.0.5? link

Support subscription activation and updates

Now that the basic pieces are in place and will not need a reinstall, it is time to enable the subscription on each node.

Go to each node to the subscription section and upload the subscription key.

Now run updates against the enterprise repo and reboot each machine.

Configure the ceph network

• Assign 10.201.0.5?/24 to the nicceph0 interface
• Enable autostart
• Set comment "Ceph Network"
• Set mtu 900
• Apply the configuration

Repeat for all nodes

Install Ceph

Go to datacenter / ceph page.  When prompted, select install ceph

• Install squid (Ceph 19.2)
• Use the enterprise repositories
• Do the install
• Set the configuration
  • Select the 10.201... interface
  • Use the public network for the cluster network
  • Set the initial monitor node as pve-node0
  • Leave the replicas and minimum replicas values at default.
• Go to the other nodes
  • Only the install process is needed, not the config

Clean up disks

We had used these machines for other testing in the test stand, we will go through and clean up the disks prior to use.

• remove partitions from the disks w/ fdisk (as needed)
• then clean up the disk with "ceph-volume lvm zap /dev/sd[X] --destroy"

This was done on /dev/sdc and /dev/sdd on the systems

Create the Ceph OSD (Object Storage Deamon)

On each machine go to ceph/OSD

• Create OSD
• Disk /dev/sdc
• DB Disk use OSD Disk
• Encrypt OSD
• WAL Disk use OSD/DB disk

Repeat for /dev/sdd.

After a minute they should show up on the ui.

Add more Ceph monitors

Add pve-node1 and pve-node2 as ceph monitors and managers.

Add a Ceph pool

pve-node0 /ceph / pool

• Create Pool
• name StoragePool

Setting up the VM data links, a bonded bridged network

Go to network / Create / Linux Bond

• Name bond0
• autostart
• slaves nicdata0 nicdata1
• mode LACP (802.3ad)
• Comment VM Data Bond
• mtu 9000
• Apply Configuration

Create a data bridge, network / Create / Linux Bridge

• Name vmbr1
• autostart
• vlan aware
• Bridge ports bond0
• Comment VM Data Bridge
• mtu 9000
• VLAN IDs 2 3 20 21 22 98 99 101 105 106

Do this on each of the nodes.

Notes

When creating VMs we want to connect the network to vmbr1 and specify the vlan tag that should be used as this setup gives us access to more than one vlan.