Per WP 13396 I am upgrading the proxmox vm cluster ceph storage from v19 to v20 before it goes end of support.
This has been done in the test stand, and with other groups in LLO & LHO without any service interruption.
I am following the directions in the proxmox wiki: https://pve.proxmox.com/wiki/Ceph_Squid_to_Tentacle
I am starting with an up to date cluster with pve-manager 9.2.3 and ceph: 19.2.3-pve4 which is above the minimum requirements for this upgrade.
Steps (as noted in the wiki all ceph commands are done on only one box):
1. On each box update the apt sources to use the ceph-tentacle repository (v20) instead of the squid (v19) repositories and run apt-update
2. Set the 'noout' flag on ceph.
ceph osd set noout
3. On each box run
apt full-upgrade
4. On each box, one at a time, restart the monitor daemon, checking health with 'ceph -s' (HEALTH_WARN with a note about noout flag set is ok. Watching the gui summary gives near instant feedback too).
systemctl restart ceph-mon.target
5. verify that Tentacle is in the output of:
ceph mon dump | grep min_mon_release
6. On each box, one at a time, restart the manager daemon (this happened automatcially with the restart of the monitor).
systemctl restart ceph-mgr.target
7. On each box, one at a time, restart the OSD daemons. (check ceph status before proceeding on each box)
systemctl restart ceph-osd.target
8. Upgrade all CephFS MDS daemons - skipped
9. Disallow pre-squid OSDs and enable all new tentacle only functionality
ceph osd require-osd-release tentacle
10. clear the noout flag
ceph osd unset noout
At this point the cpeh is health.