Supported by the National Science Foundation Collaborator: University of Michigan Collaborator: Michigan State University Collaborator: Wayne State University Collaborator: Indiana University

Ceph Upgrade from Nautilus to Octopus

17 July 2020

The OSiRIS team updated the ceph cluster from Nautilus 14.2.9 to Octopus 15.2.4 which is the latest release of Ceph as of August 2020 and it is the fourth release of the Ceph Octopus stable release series.

Major Changes from Nautilus

  • A new deployment tool called cephadm has been introduced that integrates Ceph daemon deployment and management via containers into the orchestration layer. For more information see Cephadm
  • Health alerts can now be muted, either temporarily or permanently.
  • Health alerts are now raised for recent Ceph daemons crashes.

CentOS 7 Dependencies

The dashboard, prometheus, and restful manager modules do not work on CentOS 7 build due to Python 3 module dependencies that are missing in CentOS 7. To resolve the dependency issue and make sure the dashboard, prometheus and restful manager modules function as desinged in Octopus we installed three python3 rpm packages in our repo server by adding them to um-repo. The 3 packages are i) python3-cherrypy ii) python3-routes iii) python3-jwt. Please install these python3 packages in CentOS7 before proceeding with upgrade to Ceph Octopus if you use the ceph-mgr-dashboard, prometheus and restful manager modules.

Upgrade Instructions

  1. Make sure your cluster is stable and healthy (no down or recovering OSDs) 
    2. ceph health detail
  2. Set the noout flag for the duration of the upgrade 
    3. ceph osd set noout
  3. Upgrade monitors by installing the new packages and restarting the monitor daemons 
    4. systemctl restart ceph-mon.target
    Once all monitors are up, verify that the monitor upgrade is complete by looking for the octopus string in the mon map 
    ceph mon dump | grep min_mon_release
    The output should show : 
    min_mon_release 15 (octopus)
  4. Upgrade ceph-mgr daemons by installing the new packages and restarting all manager daemons 
    5. systemctl restart ceph-mgr.target
    Verify that ceph-mgr deamons are running by executing 
    ceph -s
  5. Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD hosts 
    6. systemctl restart ceph-osd.target
    The progress of the OSD upgrades can be monitored using the either of the commands 
    ceph versions 
ceph osd versions
  6. Upgrade all CephFS MDS daemons

  7. Upgrade all radosgw daemons by upgrading packages and restarting daemons on all hosts 
    8. systemctl restart ceph-radosgw.target
  8. Complete the upgrade by disallowing pre-Octopus OSDs and enabling all new Octopus-only functionality 
    9. ceph osd require-osd-release octopus
  9. If noout was set at the beginning be sure to clear it with
    10. ceph osd unset noout
  10. Verify the cluster is healthy by running
    11. ceph health

For the complete information about upgrading Ceph from Nautilus to Octopus including new features, prerequisties and compatibility deatils , please refer to the Octopus upgrade documentation on the ceph website

Tags