3

My cluster runs on Cassandra 3.

Upgrading of my cluster involves many steps, included switching to Cassandra 4.Upgrading process goes node by node, to provide High Availability. So if one of the nodes fails, I need revert already upgraded nodes back. It means some of the nodes already running in Cassandra 4, where new dataformat nb-*.db. Cassandra 3 fails on startup if data format is nb-*db (it's expected behavior).

It looks nodetool snapshot -t before-upgrade command is the solution (run before upgrading). And command sstableloader to restore from snapshots, to bring data back in proper format, but sstableloader run on running cluster, which is not my case.

What is the correct procedure for this situation ?

1 Answer 1

1

First of all, you need to change your mindset when it comes to Cassandra. We understand that it is a difficult thing to do when you are used to running old, monolithic relational databases which are not designed for true high availability.

Cassandra is an always-on database where all operations are online -- even upgrades. If the upgrade on one node fails (for whatever reason), you need to investigate why it "failed" and resolve it then continue with the rest of the upgrades.

You should not look to rollback as your first option. In my experience working on hundreds of clusters across hundreds of companies, rollback is not even an option. If a rollback is required, some catastrophic event has taken place and it is usually due to human error.

If you must rollback, the procedure is:

  1. Uninstall Cassandra from ALL the nodes.
  2. Remove the contents of data/, commitlog/, saved_caches and all other directories.
  3. Delete C* configuration files.
  4. Perform a fresh installation of Cassandra.
  5. Configure Cassandra using backups of configuration files.
  6. Restore data from snapshots.

But again, rolling back is a last resort option when everything else has failed. I've never had to do this because there isn't a need for it in Cassandra. Cheers!

2
  • 1
    How does "all operations are online" coincide with your suggestion to "Uninstall Cassandra from ALL the nodes" for doing a rollback? Sep 7, 2022 at 12:22
  • 1
    It is necessary to start over in order to "rollback" upgraded nodes because older versions of Cassandra will not be able to read the file formats of newer versions. As I said, rollbacks are NOT part of the normal operation of Cassandra. I've worked with hundreds of clusters for enterprises and I've never had to rollback. Cheers! Sep 7, 2022 at 12:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.