Last week I asked a question that was answered below... I need to improve the question I asked earlier (in quotes below) and give a little more context. The answer below did help the repair performance but I still believe we may have done something wrong or missed a step that is causing the full repair to go very slow.
"We have a 32 node cluster with an average of approximately 150GB of data on each node. When running a full repair (not doing primary range) it is taking approximately 26 hours to complete. This seems very slow to complete.. any ideas on why this type of repair is going so slow? Are there any settings I may tweak to make it go faster?"
More detail about question:
- Our system is running on Cassandra 3.x
- 16x16 nodes
- Replication 3x3
- Previously doing incremental repairs
- 150GB per node
What happened was the following:
- node bootstrapped on 3.x
- started repair ( incremental ) : assuming this was full repair first time
- lost 2 nodes -- (VMs crashed)
- removed the two nodes from the cluster.
- started full repair on every node (not doing primary range).
Yes running the full repair (primary range) did improve the total time.
- Did we miss a step (see above) in bringing our cluster back to normal.. such that it caused our full repair (not doing primary range) to take an extremely long time?
- Is it typical that full repair (not doing primary range) takes more than twice the time to complete than a full primary range repair?
Answer to previous question:
How to make repairs go faster:
Use the -pr flag. This repairs only the primary range, which prevents the unnecessary repairs of the additional ranges. If you run your repairs on all nodes (and you should) then you'll repair everything anyway, but without repairing everything an extra time for each replica.
Focus repairs only on the tables/keyspaces that need it. Some tables may be static enough that you don't really need to repair them. For instance, if you don't regularly add new users, you probably don't have to repair system_auth. Nodetool repair allows you to specify keyspaces and tables to run a repair on, and you should definitely use it.
Avoid streaming repairs from different physical data centers. This was a big "win" for us. Nodetool repair allows you to specify a list of hosts to include in the repairs. If you have multiple data centers, you should limit your node to only repair itself from nodes which are "local" to it.
Focus repairs only on specific token ranges. You probably won't always have to do this, but sometimes you only have a window that allows you repair for so long. To programmatically repair say only one third or quarter of your data, you can further focus your repair tasks by specifying certain token ranges. I don't typically like to do this one, because repairing all data for a table can stretch out for a few days. But if you have only a small window to run a repair, this might help.
Don't repair the same data multiple times. Repair only where you need to. Stream repairs only from specific, local nodes. Focus your repairs as much as you can.