We have two nodes. Node1 was down for a long time. During this time Node2 raised 1 TB capacity when Node1 has 100 GB.

We tried to repair Node1 with nodetool repair but nothing changed. After that we started nodetool repair in Node2, it took 5 days to compaction but nothing changed either.

Actual status here:

Datacenter: dc1

|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID    Rack

UN  172.x.y.z     149.46 GB  256          100.0%            xxx  rack1
UN  172.x.y.k     1.04 TB    256          100.0%            xyz  rack1

Nodes are in AWS. What should we do?

  • What replication factor are you using? – Christophe Schmitz Mar 28 '18 at 7:33
  • @ChristopheSchmitz cqlsh:5.0.1 cassandra: 3.0.9 CQL Spec: 3.4.0, Replication factor = 2 – Ekrem Gurdal Mar 28 '18 at 7:41
  • Are you doing any delete queries? Are you using TTL? – Christophe Schmitz Mar 28 '18 at 7:42
  • @ChristopheSchmitz We have never done delete queries yet and we are not using TTL. We only use store data and do some backward processing jobs. – Ekrem Gurdal Mar 28 '18 at 7:56

We had a solution but don't know how it happened. Here what we did:

  • Took snapshot of both servers in case of data loss over AWS
  • We detached Volume where data is stored. (for instance 172.x.y.k)
  • We formatted instance and loaded Cassandra with new version. Then we attached Volume where data is stored.
  • Finally we started nodetool repair --full in terminal and it took 4 days.

Now our nodes are equal.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.