1

Regarding failed with error aborted will we lose the chunks

enter image description here

In logs I found this:

2017-09-14T01:35:21.533+0900 W SHARDING [conn38309]
Chunk move failed :: caused by :: ChunkRangeCleanupPending: 
can't accept new chunks because there are still 4 deletes from previous migration
2017-09-14T01:35:11.439+0900 I COMMAND  [conn38309] command admin.$cmd
command: moveChunk { moveChunk: "transam_db.panelist", 
shardVersion: [ Timestamp 762000|1, ObjectId('590b4564c5ac1ee3b6e9050d') ],
epoch: ObjectId('590b4564c5ac1ee3b6e9050d'), 
configdb: "configReplSet/mgdb07:27019,mgdb08:27019,mgdb09:27019", 
fromShard: "rs1", toShard: "rs4", 
min: { panelist_id: 407157 }, 
max: { panelist_id: 416836 }, 
chunkVersion: [ Timestamp 762000|1, ObjectId('590b4564c5ac1ee3b6e9050d') ],
maxChunkSizeBytes: 67108864, waitForDelete: false, takeDistLock: false }
exception: can't accept new chunks because there are still 4 deletes from previous migration 
code:200 numYields:75 reslen:278 locks:{ Global: { acquireCount: { r: 161, w: 3 } }, 
Database: { acquireCount: { r: 79, w: 3 } }, 
Collection: { acquireCount: { r: 79, W: 3 } } } 
protocol:op_command 2622ms

Due to this, it's unable to move the chunks. Please give a solution.

Improve this question
0

1 Answer 1

Reset to default
1

Answer is NO.

At 3.4 mongodb we got multi threaded chunk balancing and it is "little bit" stupid that way that it can try to move chunk (from or to) shard what is currently already participating in some other chunk move operation. Of course, that is not possible and then that new move operation is aborted.

If you start seeing huge aborted values, like thousands, check log files of those shards whose name is most listed on error list. Like rs4 there. From log files you see what was "reason" why move was aborted. There can be be cursor with no timeout and previous move operations remove is hanging there.

But, aborted chunk move does not lose data, ever. Data is never removed until it is copied to the new location and then mark to be there.

To find a possible reason for those aborts, all (primary) mongod.log files must be checked. Without them, it's impossible. There is always from node and to node, both log files have information about "that" movechunk operation what was aborted.

Improve this answer
7
  • I have atached above the log related to it please give a solution what to do JJjussi
    – ashk
    Commented Sep 13, 2017 at 18:36
  • As log says, there is still 4 deletes pending. Probably you have queries what had "no cursor timeout" and then those queries just died --> cursor is still open and won't close. What you can do, is rs.stepDown() all current PRIMARY nodes. After stepping down, you need restart those same mongod -processes. After that there is no locks or traces of those cursors. To prevent this happening again, check your application code for 'no cursor timeout'.. Every cursor (query) must have timeout value to prevent this happening again.
    – JJussi
    Commented Sep 17, 2017 at 9:46
  • 7611 : Failed with error 'aborted', from rs4 to rs3 the aborted values reached to thousands and the cpu is reaching 100% in the rs4 shard whats the solution
    – ashk
    Commented Nov 2, 2017 at 7:57
  • Check mongod.log at both primaries to find out what is the reason for moveChunk abort.
    – JJussi
    Commented Nov 2, 2017 at 9:54
  • i have checked the logs in both primary in rs4 chunks were not moving because 1 delete from previous migration and in rs3 cant accept new chunks because 1 delete from previous migration and chunks have not been balanced since last week rs0:-403 rs1:-404 rs3:-402 & rs:- 416 but rs1 is primary of 4 shard how we can balance the chunks blanacer is not starting due to these deletes
    – ashk
    Commented Nov 3, 2017 at 5:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.