2

Situation

I have 3 nodes on a Replica Set (P+S+S). Mongodb version 2.6.9. I would like to migrate all the nodes to 3.0.4 wiredtiger.

When I need to do that without downtime, I do the following workflow:

  1. setup a new server with 3.0.4
  2. add the node to the replica set
  3. remove one of the old nodes
  4. go to step 1

Problem is that with my set of data, adding a node with the "rs.addnode" tactic just doesn't work. The process takes forever and starts the initial syncronization again and again. To walkaround this, I normally do a cold backup from one of the old nodes to the new one. That takes the data and the local database, the node joins the replica set without initial syncronization.

Problem is that I cannot do that because the file system is not the same. If I can put one with wiredtiger, I can do the cold backup trick with the other ones.

Attempts I made with no luck:

  1. reduce size of my database
  2. reduce the Oplog size
  3. try with a dump backup / restore
  4. rs.add

I know there is a way to add a node with a setting --fastsync. But to do that, I need to have a copy from the primary, and the files are not the same. Not sure if that will do the trick with dump backup / restore. I would know if when the node takes the new member as a valid secondary, I can restart the node without the fastsync or I should put that setting forever.

This is my mongod.log when it fails with the initial sync:

I INDEX    [rsSync]         building index using bulk method
I INDEX    [rsSync] build index done.  scanned 3 total records. 0 secs
I REPL     [rsSync] initial sync cloning db: config
I REPL     [rsSync] initial sync data copy, starting syncup
I REPL     [rsSync] oplog sync 1 of 3
I NETWORK  [rsSync] Socket recv() timeout  x.x.x.x:x
I NETWORK  [rsSync] SocketException: remote: x.x.x.x:x error: 9001 socket exception [RECV_TIMEOUT] server [x.x.x.x:x]
I NETWORK  [rsSync] DBClientCursor::init call() failed
E REPL     [rsSync] 10276 DBClientBase::findN: transport error: x:x ns: local.oplog.rs query: { query: {}, orderby: { $natural: -1 } }
E REPL     [rsSync] initial sync attempt failed, 9 attempts remaining

SOLUTION (see comments)

configure ulimits:

sudo nano /etc/security/limits.conf
# add at the bottom
mongod       soft    nproc   64000
mongod       hard    nproc   64000
mongod       soft    nofile  64000
mongod       hard    nofile  64000
mongod       soft    fsize   unlimited
mongod       hard    fsize   unlimited
mongod       soft    cpu     unlimited
mongod       hard    cpu     unlimited
mongod       soft    as      unlimited
mongod       hard    as      unlimited

configure keep alive

sudo nano /etc/sysctl.conf
# add at the bottom
net.ipv4.tcp_keepalive_time = 120

Note: I did that on the new member and on the existing ones

  • You have to resync the data because the storage formats on disk are different. Are you getting a specific error during the initial sync? What is your specific O/S version and hosting type (eg AWS, Azure, bare metal, ...)? – Stennie Jul 3 '15 at 12:43
  • the initial sync starts again and again until consumes the number of attempts. I am using Centos 7.1 on a Azure instance. – Jordi Jul 3 '15 at 13:09
  • 1
    Have you reviewed the production notes for Linux & Azure? In particular I would make sure you have reduced the tcp_keep_alive timing to 120 seconds and increased your ulimits from the default before doing the initial sync. – Stennie Jul 3 '15 at 13:18
  • My ulimits are: /etc/security/limits.conf mongod soft nproc 64000 mongod hard nproc 64000 mongod soft nofile 64000 mongod hard nofile 64000 It doesn't look like a linux limitation, it looks like mongodb limitation/timeout – Jordi Jul 3 '15 at 13:28
  • 1
    Setting the tcp_keepalive_time has definitely fixed the problem for other users with the same issue. I don't think you have a choice of not going through the Azure balancer or some other network gear that has the keepalive issue, so it's definitely worth a try. Note: In case you didn't check my profile .. I work for MongoDB ;-). – Stennie Jul 6 '15 at 11:27
2

To migrate data between different storage engines in MongoDB 3.0 you will need to perform an initial sync to the new replica set node or use mongorestore to seed a new replica set with your data.

If you want to add a node with a new storage engine to a live production replica set, rs.add() is the correct approach. To ensure a smooth upgrade I would also upgrade all existing nodes to the latest version of MongoDB 3.0.x with MMAP storage (the default) before adding your new WiredTiger node(s).

The key information in your question (and comments) is:

  • socket exceptions related to networking timeouts:

    I NETWORK  [rsSync] Socket recv() timeout  x.x.x.x:x
    I NETWORK  [rsSync] SocketException: remote: x.x.x.x:x error: 9001 socket exception [RECV_TIMEOUT] server [x.x.x.x:x]
    
  • deploying on Azure, which has some known issues with timeouts

As per the MongoDB production notes on Azure, you should reduce the TCP keepalive setting to avoid network timeouts:

The TCP keepalive on the Azure load balancer is 240 seconds by default, which can cause it to silently drop connections if the TCP keepalive on your Azure systems is greater than this value. You should set tcp_keepalive_time to 120 to ameliorate this problem.

As further preparation for successful initial sync with WiredTiger you should review the MongoDB production notes for your operating system.

Since you mention using Linux, I would recommend checking:

  • ulimits are appropriately set. WiredTiger creates more files than MMAP (separate data & index files per collection rather than per database) so will use more file handles than MMAP nodes in the same replica set. If you don't increase the ulimits from the default you may find the initial data transfer completes but initial sync fails at the index build stage with a fatal error like Too many open files.

  • the WiredTiger data directory is on a volume using the XFS file system. As at the time of writing, there are known performance issues (such as throughput stalls) with EXT4.

The production notes in the MongoDB manual are updated regularly based on issues and experiences reported by MongoDB users, so should be part of your preflight checklist for production O/S deployments or upgrades.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.