Can arbiterOnly replica in MongoDB become SECONDARY and what it means?

Question

The MongoDB documentation says that when a replica is added to the set as arbiterOnly=true it will not become Primary. But can Arbiter become a secondary and what does it when it does?

I have a replica set consisting of two data replicas and one arbiterOnly.

Some time ago my one of my data replicas which was secondary at a time crushed due to hard drive failure. After I fixed that problem and restarted secondary it went into “Recovering” state. But my arbiter is now “Secondary”. (That is shown by rs.status() command)

Furthermore, the data directory for this Arbiter shows bunch of files of total size >10GB. They indeed look to me like data files. Are they? What is going to happen to these files when Recovery completes?

If Arbiter can become truly a data node (even secondary) that opens up the whole new bag of worms: I need to have enough free disks available to Arbiter, right now I don't. Is there option to prevent arbiterOnly from becoming secondary?

Answer 1

Some time ago my one of my data replicas which was secondary at a time crushed due to hard drive failure. After I fixed that problem and restarted secondary it went into “Recovering” state. But my arbiter is now “Secondary”

A MongoDB arbiter cannot automatically become a secondary or a primary node, as it does not have a copy of the data set.

If you try to manually reconfigure the arbiter as a regular node via rs.reconfig() you should get an exception similar to:

{
    "errmsg" : "exception: arbiterOnly may not change for members",
    "code" : 13510,
    "ok" : 0
}

Furthermore, the data directory for this Arbiter shows bunch of files of total size >10GB. They indeed look to me like data files. Are they? What is going to happen to these files when Recovery completes?

Assuming this node is definitely an arbiter, I would expect those files are unused (check the timestamps?) and are either:

• a local database created if this node was incorrectly initialized with an oplog
• unused copy of data directory if this node was copied from another secondary or used as a standalone before

You can always log into the arbiter mongod directly to see what data it appears to have.