Sharding is the practice of logically dividing or partitioning data, usually using a sprecific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more ...

learn more… | top users | synonyms

0
votes
1answer
13 views

Compound shard key with _id field

I have documents like: {_id: "someid1", "bar": "somevaluebar1"} {_id: "someid2", "foo": "somevaluefoo2", "bar": "somevaluebar2"} {_id: "someid3", "foo": "somevaluefoo3", "zoo": "somevaluezoo3"} {_id: ...
0
votes
0answers
12 views

MongoDB DUPLICATE non-sharded collection was created when running mongoimport on 2 different query routers

I am running multiple mongoimport processes to import a collection lc_data into a mongo cluster. This collection is NOT sharded and its primary server is the shard-01 server. mongos> db.lc_data....
1
vote
0answers
23 views

Mongos process memory usage and crashes

So our MongoS instances use up A LOT of memory. More than even the shards and sometimes crash because of it. Cluster structure 3x mongos 3x mongod Shards 3x mongod Replicas (one for each shard) 3x ...
6
votes
0answers
102 views

SQL Server Sharding strategy optimization

I have implemented a sharding solution for our SQL Server databases as per the MSDN documentation here: https://msdn.microsoft.com/en-gb/library/dn589797.aspx This has been running for around 6 ...
0
votes
0answers
23 views

MongoS server not starting also not balancing Server's sharding metadata manager failed asking for instance is manually reset

My MongoS servers are not staring they are sending this error in logs. SHARDING [Balancer] caught exception while doing balance: Server's sharding metadata manager failed to initialize and will ...
0
votes
1answer
37 views

Table/database design question

Suppose you have these fields to keep track of in records: (first_name, last_name, state_id, stateid_status, day, month, year, id_status, id, criminal_status, charge, plea, sentence, create_time, ...
0
votes
2answers
392 views

mongodb could not find host matching read preference { mode: \“primary\” } for set?

I am deploy a monodb sharding, I have deploy a replica set in three machines: dev41:27017,dev42:27017,dev193:27017 and configsvr in three machine: dev41:27019,dev42:27019,dev193:27019 and also a ...
1
vote
1answer
172 views

Postgres sharding installation with citusdb

We are developing a prototype for a BIG data product. We have almost 2 billion records. We have used PostgresSQL 9.5 as a back-end and Python as front-end. We are using a 16*2.4 GHz processors with ...
0
votes
0answers
23 views

MongoDB: Splitting empty collection with hashed sharding key

I'm currently trying to pre-split an empty collection. According the documentation (found here: https://docs.mongodb.org/manual/reference/command/split/): As such, the command cannot use the find ...
0
votes
1answer
41 views

Do SQL Sharding or Cluster help speed up slow queries, or just concurency?

I am having a performance problem with my current single instance database as the volumes of data grow. The application I am working on receives inserts once a month, and the oldest month is deleted ...
0
votes
1answer
34 views

MongoDB document found on replica set but not in mongos

I have a sharded cluster: two shards - each a replica set with 3 nodes, and a config replica set (with 3 nodes). I sharded my cluster by two collections in the same database: - players.user_id - it'...
0
votes
0answers
22 views

Rebalancing fast with mongodb

I am implementing a sharding with MongoDB and have recently added another node, but the rebalancing of data is very slow. Servers have 64 Mac OS with RAM. I need help on how I can run the mongod ...
0
votes
0answers
39 views

MongoDB slow aggregation for 30 millions of documents

I have MongoDB cluster that include 3 Shard servers based on m3.large type, config server and mongos. There I have a database on 30G with one shard collection with 30 millions of documents. Each ...
0
votes
1answer
34 views

MySQL Cluster - How the Sharding works?

Right now I have a MySQL database with a couple of tables with around 20MM rows each; This DB is located on a single server ( 4gb RAM, 60GB storage, 2 CPUs) each table size is 4gb +-, but my server ...
0
votes
0answers
40 views

MongoDB sharded Replica cluster AWS

I am currently evaluating where to run a sharded&replica MongoDB cluster right now I am looking into: self hosted, or either self made on AWS with a bunch of EC2 instances AWS cloudformation ...
0
votes
2answers
23 views

MongoDB shard keeps growing, I don't know why?

I've had a project dumped on me and I'm trying to get up to speed withMongo. But I have an immediate firefight, in that one shard in my Mongo cluster just keeps growing and growing, adding more ...
1
vote
0answers
19 views

ConfigServersInconsistent:How to correct the inconsistency?

Detected that the one of the config servers is inconsistent with rest two, in a 3 config-server environment. But not sure how to set this right. I have gone through mongodb config servers not in sync ...
1
vote
1answer
153 views

MongoDB allocates chunks only to Primary when sharding is enabled

I'm setting up an instance of MongoDB with 3 shards. I've setup everything seemingly correctly by following the docs and comparing my config with that of a friend who has a similar setup but with a ...
1
vote
1answer
93 views

MongoDB with Sharding has slow performance

I have the following setup for my MongoDB Sharded Cluster (everything is running on the same machine on localhost, just different port numbers): ConfigServers : 1 ReplicaSet consisting of 3 mongod ...
1
vote
0answers
77 views

Adding a Mongo Shard decreases the insert performance

For a mobile application which will hold quite some documents, we've chosen Mongo as a database. Let me explain the situation first: Every day, I have a set of documents that needs to be inserted, ...
1
vote
0answers
62 views

MaxScale vs Fabric [closed]

I am looking to expand my single MySQL database into multiple instances to provide High Availability. As well, I am looking to be able to have read replicas and to shard in the future. I have looked ...
0
votes
0answers
29 views

How to query MongoDB sharding and replica set configuration from remote machine

I have to query to get MongoDB cluster configuration from remote machine through programming code. I can get these details from Mongo shell using sh.status() and rs.status() command. I am looking ...
0
votes
1answer
45 views

MySQL - design help for scale

I will be using Amazon Web Services RDS service to host a MySQL 5.7 server (Currently AWS do not support 5.7, but they have announced they are working on implementing this version and it will be out ...
2
votes
0answers
86 views

Expanding MariaDB cluster (sharding)

I have an expanding MariaDB 3 node cluster and need to expand the available storage. Each node has (to make the maths slightly easier) 120Gb of database storage. As this is replicated, the total DB ...
0
votes
1answer
43 views

Adding more space + bandwidth to a MongoDB cluster

I've read a lot about sharding and have gone through cluster setup process as well. What I can't get my head around is the actual database capacity after its sharded. So say for instance I have one ...
2
votes
1answer
41 views

How to make most of aws?

I'm new to aws (just one day old). I'm impressed with aws free tier and want to utilize to its best. As of now I'm done with installing mongodb on aws micro instances and mongod EC2 instances are ...
0
votes
0answers
19 views

mongodb balancer exception

We are using mongodb having 2 shards.But we realized that chunks not moving between shards.When I look at logs of servers running mongos we are seeing the message: SHARDING [Balancer] balancer move ...
0
votes
1answer
59 views

Must data be evenly distributed among Mongo shards?

According to this article: https://www.mongodb.com/blog/post/on-selecting-a-shard-key-for-mongodb. Assuming I have my schema as: { time_posted: ..., userid: ..., content: ... } The ...
1
vote
0answers
34 views

How does defining a MongoDB shard key influence the connectivity between my mongo(s/d)s?

I am trying to test mongoDB sharding, but when following the basic steps in common tutorials, I am facing a CONNECT_ERROR issue that really doesn't make sense to me. I use four machines for this test ...
7
votes
1answer
41 views

Mongos not returning existing data in shards

So we have a configuration of 2 shard (each is a replica set) and one sharded collection. There are around 10 million records in each shard. We are using mongo 2.6.4 version. We have sharded with ...
1
vote
0answers
78 views

Securing High-Availability on Partitioned Database

I have a 90GB SQL 2012 database I am sharding over 3 database and I plan on growing as more clients are coming. A new customer came to us with a demand to know on we secure high-availability in our ...
0
votes
1answer
39 views

How set balancing as main task for sharder MongoDB cluster?

I have a big collection (over 400 millions records) which I want to divided between 4 shards. I created infrastructure (4 shards, config server, mongos) and all work well (balancer is working and data ...
0
votes
0answers
38 views

mongodb: fail for removing shard from shardings

When I try to remove a shard from my mongodb sharding, I got the following error: Sun Sep 6 11:30:09 [Balancer] going to move { _id: "hydra.user_short_message-...
0
votes
0answers
16 views

mongodb sharding by id, based on the first 8 characters

we have a database full of grams,and they're organised in a tree fashion, with parent references, like so: _id: gram:a;p1:as;p2;astr;p3;astrot;p4:astrotur;word:astroturf we would like to chunk the ...
2
votes
1answer
65 views

How to deploy MongoDB config servers to achieve maximum availability during network partitions?

What's the recommended way to deploy config servers? Assume, we have two shards, each one is a 3-nodes replica set. If I deploy all config servers together with one of these shards, a single network ...
0
votes
1answer
67 views

No Data Replication in Progress mongodb

Setup Information One ConfigServer mongod.exe --configsvr --dbpath C:\MongoDBCluster\ConfigSrv --Mongos Server-- mongos.exe --configdb 10.133.1.130) 2 Shard Servers mongod.exe --shardsvr --...
1
vote
1answer
286 views

MongoDB: movechunk failed after adding a new share to cluster

I have a MongoDB cluster version 2.6.8, which already has two shares: share1 and share2. When I add a new share3 to the cluster, data chunk can't migrate to share3. This is from the mongos log: ...
2
votes
2answers
276 views

Sizing a MongoDB shard?

I am in the process of planning a MongoDB deployment, this will be my first to production and I have a question regarding the disk size of each shard in a cluster. This cluster itself will be for ...
1
vote
0answers
45 views

Database sharding, denormalization and synchronization

In sharding, it is advisable to follow a share nothing, shard everything approach. Essentially, this translates to: data that are to be used together are to be stored together (i.e. in a single shard)....
2
votes
1answer
33 views

Sharding Mongodb while limiting the number of documents to be migrated

I have little experience working with databases, but at my work i was tasked to deploy a sharded Mongodb cluster and the constrain on the shard is: database with multiple colections, 7 collections ...
0
votes
2answers
53 views

Wrong sharding on mongoDB 3.0

I have a mongos with 2 shard servers (and 3 config servers) sh.status() --- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : ...
2
votes
1answer
90 views

MongoDB sharding

I have a questions about mongodb sharding. I have 7 VM with 1 master mongos and 3 config server en 3 shard servers. Mongo Main - 192.168.212.173 poort 27020 Mongo Shard1 - 192.168....
0
votes
0answers
200 views

Mongodb does not contain shardkey for pattern

I've created a database with 3 sharded clusters using the wiredTiger engine and I've enable the storage engine option for replicas. Now after enabling sharding for collection, importing a csv to mongo ...
0
votes
1answer
69 views

updating single row in MYSQL

We have a situation where we need to update on a single row several thousands time a second and this requirement lasts continuously for 5 hours every day due to the nature of the business. Design ...
6
votes
1answer
140 views

Remove a shard in MongoDB

I have a MongoDB cluster, I accidentally hosed a shard, I don't have a backup. It's pretty easy to recover the data, but after issuing the removeShard command, it says it's draining. However, the ...
0
votes
1answer
107 views

Why sharding does not show any improvement?

Here is what I did to test sharding performance. I have 4 servers, I ran config server and router on a server. 3 others are meant to be shards. First, I only added one shard to router, then I ran a ...
2
votes
1answer
187 views

MongoDB : How to estimate max number of connections supported?

We have a sharded cluster with 2 replica sets. Each replica set is a 3 member cluster with one primary and 2 secondaries. Each instance is a r3.large (aws-ec2) instance with 15G ram. mongos's are in ...
2
votes
2answers
2k views

Mongos version 3.0.1 Split Chunk Error with Sharding

I have 3 config server and two shards. Each shard has a primary, secondary and an arbiter. This is running with the following mongod settings: storageEngine=wiredTiger wiredTigerJournalCompressor=...
0
votes
1answer
155 views

MongoDB Hashed Shard Keys Selection

I have a DB which holds very simple documents called "Item": {_id:<ObjectId>, playerId:int, isRead:bool, content:string} containing about 80M documents. Each Item belongs to a certain player, ...
1
vote
1answer
51 views

How to shard a database like that of SO?

I know a bit about sharding but I cannot wrap my head around that fact that say eventually SO cannot scale vertically then how would they shard their SQL DB? Even if we assume SO will not reach that ...