Skip to main content

Questions tagged [sharding]

Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.

Filter by
Sorted by
Tagged with
15 votes
3 answers
10k views

Should a multi tenant system with SQL Server 2016, Shard or have Tenant isolation via separate database per tenant?

Given the use case: Tenant data should not cross talk, one tenant does not need another tenant's data. Each tenant could potentially have large historical data volume. SQL Server is hosted in AWS ...
D.S.'s user avatar
  • 365
15 votes
2 answers
44k views

Effectively handle 10-100 millions row table of unrelated data

What are the common approaches to boost read/write performance of table with up to 100 millions of rows? Table has columnSEGMENT_ID INT NOT NULL, where each segment has about 100.000-1.000.000 rows. ...
VB_'s user avatar
  • 407
12 votes
2 answers
7k views

MongoDB: co-locate the mongos process on application servers

I would like to ask a question about a best practice described in this document: http://info.mongodb.com/rs/mongodb/images/MongoDB-Performance-Best-Practices.pdf Use multiple query routers. Use ...
tenshi's user avatar
  • 173
10 votes
11 answers
37k views

mongodb could not find host matching read preference { mode: \"primary\" } for set?

I am deploy a monodb sharding, I have deploy a replica set in three machines: dev41:27017,dev42:27017,dev193:27017 and configsvr in three machine: dev41:27019,dev42:27019,dev193:27019 and also a ...
roger's user avatar
  • 213
9 votes
1 answer
5k views

mongodb shard chunk migration 500GB takes 13 days - Is this slow or normal?

I have mongodb shard cluster, shard key is hashed. It has 2 shard replica sets. Each replica set has 2 machines. I did an experiment by adding another 2 shard replica sets, and it starts to rebalance....
rendybjunior's user avatar
9 votes
5 answers
48k views

MongoDB --- Failed global initialization: Failed to open "/var/log/mongodb/mongod-config.log"

I'm trying to setup the config servers for mongodb sharding. I created a specific config file that is set to log to /var/log/mongodb/mongod-config.log. When I run mongod --config <path-to-config-...
rapidDev's user avatar
  • 131
9 votes
2 answers
3k views

verify that mongos server is connected to config servers

I've been writing a backup script for sharded replica-sets and it's almost done. Except I can't seem to get it to successfully start the balancer backup after everything's all said and done. Here's ...
Alexej Magura's user avatar
7 votes
1 answer
19k views

Horizontally scaling SQL Server, distributing the database with sharding

I wanted to know if there is any way to distribute a SQL Server (I'm using 2012 version) database accross multiple nodes. I'm trying to compare READ queries performance between SQL Server and MongoDB. ...
user98851's user avatar
7 votes
2 answers
562 views

Mongos not returning existing data in shards

So we have a configuration of 2 shard (each is a replica set) and one sharded collection. There are around 10 million records in each shard. We are using mongo 2.6.4 version. We have sharded with ...
Ivan Longin's user avatar
7 votes
3 answers
4k views

Scaling out SQL Server and syncing data across multiple machines

I don't have expertise in architecting databases, and I've been teaching myself new stuff every day. I'd like to make an Internet-scale application using SQL Server as the data store. I haven't found ...
Mark13426's user avatar
  • 579
6 votes
1 answer
832 views

Remove a shard in MongoDB

I have a MongoDB cluster, I accidentally hosed a shard, I don't have a backup. It's pretty easy to recover the data, but after issuing the removeShard command, it says it's draining. However, the ...
Kevin's user avatar
  • 61
5 votes
1 answer
2k views

MongoDB presplitting chunks for compound shard key

In my mongodb setup I have a compound shard key {"region" : 1, "foo" : 1, "bar" : 1} and I know the values region can be and that each region should be on one chunk. Therefore I'd like to pre-split ...
Nils's user avatar
  • 153
5 votes
1 answer
5k views

Sharding Postgres Database

I have a Postgres database that has grown to the size where it is no longer feasible to store everything on a single database node. There is a Customer table in my schema where each row represents a (...
CadentOrange's user avatar
5 votes
2 answers
1k views

Chunk Size is shown as 1 KB in mongo db logs even it is set to 300 MB [closed]

I am running a mongo cluster. Chunk size set is 300 MB but for today morning it is showing me in logs that chunk size is 1024 Byte. I checked in current op there also it is showing chunks of 1024 byte....
viren's user avatar
  • 511
4 votes
2 answers
10k views

How to Determine Chunk Distribution (data and number of docs) in a Sharded MongoDB Cluster?

Having done all the right things - picked an appropriate shard key, scaled horizontally, distributed my data across several shards, I find I now have no real way to determine how well balanced the ...
Adam C's user avatar
  • 9,245
4 votes
1 answer
4k views

How do I get PostgreSQL FDW to push down the LIMIT to the (single) backend server?

I've set up a PostgreSQL FDW server with the following table, sharded by user_id over four servers: CREATE TABLE my_big_table ( user_id bigint NOT NULL, serial bigint NOT NULL, -- ...
gertvdijk's user avatar
  • 227
4 votes
1 answer
5k views

MongoDB crashes with out-of-memory or is being killed by oom-killer

A two shards MongoDB database regularly crashes with out-of-memory error or is being killed by the oom-killer. The system runs on GCE Debian 9.4 with MongoDB v3.6.5, WiredTiger storage engine and ...
ssasa's user avatar
  • 241
4 votes
1 answer
4k views

Mariadb sharding?

I'm trying to implement a mariadb sharding for learning purpose. I have looked into database sharding in general and found mariadb spider storage engine support sharding. I do not want spider storage ...
Viraj's user avatar
  • 323
3 votes
1 answer
5k views

Mongo large chunks will not split

I had a 3 shard set up, and ran out of capacity, so I added 3 more shards. (Each shard is a replica set). But the data isn't spread across the cluster evenly. I have my chunkSize set to the standard ...
Landon's user avatar
  • 303
3 votes
2 answers
12k views

How to activate shard balancer in sharded MongoDB

I have a very simple MongoDB database structure. Also I have 5 shards in it (3 of them are replicasets). But we are testing DB loading and it seems that shard balancing isn't currently enabled. On ...
bronze's user avatar
  • 57
3 votes
1 answer
5k views

MongoDB - sharding an collection with data already in it

I'm using Mongo (shell version 2.6.12) in a clustered configuration. Right now I have a few collections sharded, and was looking to shard another collection. This collection already has data in it. ...
SteveC's user avatar
  • 33
3 votes
1 answer
4k views

Is it correct to use IP addresses (instead of hostnames) while configuring Mongo DB cluster? Does it cause any issues?

I have been using Mongo DB Cluster for some data analysis (Mongo DB version 2.4.5). My configuration is as follows: 1) Node1 - mongoconfig server, mongod shard 2) Node2 - mongoconfig server, mongod ...
ShekharM's user avatar
3 votes
1 answer
752 views

GridFS: why can't I shard chunks with a hashed key?

There is a statement in the MongoDb documentation: You cannot use Hashed Sharding when sharding the chunks collection. It does not say that, but I guess it means that I can only use ranged ...
nagylzs's user avatar
  • 411
3 votes
1 answer
214 views

MongoDB index strategy

I'm developing a system to back-up computers and store the files / data with retention for a while. The actual files are stored in an object store, while I'm planning to keep the metadata of the files ...
Grotehuiskameel's user avatar
3 votes
1 answer
3k views

MongoDB Shard - Disable Cursor timeout

I'm using an sharded-cluster(Mongo version 2.4.1) with a java application. After a specific time (10 minutes or more) I got the following exception: com.mongodb.MongoException$Network: Read operation ...
DanielH's user avatar
  • 35
3 votes
1 answer
5k views

Removing Database from sharded cluster in Mongo DB

I Have a mongo shard db system with two shards and and three mongos servers.My application creates DB on daily basis as per our business requirements. We have some sharded collection in each day DB. ...
viren's user avatar
  • 511
3 votes
1 answer
91 views

What inefficiency does sharding & partitioning attempt to address? [closed]

I have never implemented sharding or partitioning, but from what I understand, multiple servers are used to distribute large amounts of data. Is this because of a limited hardware budget or to ...
user avatar
3 votes
1 answer
817 views

MongoDB sharding by monotonic id with zones for archiving

Situation: I have an ever-growing collection of documents which do have monotonically increasing unique id. All queries are direct lookup for documents with specified _id (no range queries nor ...
Antonio Tomac's user avatar
3 votes
0 answers
336 views

FDW-based Sharding with PostgreSQL 10: Multiple coordinator nodes?

Context I'm currently looking for and comparing different options for sharding data from a Postgres database into multiple ones. The end result may be something like having instances of a web ...
ginmrt's user avatar
  • 31
3 votes
0 answers
2k views

Mongos process memory usage and crashes

So our MongoS instances use up A LOT of memory. More than even the shards and sometimes crash because of it. Cluster structure 3x mongos 3x mongod Shards 3x mongod Replicas (one for each shard) 3x ...
Sam's user avatar
  • 111
2 votes
1 answer
638 views

Are multiple databases sharding for read on docker as a good idea?

So I have a server with multiple CPU cores and it installed a database. Do you think that if we install multiple databases (data sharding) with docker, then each request will go to different databases ...
Bằng Rikimaru's user avatar
2 votes
1 answer
2k views

What is the new JDBC 4.3 Connection Request feature with `beginRequest` & `endRequest` methods?

The JDBC 4.3 update arrived with Java 9 2017-10. Amongst its few changes is the new feature on Connection where the beginRequest and endRequest methods signals… Hints to the driver that a request, ...
Basil Bourque's user avatar
2 votes
2 answers
5k views

Sizing a MongoDB shard?

I am in the process of planning a MongoDB deployment, this will be my first to production and I have a question regarding the disk size of each shard in a cluster. This cluster itself will be for ...
tomstephens89's user avatar
2 votes
1 answer
427 views

How to implement High Availability for all nodes in a sharded database?

I'm doing some research into implementing sharding for one of my large (~2 TB) production databases. Currently the database has HADR via a multi-replica AlwaysOn Availability Group. Since sharding a ...
J.D.'s user avatar
  • 38.1k
2 votes
1 answer
717 views

What is the meaning of "Monotonically Changing Shard Keys"?

I am writing my first pubblication concerning MongoDB and more specific MongoDB Sharded clusters. On the documentantion I found that the shard key in case of ranged sharding should be "Non-...
Massimo Lavermicocca's user avatar
2 votes
2 answers
2k views

MongoDB Cluster - Can't access member of shard with credentials

I have a Sharded Cluster and I am trying to access a specific member of a replica set within a shard. When connecting to the mongos it is fine, but when using the same credentials on a member of the ...
Alex Johnston's user avatar
2 votes
1 answer
555 views

mongo shard + maxSize

I have a sharded cluster (mongo 3.2). I'm trying to set a Maximum Storage Size for a Given Shard, but it fails. mongos> show dbs DB_files 820.835GB config 0.012GB mongos> config = db....
user183001's user avatar
2 votes
1 answer
4k views

Multiple MongoDB databases (1 per client) and sharding - are they compatible?

Currently I'm working with one MongoDB database per client, on a single replica set. We would like to keep one database per client because of security concerns. Each client has the same 15 ...
clopez's user avatar
  • 213
2 votes
1 answer
2k views

MongoDB : How to estimate max number of connections supported?

We have a sharded cluster with 2 replica sets. Each replica set is a 3 member cluster with one primary and 2 secondaries. Each instance is a r3.large (aws-ec2) instance with 15G ram. mongos's are in ...
titogeo's user avatar
  • 230
2 votes
1 answer
234 views

Mongodb sharded cluster backup user issue

We're using a simple python script to backup our mongodb cluster. the script connects to each mongod instance via ssh, and performs mongodump, all without requiring user/pass ever since we started ...
Meny Issakov's user avatar
2 votes
1 answer
611 views

MongoDB sharded cluster chunks distribution

I have deployed MongoDB cluster with two shards (two replica sets) with docker-compose. Unfortunately I'm having problems with chunks distribution between shards, I'm new to MongoDB and database ...
Tomek's user avatar
  • 23
2 votes
1 answer
3k views

One of three Mongo Shard node is running with very high CPU LOAD/USAGE

We have three mongo shards with db version v3.2.10.The primary shard is running with very high CPU usage ~675% while other 2 shard node are running ~12% CPU usage. Here is output for $top command of ...
Afsar's user avatar
  • 21
2 votes
1 answer
784 views

ProxySQL equivalent of MaxScale schemarouter

background My employer developed a web application we provide on software-as-a-service terms to our customers. To allow for multiple customers with a huge mass of data to be stored in a database, we ...
Phillip -Zyan K Lee- Stockmann's user avatar
2 votes
2 answers
5k views

Mongos version 3.0.1 Split Chunk Error with Sharding

I have 3 config server and two shards. Each shard has a primary, secondary and an arbiter. This is running with the following mongod settings: storageEngine=wiredTiger wiredTigerJournalCompressor=...
klall's user avatar
  • 121
2 votes
1 answer
2k views

MongoDB Stuck with Jumbo chunks while exhausting a shard

I'm running a shared MongoDB 2.4 cluster. I'm currently in a process of exhausting a shard and I'm down to the last 15 chunks needs to be migrated out of 9660. My issue is that these chunks are too ...
Meny Issakov's user avatar
2 votes
1 answer
332 views

Cannot connect to MongoDB router in sharded cluster

I followed this guide on how to setup a sharded MongoDB cluster with Docker containers. It all seemed to work as it should because there were no error messages in the logs and the sh.status() command ...
Tobias S.'s user avatar
  • 131
2 votes
1 answer
1k views

MongoDB sharding multiple collections with the same shard key

Lets imagine I have two mongodb sharded collection: likes & comments. Both collections are sharded, both collections have userId as a sharded key. Lets imagine that likes collection is huge and ...
user3489820's user avatar
2 votes
1 answer
1k views

Kafka Partitions vs Mongo Sharding which one is better throughput [closed]

I am using Mongo Sharding to register page views on my website. We have hashed shard key to evenly distribute data in multiple shards. Then our aggregation queries run over time range at interval to ...
viren's user avatar
  • 511
2 votes
1 answer
221 views

How can we restrict collections on specific shards of Mongodb while using Tag Awareness Sharding?

We have a Mongodb cluster which has 10 shards (0-9) and two different applications supposedly as APP1 and APP2. Both APP1 and APP2 have different data model and shard keys. We want that to assign ...
Rajat Mishra's user avatar
2 votes
1 answer
632 views

When converting a replica set with an arbiter to a shard, does the arbiter also get added to the shard?

I'm converting an existing replica set that has an arbiter to a shard as shown in this guide: https://docs.mongodb.com/v3.2/tutorial/convert-replica-set-to-replicated-shard-cluster/ and it's ...
elfick's user avatar
  • 23

1
2 3 4 5
7