Skip to main content

Questions tagged [sharding]

Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.

110 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
9 votes
2 answers
3k views

verify that mongos server is connected to config servers

I've been writing a backup script for sharded replica-sets and it's almost done. Except I can't seem to get it to successfully start the balancer backup after everything's all said and done. Here's ...
Alexej Magura's user avatar
3 votes
1 answer
817 views

MongoDB sharding by monotonic id with zones for archiving

Situation: I have an ever-growing collection of documents which do have monotonically increasing unique id. All queries are direct lookup for documents with specified _id (no range queries nor ...
Antonio Tomac's user avatar
3 votes
0 answers
336 views

FDW-based Sharding with PostgreSQL 10: Multiple coordinator nodes?

Context I'm currently looking for and comparing different options for sharding data from a Postgres database into multiple ones. The end result may be something like having instances of a web ...
ginmrt's user avatar
  • 31
3 votes
0 answers
2k views

Mongos process memory usage and crashes

So our MongoS instances use up A LOT of memory. More than even the shards and sometimes crash because of it. Cluster structure 3x mongos 3x mongod Shards 3x mongod Replicas (one for each shard) 3x ...
Sam's user avatar
  • 111
2 votes
0 answers
1k views

MongoDB Chunk Migration fails with "error 'aborted'"

I have a simple setup: 2 Mongos 3 Shards, deployed as a Replica Set 3 Config Servers, deployed as a Replica Set Each replica set consists of 1 Primary, 1 Secondary and 1 Arbiter. (Except the Config ...
Snowball's user avatar
  • 121
2 votes
1 answer
471 views

MONGOS uselessly scanning indexed documents when counting on shards

I have a simple collection with 40M documents distributed over 2 shards. The documents contain a field named COUNTRY with a sparse index on it. db.contacts.count({ "COUNTRY": "US" }) I am trying to ...
Julien's user avatar
  • 21
2 votes
0 answers
209 views

Choosing the correct shard key for MongoDB

I'm new to MongoDB. From the mongod documentation, I know that when choosing shard keys the key should always have high cardinality, so write operations are distributed to different shards (a kind of ...
siv's user avatar
  • 21
2 votes
1 answer
2k views

MongoDB Using String-Type Shard Key for Zone Ranges

Mongo lets any field type be used to set ranges for Shard Zones, but doesn't specify how analyses are conducted. My goal is to point queries and writes at the exact right server, WITHOUT using a ...
Gukki5's user avatar
  • 121
2 votes
1 answer
788 views

Huge logs are generated in mongo db shard logs

We are using Mongo db Shard set up. What i see that on primary of each shard log file keep growing to GBs. Even in a day it reaches to 10GB for single day. This appears to be slowing down Mongo ...
viren's user avatar
  • 511
2 votes
1 answer
1k views

How to properly use moveChunk when chunks of certain range needs to be moved?

A MongoDB 3.6.3 database has two shards and a mongodump file with partial data of one collection needs to be restored. The first shard is named "fast" while second "slow". The idea is to restore the ...
ssasa's user avatar
  • 241
2 votes
1 answer
524 views

MySQL InnoDB or MySQL NDB Cluster which one to use

We have an application with 8000 users accessing in parallel(max). We are trying to bring in horizontal scaling. I came across MySQL NDB cluster. Should I go with it? Just for auto sharding and other ...
Mani's user avatar
  • 121
2 votes
1 answer
757 views

Chunks not splitting on MongoDB sharded clusters

I have created 2 shard servers named rs1 and rs2. Now in the database collection there are multiple records those should split on these 2 servers in chunks. The sharding key is TxnMonth. It is not ...
Venus Garg's user avatar
2 votes
1 answer
5k views

Read concern level of majority error while upgrading Sharded Cluster from 3.2 to 3.4

I have upgraded MongodDB sharded cluster having two replica sets from 3.2 to 3.4. Current storage engine is MMAPv1. After successfully upgrading all the secondary, primary, config server and mongos to ...
umarxe's user avatar
  • 21
2 votes
0 answers
729 views

Adding a Mongo Shard decreases the insert performance

For a mobile application which will hold quite some documents, we've chosen Mongo as a database. Let me explain the situation first: Every day, I have a set of documents that needs to be inserted, ...
Complexity's user avatar
2 votes
0 answers
1k views

Expanding MariaDB cluster (sharding)

I have an expanding MariaDB 3 node cluster and need to expand the available storage. Each node has (to make the maths slightly easier) 120Gb of database storage. As this is replicated, the total DB ...
user3566845's user avatar
2 votes
0 answers
432 views

Database sharding, denormalization and synchronization

In sharding, it is advisable to follow a share nothing, shard everything approach. Essentially, this translates to: data that are to be used together are to be stored together (i.e. in a single shard)....
ahron's user avatar
  • 813
2 votes
0 answers
3k views

Error to connect mongos when trying to create replicated sharded cluster

I'm trying to create an replicated sharded cluster in mongodb. Initially I've created two shards and there are a replica set with three members in each shard. And all the shards and replicasets run in ...
Kaushick Gope's user avatar
2 votes
0 answers
127 views

Modulus calculation for compound primary keys in MySQL Cluster (and other sharded databases)

In MySQL Cluster, and generally in sharded databases, how is a compound primary key's modulus calculated? Say, for example, there are two tables that have primary keys like so: CREATE TABLE Orders (...
Juan Carlos Coto's user avatar
2 votes
0 answers
607 views

MongoDB Sharding and Replica Set Allocation

I am going to setup MongoDB cluster: 3 config servers, 3 routing servers, 2 shards (replica set per shard). A replica set is just simply 1 primary, 1 secondary and 1 arbiter. Therefore, in my ...
fmchan's user avatar
  • 121
2 votes
0 answers
211 views

how to start sharding in mongodb

I am trying to learn sharding in MongoDb and here is my plan: I'm trying to shard the database S001 into 2 different machines. 192.xx.xx.60>mongod -configsvr --dbpath S001 ..Now running in ...
Praveen's user avatar
  • 121
2 votes
1 answer
1k views

MongoDB cluster setup

I am trying to set up a 3 node MongoDB cluster. 1) Started mongodb in all 3 nodes with the below config file. net: bindIp: 0.0.0.0 port: 10901 setParameter: ...
user1578872's user avatar
1 vote
0 answers
36 views

Primary MongoD hit 1M active sessions in cache

Overview We're running a sharded MongoDB cluster, using community edition v4.4. The primary mongod in one of the data shards ('shardX' for short) has hit active sessions in cache limit of 1M and new ...
user3752071's user avatar
1 vote
0 answers
489 views

Multiple databases vs partitioning in PostgreSQL

Consider following database schema for PostgreSQL (I'm using 13.x.x) Users table - just place to store users and do authentication Projects - projects which are created by specific user Driver/Event/...
DocC's user avatar
  • 111
1 vote
1 answer
470 views

Thousands foreign keys referenced to one table and other postgres scaling philosophy

I'm inspecting one legacy and quite big database which is growing faster nowadays than before. Following query select count(*) from information_schema.tables where table_schema = 'public' returns a ...
danilabagroff's user avatar
1 vote
0 answers
466 views

how to maximize usage of number of databases on a postgres server while limited by connections

We're building a system where ideally we'd give a database per client in a multi-tenant setup. However, although a single Postgres install supports 4,294,950,911 databases on a server, the suggested ...
pyramation's user avatar
1 vote
2 answers
3k views

mongodb conf node won't start - compatibility error

I recently upgraded a dev sharded-cluster from 3.6 to 4.0 - one of the conf servers is now failing to start: 2020-01-30T13:19:02.972-0600 F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM:...
Micheal Shallop's user avatar
1 vote
1 answer
486 views

Schema changes on sharded database

I have performance issues with one particulary large table, 500+ Million rows, 300Gb data, Postgres 10.5. It is already partitioned. I am working on optimising it here and there, but that is not ...
k0ndaa's user avatar
  • 11
1 vote
2 answers
467 views

Sharding standalone MongoDB

I've got a standalone MongoDB server with a lot amount of data on it. Let's say, it's srv1 server. I've decided to make a sharded cluster, so I've created additional servers srv2 and srv3. We have: ...
Mikhail N.'s user avatar
1 vote
0 answers
233 views

How to scale MongoDB/GridFS for fast ingestion on limited timescale

I'm working on a cloud-agnostic service which must allow the ingestion of a lot of media files in a very limited timeframe. Some numbers: 3-4 hours total of workload: the service will be active ...
Federico Bonelli's user avatar
1 vote
1 answer
375 views

Sharded Mongodb stalls randomly

I have setup Sharded MongoDB cluster using hashed sharding in kuberenetes.I first created the config server Replicaset and then created 2 shard replicasets. Finally created mongos to connect to the ...
Prajwal M's user avatar
1 vote
2 answers
551 views

MongoDB specifying shard key and sharding strategy

What is the syntax to state that a collection is to be sharded as location based, while using the range based sharding strategy. I know you have to use the the sh.shardCollection() command, but what ...
tom dinh's user avatar
1 vote
0 answers
110 views

adding documents to shard collection results in incorrect amount

I have added 2 shards to my cluster: sh.addShard("127.0.0.1:27023") { "shardAdded" : "shard0000", "ok" : 1, "operationTime" : Timestamp(1552115819, 4), "$clusterTime" : { "...
tom dinh's user avatar
1 vote
0 answers
767 views

MongoDB ticket availability reaching zero

I recently came across intresting issue. I have a sharded cluster which has two replication sets with 2 secondaries. On my primaries, there were 300-400 connections at a point in time, shortly after ...
user173029's user avatar
1 vote
0 answers
71 views

One of the mongo shards consuming too much disk space.

We have mongo cluster with 10 shards of each having 3 replicas. Recently we are facing an issue, exceptionally in a couple of shards, the disk space among all the 3 replicas has reached above 90%. ...
kolli saisriharsha's user avatar
1 vote
0 answers
743 views

Sharding PostgreSQL with postgres_fdw (Foreign Data Wrapper)

We want to shard a single PostgreSQL 10.2 database by tenant (client id) to multiple servers. One of the easiest approach is to use Foreign Data Wrapper (postgres_fdw extension). It will looks like: ...
Philipp Bocharov's user avatar
1 vote
1 answer
268 views

MongoDB Shard distribution issue

I am facing an issue in the shard distribution. MongoDB Version: 3.0.15 I have a collection that is sharded. I have two shards. Output of db.getCollection('event').stats() { "sharded" : true, "...
user3138864's user avatar
1 vote
1 answer
220 views

MongoDB as a log storage. Choosing shard key

I'm designing a log storage system based on MongoDB. I want to shard a log collection to increase ingestion and capacity (distribute writes to several machines) while allow fast search. I should be ...
Philipp Bocharov's user avatar
1 vote
1 answer
846 views

MongoDB 3.6 create local Shard db without replica

This is what I want to do: Create a Config Server Create 2 Shard Create a Mongos Router I don't want to replicate my data using replica set, I want to simulate that each shard has only a part of ...
ow-me's user avatar
  • 11
1 vote
1 answer
592 views

Why findOne() hangs on a sharded collection?

A MongoDB v3.6.4 database with a large sharded collection and WiredTiger engine hangs when issuing a findOne() or find() command without any parameters. The collection has only the default index on ...
ssasa's user avatar
  • 241
1 vote
1 answer
2k views

Mongodb: Determining shard key strategy on compound index

I have a collection with 170 millions+ documents and it is only going to increase. The size of the collection is not that huge, currently around 70 GB. The collection has two fields indexed on: {...
Sameer's user avatar
  • 123
1 vote
0 answers
21 views

How do I control the data that goes into each data node in NDBCLUSTER

I am currently working with NDBCLUSTER. I have the following table(a dummy example, since I cannot disclose details about actual application, but the structure is similar): create table student( ...
cazaimi's user avatar
  • 11
1 vote
1 answer
125 views

MongoDB replicate data item across multiple shards

Is it possible to replicate a data item across multiple shards? My end goal is to determine how up-to-date the data in a replica slave is. I'm only concerned with subsets of the data, so I don't ...
Azmisov's user avatar
  • 111
1 vote
0 answers
396 views

Error adding replication set due to invalid election id and config version

We are in the process of moving our MongoDb replication sets to new hardware but is encountering a problem when adding our new replication set to the cluster. 2017-03-28T20:07:18.434+0200 W NETWORK [...
m__'s user avatar
  • 529
1 vote
1 answer
47 views

Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database?

Is it possible to delegate responsibility for specific indexes for a collection to specific shards in a mongoDB database? For instance, say I have a collection called 'books': { title: <string>...
Adé's user avatar
  • 119
1 vote
0 answers
323 views

MongoDB: removeShard when dbsToMove is an empty array

Background: Two empty shards with no explicit sharding scheme - shard0000 and shard0001. mongorestore a single database, all of which defaults to shard0000 Add a shard key to one collection and let ...
WAF's user avatar
  • 329
1 vote
0 answers
421 views

mongs failed to findone when collection is under sharding migration

I have a collection which enabled sharding on 2 replica sets, rs01 and rs02. I added a new replica set rs03, it started to migrating chunks to rs03 slowly(it only migrated about 300 chunks in 4 days......
CSJ's user avatar
  • 155
1 vote
0 answers
102 views

ConfigServersInconsistent:How to correct the inconsistency?

Detected that the one of the config servers is inconsistent with rest two, in a 3 config-server environment. But not sure how to set this right. I have gone through mongodb config servers not in sync ...
sreeraag's user avatar
  • 131
1 vote
0 answers
114 views

How does defining a MongoDB shard key influence the connectivity between my mongo(s/d)s?

I am trying to test mongoDB sharding, but when following the basic steps in common tutorials, I am facing a CONNECT_ERROR issue that really doesn't make sense to me. I use four machines for this test ...
Vongo's user avatar
  • 171
1 vote
0 answers
110 views

Securing High-Availability on Partitioned Database

I have a 90GB SQL 2012 database I am sharding over 3 database and I plan on growing as more clients are coming. A new customer came to us with a demand to know on we secure high-availability in our ...
Michael Jerrard's user avatar
1 vote
0 answers
481 views

Problems in sharding in mongodb

I am currently working on sharding. I have certain issues related to sharding : 1.If I load data again to a collection which is already sharded, the data stays only in one shard and not migrating to ...
Priya's user avatar
  • 139