Questions tagged [sharding]

Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.

Filter by
Sorted by
Tagged with
0 votes
0 answers
22 views

why mongodb does not create chunk for a sharded cluster?

I had a mongo instance running that was the only node in its replicaset. It had about 33G of data. Now I added 2 other shards each only one node in their replicaset and added them to cluster using db....
ssaf's user avatar
  • 1
-2 votes
1 answer
29 views

Solution to query that must examine all rows / documents in worst case

The logic of my query goes like this: Filter documents based on arbitrarily many and non-predetermined fields. The filter conditions can total to hundreds. But the worst case is that there are no ...
Bear Bile Farming is Torture's user avatar
-1 votes
2 answers
71 views

How many shards is Facebook's user table partitioned across?

Scaling services with Shard Manager on the Engineering at Meta site says: Shard Manager manages tens of millions of shards hosted on hundreds of thousands of servers across hundreds of applications ...
Bear Bile Farming is Torture's user avatar
0 votes
1 answer
42 views

NoSQL vs SQL during sharding when a single read operation only hits one shard

I've a use case where I'm stuck between figuring out whether to use SQL or NoSQL db. The db has 2 fields - A (PK), B A is 8 char-long, while B is 100-400 char long. The operations are simply: Write: ...
asn's user avatar
  • 117
0 votes
0 answers
12 views

How to create zones in mongodb

i have deployed a shard cluster with 2 shard replica sets and i want shardA replica set primary in one zone and secondaries in another zone same way shardB replica set primary in one zone and ...
Durga prasad's user avatar
0 votes
1 answer
48 views

Shard queries timing out - Sharding status not available

Queries to one of our shards are regularly timing out with the following error: “sharding status of collection db.collection is not currently available for description and needs to be recovered from ...
Thiatt's user avatar
  • 123
0 votes
1 answer
62 views

What strategy should I use for my gigant MYSQL Tables?

I'm working in a system that is using MySQL as primary DB. One of the biggest tables is almost 1TB in size, others are around 100-300 GBs. Queries started to be really slow. I search some info, and I ...
Danilo Bassi's user avatar
0 votes
0 answers
24 views

how to build "mongos" server in mongodb for sharding?

I have searched "how to build a mongos server in mongodb" because I am building a simple sharding system locally. No I want to use the "addShard" function. It seems it would be the ...
aoc2006's user avatar
0 votes
1 answer
105 views

MongoDB cluster Shard connection string suddenly misconfigured

A few hours ago, we had an outage in a server which was running a mongos instance. After resolving the underlying issue, the mongos has been unable to boot with the following error: "ctx":&...
Thiatt's user avatar
  • 123
0 votes
0 answers
45 views

mongodb is not distributing data evenly among shards

I have a database with 800k objects and I defined about 13 shard servers to access the data quickly. I assigned a letter to each object for use in the sharding process, for example, shards: 'a' for ...
BayGold's user avatar
0 votes
0 answers
120 views

Mongo Atlas conversion for Replica Set to Shard without downtime

Need to convert live mongo db atlas replica set to sharded. But seems it will have downtime. Because its changing structure of cluster. I tried with username/password and certificate based auth for ...
Yasar Arafath's user avatar
0 votes
0 answers
95 views

Accidentally drop of collection on shard of sharded MongoDB cluster

I have accidentally triggered a drop collection command on one replica set (primary-shard) of a MongoDB sharded cluster (no secondary shards). Unfortunately now I can not access the data of this shard ...
honigbrot's user avatar
0 votes
0 answers
175 views

mongodb - composite shard key involving identifier and date

We have a collection of data which records the activities performed over a range of time period. the fields of importance are userid and created_at many of the users are pretty important in our system ...
asgs's user avatar
  • 67
0 votes
1 answer
549 views

mongodb use _id field as shard key for time seires collection

I've successfully created and deployed a self-managed MongoDB cluster that consists of 3 shards,1 router(mongos), and 1 config-server replica set, using this tutorial. I want to create a shareded ...
Eliav Louski's user avatar
1 vote
0 answers
319 views

Multiple databases vs partitioning in PostgreSQL

Consider following database schema for PostgreSQL (I'm using 13.x.x) Users table - just place to store users and do authentication Projects - projects which are created by specific user Driver/Event/...
DocC's user avatar
  • 111
0 votes
3 answers
587 views

MongoDB - query performance when sharding by _id

When we shard a collection by the MongoDB's _id field, we could achieve the most uniform distribution there is but the docs indicate query performance doesn't scale well due to the scatter gather ...
asgs's user avatar
  • 67
0 votes
0 answers
134 views

what causes sudded increase in size of WiredTigerHS.wt in mongodb5.0

I have mongodb 5.0.8 on linux We run multiple clusters but I see that on one cluster , with 3 shards there is a simultaneous increase on all shard servers in WiredTigerHS.wt file at times in my ...
Ram's user avatar
  • 103
0 votes
1 answer
128 views

MongoDB: slow removeShard

I have a MongoDB cluster with 9 nodes (3 shards, 3 nodes each). I'm now removing one shard, but the process itself is running extremely slow. Each node of the shard being deleted stores ~400Gb of data,...
Pavel Sapezhko's user avatar
1 vote
1 answer
41 views

Storing and Querying User Defined Data

I'm working on a system where each user defines a strongly typed definition of data they can store, consisting of classes with specified fields. Embedded classes are also supported. My first instinct ...
I_A's user avatar
  • 111
0 votes
0 answers
316 views

Disadvantages of sharding PostgreSQL in 2022

I'm in the phase of choosing a database for a project, PostgreSQL seems to be the best choice. I read many questions at stackoverflow and other websites about Sharding database, they were all about ...
Jacqueline's user avatar
1 vote
1 answer
268 views

Cannot connect to MongoDB router in sharded cluster

I followed this guide on how to setup a sharded MongoDB cluster with Docker containers. It all seemed to work as it should because there were no error messages in the logs and the sh.status() command ...
Tobias S.'s user avatar
  • 111
0 votes
0 answers
64 views

If I have only 1 single type of record, but trillions of them, what is a sharding scheme that would scale?

Say I have 1 type of record, object, with title/slug/description fields, but I have trillions of them. Well, I start off with 1,000 records on day one, by end of year one I have 1M records, by year 2 ...
Lance's user avatar
  • 187
0 votes
1 answer
668 views

Is ObjectId as shard key unique across shards or just within a shard?

In my sharded cluster of version 4.2, one collections is sharded on _id where _id is just a default ObjectId. I'm seeing multiple documents with same _id on different shards' primaries. When I specify ...
corgicorgi's user avatar
0 votes
2 answers
258 views

Not able to shard collection

I have enabled sharding using the below commands. mongos> use news switched to db news mongos> sh.enableSharding("news"); { "ok" : 1, "$clusterTime" : { ...
CrazyCoder's user avatar
0 votes
0 answers
419 views

Enable username and password authentication on mongodb sharded cluster

I have a mongodb cluster(1 mongos, mongo config replica set with 2 members, 1 shard with 3 members) and I have enabled internal authentication in config replica set. Also my backend code connects to ...
Mohammad Hossein Shojaeinia's user avatar
0 votes
1 answer
99 views

MySQL Sharding with org id

I have a multi tenant application and each tenant will have an orgId. So, all tables will have orgid column. Each org will have clients, projects, ... Tables org client project Now, I am trying to ...
user1578872's user avatar
0 votes
1 answer
168 views

Sharding AWS MySQL data

I am working on a SaaS application using AWS MySQL. The application is for multiple organizations. We use the same database and tables to store multiple org data. We store org_id in most of the tables ...
user1578872's user avatar
0 votes
1 answer
231 views

Zoned and hashed sharding of existing, non-empty collection

I am trying to shard a existing collection. Collection is pretty big (about 100gb). Sharding should be done by 2 fields. First one is zone ['Europe', 'America'] and second one is _id. So my sharding ...
Denino1234's user avatar
0 votes
0 answers
198 views

My Mongos can't authenticate to my replica set

I have 3 config servers in a replica set and when I use the command: sudo mongos --configdb crs/the 3 config servers:port# --keyfile "Keyfile", I keep getting this error: [mongosMain] Unable ...
Awesomekingz Liang's user avatar
0 votes
0 answers
318 views

Multi level Sharding in Cassandra

I am sorry if this is basic question. I am new to Cassandra and need help on a query. Assume I want to host a service which accepts SurveyMonkey , now users are given a URL to take the survey and at ...
Zoro99's user avatar
  • 1
1 vote
1 answer
402 views

Thousands foreign keys referenced to one table and other postgres scaling philosophy

I'm inspecting one legacy and quite big database which is growing faster nowadays than before. Following query select count(*) from information_schema.tables where table_schema = 'public' returns a ...
danilabagroff's user avatar
1 vote
1 answer
685 views

Consistent Prefix Reads

In the book Designing Data-Intensive Applications, the author suggests that replication lag can lead to an anomaly were causality is violated. In the example that the author uses, the system is both ...
Steven L.'s user avatar
  • 165
0 votes
1 answer
202 views

Can sharding speed up insertion performance in postgresql

Referenced from this question, the most popular answer does not mention sharding. As far as I know, postgresql will acquire auto-inc lock every time it inserts, so I think sharding can improve ...
spike 王建's user avatar
0 votes
1 answer
98 views

Config enable sharding collection in MongoDB cluster

I'm a new in MongoDB cluster. I have a question about config enable shading collection in MongoDB: I have a cluster : Shard1: replical01/node1:27017, node2:27017 Shard2: replical02/node3:27017, ...
muon.ha's user avatar
-1 votes
1 answer
309 views

Fault tolerance for Database sharding and Database partitioning

I'm aware that database sharding is splitting up of datasets horizontally into various database instances, whereas database partitioning uses one single instance. In Database Sharding, what if one of ...
MrRobot9's user avatar
  • 143
0 votes
1 answer
466 views

MongoDB multi-datacenter deployment and shard keys

I am trying to check my understanding after reading docs and some older questions, so my question is quite loaded: In a sharded cluster, each shard should (and as of 3.6 must?) reside on its own ...
kot's user avatar
  • 3
0 votes
1 answer
139 views

How to have different server config on replication server for postgres_fdw?

I have a postgres sharding based on postgres_fdw and postgres partitioning, lets say 3 machine (1 coordinator and 2 Shard machines). What I want to do is have read replica version of same? How should ...
hardik24's user avatar
0 votes
1 answer
480 views

Slow insert in Postgresql when sharding using declarative partitioning and postgres_fdw?

We have been trying to partition a Postgres database on google cloud using the built-in Postgres declarative partitioning and postgres_fdw as explained here. We are running commands as follow: Shard ...
hardik24's user avatar
2 votes
1 answer
358 views

How to implement High Availability for all nodes in a sharded database?

I'm doing some research into implementing sharding for one of my large (~2 TB) production databases. Currently the database has HADR via a multi-replica AlwaysOn Availability Group. Since sharding a ...
J.D.'s user avatar
  • 34k
1 vote
0 answers
364 views

how to maximize usage of number of databases on a postgres server while limited by connections

We're building a system where ideally we'd give a database per client in a multi-tenant setup. However, although a single Postgres install supports 4,294,950,911 databases on a server, the suggested ...
pyramation's user avatar
0 votes
0 answers
29 views

Mongodb replication breaks if balancer started

have a 10 shard ( with 3 replicaset) cluster with very heavy traffic. Storage almost 1.5TB per shard in 2000 databases MongoDB server version is 4.0.3 Since these servers were added at different ...
Ram's user avatar
  • 103
0 votes
0 answers
46 views

Result rows duplicated by spider engine

I've two mysql server 'G0' and 'G1'. And there is a one row test table on them respectively. G0: mysql> select * from test.t1; +----+--------+ | id | node | +----+--------+ | 1 | DB-G0 | +----...
ASBai's user avatar
  • 103
1 vote
1 answer
248 views

Chunk size bigger than configured

We have a sharded mongo cluster. We have configured the chunk size to be 256M but they are ~1.5G each. There are no jumbo chunks as our shard key is well balanced (we have also checked for them ...
khajvah's user avatar
  • 111
0 votes
0 answers
102 views

Distribute data between MongoDb Shards

I have a development environment with two shards, one configuration server and one router as illustrated below. I have a sharded database "mydb2" and a sharded colleciton "mycollection2". When I ...
abdllhcay's user avatar
0 votes
2 answers
3k views

mongodb conf node won't start - compatibility error

I recently upgraded a dev sharded-cluster from 3.6 to 4.0 - one of the conf servers is now failing to start: 2020-01-30T13:19:02.972-0600 F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM:...
Micheal Shallop's user avatar
0 votes
2 answers
125 views

MongoDB Shard Zone Range Overlapping

I am configuring a test mongo setup with 2 shards, and I'm trying to add zone ranges to each shard: sh.updateZoneKeyRange('mydb.test', {id:MinKey,ts:MinKey}, {id:MaxKey,ts:1548787704000}, 'cold') sh....
Thiatt's user avatar
  • 123
0 votes
0 answers
390 views

Is there any way to move expired data from one collection to another periodly?

We have a very big collection, and for getting a better performance, we want to limit the size of "active" collection. That means, we have two collection: "active" and "inactive", we need to keep the ...
Woods Chen's user avatar
0 votes
1 answer
331 views

Ranged Sharding - Logic behind forming ranges for hexadecimal string

I have a sharded collection where the shard key is a field called "uuid". This field's value is of type string and represents hexadecimal values i.e a hexadecimal string. For each document this "uuid" ...
Raghu's user avatar
  • 103
2 votes
0 answers
1k views

MongoDB Chunk Migration fails with "error 'aborted'"

I have a simple setup: 2 Mongos 3 Shards, deployed as a Replica Set 3 Config Servers, deployed as a Replica Set Each replica set consists of 1 Primary, 1 Secondary and 1 Arbiter. (Except the Config ...
Snowball's user avatar
  • 121
0 votes
1 answer
835 views

FDW use in postgres

I need to create a sequence and need to use the sequence for auto increment id for more than 2 databases on 2 separate servers. I'm referring below doc for that(for test purpose): https://paquier.xyz/...
Channa's user avatar
  • 348

1
2 3 4 5 6