Questions tagged [sharding]
Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.
300
questions
0
votes
0
answers
22
views
why mongodb does not create chunk for a sharded cluster?
I had a mongo instance running that was the only node in its replicaset. It had about 33G of data. Now I added 2 other shards each only one node in their replicaset and added them to cluster using db....
- 1
-2
votes
1
answer
29
views
Solution to query that must examine all rows / documents in worst case
The logic of my query goes like this:
Filter documents based on arbitrarily many and non-predetermined fields. The filter conditions can total to hundreds. But the worst case is that there are no ...
-1
votes
2
answers
71
views
How many shards is Facebook's user table partitioned across?
Scaling services with Shard Manager on the Engineering at Meta site says:
Shard Manager manages tens of millions of shards hosted on hundreds of thousands of servers across hundreds of applications ...
0
votes
1
answer
42
views
NoSQL vs SQL during sharding when a single read operation only hits one shard
I've a use case where I'm stuck between figuring out whether to use SQL or NoSQL db.
The db has 2 fields - A (PK), B
A is 8 char-long, while B is 100-400 char long.
The operations are simply:
Write: ...
- 117
0
votes
0
answers
12
views
How to create zones in mongodb
i have deployed a shard cluster with 2 shard replica sets and i want shardA replica set primary in one zone and secondaries in another zone same way shardB replica set primary in one zone and ...
- 11
0
votes
1
answer
48
views
Shard queries timing out - Sharding status not available
Queries to one of our shards are regularly timing out with the following error:
“sharding status of collection db.collection is not currently available for description and needs to be recovered from ...
- 123
0
votes
1
answer
62
views
What strategy should I use for my gigant MYSQL Tables?
I'm working in a system that is using MySQL as primary DB.
One of the biggest tables is almost 1TB in size, others are around 100-300 GBs.
Queries started to be really slow.
I search some info, and I ...
0
votes
0
answers
24
views
how to build "mongos" server in mongodb for sharding?
I have searched "how to build a mongos server in mongodb" because I am building a simple sharding system locally. No I want to use the "addShard" function. It seems it would be the ...
- 1
0
votes
1
answer
105
views
MongoDB cluster Shard connection string suddenly misconfigured
A few hours ago, we had an outage in a server which was running a mongos instance. After resolving the underlying issue, the mongos has been unable to boot with the following error:
"ctx":&...
- 123
0
votes
0
answers
45
views
mongodb is not distributing data evenly among shards
I have a database with 800k objects and I defined about 13 shard servers to access the data quickly. I assigned a letter to each object for use in the sharding process, for example, shards: 'a' for ...
- 1
0
votes
0
answers
120
views
Mongo Atlas conversion for Replica Set to Shard without downtime
Need to convert live mongo db atlas replica set to sharded. But seems it will have downtime. Because its changing structure of cluster.
I tried with username/password and certificate based auth for ...
- 101
0
votes
0
answers
95
views
Accidentally drop of collection on shard of sharded MongoDB cluster
I have accidentally triggered a drop collection command on one replica set (primary-shard) of a MongoDB sharded cluster (no secondary shards). Unfortunately now I can not access the data of this shard ...
0
votes
0
answers
175
views
mongodb - composite shard key involving identifier and date
We have a collection of data which records the activities performed over a range of time period. the fields of importance are userid and created_at
many of the users are pretty important in our system ...
- 67
0
votes
1
answer
549
views
mongodb use _id field as shard key for time seires collection
I've successfully created and deployed a self-managed MongoDB cluster that consists of 3 shards,1 router(mongos), and 1 config-server replica set, using this tutorial.
I want to create a shareded ...
- 101
1
vote
0
answers
319
views
Multiple databases vs partitioning in PostgreSQL
Consider following database schema for PostgreSQL (I'm using 13.x.x)
Users table - just place to store users and do authentication
Projects - projects which are created by specific user
Driver/Event/...
- 111
0
votes
3
answers
587
views
MongoDB - query performance when sharding by _id
When we shard a collection by the MongoDB's _id field, we could achieve the most uniform distribution there is but the docs indicate query performance doesn't scale well due to the scatter gather ...
- 67
0
votes
0
answers
134
views
what causes sudded increase in size of WiredTigerHS.wt in mongodb5.0
I have mongodb 5.0.8 on linux
We run multiple clusters but I see that on one cluster , with 3 shards there is a simultaneous increase on all shard servers in WiredTigerHS.wt file at times in my ...
- 103
0
votes
1
answer
128
views
MongoDB: slow removeShard
I have a MongoDB cluster with 9 nodes (3 shards, 3 nodes each). I'm now removing one shard, but the process itself is running extremely slow. Each node of the shard being deleted stores ~400Gb of data,...
- 183
1
vote
1
answer
41
views
Storing and Querying User Defined Data
I'm working on a system where each user defines a strongly typed definition of data they can store, consisting of classes with specified fields. Embedded classes are also supported. My first instinct ...
- 111
0
votes
0
answers
316
views
Disadvantages of sharding PostgreSQL in 2022
I'm in the phase of choosing a database for a project, PostgreSQL seems to be the best choice.
I read many questions at stackoverflow and other websites about Sharding database, they were all about ...
- 57
1
vote
1
answer
268
views
Cannot connect to MongoDB router in sharded cluster
I followed this guide on how to setup a sharded MongoDB cluster with Docker containers.
It all seemed to work as it should because there were no error messages in the logs and the sh.status() command ...
- 111
0
votes
0
answers
64
views
If I have only 1 single type of record, but trillions of them, what is a sharding scheme that would scale?
Say I have 1 type of record, object, with title/slug/description fields, but I have trillions of them. Well, I start off with 1,000 records on day one, by end of year one I have 1M records, by year 2 ...
- 187
0
votes
1
answer
668
views
Is ObjectId as shard key unique across shards or just within a shard?
In my sharded cluster of version 4.2, one collections is sharded on _id where _id is just a default ObjectId.
I'm seeing multiple documents with same _id on different shards' primaries.
When I specify ...
0
votes
2
answers
258
views
Not able to shard collection
I have enabled sharding using the below commands.
mongos> use news
switched to db news
mongos> sh.enableSharding("news");
{
"ok" : 1,
"$clusterTime" : {
...
- 101
0
votes
0
answers
419
views
Enable username and password authentication on mongodb sharded cluster
I have a mongodb cluster(1 mongos, mongo config replica set with 2 members, 1 shard with 3 members) and I have enabled internal authentication in config replica set. Also my backend code connects to ...
0
votes
1
answer
99
views
MySQL Sharding with org id
I have a multi tenant application and each tenant will have an orgId. So, all tables will have orgid column.
Each org will have clients, projects, ...
Tables
org
client
project
Now, I am trying to ...
- 69
0
votes
1
answer
168
views
Sharding AWS MySQL data
I am working on a SaaS application using AWS MySQL.
The application is for multiple organizations. We use the same database and tables to store multiple org data. We store org_id in most of the tables ...
- 69
0
votes
1
answer
231
views
Zoned and hashed sharding of existing, non-empty collection
I am trying to shard a existing collection. Collection is pretty big (about 100gb). Sharding should be done by 2 fields. First one is zone ['Europe', 'America'] and second one is _id.
So my sharding ...
0
votes
0
answers
198
views
My Mongos can't authenticate to my replica set
I have 3 config servers in a replica set and when I use the command: sudo mongos --configdb crs/the 3 config servers:port# --keyfile "Keyfile",
I keep getting this error:
[mongosMain] Unable ...
0
votes
0
answers
318
views
Multi level Sharding in Cassandra
I am sorry if this is basic question. I am new to Cassandra and need help on a query.
Assume I want to host a service which accepts SurveyMonkey , now users are given a URL to take the survey and at ...
- 1
1
vote
1
answer
402
views
Thousands foreign keys referenced to one table and other postgres scaling philosophy
I'm inspecting one legacy and quite big database which is growing faster nowadays than before.
Following query
select count(*) from information_schema.tables where table_schema = 'public' returns a ...
- 131
1
vote
1
answer
685
views
Consistent Prefix Reads
In the book Designing Data-Intensive Applications, the author suggests that replication lag can lead to an anomaly were causality is violated. In the example that the author uses, the system is both ...
- 165
0
votes
1
answer
202
views
Can sharding speed up insertion performance in postgresql
Referenced from this question, the most popular answer does not mention sharding. As far as I know, postgresql will acquire auto-inc lock every time it inserts, so I think sharding can improve ...
- 101
0
votes
1
answer
98
views
Config enable sharding collection in MongoDB cluster
I'm a new in MongoDB cluster. I have a question about config enable shading collection in MongoDB:
I have a cluster :
Shard1: replical01/node1:27017, node2:27017
Shard2: replical02/node3:27017, ...
- 1
-1
votes
1
answer
309
views
Fault tolerance for Database sharding and Database partitioning
I'm aware that database sharding is splitting up of datasets horizontally into various database instances, whereas database partitioning uses one single instance.
In Database Sharding, what if one of ...
- 143
0
votes
1
answer
466
views
MongoDB multi-datacenter deployment and shard keys
I am trying to check my understanding after reading docs and some older questions, so my question is quite loaded:
In a sharded cluster, each shard should (and as of 3.6 must?) reside on its own ...
- 3
0
votes
1
answer
139
views
How to have different server config on replication server for postgres_fdw?
I have a postgres sharding based on postgres_fdw and postgres partitioning, lets say 3 machine (1 coordinator and 2 Shard machines). What I want to do is have read replica version of same? How should ...
- 93
0
votes
1
answer
480
views
Slow insert in Postgresql when sharding using declarative partitioning and postgres_fdw?
We have been trying to partition a Postgres database on google cloud using the built-in Postgres declarative partitioning and postgres_fdw as explained here.
We are running commands as follow:
Shard ...
- 93
2
votes
1
answer
358
views
How to implement High Availability for all nodes in a sharded database?
I'm doing some research into implementing sharding for one of my large (~2 TB) production databases.
Currently the database has HADR via a multi-replica AlwaysOn Availability Group.
Since sharding a ...
- 34k
1
vote
0
answers
364
views
how to maximize usage of number of databases on a postgres server while limited by connections
We're building a system where ideally we'd give a database per client in a multi-tenant setup.
However, although a single Postgres install supports 4,294,950,911 databases on a server, the suggested ...
- 115
0
votes
0
answers
29
views
Mongodb replication breaks if balancer started
have a 10 shard ( with 3 replicaset) cluster with very heavy traffic. Storage almost 1.5TB per shard in 2000 databases
MongoDB server version is 4.0.3
Since these servers were added at different ...
- 103
0
votes
0
answers
46
views
Result rows duplicated by spider engine
I've two mysql server 'G0' and 'G1'. And there is a one row test table on them respectively.
G0:
mysql> select * from test.t1;
+----+--------+
| id | node |
+----+--------+
| 1 | DB-G0 |
+----...
- 103
1
vote
1
answer
248
views
Chunk size bigger than configured
We have a sharded mongo cluster. We have configured the chunk size to be 256M but they are ~1.5G each. There are no jumbo chunks as our shard key is well balanced (we have also checked for them ...
- 111
0
votes
0
answers
102
views
Distribute data between MongoDb Shards
I have a development environment with two shards, one configuration server and one router as illustrated below. I have a sharded database "mydb2" and a sharded colleciton "mycollection2". When I ...
0
votes
2
answers
3k
views
mongodb conf node won't start - compatibility error
I recently upgraded a dev sharded-cluster from 3.6 to 4.0 - one of the conf servers is now failing to start:
2020-01-30T13:19:02.972-0600 F CONTROL [initandlisten] ** IMPORTANT:
UPGRADE PROBLEM:...
- 101
0
votes
2
answers
125
views
MongoDB Shard Zone Range Overlapping
I am configuring a test mongo setup with 2 shards, and I'm trying to add zone ranges to each shard:
sh.updateZoneKeyRange('mydb.test', {id:MinKey,ts:MinKey}, {id:MaxKey,ts:1548787704000}, 'cold')
sh....
- 123
0
votes
0
answers
390
views
Is there any way to move expired data from one collection to another periodly?
We have a very big collection, and for getting a better performance, we want to limit the size of "active" collection.
That means, we have two collection: "active" and "inactive", we need to keep the ...
- 111
0
votes
1
answer
331
views
Ranged Sharding - Logic behind forming ranges for hexadecimal string
I have a sharded collection where the shard key is a field called "uuid". This field's value is of type string and represents hexadecimal values i.e a hexadecimal string. For each document this "uuid" ...
- 103
2
votes
0
answers
1k
views
MongoDB Chunk Migration fails with "error 'aborted'"
I have a simple setup:
2 Mongos
3 Shards, deployed as a Replica Set
3 Config Servers, deployed as a Replica Set
Each replica set consists of 1 Primary, 1 Secondary and 1 Arbiter. (Except the Config ...
- 121
0
votes
1
answer
835
views
FDW use in postgres
I need to create a sequence and need to use the sequence for auto increment id for more than 2 databases on 2 separate servers. I'm referring below doc for that(for test purpose): https://paquier.xyz/...
- 348