Questions tagged [sharding]
Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.
310
questions
0
votes
0
answers
32
views
Getting a good shard key for mongodb for billions of records
At my company we have a mongo collection that has reached 120 million records that is currently sharded. Sadly and painfully, the chosen sharding strategy has been shown to be really bad: suffers both ...
0
votes
1
answer
39
views
Sharding in Timescaledb (Postgres) Opensource
If I am wrong in my understanding, please feel free to correct me
In my project, we have a timeseries database. It is setup as 3-node (One leader, 2 read-replicas) patroni cluster. Each node is an AWS ...
0
votes
1
answer
121
views
How to move partitions to other pg server?
I have a table with 1b rows and it continues growing. Currently, I use pg_partman for partitioning by month. There are a lot of select queries only for current or previous months. Is it possible to ...
- 103
0
votes
0
answers
27
views
Mysql sharding in depth
I have a table that looks like this. Learning sql recently,
USERID|MATCHID|TOURNAMENTID|SCORE|COUNTRY
There are different tournaments running day to day and each tournament has many matches. some ...
- 37
0
votes
1
answer
43
views
Sharded Cluster - Why does my data not splitted among nodes?
I created a cluster with two shards. Or so I thought. In my design, first server has mongos_router shard1 (3 nodes) and config servers. Second server only has shard 2 nodes. After I configured shards ...
0
votes
1
answer
53
views
Temporal service failed with "Persistent store operation failure", "Failed to update shard"
Hi i'm using Temporal service framework to orchestrate workflow process and cassandra database as datastore to record the workflow stateful process.
My temporal is getting error when trying to update ...
0
votes
0
answers
35
views
MongoDB movePrimary doesn't work when the primary shard is down
We have a sharded cluster with several collections sharded. There are two collections that cannot be sharded because there is a unique index on the collection. So these two collections are on the ...
- 101
0
votes
1
answer
31
views
MongoDB composite shard key including timestamp write behavior
I am investigating MongoDB's appropriateness for our use case. We have clients that generate log data, and most of our use cases for this DB involve querying by Client ID and a time range of when the ...
0
votes
1
answer
48
views
Postgres Sharding
As a hypothetical example, say I wanted to shard a PostgreSQL database for a multi-tenant setup. Each of the tenants has an ID that is stored as a foreign key in a column on every record of every ...
- 11
0
votes
2
answers
100
views
Storing terabytes of data in Azure SQL Server
I need to develop a Azure SQL server production system capable of storing a user item recommendations based on the user's purchase history, user health profile and item recommendations(around 415 MB). ...
- 103
0
votes
0
answers
24
views
Mongodb speed up new shards balancing
We have huge collections ( around 3.5 TB ) that is sharded with 2 shards on mongodb. Since the collection is growing and load is getting increased on existing shards we decided to add 2 new shards for ...
0
votes
1
answer
102
views
How to check mongodb time series collection shard distribution
[direct: mongos] test> db.weather.getShardDistribution()
MongoshInvalidInputError: [SHAPI-10001] Collection weather is not sharded
we are getting this error and how to check the sharded collection ...
santhosha MS
1
vote
0
answers
55
views
Primary MongoD hit 1M active sessions in cache
Overview
We're running a sharded MongoDB cluster, using community edition v4.4.
The primary mongod in one of the data shards ('shardX' for short) has hit active sessions in cache limit of 1M and new ...
- 11
0
votes
0
answers
118
views
when start the mongos for sharding Getting error
Quickly I giving the details, I have 4 Linux VMs with MongoDB installation. 1. Mongodb is for config so in Mongod.conf file enabled replication with shading like below
replication:
replSetName: "...
0
votes
0
answers
51
views
why mongodb does not create chunk for a sharded cluster?
I had a mongo instance running that was the only node in its replicaset. It had about 33G of data. Now I added 2 other shards each only one node in their replicaset and added them to cluster using db....
- 11
-2
votes
1
answer
33
views
Solution to query that must examine all rows / documents in worst case
The logic of my query goes like this:
Filter documents based on arbitrarily many and non-predetermined fields. The filter conditions can total to hundreds. But the worst case is that there are no ...
0
votes
2
answers
247
views
How many shards is Facebook's user table partitioned across?
Scaling services with Shard Manager on the Engineering at Meta site says:
Shard Manager manages tens of millions of shards hosted on hundreds of thousands of servers across hundreds of applications ...
0
votes
1
answer
355
views
NoSQL vs SQL during sharding when a single read operation only hits one shard
I've a use case where I'm stuck between figuring out whether to use SQL or NoSQL db.
The db has 2 fields - A (PK), B
A is 8 char-long, while B is 100-400 char long.
The operations are simply:
Write: ...
- 127
0
votes
0
answers
26
views
How to create zones in mongodb
i have deployed a shard cluster with 2 shard replica sets and i want shardA replica set primary in one zone and secondaries in another zone same way shardB replica set primary in one zone and ...
- 11
0
votes
1
answer
277
views
Shard queries timing out - Sharding status not available
Queries to one of our shards are regularly timing out with the following error:
“sharding status of collection db.collection is not currently available for description and needs to be recovered from ...
- 123
0
votes
1
answer
75
views
What strategy should I use for my gigant MYSQL Tables?
I'm working in a system that is using MySQL as primary DB.
One of the biggest tables is almost 1TB in size, others are around 100-300 GBs.
Queries started to be really slow.
I search some info, and I ...
- 109
0
votes
0
answers
26
views
how to build "mongos" server in mongodb for sharding?
I have searched "how to build a mongos server in mongodb" because I am building a simple sharding system locally. No I want to use the "addShard" function. It seems it would be the ...
- 1
0
votes
1
answer
277
views
MongoDB cluster Shard connection string suddenly misconfigured
A few hours ago, we had an outage in a server which was running a mongos instance. After resolving the underlying issue, the mongos has been unable to boot with the following error:
"ctx":&...
- 123
0
votes
0
answers
136
views
Accidentally drop of collection on shard of sharded MongoDB cluster
I have accidentally triggered a drop collection command on one replica set (primary-shard) of a MongoDB sharded cluster (no secondary shards). Unfortunately now I can not access the data of this shard ...
0
votes
0
answers
228
views
mongodb - composite shard key involving identifier and date
We have a collection of data which records the activities performed over a range of time period. the fields of importance are userid and created_at
many of the users are pretty important in our system ...
- 67
0
votes
1
answer
911
views
mongodb use _id field as shard key for time seires collection
I've successfully created and deployed a self-managed MongoDB cluster that consists of 3 shards,1 router(mongos), and 1 config-server replica set, using this tutorial.
I want to create a shareded ...
- 101
1
vote
0
answers
529
views
Multiple databases vs partitioning in PostgreSQL
Consider following database schema for PostgreSQL (I'm using 13.x.x)
Users table - just place to store users and do authentication
Projects - projects which are created by specific user
Driver/Event/...
- 111
0
votes
3
answers
863
views
MongoDB - query performance when sharding by _id
When we shard a collection by the MongoDB's _id field, we could achieve the most uniform distribution there is but the docs indicate query performance doesn't scale well due to the scatter gather ...
- 67
0
votes
0
answers
229
views
what causes sudded increase in size of WiredTigerHS.wt in mongodb5.0
I have mongodb 5.0.8 on linux
We run multiple clusters but I see that on one cluster , with 3 shards there is a simultaneous increase on all shard servers in WiredTigerHS.wt file at times in my ...
- 103
0
votes
1
answer
206
views
MongoDB: slow removeShard
I have a MongoDB cluster with 9 nodes (3 shards, 3 nodes each). I'm now removing one shard, but the process itself is running extremely slow. Each node of the shard being deleted stores ~400Gb of data,...
- 183
1
vote
1
answer
60
views
Storing and Querying User Defined Data
I'm working on a system where each user defines a strongly typed definition of data they can store, consisting of classes with specified fields. Embedded classes are also supported. My first instinct ...
- 111
0
votes
0
answers
366
views
Disadvantages of sharding PostgreSQL in 2022
I'm in the phase of choosing a database for a project, PostgreSQL seems to be the best choice.
I read many questions at stackoverflow and other websites about Sharding database, they were all about ...
- 77
2
votes
1
answer
337
views
Cannot connect to MongoDB router in sharded cluster
I followed this guide on how to setup a sharded MongoDB cluster with Docker containers.
It all seemed to work as it should because there were no error messages in the logs and the sh.status() command ...
- 131
0
votes
0
answers
73
views
If I have only 1 single type of record, but trillions of them, what is a sharding scheme that would scale?
Say I have 1 type of record, object, with title/slug/description fields, but I have trillions of them. Well, I start off with 1,000 records on day one, by end of year one I have 1M records, by year 2 ...
- 211
0
votes
1
answer
921
views
Is ObjectId as shard key unique across shards or just within a shard?
In my sharded cluster of version 4.2, one collections is sharded on _id where _id is just a default ObjectId.
I'm seeing multiple documents with same _id on different shards' primaries.
When I specify ...
0
votes
2
answers
357
views
Not able to shard collection
I have enabled sharding using the below commands.
mongos> use news
switched to db news
mongos> sh.enableSharding("news");
{
"ok" : 1,
"$clusterTime" : {
...
- 101
0
votes
0
answers
489
views
Enable username and password authentication on mongodb sharded cluster
I have a mongodb cluster(1 mongos, mongo config replica set with 2 members, 1 shard with 3 members) and I have enabled internal authentication in config replica set. Also my backend code connects to ...
0
votes
1
answer
134
views
MySQL Sharding with org id
I have a multi tenant application and each tenant will have an orgId. So, all tables will have orgid column.
Each org will have clients, projects, ...
Tables
org
client
project
Now, I am trying to ...
- 79
0
votes
1
answer
199
views
Sharding AWS MySQL data
I am working on a SaaS application using AWS MySQL.
The application is for multiple organizations. We use the same database and tables to store multiple org data. We store org_id in most of the tables ...
- 79
0
votes
1
answer
305
views
Zoned and hashed sharding of existing, non-empty collection
I am trying to shard a existing collection. Collection is pretty big (about 100gb). Sharding should be done by 2 fields. First one is zone ['Europe', 'America'] and second one is _id.
So my sharding ...
0
votes
0
answers
225
views
My Mongos can't authenticate to my replica set
I have 3 config servers in a replica set and when I use the command: sudo mongos --configdb crs/the 3 config servers:port# --keyfile "Keyfile",
I keep getting this error:
[mongosMain] Unable ...
0
votes
0
answers
391
views
Multi level Sharding in Cassandra
I am sorry if this is basic question. I am new to Cassandra and need help on a query.
Assume I want to host a service which accepts SurveyMonkey , now users are given a URL to take the survey and at ...
- 1
1
vote
1
answer
492
views
Thousands foreign keys referenced to one table and other postgres scaling philosophy
I'm inspecting one legacy and quite big database which is growing faster nowadays than before.
Following query
select count(*) from information_schema.tables where table_schema = 'public' returns a ...
- 131
1
vote
1
answer
940
views
Consistent Prefix Reads
In the book Designing Data-Intensive Applications, the author suggests that replication lag can lead to an anomaly were causality is violated. In the example that the author uses, the system is both ...
- 165
0
votes
1
answer
318
views
Can sharding speed up insertion performance in postgresql
Referenced from this question, the most popular answer does not mention sharding. As far as I know, postgresql will acquire auto-inc lock every time it inserts, so I think sharding can improve ...
- 101
0
votes
1
answer
148
views
Config enable sharding collection in MongoDB cluster
I'm a new in MongoDB cluster. I have a question about config enable shading collection in MongoDB:
I have a cluster :
Shard1: replical01/node1:27017, node2:27017
Shard2: replical02/node3:27017, ...
- 1
-1
votes
1
answer
398
views
Fault tolerance for Database sharding and Database partitioning
I'm aware that database sharding is splitting up of datasets horizontally into various database instances, whereas database partitioning uses one single instance.
In Database Sharding, what if one of ...
- 143
0
votes
1
answer
518
views
MongoDB multi-datacenter deployment and shard keys
I am trying to check my understanding after reading docs and some older questions, so my question is quite loaded:
In a sharded cluster, each shard should (and as of 3.6 must?) reside on its own ...
- 3
0
votes
1
answer
196
views
How to have different server config on replication server for postgres_fdw?
I have a postgres sharding based on postgres_fdw and postgres partitioning, lets say 3 machine (1 coordinator and 2 Shard machines). What I want to do is have read replica version of same? How should ...
- 93
0
votes
1
answer
569
views
Slow insert in Postgresql when sharding using declarative partitioning and postgres_fdw?
We have been trying to partition a Postgres database on google cloud using the built-in Postgres declarative partitioning and postgres_fdw as explained here.
We are running commands as follow:
Shard ...
- 93