Skip to main content

Questions tagged [sharding]

Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.

Filter by
Sorted by
Tagged with
0 votes
1 answer
56 views

How to move partitions to other pg server?

I have a table with 1b rows and it continues growing. Currently, I use pg_partman for partitioning by month. There are a lot of select queries only for current or previous months. Is it possible to ...
Roman N's user avatar
  • 103
0 votes
0 answers
23 views

Mysql sharding in depth

I have a table that looks like this. Learning sql recently, USERID|MATCHID|TOURNAMENTID|SCORE|COUNTRY There are different tournaments running day to day and each tournament has many matches. some ...
umarkaa's user avatar
  • 37
0 votes
1 answer
32 views

Sharded Cluster - Why does my data not splitted among nodes?

I created a cluster with two shards. Or so I thought. In my design, first server has mongos_router shard1 (3 nodes) and config servers. Second server only has shard 2 nodes. After I configured shards ...
Furkan YIlmaZ's user avatar
0 votes
1 answer
27 views

Temporal service failed with "Persistent store operation failure", "Failed to update shard"

Hi i'm using Temporal service framework to orchestrate workflow process and cassandra database as datastore to record the workflow stateful process. My temporal is getting error when trying to update ...
Ahmad Ricky Nazarrudin's user avatar
0 votes
0 answers
27 views

MongoDB movePrimary doesn't work when the primary shard is down

We have a sharded cluster with several collections sharded. There are two collections that cannot be sharded because there is a unique index on the collection. So these two collections are on the ...
user2704732's user avatar
0 votes
1 answer
15 views

MongoDB composite shard key including timestamp write behavior

I am investigating MongoDB's appropriateness for our use case. We have clients that generate log data, and most of our use cases for this DB involve querying by Client ID and a time range of when the ...
threewordphrase's user avatar
0 votes
1 answer
37 views

Postgres Sharding

As a hypothetical example, say I wanted to shard a PostgreSQL database for a multi-tenant setup. Each of the tenants has an ID that is stored as a foreign key in a column on every record of every ...
lucifer34's user avatar
0 votes
2 answers
71 views

Storing terabytes of data in Azure SQL Server

I need to develop a Azure SQL server production system capable of storing a user item recommendations based on the user's purchase history, user health profile and item recommendations(around 415 MB). ...
Ajit Goel's user avatar
  • 103
0 votes
0 answers
22 views

Mongodb speed up new shards balancing

We have huge collections ( around 3.5 TB ) that is sharded with 2 shards on mongodb. Since the collection is growing and load is getting increased on existing shards we decided to add 2 new shards for ...
Yulian Oifa's user avatar
0 votes
1 answer
70 views

How to check mongodb time series collection shard distribution

[direct: mongos] test> db.weather.getShardDistribution() MongoshInvalidInputError: [SHAPI-10001] Collection weather is not sharded we are getting this error and how to check the sharded collection ...
user avatar
1 vote
0 answers
37 views

Primary MongoD hit 1M active sessions in cache

Overview We're running a sharded MongoDB cluster, using community edition v4.4. The primary mongod in one of the data shards ('shardX' for short) has hit active sessions in cache limit of 1M and new ...
user3752071's user avatar
0 votes
0 answers
67 views

when start the mongos for sharding Getting error

Quickly I giving the details, I have 4 Linux VMs with MongoDB installation. 1. Mongodb is for config so in Mongod.conf file enabled replication with shading like below replication: replSetName: "...
Aravind rajamani's user avatar
0 votes
0 answers
45 views

why mongodb does not create chunk for a sharded cluster?

I had a mongo instance running that was the only node in its replicaset. It had about 33G of data. Now I added 2 other shards each only one node in their replicaset and added them to cluster using db....
Ssaf's user avatar
  • 11
-2 votes
1 answer
33 views

Solution to query that must examine all rows / documents in worst case

The logic of my query goes like this: Filter documents based on arbitrarily many and non-predetermined fields. The filter conditions can total to hundreds. But the worst case is that there are no ...
Bear Bile Farming is Torture's user avatar
-1 votes
2 answers
183 views

How many shards is Facebook's user table partitioned across?

Scaling services with Shard Manager on the Engineering at Meta site says: Shard Manager manages tens of millions of shards hosted on hundreds of thousands of servers across hundreds of applications ...
Bear Bile Farming is Torture's user avatar
0 votes
1 answer
280 views

NoSQL vs SQL during sharding when a single read operation only hits one shard

I've a use case where I'm stuck between figuring out whether to use SQL or NoSQL db. The db has 2 fields - A (PK), B A is 8 char-long, while B is 100-400 char long. The operations are simply: Write: ...
asn's user avatar
  • 127
0 votes
0 answers
24 views

How to create zones in mongodb

i have deployed a shard cluster with 2 shard replica sets and i want shardA replica set primary in one zone and secondaries in another zone same way shardB replica set primary in one zone and ...
Durga prasad's user avatar
0 votes
1 answer
222 views

Shard queries timing out - Sharding status not available

Queries to one of our shards are regularly timing out with the following error: “sharding status of collection db.collection is not currently available for description and needs to be recovered from ...
Thiatt's user avatar
  • 123
0 votes
1 answer
68 views

What strategy should I use for my gigant MYSQL Tables?

I'm working in a system that is using MySQL as primary DB. One of the biggest tables is almost 1TB in size, others are around 100-300 GBs. Queries started to be really slow. I search some info, and I ...
Danilo Bassi's user avatar
0 votes
0 answers
25 views

how to build "mongos" server in mongodb for sharding?

I have searched "how to build a mongos server in mongodb" because I am building a simple sharding system locally. No I want to use the "addShard" function. It seems it would be the ...
aoc2006's user avatar
0 votes
1 answer
232 views

MongoDB cluster Shard connection string suddenly misconfigured

A few hours ago, we had an outage in a server which was running a mongos instance. After resolving the underlying issue, the mongos has been unable to boot with the following error: "ctx":&...
Thiatt's user avatar
  • 123
0 votes
0 answers
124 views

Accidentally drop of collection on shard of sharded MongoDB cluster

I have accidentally triggered a drop collection command on one replica set (primary-shard) of a MongoDB sharded cluster (no secondary shards). Unfortunately now I can not access the data of this shard ...
honigbrot's user avatar
0 votes
0 answers
219 views

mongodb - composite shard key involving identifier and date

We have a collection of data which records the activities performed over a range of time period. the fields of importance are userid and created_at many of the users are pretty important in our system ...
asgs's user avatar
  • 67
0 votes
1 answer
852 views

mongodb use _id field as shard key for time seires collection

I've successfully created and deployed a self-managed MongoDB cluster that consists of 3 shards,1 router(mongos), and 1 config-server replica set, using this tutorial. I want to create a shareded ...
Eliav Louski's user avatar
1 vote
0 answers
490 views

Multiple databases vs partitioning in PostgreSQL

Consider following database schema for PostgreSQL (I'm using 13.x.x) Users table - just place to store users and do authentication Projects - projects which are created by specific user Driver/Event/...
DocC's user avatar
  • 111
0 votes
3 answers
815 views

MongoDB - query performance when sharding by _id

When we shard a collection by the MongoDB's _id field, we could achieve the most uniform distribution there is but the docs indicate query performance doesn't scale well due to the scatter gather ...
asgs's user avatar
  • 67
0 votes
0 answers
210 views

what causes sudded increase in size of WiredTigerHS.wt in mongodb5.0

I have mongodb 5.0.8 on linux We run multiple clusters but I see that on one cluster , with 3 shards there is a simultaneous increase on all shard servers in WiredTigerHS.wt file at times in my ...
Ram's user avatar
  • 103
0 votes
1 answer
194 views

MongoDB: slow removeShard

I have a MongoDB cluster with 9 nodes (3 shards, 3 nodes each). I'm now removing one shard, but the process itself is running extremely slow. Each node of the shard being deleted stores ~400Gb of data,...
Pavel Sapezhko's user avatar
1 vote
1 answer
55 views

Storing and Querying User Defined Data

I'm working on a system where each user defines a strongly typed definition of data they can store, consisting of classes with specified fields. Embedded classes are also supported. My first instinct ...
I_A's user avatar
  • 111
0 votes
0 answers
355 views

Disadvantages of sharding PostgreSQL in 2022

I'm in the phase of choosing a database for a project, PostgreSQL seems to be the best choice. I read many questions at stackoverflow and other websites about Sharding database, they were all about ...
Jacqueline's user avatar
2 votes
1 answer
333 views

Cannot connect to MongoDB router in sharded cluster

I followed this guide on how to setup a sharded MongoDB cluster with Docker containers. It all seemed to work as it should because there were no error messages in the logs and the sh.status() command ...
Tobias S.'s user avatar
  • 131
0 votes
0 answers
72 views

If I have only 1 single type of record, but trillions of them, what is a sharding scheme that would scale?

Say I have 1 type of record, object, with title/slug/description fields, but I have trillions of them. Well, I start off with 1,000 records on day one, by end of year one I have 1M records, by year 2 ...
Lance's user avatar
  • 209
0 votes
1 answer
879 views

Is ObjectId as shard key unique across shards or just within a shard?

In my sharded cluster of version 4.2, one collections is sharded on _id where _id is just a default ObjectId. I'm seeing multiple documents with same _id on different shards' primaries. When I specify ...
corgicorgi's user avatar
0 votes
2 answers
341 views

Not able to shard collection

I have enabled sharding using the below commands. mongos> use news switched to db news mongos> sh.enableSharding("news"); { "ok" : 1, "$clusterTime" : { ...
CrazyCoder's user avatar
0 votes
0 answers
473 views

Enable username and password authentication on mongodb sharded cluster

I have a mongodb cluster(1 mongos, mongo config replica set with 2 members, 1 shard with 3 members) and I have enabled internal authentication in config replica set. Also my backend code connects to ...
Mohammad Hossein Shojaeinia's user avatar
0 votes
1 answer
127 views

MySQL Sharding with org id

I have a multi tenant application and each tenant will have an orgId. So, all tables will have orgid column. Each org will have clients, projects, ... Tables org client project Now, I am trying to ...
user1578872's user avatar
0 votes
1 answer
190 views

Sharding AWS MySQL data

I am working on a SaaS application using AWS MySQL. The application is for multiple organizations. We use the same database and tables to store multiple org data. We store org_id in most of the tables ...
user1578872's user avatar
0 votes
1 answer
298 views

Zoned and hashed sharding of existing, non-empty collection

I am trying to shard a existing collection. Collection is pretty big (about 100gb). Sharding should be done by 2 fields. First one is zone ['Europe', 'America'] and second one is _id. So my sharding ...
Denino1234's user avatar
0 votes
0 answers
220 views

My Mongos can't authenticate to my replica set

I have 3 config servers in a replica set and when I use the command: sudo mongos --configdb crs/the 3 config servers:port# --keyfile "Keyfile", I keep getting this error: [mongosMain] Unable ...
Awesomekingz Liang's user avatar
0 votes
0 answers
366 views

Multi level Sharding in Cassandra

I am sorry if this is basic question. I am new to Cassandra and need help on a query. Assume I want to host a service which accepts SurveyMonkey , now users are given a URL to take the survey and at ...
Zoro99's user avatar
  • 1
1 vote
1 answer
474 views

Thousands foreign keys referenced to one table and other postgres scaling philosophy

I'm inspecting one legacy and quite big database which is growing faster nowadays than before. Following query select count(*) from information_schema.tables where table_schema = 'public' returns a ...
danilabagroff's user avatar
1 vote
1 answer
885 views

Consistent Prefix Reads

In the book Designing Data-Intensive Applications, the author suggests that replication lag can lead to an anomaly were causality is violated. In the example that the author uses, the system is both ...
Steven L.'s user avatar
  • 165
0 votes
1 answer
304 views

Can sharding speed up insertion performance in postgresql

Referenced from this question, the most popular answer does not mention sharding. As far as I know, postgresql will acquire auto-inc lock every time it inserts, so I think sharding can improve ...
spike 王建's user avatar
0 votes
1 answer
142 views

Config enable sharding collection in MongoDB cluster

I'm a new in MongoDB cluster. I have a question about config enable shading collection in MongoDB: I have a cluster : Shard1: replical01/node1:27017, node2:27017 Shard2: replical02/node3:27017, ...
muon.ha's user avatar
-1 votes
1 answer
382 views

Fault tolerance for Database sharding and Database partitioning

I'm aware that database sharding is splitting up of datasets horizontally into various database instances, whereas database partitioning uses one single instance. In Database Sharding, what if one of ...
MrRobot9's user avatar
  • 143
0 votes
1 answer
507 views

MongoDB multi-datacenter deployment and shard keys

I am trying to check my understanding after reading docs and some older questions, so my question is quite loaded: In a sharded cluster, each shard should (and as of 3.6 must?) reside on its own ...
kot's user avatar
  • 3
0 votes
1 answer
189 views

How to have different server config on replication server for postgres_fdw?

I have a postgres sharding based on postgres_fdw and postgres partitioning, lets say 3 machine (1 coordinator and 2 Shard machines). What I want to do is have read replica version of same? How should ...
hardik24's user avatar
0 votes
1 answer
547 views

Slow insert in Postgresql when sharding using declarative partitioning and postgres_fdw?

We have been trying to partition a Postgres database on google cloud using the built-in Postgres declarative partitioning and postgres_fdw as explained here. We are running commands as follow: Shard ...
hardik24's user avatar
2 votes
1 answer
429 views

How to implement High Availability for all nodes in a sharded database?

I'm doing some research into implementing sharding for one of my large (~2 TB) production databases. Currently the database has HADR via a multi-replica AlwaysOn Availability Group. Since sharding a ...
J.D.'s user avatar
  • 38.1k
1 vote
0 answers
468 views

how to maximize usage of number of databases on a postgres server while limited by connections

We're building a system where ideally we'd give a database per client in a multi-tenant setup. However, although a single Postgres install supports 4,294,950,911 databases on a server, the suggested ...
pyramation's user avatar

1
2 3 4 5
7