Questions tagged [sharding]

Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.

Filter by
Sorted by
Tagged with
0 votes
1 answer
39 views

MongoDB: slow removeShard

I have a MongoDB cluster with 9 nodes (3 shards, 3 nodes each). I'm now removing one shard, but the process itself is running extremely slow. Each node of the shard being deleted stores ~400Gb of data,...
user avatar
0 votes
0 answers
16 views

How has sharding become easier in mysql now(in 2022) as compared to before?

I was currently implementing a side project where I had to make a choice for database. The basic considerations are listed below: The schema is not strict and could have many empty fields. There ...
user avatar
1 vote
1 answer
27 views

Storing and Querying User Defined Data

I'm working on a system where each user defines a strongly typed definition of data they can store, consisting of classes with specified fields. Embedded classes are also supported. My first instinct ...
user avatar
  • 111
0 votes
0 answers
65 views

Disadvantages of sharding PostgreSQL in 2022

I'm in the phase of choosing a database for a project, PostgreSQL seems to be the best choice. I read many questions at stackoverflow and other websites about Sharding database, they were all about ...
user avatar
0 votes
0 answers
14 views

MongoDB : How to set persisent mongos and shard configuration

I have a test environment running on Windows 10 as follows : A replica set "rs0" with 3 nodes : MONGO2, MONGO3, MONGOARB1 each running on port 27017. A config server in a replicaSet "...
user avatar
  • 283
1 vote
1 answer
27 views

Cannot connect to MongoDB router in sharded cluster

I followed this guide on how to setup a sharded MongoDB cluster with Docker containers. It all seemed to work as it should because there were no error messages in the logs and the sh.status() command ...
user avatar
  • 111
0 votes
0 answers
28 views

If I have only 1 single type of record, but trillions of them, what is a sharding scheme that would scale?

Say I have 1 type of record, object, with title/slug/description fields, but I have trillions of them. Well, I start off with 1,000 records on day one, by end of year one I have 1M records, by year 2 ...
user avatar
  • 209
0 votes
1 answer
99 views

Is ObjectId as shard key unique across shards or just within a shard?

In my sharded cluster of version 4.2, one collections is sharded on _id where _id is just a default ObjectId. I'm seeing multiple documents with same _id on different shards' primaries. When I specify ...
user avatar
0 votes
2 answers
34 views

Not able to shard collection

I have enabled sharding using the below commands. mongos> use news switched to db news mongos> sh.enableSharding("news"); { "ok" : 1, "$clusterTime" : { ...
user avatar
0 votes
0 answers
55 views

Enable username and password authentication on mongodb sharded cluster

I have a mongodb cluster(1 mongos, mongo config replica set with 2 members, 1 shard with 3 members) and I have enabled internal authentication in config replica set. Also my backend code connects to ...
user avatar
0 votes
1 answer
31 views

MySQL Sharding with org id

I have a multi tenant application and each tenant will have an orgId. So, all tables will have orgid column. Each org will have clients, projects, ... Tables org client project Now, I am trying to ...
user avatar
0 votes
1 answer
59 views

Sharding AWS MySQL data

I am working on a SaaS application using AWS MySQL. The application is for multiple organizations. We use the same database and tables to store multiple org data. We store org_id in most of the tables ...
user avatar
0 votes
1 answer
87 views

Zoned and hashed sharding of existing, non-empty collection

I am trying to shard a existing collection. Collection is pretty big (about 100gb). Sharding should be done by 2 fields. First one is zone ['Europe', 'America'] and second one is _id. So my sharding ...
user avatar
0 votes
0 answers
37 views

My Mongos can't authenticate to my replica set

I have 3 config servers in a replica set and when I use the command: sudo mongos --configdb crs/the 3 config servers:port# --keyfile "Keyfile", I keep getting this error: [mongosMain] Unable ...
user avatar
0 votes
0 answers
123 views

Multi level Sharding in Cassandra

I am sorry if this is basic question. I am new to Cassandra and need help on a query. Assume I want to host a service which accepts SurveyMonkey , now users are given a URL to take the survey and at ...
user avatar
  • 1
0 votes
1 answer
142 views

Thousands foreign keys referenced to one table and other postgres scaling philosophy

I'm inspecting one legacy and quite big database which is growing faster nowadays than before. Following query select count(*) from information_schema.tables where table_schema = 'public' returns a ...
user avatar
0 votes
1 answer
388 views

Consistent Prefix Reads

In the book Designing Data-Intensive Applications, the author suggests that replication lag can lead to an anomaly were causality is violated. In the example that the author uses, the system is both ...
user avatar
  • 155
0 votes
1 answer
61 views

Can sharding speed up insertion performance in postgresql

Referenced from this question, the most popular answer does not mention sharding. As far as I know, postgresql will acquire auto-inc lock every time it inserts, so I think sharding can improve ...
user avatar
0 votes
1 answer
33 views

Config enable sharding collection in MongoDB cluster

I'm a new in MongoDB cluster. I have a question about config enable shading collection in MongoDB: I have a cluster : Shard1: replical01/node1:27017, node2:27017 Shard2: replical02/node3:27017, ...
user avatar
-1 votes
1 answer
168 views

Fault tolerance for Database sharding and Database partitioning

I'm aware that database sharding is splitting up of datasets horizontally into various database instances, whereas database partitioning uses one single instance. In Database Sharding, what if one of ...
user avatar
  • 143
0 votes
1 answer
338 views

MongoDB multi-datacenter deployment and shard keys

I am trying to check my understanding after reading docs and some older questions, so my question is quite loaded: In a sharded cluster, each shard should (and as of 3.6 must?) reside on its own ...
user avatar
  • 3
0 votes
1 answer
64 views

How to have different server config on replication server for postgres_fdw?

I have a postgres sharding based on postgres_fdw and postgres partitioning, lets say 3 machine (1 coordinator and 2 Shard machines). What I want to do is have read replica version of same? How should ...
user avatar
0 votes
1 answer
225 views

Slow insert in Postgresql when sharding using declarative partitioning and postgres_fdw?

We have been trying to partition a Postgres database on google cloud using the built-in Postgres declarative partitioning and postgres_fdw as explained here. We are running commands as follow: Shard ...
user avatar
2 votes
1 answer
261 views

How to implement High Availability for all nodes in a sharded database?

I'm doing some research into implementing sharding for one of my large (~2 TB) production databases. Currently the database has HADR via a multi-replica AlwaysOn Availability Group. Since sharding a ...
user avatar
  • 24.2k
1 vote
0 answers
70 views

how to maximize usage of number of databases on a postgres server while limited by connections

We're building a system where ideally we'd give a database per client in a multi-tenant setup. However, although a single Postgres install supports 4,294,950,911 databases on a server, the suggested ...
user avatar
0 votes
0 answers
19 views

Mongodb replication breaks if balancer started

have a 10 shard ( with 3 replicaset) cluster with very heavy traffic. Storage almost 1.5TB per shard in 2000 databases MongoDB server version is 4.0.3 Since these servers were added at different ...
user avatar
  • 101
0 votes
0 answers
37 views

Result rows duplicated by spider engine

I've two mysql server 'G0' and 'G1'. And there is a one row test table on them respectively. G0: mysql> select * from test.t1; +----+--------+ | id | node | +----+--------+ | 1 | DB-G0 | +----...
user avatar
  • 101
0 votes
1 answer
70 views

Chunk size bigger than configured

We have a sharded mongo cluster. We have configured the chunk size to be 256M but they are ~1.5G each. There are no jumbo chunks as our shard key is well balanced (we have also checked for them ...
user avatar
  • 101
0 votes
0 answers
60 views

Distribute data between MongoDb Shards

I have a development environment with two shards, one configuration server and one router as illustrated below. I have a sharded database "mydb2" and a sharded colleciton "mycollection2". When I ...
user avatar
0 votes
2 answers
2k views

mongodb conf node won't start - compatibility error

I recently upgraded a dev sharded-cluster from 3.6 to 4.0 - one of the conf servers is now failing to start: 2020-01-30T13:19:02.972-0600 F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM:...
user avatar
0 votes
2 answers
78 views

MongoDB Shard Zone Range Overlapping

I am configuring a test mongo setup with 2 shards, and I'm trying to add zone ranges to each shard: sh.updateZoneKeyRange('mydb.test', {id:MinKey,ts:MinKey}, {id:MaxKey,ts:1548787704000}, 'cold') sh....
user avatar
  • 113
0 votes
0 answers
75 views

Is there any way to move expired data from one collection to another periodly?

We have a very big collection, and for getting a better performance, we want to limit the size of "active" collection. That means, we have two collection: "active" and "inactive", we need to keep the ...
user avatar
0 votes
1 answer
242 views

Ranged Sharding - Logic behind forming ranges for hexadecimal string

I have a sharded collection where the shard key is a field called "uuid". This field's value is of type string and represents hexadecimal values i.e a hexadecimal string. For each document this "uuid" ...
user avatar
  • 103
2 votes
0 answers
659 views

MongoDB Chunk Migration fails with "error 'aborted'"

I have a simple setup: 2 Mongos 3 Shards, deployed as a Replica Set 3 Config Servers, deployed as a Replica Set Each replica set consists of 1 Primary, 1 Secondary and 1 Arbiter. (Except the Config ...
user avatar
  • 121
-1 votes
1 answer
523 views

FDW use in postgres

I need to create a sequence and need to use the sequence for auto increment id for more than 2 databases on 2 separate servers. I'm referring below doc for that(for test purpose): https://paquier.xyz/...
user avatar
-1 votes
1 answer
772 views

How sharding of databases effect ACID property or transactions?

MySQL (and other Relational Databases) are known to be ACID compliant with respect to transactions. However, when databases are sharded, does the ACID property of transactions break since now ...
user avatar
0 votes
1 answer
71 views

Mysql Sharding. Application Layer vs Mysql Layer

I am going to create shards in mysql table and I am stuck at a dileman whether to go for application layer sharding or let MySQL handle it. I mean, wether I should be figuring out which shard to go ...
user avatar
1 vote
1 answer
200 views

Schema changes on sharded database

I have performance issues with one particulary large table, 500+ Million rows, 300Gb data, Postgres 10.5. It is already partitioned. I am working on optimising it here and there, but that is not ...
user avatar
  • 11
1 vote
1 answer
328 views

Mongodb - Best Practice for TTL index on huge dataset

I want to create a TTL index for a table. Table size 1.2TB I have a shard infra. Each shard is having 3 node replica set. For creating this index, can I go with the rolling index creation method? ...
user avatar
  • 1,688
0 votes
1 answer
120 views

PostgreSQL autoincrement up to certain number then reset for sharding

I've was recently reading an article around how Instgram used to shard their IDs and they use a unique function to generate ids on their servers https://instagram-engineering.com/sharding-ids-at-...
user avatar
  • 115
0 votes
0 answers
102 views

MongoDB uneven document sharding - and Migration Windows

I created a sharded mongodb cluster with 2 shards. Then I created a sharded collection by range and inserted 10000 documents with random documents with random values for the sharded key using Java. ...
user avatar
  • 101
1 vote
1 answer
271 views

Sharding standalone MongoDB

I've got a standalone MongoDB server with a lot amount of data on it. Let's say, it's srv1 server. I've decided to make a sharded cluster, so I've created additional servers srv2 and srv3. We have: ...
user avatar
2 votes
1 answer
586 views

MongoDB sharding multiple collections with the same shard key

Lets imagine I have two mongodb sharded collection: likes & comments. Both collections are sharded, both collections have userId as a sharded key. Lets imagine that likes collection is huge and ...
user avatar
1 vote
0 answers
142 views

How to scale MongoDB/GridFS for fast ingestion on limited timescale

I'm working on a cloud-agnostic service which must allow the ingestion of a lot of media files in a very limited timeframe. Some numbers: 3-4 hours total of workload: the service will be active ...
user avatar
1 vote
1 answer
302 views

Sharded Mongodb stalls randomly

I have setup Sharded MongoDB cluster using hashed sharding in kuberenetes.I first created the config server Replicaset and then created 2 shard replicasets. Finally created mongos to connect to the ...
user avatar
0 votes
1 answer
899 views

Sharding a MongoDB instance by a spatial key and a date

This may be more an advice than anything else, but I'm designing an app that stores in a table objects that have a 2dsphere index and they also have a date (+time), both of them being part of queries ...
user avatar
0 votes
0 answers
134 views

Mysql auto sharding

I am trying to understand the automatic way of Sharding. I understand there is a way to do sharding through application logic (for example, id > 100 go to DB B, < 100 go to DB A) However, is there ...
user avatar
0 votes
2 answers
252 views

Why does PostgreSQL FDW talk to multiple backend servers for this point query?

I've set up a PostgreSQL FDW server with the following table, sharded by user_id over four servers: CREATE TABLE my_big_table ( user_id bigint NOT NULL, serial bigint NOT NULL, -- ...
user avatar
  • 197
3 votes
1 answer
2k views

How do I get PostgreSQL FDW to push down the LIMIT to the (single) backend server?

I've set up a PostgreSQL FDW server with the following table, sharded by user_id over four servers: CREATE TABLE my_big_table ( user_id bigint NOT NULL, serial bigint NOT NULL, -- ...
user avatar
  • 197
1 vote
1 answer
780 views

How to shutdown mongodb sharded cluster temporary

I have setup mongodb sharded cluster with following topology 3x3 shards (each have 2 replica sets) 1x3 config server replica cluster 2 mongos routers I want to shutdown whole cluster temporary ...
user avatar

1
2 3 4 5 6