Questions tagged [sharding]

Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.

Filter by
Sorted by
Tagged with
0
votes
1answer
11 views

Fault tolerance for Database sharding and Database partitioning

I'm aware that database sharding is splitting up of datasets horizontally into various database instances, whereas database partitioning uses one single instance. In Database Sharding, what if one of ...
0
votes
1answer
26 views

MongoDB multi-datacenter deployment and shard keys

I am trying to check my understanding after reading docs and some older questions, so my question is quite loaded: In a sharded cluster, each shard should (and as of 3.6 must?) reside on its own ...
0
votes
1answer
12 views

How to have different server config on replication server for postgres_fdw?

I have a postgres sharding based on postgres_fdw and postgres partitioning, lets say 3 machine (1 coordinator and 2 Shard machines). What I want to do is have read replica version of same? How should ...
0
votes
1answer
39 views

Slow insert in Postgresql when sharding using declarative partitioning and postgres_fdw?

We have been trying to partition postgresql database on google cloud using the inbuilt Postgresql declarative partitioning and postgres_fdw as explained here. We are running commands as follow: Shard ...
2
votes
1answer
72 views

How to implement High Availability for all nodes in a sharded database?

I'm doing some research into implementing sharding for one of my large (~2 TB) production databases. Currently the database has HADR via a multi-replica AlwaysOn Availability Group. Since sharding a ...
1
vote
0answers
21 views

how to maximize usage of number of databases on a postgres server while limited by connections

We're building a system where ideally we'd give a database per client in a multi-tenant setup. However, although a single Postgres install supports 4,294,950,911 databases on a server, the suggested ...
0
votes
0answers
16 views

Mongodb replication breaks if balancer started

have a 10 shard ( with 3 replicaset) cluster with very heavy traffic. Storage almost 1.5TB per shard in 2000 databases MongoDB server version is 4.0.3 Since these servers were added at different ...
0
votes
0answers
11 views

Mongo DB Cluster activesessions skyrocketing

I have a forked instance of mongod running with the NUMACTL options right now. It is handling 40k - 50k updates per second without issue. I tried migrating this to a sharded cluster and started ...
0
votes
0answers
19 views

Result rows duplicated by spider engine

I've two mysql server 'G0' and 'G1'. And there is a one row test table on them respectively. G0: mysql> select * from test.t1; +----+--------+ | id | node | +----+--------+ | 1 | DB-G0 | +----...
0
votes
1answer
20 views

Chunk size bigger than configured

We have a sharded mongo cluster. We have configured the chunk size to be 256M but they are ~1.5G each. There are no jumbo chunks as our shard key is well balanced (we have also checked for them ...
0
votes
0answers
29 views

Distribute data between MongoDb Shards

I have a development environment with two shards, one configuration server and one router as illustrated below. I have a sharded database "mydb2" and a sharded colleciton "mycollection2". When I ...
0
votes
1answer
903 views

mongodb conf node won't start - compatibility error

I recently upgraded a dev sharded-cluster from 3.6 to 4.0 - one of the conf servers is now failing to start: 2020-01-30T13:19:02.972-0600 F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM:...
0
votes
1answer
25 views

MongoDB Shard Zone Range Overlapping

I am configuring a test mongo setup with 2 shards, and I'm trying to add zone ranges to each shard: sh.updateZoneKeyRange('mydb.test', {id:MinKey,ts:MinKey}, {id:MaxKey,ts:1548787704000}, 'cold') sh....
0
votes
0answers
28 views

Is there any way to move expired data from one collection to another periodly?

We have a very big collection, and for getting a better performance, we want to limit the size of "active" collection. That means, we have two collection: "active" and "inactive", we need to keep the ...
0
votes
1answer
54 views

Ranged Sharding - Logic behind forming ranges for hexadecimal string

I have a sharded collection where the shard key is a field called "uuid". This field's value is of type string and represents hexadecimal values i.e a hexadecimal string. For each document this "uuid" ...
2
votes
0answers
154 views

MongoDB Chunk Migration fails with “error 'aborted'”

I have a simple setup: 2 Mongos 3 Shards, deployed as a Replica Set 3 Config Servers, deployed as a Replica Set Each replica set consists of 1 Primary, 1 Secondary and 1 Arbiter. (Except the Config ...
0
votes
0answers
11 views

Decision on the shard key in MongoDB 3.4

I have _id column in the following format "date-machineNumber-eventgroup eventid-date-Timestamp-0" Can this key be used as shard key? shard key: { "_id" : 1 } unique: false ...
-1
votes
1answer
199 views

FDW use in postgres

I need to create a sequence and need to use the sequence for auto increment id for more than 2 databases on 2 separate servers. I'm referring below doc for that(for test purpose): https://paquier.xyz/...
-2
votes
1answer
133 views

How sharding of databases effect ACID property or transactions?

MySQL (and other Relational Databases) are known to be ACID compliant with respect to transactions. However, when databases are sharded, does the ACID property of transactions break since now ...
0
votes
0answers
26 views

Mysql Sharding. Application Layer vs Mysql Layer

I am going to create shards in mysql table and I am stuck at a dileman whether to go for application layer sharding or let MySQL handle it. I mean, wether I should be figuring out which shard to go ...
1
vote
1answer
43 views

Schema changes on sharded database

I have performance issues with one particulary large table, 500+ Million rows, 300Gb data, Postgres 10.5. It is already partitioned. I am working on optimising it here and there, but that is not ...
1
vote
1answer
75 views

Mongodb - Best Practice for TTL index on huge dataset

I want to create a TTL index for a table. Table size 1.2TB I have a shard infra. Each shard is having 3 node replica set. For creating this index, can I go with the rolling index creation method? ...
0
votes
0answers
32 views

MongoDB Move data to another shard

I have a MongoDB with 3 shards: A, B, C. B's disk is full so everytime I start mongod it eventually stops. However, A and C still have GB of space. Hiring a new server with more space than B and ...
0
votes
1answer
28 views

PostgreSQL autoincrement up to certain number then reset for sharding

I've was recently reading an article around how Instgram used to shard their IDs and they use a unique function to generate ids on their servers https://instagram-engineering.com/sharding-ids-at-...
0
votes
0answers
56 views

MongoDB uneven document sharding - and Migration Windows

I created a sharded mongodb cluster with 2 shards. Then I created a sharded collection by range and inserted 10000 documents with random documents with random values for the sharded key using Java. ...
1
vote
1answer
93 views

Sharding standalone MongoDB

I've got a standalone MongoDB server with a lot amount of data on it. Let's say, it's srv1 server. I've decided to make a sharded cluster, so I've created additional servers srv2 and srv3. We have: ...
2
votes
1answer
220 views

MongoDB sharding multiple collections with the same shard key

Lets imagine I have two mongodb sharded collection: likes & comments. Both collections are sharded, both collections have userId as a sharded key. Lets imagine that likes collection is huge and ...
1
vote
0answers
49 views

How to scale MongoDB/GridFS for fast ingestion on limited timescale

I'm working on a cloud-agnostic service which must allow the ingestion of a lot of media files in a very limited timeframe. Some numbers: 3-4 hours total of workload: the service will be active ...
1
vote
1answer
97 views

Sharded Mongodb stalls randomly

I have setup Sharded MongoDB cluster using hashed sharding in kuberenetes.I first created the config server Replicaset and then created 2 shard replicasets. Finally created mongos to connect to the ...
0
votes
1answer
298 views

Sharding a MongoDB instance by a spatial key and a date

This may be more an advice than anything else, but I'm designing an app that stores in a table objects that have a 2dsphere index and they also have a date (+time), both of them being part of queries ...
0
votes
0answers
98 views

Mysql auto sharding

I am trying to understand the automatic way of Sharding. I understand there is a way to do sharding through application logic (for example, id > 100 go to DB B, < 100 go to DB A) However, is there ...
0
votes
2answers
111 views

Why does PostgreSQL FDW talk to multiple backend servers for this point query?

I've set up a PostgreSQL FDW server with the following table, sharded by user_id over four servers: CREATE TABLE my_big_table ( user_id bigint NOT NULL, serial bigint NOT NULL, -- ...
2
votes
1answer
518 views

How do I get PostgreSQL FDW to push down the LIMIT to the (single) backend server?

I've set up a PostgreSQL FDW server with the following table, sharded by user_id over four servers: CREATE TABLE my_big_table ( user_id bigint NOT NULL, serial bigint NOT NULL, -- ...
1
vote
1answer
344 views

How to shutdown mongodb sharded cluster temporary

I have setup mongodb sharded cluster with following topology 3x3 shards (each have 2 replica sets) 1x3 config server replica cluster 2 mongos routers I want to shutdown whole cluster temporary ...
0
votes
1answer
1k views

Limits of table partitioning (in PostgreSQL)

I have the following use case (PostgreSQL, currently v9 but I'm planning to upgrade to the latest version). I have a data warehouse application with a central table that has a sizeable number of ...
0
votes
0answers
410 views

is it possible to make multiple mongo routers connected to just one shard on a mongoDB shard cluster?

I'd like to make a mongo cluster with multiple mongo routers(mongos) with just one shard(mongod) like thses figures. So I made two mongo routers named 'mongorouter-1', 'mongorouter-2', and also ...
1
vote
2answers
62 views

Can sharding help in this scenario? [closed]

I will have around 600 GB of data in single table with multiple columns in MySQL. Basically this table will store user events from application. Indexes are there on required columns. Question: If I ...
1
vote
2answers
399 views

Vertical / Horizontal Scaling Disk Size MySQL or MariaDB

I have been looking into this for a while and cannot find a suitable solution so would like to know if someone here can please help. I am trying to set up database back-end that will assist with ...
0
votes
0answers
40 views

Mongo - Find is returning differents results with each request

Currently in my collection has the following spec: Version: MongoDb - 4.0.6 Sharding: 3 shards Shard key: Publication identifier The shards are consistent The problem is due using the same operation ...
0
votes
1answer
58 views

Mongo zoned sharding with a single value

We have a database with around 50 collections, and 70 million records. All records belong to specific customers and have a customerId property. Currently all clients are in the USA. However, we're in ...
0
votes
1answer
22 views

shard key specification

I have a document like so: { "_id" : ObjectId("52f29f91ddbd75540aba6dae"), "teams" : [ { "name" : "Washington Bullets", "score" : 100, "home" : ...
1
vote
1answer
286 views

MongoDB specifying shard key and sharding strategy

What is the syntax to state that a collection is to be sharded as location based, while using the range based sharding strategy. I know you have to use the the sh.shardCollection() command, but what ...
2
votes
1answer
109 views

What is the meaning of “Monotonically Changing Shard Keys”?

I am writing my first pubblication concerning MongoDB and more specific MongoDB Sharded clusters. On the documentantion I found that the shard key in case of ranged sharding should be "Non-...
0
votes
0answers
96 views

Reading from two shards using ProxySQL

I have a table that has been sharded between two machines, and the same table on each machine holds different data (not replicated). With ProxySQL, how can I create a mysql_query_rule that allows a ...
0
votes
1answer
76 views

documents don't go into newly added shard

I have added a shard to an existing cluster, and when I do db.printShardingStatus() I see a distribution: mongos> db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" :...
1
vote
0answers
72 views

adding documents to shard collection results in incorrect amount

I have added 2 shards to my cluster: sh.addShard("127.0.0.1:27023") { "shardAdded" : "shard0000", "ok" : 1, "operationTime" : Timestamp(1552115819, 4), "$clusterTime" : { "...
1
vote
1answer
198 views

mongodb sharding collection

Im trying to shard my collection. when I execute the command: sh.enableSharding("testdb") I get: { "ok" : 1, "operationTime" : Timestamp(1552083125, 2), "$clusterTime" : { "...
-3
votes
2answers
233 views

Replication and sharding combinations

In modern relational databases, what combinations of replication and sharding are possible? Actually, I'm interested in the following strategy. Is it possible to replicate a small table in order to ...
1
vote
1answer
65 views

MongoDB sharding and zone tag that has a range of 1 key value

Suppose I have a zone tag that has a range of exactly 1 key value, such as the letter A. Further, suppose I assign this zone to two shards. Will the data with a shard key of A be put on both shards? ...
1
vote
0answers
123 views

MongoDB ticket availability reaching zero

I recently came across intresting issue. I have a sharded cluster which has two replication sets with 2 secondaries. On my primaries, there were 300-400 connections at a point in time, shortly after ...

1
2 3 4 5 6