Questions tagged [sharding]

Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.

Filter by
Sorted by
Tagged with
0
votes
0answers
24 views

Is there any way to move expired data from one collection to another periodly?

We have a very big collection, and for getting a better performance, we want to limit the size of "active" collection. That means, we have two collection: "active" and "inactive", we need to keep the ...
0
votes
1answer
32 views

Ranged Sharding - Logic behind forming ranges for hexadecimal string

I have a sharded collection where the shard key is a field called "uuid". This field's value is of type string and represents hexadecimal values i.e a hexadecimal string. For each document this "uuid" ...
1
vote
0answers
46 views

MongoDB Chunk Migration fails with “error 'aborted'”

I have a simple setup: 2 Mongos 3 Shards, deployed as a Replica Set 3 Config Servers, deployed as a Replica Set Each replica set consists of 1 Primary, 1 Secondary and 1 Arbiter. (Except the Config ...
0
votes
0answers
7 views

Decision on the shard key in MongoDB 3.4

I have _id column in the following format "date-machineNumber-eventgroup eventid-date-Timestamp-0" Can this key be used as shard key? shard key: { "_id" : 1 } unique: false ...
-1
votes
1answer
68 views

FDW use in postgres

I need to create a sequence and need to use the sequence for auto increment id for more than 2 databases on 2 separate servers. I'm referring below doc for that(for test purpose): https://paquier.xyz/...
-2
votes
1answer
48 views

How sharding of databases effect ACID property or transactions?

MySQL (and other Relational Databases) are known to be ACID compliant with respect to transactions. However, when databases are sharded, does the ACID property of transactions break since now ...
0
votes
0answers
18 views

Mysql Sharding. Application Layer vs Mysql Layer

I am going to create shards in mysql table and I am stuck at a dileman whether to go for application layer sharding or let MySQL handle it. I mean, wether I should be figuring out which shard to go ...
1
vote
1answer
20 views

Schema changes on sharded database

I have performance issues with one particulary large table, 500+ Million rows, 300Gb data, Postgres 10.5. It is already partitioned. I am working on optimising it here and there, but that is not ...
1
vote
1answer
33 views

Mongodb - Best Practice for TTL index on huge dataset

I want to create a TTL index for a table. Table size 1.2TB I have a shard infra. Each shard is having 3 node replica set. For creating this index, can I go with the rolling index creation method? ...
0
votes
0answers
22 views

MongoDB Move data to another shard

I have a MongoDB with 3 shards: A, B, C. B's disk is full so everytime I start mongod it eventually stops. However, A and C still have GB of space. Hiring a new server with more space than B and ...
0
votes
1answer
17 views

PostgreSQL autoincrement up to certain number then reset for sharding

I've was recently reading an article around how Instgram used to shard their IDs and they use a unique function to generate ids on their servers https://instagram-engineering.com/sharding-ids-at-...
0
votes
0answers
42 views

MongoDB uneven document sharding - and Migration Windows

I created a sharded mongodb cluster with 2 shards. Then I created a sharded collection by range and inserted 10000 documents with random documents with random values for the sharded key using Java. ...
1
vote
1answer
61 views

Sharding standalone MongoDB

I've got a standalone MongoDB server with a lot amount of data on it. Let's say, it's srv1 server. I've decided to make a sharded cluster, so I've created additional servers srv2 and srv3. We have: ...
1
vote
1answer
112 views

MongoDB sharding multiple collections with the same shard key

Lets imagine I have two mongodb sharded collection: likes & comments. Both collections are sharded, both collections have userId as a sharded key. Lets imagine that likes collection is huge and ...
1
vote
0answers
23 views

How to scale MongoDB/GridFS for fast ingestion on limited timescale

I'm working on a cloud-agnostic service which must allow the ingestion of a lot of media files in a very limited timeframe. Some numbers: 3-4 hours total of workload: the service will be active ...
0
votes
0answers
42 views

What does config shard mean here?

I created a MongoDB sharded cluster with replica set rsa, rsb and rsc. Configuration server replica set is configRS. Initially shard status listed only these shards as primary shard for the sharded ...
1
vote
1answer
40 views

Sharded Mongodb stalls randomly

I have setup Sharded MongoDB cluster using hashed sharding in kuberenetes.I first created the config server Replicaset and then created 2 shard replicasets. Finally created mongos to connect to the ...
0
votes
1answer
113 views

Sharding a MongoDB instance by a spatial key and a date

This may be more an advice than anything else, but I'm designing an app that stores in a table objects that have a 2dsphere index and they also have a date (+time), both of them being part of queries ...
0
votes
0answers
55 views

Mysql auto sharding

I am trying to understand the automatic way of Sharding. I understand there is a way to do sharding through application logic (for example, id > 100 go to DB B, < 100 go to DB A) However, is there ...
0
votes
2answers
62 views

Why does PostgreSQL FDW talk to multiple backend servers for this point query?

I've set up a PostgreSQL FDW server with the following table, sharded by user_id over four servers: CREATE TABLE my_big_table ( user_id bigint NOT NULL, serial bigint NOT NULL, -- ...
2
votes
1answer
222 views

How do I get PostgreSQL FDW to push down the LIMIT to the (single) backend server?

I've set up a PostgreSQL FDW server with the following table, sharded by user_id over four servers: CREATE TABLE my_big_table ( user_id bigint NOT NULL, serial bigint NOT NULL, -- ...
1
vote
1answer
219 views

How to shutdown mongodb sharded cluster temporary

I have setup mongodb sharded cluster with following topology 3x3 shards (each have 2 replica sets) 1x3 config server replica cluster 2 mongos routers I want to shutdown whole cluster temporary ...
0
votes
1answer
305 views

Limits of table partitioning (in PostgreSQL)

I have the following use case (PostgreSQL, currently v9 but I'm planning to upgrade to the latest version). I have a data warehouse application with a central table that has a sizeable number of ...
0
votes
0answers
242 views

is it possible to make multiple mongo routers connected to just one shard on a mongoDB shard cluster?

I'd like to make a mongo cluster with multiple mongo routers(mongos) with just one shard(mongod) like thses figures. So I made two mongo routers named 'mongorouter-1', 'mongorouter-2', and also ...
1
vote
2answers
53 views

Can sharding help in this scenario? [closed]

I will have around 600 GB of data in single table with multiple columns in MySQL. Basically this table will store user events from application. Indexes are there on required columns. Question: If I ...
1
vote
2answers
208 views

Vertical / Horizontal Scaling Disk Size MySQL or MariaDB

I have been looking into this for a while and cannot find a suitable solution so would like to know if someone here can please help. I am trying to set up database back-end that will assist with ...
0
votes
0answers
38 views

Mongo - Find is returning differents results with each request

Currently in my collection has the following spec: Version: MongoDb - 4.0.6 Sharding: 3 shards Shard key: Publication identifier The shards are consistent The problem is due using the same operation ...
0
votes
1answer
38 views

Mongo zoned sharding with a single value

We have a database with around 50 collections, and 70 million records. All records belong to specific customers and have a customerId property. Currently all clients are in the USA. However, we're in ...
0
votes
1answer
402 views

Replication or Sharding (MongoDB)

I understand each of the terms but I do not understand very well if they can work simultaneously. In my case, for example, I have a machine with 20GB and I would like to know which is the best ...
0
votes
1answer
21 views

shard key specification

I have a document like so: { "_id" : ObjectId("52f29f91ddbd75540aba6dae"), "teams" : [ { "name" : "Washington Bullets", "score" : 100, "home" : ...
1
vote
1answer
197 views

MongoDB specifying shard key and sharding strategy

What is the syntax to state that a collection is to be sharded as location based, while using the range based sharding strategy. I know you have to use the the sh.shardCollection() command, but what ...
2
votes
1answer
75 views

What is the meaning of “Monotonically Changing Shard Keys”?

I am writing my first pubblication concerning MongoDB and more specific MongoDB Sharded clusters. On the documentantion I found that the shard key in case of ranged sharding should be "Non-...
0
votes
0answers
66 views

Reading from two shards using ProxySQL

I have a table that has been sharded between two machines, and the same table on each machine holds different data (not replicated). With ProxySQL, how can I create a mysql_query_rule that allows a ...
0
votes
1answer
58 views

documents don't go into newly added shard

I have added a shard to an existing cluster, and when I do db.printShardingStatus() I see a distribution: mongos> db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" :...
1
vote
0answers
47 views

adding documents to shard collection results in incorrect amount

I have added 2 shards to my cluster: sh.addShard("127.0.0.1:27023") { "shardAdded" : "shard0000", "ok" : 1, "operationTime" : Timestamp(1552115819, 4), "$clusterTime" : { "...
1
vote
1answer
127 views

mongodb sharding collection

Im trying to shard my collection. when I execute the command: sh.enableSharding("testdb") I get: { "ok" : 1, "operationTime" : Timestamp(1552083125, 2), "$clusterTime" : { "...
0
votes
0answers
36 views

Mongodb - add new node secondary in existing replica set shard

my mongodb configuration which i am trying to add. systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log storage: dbPath: /var/lib/mongo/mongo journal: enabled: ...
-3
votes
2answers
188 views

Replication and sharding combinations

In modern relational databases, what combinations of replication and sharding are possible? Actually, I'm interested in the following strategy. Is it possible to replicate a small table in order to ...
1
vote
1answer
53 views

MongoDB sharding and zone tag that has a range of 1 key value

Suppose I have a zone tag that has a range of exactly 1 key value, such as the letter A. Further, suppose I assign this zone to two shards. Will the data with a shard key of A be put on both shards? ...
0
votes
0answers
2 views

MongoDB, sharding with multi DB instance

I've a MongoDB instance with 40 DBs with same collections, every DB has a name like AWS1010, AWS1011 and so on. Now I've to activate sharding and I'd like to know if it is possible with that situation ...
1
vote
0answers
62 views

MongoDB ticket availability reaching zero

I recently came across intresting issue. I have a sharded cluster which has two replication sets with 2 secondaries. On my primaries, there were 300-400 connections at a point in time, shortly after ...
0
votes
3answers
112 views

MySQL insert into different databases tables based on PK value

Lets assume i have two databases, both having a table 'user_feedback', sharding data between the two databases. My goal is to insert all records where Primary key Id%2=0 goes to db1.user_feedback, and ...
0
votes
0answers
17 views

Syncing DDL accross Postgres shards

Our database writing needs exceed the capacity of a single Postgres cluster (since each cluster can only support 1 writer server). To combat this, we've sharded our data across 8 separate Postgres ...
1
vote
1answer
54 views

Managing 100s of similar databases

We are migrating to a model where every "customer" lives in its own dedicated schema (database per tenant). This will allow us to later move some accounts to different DB clusters or upgrade specific ...
1
vote
1answer
1k views

No shardIdentity document - MongoDB

Having a sharded cluster in production which has been running for a good year now without issues. Everything is on 4.0, classic 3-3-3(config) + 2 mongos. While revisiting our back-up strategy i ...
0
votes
0answers
38 views

[MongoDB]See Collections (and Files) created in a Shard on the Cluster

My structure: A Sharded Cluster with multiple Shards. Each Shard contains 1 Collection, which is pinned to it by tags. Every Shard is at the same time a ReplicaSet Secondary. It receives data from a ...
2
votes
1answer
101 views

MONGOS uselessly scanning indexed documents when counting on shards

I have a simple collection with 40M documents distributed over 2 shards. The documents contain a field named COUNTRY with a sparse index on it. db.contacts.count({ "COUNTRY": "US" }) I am trying to ...
1
vote
0answers
53 views

One of the mongo shards consuming too much disk space.

We have mongo cluster with 10 shards of each having 3 replicas. Recently we are facing an issue, exceptionally in a couple of shards, the disk space among all the 3 replicas has reached above 90%. ...
2
votes
1answer
78 views

Getting exception on running sh.status() command on MongoDB sharded cluster

I am running sh.status() command on my mongos instance of sharded Mongo cluster and I am getting below exception. mongos> sh.status() assert: command failed: { "ok" : 0, "errmsg" : "error ...
-1
votes
1answer
66 views

What is a beachhead? (replication / sharding)

I quote the following from this blog post from AWS: Because you’re running on MySQL, you can use one of the methods listed in the Aurora documentation to establish a beachhead in Aurora. What ...