Questions tagged [sharding]

Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.

Filter by
Sorted by
Tagged with
0
votes
1answer
11 views

Sharding a MongoDB instance by a spatial key and a date

This may be more an advice than anything else, but I'm designing an app that stores in a table objects that have a 2dsphere index and they also have a date (+time), both of them being part of queries ...
0
votes
0answers
14 views

Mysql auto sharding

I am trying to understand the automatic way of Sharding. I understand there is a way to do sharding through application logic (for example, id > 100 go to DB B, < 100 go to DB A) However, is there ...
0
votes
2answers
25 views

Why does PostgreSQL FDW talk to multiple backend servers for this point query?

I've set up a PostgreSQL FDW server with the following table, sharded by user_id over four servers: CREATE TABLE my_big_table ( user_id bigint NOT NULL, serial bigint NOT NULL, -- ...
2
votes
1answer
55 views

How do I get PostgreSQL FDW to push down the LIMIT to the (single) backend server?

I've set up a PostgreSQL FDW server with the following table, sharded by user_id over four servers: CREATE TABLE my_big_table ( user_id bigint NOT NULL, serial bigint NOT NULL, -- ...
1
vote
1answer
47 views

How to shutdown mongodb sharded cluster temporary

I have setup mongodb sharded cluster with following topology 3x3 shards (each have 2 replica sets) 1x3 config server replica cluster 2 mongos routers I want to shutdown whole cluster temporary ...
0
votes
1answer
28 views

Limits of table partitioning (in PostgreSQL)

I have the following use case (PostgreSQL, currently v9 but I'm planning to upgrade to the latest version). I have a data warehouse application with a central table that has a sizeable number of ...
0
votes
0answers
25 views

is it possible to make multiple mongo routers connected to just one shard on a mongoDB shard cluster?

I'd like to make a mongo cluster with multiple mongo routers(mongos) with just one shard(mongod) like thses figures. So I made two mongo routers named 'mongorouter-1', 'mongorouter-2', and also ...
1
vote
2answers
50 views

Can sharding help in this scenario? [closed]

I will have around 600 GB of data in single table with multiple columns in MySQL. Basically this table will store user events from application. Indexes are there on required columns. Question: If I ...
1
vote
2answers
64 views

Vertical / Horizontal Scaling Disk Size MySQL or MariaDB

I have been looking into this for a while and cannot find a suitable solution so would like to know if someone here can please help. I am trying to set up database back-end that will assist with ...
0
votes
0answers
38 views

Mongo - Find is returning differents results with each request

Currently in my collection has the following spec: Version: MongoDb - 4.0.6 Sharding: 3 shards Shard key: Publication identifier The shards are consistent The problem is due using the same operation ...
0
votes
0answers
24 views

Mongo zoned sharding with a single value

We have a database with around 50 collections, and 70 million records. All records belong to specific customers and have a customerId property. Currently all clients are in the USA. However, we're in ...
0
votes
1answer
149 views

Replication or Sharding (MongoDB)

I understand each of the terms but I do not understand very well if they can work simultaneously. In my case, for example, I have a machine with 20GB and I would like to know which is the best ...
0
votes
1answer
15 views

shard key specification

I have a document like so: { "_id" : ObjectId("52f29f91ddbd75540aba6dae"), "teams" : [ { "name" : "Washington Bullets", "score" : 100, "home" : ...
0
votes
1answer
58 views

MongoDB specifying shard key and sharding strategy

What is the syntax to state that a collection is to be sharded as location based, while using the range based sharding strategy. I know you have to use the the sh.shardCollection() command, but what ...
2
votes
1answer
29 views

What is the meaning of “Monotonically Changing Shard Keys”?

I am writing my first pubblication concerning MongoDB and more specific MongoDB Sharded clusters. On the documentantion I found that the shard key in case of ranged sharding should be "Non-...
0
votes
0answers
37 views

Reading from two shards using ProxySQL

I have a table that has been sharded between two machines, and the same table on each machine holds different data (not replicated). With ProxySQL, how can I create a mysql_query_rule that allows a ...
0
votes
1answer
28 views

documents don't go into newly added shard

I have added a shard to an existing cluster, and when I do db.printShardingStatus() I see a distribution: mongos> db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" :...
1
vote
0answers
39 views

adding documents to shard collection results in incorrect amount

I have added 2 shards to my cluster: sh.addShard("127.0.0.1:27023") { "shardAdded" : "shard0000", "ok" : 1, "operationTime" : Timestamp(1552115819, 4), "$clusterTime" : { "...
1
vote
1answer
55 views

mongodb sharding collection

Im trying to shard my collection. when I execute the command: sh.enableSharding("testdb") I get: { "ok" : 1, "operationTime" : Timestamp(1552083125, 2), "$clusterTime" : { "...
0
votes
0answers
26 views

Mongodb - add new node secondary in existing replica set shard

my mongodb configuration which i am trying to add. systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log storage: dbPath: /var/lib/mongo/mongo journal: enabled: ...
-3
votes
2answers
157 views

Replication and sharding combinations

In modern relational databases, what combinations of replication and sharding are possible? Actually, I'm interested in the following strategy. Is it possible to replicate a small table in order to ...
1
vote
1answer
36 views

MongoDB sharding and zone tag that has a range of 1 key value

Suppose I have a zone tag that has a range of exactly 1 key value, such as the letter A. Further, suppose I assign this zone to two shards. Will the data with a shard key of A be put on both shards? ...
0
votes
0answers
2 views

MongoDB, sharding with multi DB instance

I've a MongoDB instance with 40 DBs with same collections, every DB has a name like AWS1010, AWS1011 and so on. Now I've to activate sharding and I'd like to know if it is possible with that situation ...
1
vote
0answers
30 views

MongoDB ticket availability reaching zero

I recently came across intresting issue. I have a sharded cluster which has two replication sets with 2 secondaries. On my primaries, there were 300-400 connections at a point in time, shortly after ...
0
votes
3answers
66 views

MySQL insert into different databases tables based on PK value

Lets assume i have two databases, both having a table 'user_feedback', sharding data between the two databases. My goal is to insert all records where Primary key Id%2=0 goes to db1.user_feedback, and ...
0
votes
0answers
17 views

Syncing DDL accross Postgres shards

Our database writing needs exceed the capacity of a single Postgres cluster (since each cluster can only support 1 writer server). To combat this, we've sharded our data across 8 separate Postgres ...
0
votes
1answer
47 views

Managing 100s of similar databases

We are migrating to a model where every "customer" lives in its own dedicated schema (database per tenant). This will allow us to later move some accounts to different DB clusters or upgrade specific ...
1
vote
1answer
491 views

No shardIdentity document - MongoDB

Having a sharded cluster in production which has been running for a good year now without issues. Everything is on 4.0, classic 3-3-3(config) + 2 mongos. While revisiting our back-up strategy i ...
0
votes
0answers
38 views

[MongoDB]See Collections (and Files) created in a Shard on the Cluster

My structure: A Sharded Cluster with multiple Shards. Each Shard contains 1 Collection, which is pinned to it by tags. Every Shard is at the same time a ReplicaSet Secondary. It receives data from a ...
2
votes
0answers
53 views

MONGOS uselessly scanning indexed documents when counting on shards

I have a simple collection with 40M documents distributed over 2 shards. The documents contain a field named COUNTRY with a sparse index on it. db.contacts.count({ "COUNTRY": "US" }) I am trying to ...
1
vote
0answers
43 views

One of the mongo shards consuming too much disk space.

We have mongo cluster with 10 shards of each having 3 replicas. Recently we are facing an issue, exceptionally in a couple of shards, the disk space among all the 3 replicas has reached above 90%. ...
2
votes
1answer
44 views

Getting exception on running sh.status() command on MongoDB sharded cluster

I am running sh.status() command on my mongos instance of sharded Mongo cluster and I am getting below exception. mongos> sh.status() assert: command failed: { "ok" : 0, "errmsg" : "error ...
-1
votes
1answer
54 views

What is a beachhead? (replication / sharding)

I quote the following from this blog post from AWS: Because you’re running on MySQL, you can use one of the methods listed in the Aurora documentation to establish a beachhead in Aurora. What ...
0
votes
2answers
224 views

Migration of MariaDB/MySQL to another server without downtime

System : MariaDB (MySQL 5.5) Parameters : innodb_file_per_table=1 The database I have now is approaching its full capacity (94% full now, 6.4TB out of 6.9TB) I can't run OPTIMIZE TABLE to reclaim ...
1
vote
0answers
80 views

Choosing the correct shard key for MongoDB

I'm new to MongoDB. From the mongod documentation, I know that when choosing shard keys the key should always have high cardinality, so write operations are distributed to different shards (a kind of ...
1
vote
2answers
255 views

MySQL refuses UPDATE with (1071, 'Specified key was too long; max key length is 1024 bytes') sometimes

I'm using MySQL 5.7.2.23. I need to copy some columns between two tables (u -> m) and I am using the following update command: UPDATE m JOIN u ON u.id=m.user_id AND u.package_name=m....
1
vote
1answer
517 views

Is Postgres sharding available on Aurora (AWS)

I need to shard and/or partition my largeish Postgres db tables. The Postgres partitioning functionality seems crazy heavyweight (in terms of DDL). I see talk from <=2015 about pg_shard, but am ...
0
votes
1answer
340 views

Query sharded collection returns all the results from the primary shard AND from the other shards

I tried to set up a shard to test this functionality and maybe migrate some of our data into it. I followed the documentation guide https://docs.mongodb.com/v3.4/tutorial/convert-replica-set-to-...
1
vote
1answer
237 views

How to deal with unbounded growth in cassandra shards

Let's say we have a website like youtube where we have a comment session per e video. Users want to be able to comment and to see the comments made by other users by comment creation time. This is ...
1
vote
0answers
244 views

Sharding PostgreSQL with postgres_fdw (Foreign Data Wrapper)

We want to shard a single PostgreSQL 10.2 database by tenant (client id) to multiple servers. One of the easiest approach is to use Foreign Data Wrapper (postgres_fdw extension). It will looks like: ...
0
votes
0answers
151 views

Force remove a dead shard from mongodb

I have a cluster with 3 shards, and one of them died. There is no backup of the shard and I can live without the data. I'm trying to remove the shard from the cluster and restart with the 2 remaining ...
0
votes
1answer
118 views

what if mongos server goes down in shard cluster setup

I have followed the tuturial Deploy a Sharded Cluster I have setup config server replica set with 3 members (cfg1, cfg2, cfg3) [replica set name = configserverreplica] mongodb server replica set ...
2
votes
1answer
240 views

MongoDB sharding by monotonic id with zones for archiving

Situation: I have an ever-growing collection of documents which do have monotonically increasing unique id. All queries are direct lookup for documents with specified _id (no range queries nor ...
0
votes
1answer
35 views

MySQL- reducing memory footprint

Our InnoDB/Mysql5.6 database is growing at a fairly steady pace of doubling in size every year, currently at 22Gb. This is sales data; we have a number of merchants each generating millions of rows, ...
1
vote
1answer
83 views

MongoDB Shard distribution issue

I am facing an issue in the shard distribution. MongoDB Version: 3.0.15 I have a collection that is sharded. I have two shards. Output of db.getCollection('event').stats() { "sharded" : true, "...
1
vote
1answer
403 views

MongoDB Using String-Type Shard Key for Zone Ranges

Mongo lets any field type be used to set ranges for Shard Zones, but doesn't specify how analyses are conducted. My goal is to point queries and writes at the exact right server, WITHOUT using a ...
1
vote
1answer
44 views

MongoDB as a log storage. Choosing shard key

I'm designing a log storage system based on MongoDB. I want to shard a log collection to increase ingestion and capacity (distribute writes to several machines) while allow fast search. I should be ...
2
votes
1answer
521 views

MongoDB crashes with out-of-memory or is being killed by oom-killer

A two shards MongoDB database regularly crashes with out-of-memory error or is being killed by the oom-killer. The system runs on GCE Debian 9.4 with MongoDB v3.6.5, WiredTiger storage engine and ...
0
votes
1answer
943 views

Mongodb rs.add() new member in STARTUP state

I have been running a shell script that starts mongod instances on different machines, initializing them (the first machines) and adding them to a replica set (the others). The following is a summary ...
1
vote
1answer
196 views

Can mongorestore replay oplog via mongos?

I'm using MongoDB 3.6.2 and configured to sharded cluster. I used mongodump to dump oplog from every shard and then restored by mongorestore via mongos, but I got below error. Failed: no oplog file ...