Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [sharding]

Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.

0
votes
0answers
8 views

is it possible to make multiple mongo routers connected to just one shard on a mongoDB shard cluster?

I'd like to make a mongo cluster with multiple mongo routers(mongos) with just one shard(mongod) like thses figures. So I made two mongo routers named 'mongorouter-1', 'mongorouter-2', and also ...
1
vote
2answers
49 views

Can sharding help in this scenario? [closed]

I will have around 600 GB of data in single table with multiple columns in MySQL. Basically this table will store user events from application. Indexes are there on required columns. Question: If I ...
1
vote
2answers
49 views

Vertical / Horizontal Scaling Disk Size MySQL or MariaDB

I have been looking into this for a while and cannot find a suitable solution so would like to know if someone here can please help. I am trying to set up database back-end that will assist with ...
0
votes
0answers
37 views

Mongo - Find is returning differents results with each request

Currently in my collection has the following spec: Version: MongoDb - 4.0.6 Sharding: 3 shards Shard key: Publication identifier The shards are consistent The problem is due using the same operation ...
0
votes
0answers
19 views

Mongo zoned sharding with a single value

We have a database with around 50 collections, and 70 million records. All records belong to specific customers and have a customerId property. Currently all clients are in the USA. However, we're in ...
0
votes
1answer
79 views

Replication or Sharding (MongoDB)

I understand each of the terms but I do not understand very well if they can work simultaneously. In my case, for example, I have a machine with 20GB and I would like to know which is the best ...
0
votes
1answer
12 views

shard key specification

I have a document like so: { "_id" : ObjectId("52f29f91ddbd75540aba6dae"), "teams" : [ { "name" : "Washington Bullets", "score" : 100, "home" : ...
0
votes
1answer
30 views

MongoDB specifying shard key and sharding strategy

What is the syntax to state that a collection is to be sharded as location based, while using the range based sharding strategy. I know you have to use the the sh.shardCollection() command, but what ...
2
votes
1answer
24 views

What is the meaning of “Monotonically Changing Shard Keys”?

I am writing my first pubblication concerning MongoDB and more specific MongoDB Sharded clusters. On the documentantion I found that the shard key in case of ranged sharding should be "Non-...
0
votes
0answers
26 views

Reading from two shards using ProxySQL

I have a table that has been sharded between two machines, and the same table on each machine holds different data (not replicated). With ProxySQL, how can I create a mysql_query_rule that allows a ...
0
votes
1answer
24 views

documents don't go into newly added shard

I have added a shard to an existing cluster, and when I do db.printShardingStatus() I see a distribution: mongos> db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" :...
1
vote
0answers
36 views

adding documents to shard collection results in incorrect amount

I have added 2 shards to my cluster: sh.addShard("127.0.0.1:27023") { "shardAdded" : "shard0000", "ok" : 1, "operationTime" : Timestamp(1552115819, 4), "$clusterTime" : { "...
1
vote
1answer
29 views

mongodb sharding collection

Im trying to shard my collection. when I execute the command: sh.enableSharding("testdb") I get: { "ok" : 1, "operationTime" : Timestamp(1552083125, 2), "$clusterTime" : { "...
0
votes
0answers
25 views

Mongodb - add new node secondary in existing replica set shard

my mongodb configuration which i am trying to add. systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log storage: dbPath: /var/lib/mongo/mongo journal: enabled: ...
-3
votes
2answers
134 views

Replication and sharding combinations

In modern relational databases, what combinations of replication and sharding are possible? Actually, I'm interested in the following strategy. Is it possible to replicate a small table in order to ...
1
vote
1answer
35 views

MongoDB sharding and zone tag that has a range of 1 key value

Suppose I have a zone tag that has a range of exactly 1 key value, such as the letter A. Further, suppose I assign this zone to two shards. Will the data with a shard key of A be put on both shards? ...
0
votes
0answers
2 views

MongoDB, sharding with multi DB instance

I've a MongoDB instance with 40 DBs with same collections, every DB has a name like AWS1010, AWS1011 and so on. Now I've to activate sharding and I'd like to know if it is possible with that situation ...
1
vote
0answers
24 views

MongoDB ticket availability reaching zero

I recently came across intresting issue. I have a sharded cluster which has two replication sets with 2 secondaries. On my primaries, there were 300-400 connections at a point in time, shortly after ...
0
votes
3answers
60 views

MySQL insert into different databases tables based on PK value

Lets assume i have two databases, both having a table 'user_feedback', sharding data between the two databases. My goal is to insert all records where Primary key Id%2=0 goes to db1.user_feedback, and ...
0
votes
0answers
14 views

Syncing DDL accross Postgres shards

Our database writing needs exceed the capacity of a single Postgres cluster (since each cluster can only support 1 writer server). To combat this, we've sharded our data across 8 separate Postgres ...
0
votes
1answer
47 views

Managing 100s of similar databases

We are migrating to a model where every "customer" lives in its own dedicated schema (database per tenant). This will allow us to later move some accounts to different DB clusters or upgrade specific ...
1
vote
1answer
261 views

No shardIdentity document - MongoDB

Having a sharded cluster in production which has been running for a good year now without issues. Everything is on 4.0, classic 3-3-3(config) + 2 mongos. While revisiting our back-up strategy i ...
0
votes
0answers
38 views

[MongoDB]See Collections (and Files) created in a Shard on the Cluster

My structure: A Sharded Cluster with multiple Shards. Each Shard contains 1 Collection, which is pinned to it by tags. Every Shard is at the same time a ReplicaSet Secondary. It receives data from a ...
2
votes
0answers
45 views

MONGOS uselessly scanning indexed documents when counting on shards

I have a simple collection with 40M documents distributed over 2 shards. The documents contain a field named COUNTRY with a sparse index on it. db.contacts.count({ "COUNTRY": "US" }) I am trying to ...
1
vote
0answers
33 views

One of the mongo shards consuming too much disk space.

We have mongo cluster with 10 shards of each having 3 replicas. Recently we are facing an issue, exceptionally in a couple of shards, the disk space among all the 3 replicas has reached above 90%. ...
2
votes
1answer
34 views

Getting exception on running sh.status() command on MongoDB sharded cluster

I am running sh.status() command on my mongos instance of sharded Mongo cluster and I am getting below exception. mongos> sh.status() assert: command failed: { "ok" : 0, "errmsg" : "error ...
-1
votes
1answer
46 views

What is a beachhead? (replication / sharding)

I quote the following from this blog post from AWS: Because you’re running on MySQL, you can use one of the methods listed in the Aurora documentation to establish a beachhead in Aurora. What ...
0
votes
2answers
189 views

Migration of MariaDB/MySQL to another server without downtime

System : MariaDB (MySQL 5.5) Parameters : innodb_file_per_table=1 The database I have now is approaching its full capacity (94% full now, 6.4TB out of 6.9TB) I can't run OPTIMIZE TABLE to reclaim ...
1
vote
0answers
70 views

Choosing the correct shard key for MongoDB

I'm new to MongoDB. From the mongod documentation, I know that when choosing shard keys the key should always have high cardinality, so write operations are distributed to different shards (a kind of ...
1
vote
2answers
201 views

MySQL refuses UPDATE with (1071, 'Specified key was too long; max key length is 1024 bytes') sometimes

I'm using MySQL 5.7.2.23. I need to copy some columns between two tables (u -> m) and I am using the following update command: UPDATE m JOIN u ON u.id=m.user_id AND u.package_name=m....
1
vote
1answer
409 views

Is Postgres sharding available on Aurora (AWS)

I need to shard and/or partition my largeish Postgres db tables. The Postgres partitioning functionality seems crazy heavyweight (in terms of DDL). I see talk from <=2015 about pg_shard, but am ...
0
votes
1answer
271 views

Query sharded collection returns all the results from the primary shard AND from the other shards

I tried to set up a shard to test this functionality and maybe migrate some of our data into it. I followed the documentation guide https://docs.mongodb.com/v3.4/tutorial/convert-replica-set-to-...
1
vote
1answer
178 views

How to deal with unbounded growth in cassandra shards

Let's say we have a website like youtube where we have a comment session per e video. Users want to be able to comment and to see the comments made by other users by comment creation time. This is ...
1
vote
0answers
214 views

Sharding PostgreSQL with postgres_fdw (Foreign Data Wrapper)

We want to shard a single PostgreSQL 10.2 database by tenant (client id) to multiple servers. One of the easiest approach is to use Foreign Data Wrapper (postgres_fdw extension). It will looks like: ...
0
votes
0answers
134 views

Force remove a dead shard from mongodb

I have a cluster with 3 shards, and one of them died. There is no backup of the shard and I can live without the data. I'm trying to remove the shard from the cluster and restart with the 2 remaining ...
0
votes
1answer
91 views

what if mongos server goes down in shard cluster setup

I have followed the tuturial Deploy a Sharded Cluster I have setup config server replica set with 3 members (cfg1, cfg2, cfg3) [replica set name = configserverreplica] mongodb server replica set ...
1
vote
1answer
201 views

MongoDB sharding by monotonic id with zones for archiving

Situation: I have an ever-growing collection of documents which do have monotonically increasing unique id. All queries are direct lookup for documents with specified _id (no range queries nor ...
0
votes
1answer
34 views

MySQL- reducing memory footprint

Our InnoDB/Mysql5.6 database is growing at a fairly steady pace of doubling in size every year, currently at 22Gb. This is sales data; we have a number of merchants each generating millions of rows, ...
1
vote
1answer
59 views

MongoDB Shard distribution issue

I am facing an issue in the shard distribution. MongoDB Version: 3.0.15 I have a collection that is sharded. I have two shards. Output of db.getCollection('event').stats() { "sharded" : true, "...
1
vote
1answer
339 views

MongoDB Using String-Type Shard Key for Zone Ranges

Mongo lets any field type be used to set ranges for Shard Zones, but doesn't specify how analyses are conducted. My goal is to point queries and writes at the exact right server, WITHOUT using a ...
1
vote
1answer
40 views

MongoDB as a log storage. Choosing shard key

I'm designing a log storage system based on MongoDB. I want to shard a log collection to increase ingestion and capacity (distribute writes to several machines) while allow fast search. I should be ...
2
votes
1answer
468 views

MongoDB crashes with out-of-memory or is being killed by oom-killer

A two shards MongoDB database regularly crashes with out-of-memory error or is being killed by the oom-killer. The system runs on GCE Debian 9.4 with MongoDB v3.6.5, WiredTiger storage engine and ...
0
votes
1answer
768 views

Mongodb rs.add() new member in STARTUP state

I have been running a shell script that starts mongod instances on different machines, initializing them (the first machines) and adding them to a replica set (the others). The following is a summary ...
1
vote
1answer
167 views

Can mongorestore replay oplog via mongos?

I'm using MongoDB 3.6.2 and configured to sharded cluster. I used mongodump to dump oplog from every shard and then restored by mongorestore via mongos, but I got below error. Failed: no oplog file ...
0
votes
1answer
207 views

What's MongoDB's default behaviour when no sharded collection is specified

When we deploy a Sharded Cluster, without having a sharded collection in our database, what is the default behaviour of MongoDB? How does MongoDB, distributes the data across the shards?
0
votes
2answers
105 views

Mongodb chunksize issue

I have a collection which have size : 60 GB I want to shard this collection. Shard key is hash index and unique. I followed this document https://docs.mongodb.com/manual/reference/limits/#Sharding-...
2
votes
1answer
938 views

MongoDB - sharding an collection with data already in it

I'm using Mongo (shell version 2.6.12) in a clustered configuration. Right now I have a few collections sharded, and was looking to shard another collection. This collection already has data in it. ...
0
votes
1answer
146 views

Mongodb shard balancing failed

Data balancing fails on Mongo cluster with 1 mongos and 3 mongod instances. All the data for the collection fs.chunks stays in the same shard with the following error. Env: Ubuntu- AWS m3.large, ...
1
vote
1answer
125 views

MongoDB Zone Sharding without min & max key [closed]

I have a database structure that is similar to the following structure. User collection Sub collection0, with userId Sub collection1, with userId Sub collection2, with userId and so on Example: ...
0
votes
1answer
117 views

Handling Bus Error in Mongo Service

I have a Mongo service with which multiple hosts are interacting. The host on which the Mongo service is running is quite special -- it has 3TB of RAM. However, that host also throws intermittent Bus ...