Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [sharding]

Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.

0
votes
0answers
18 views

Replication or Sharding (MongoDB)

I understand each of the terms but I do not understand very well if they can work simultaneously. In my case, for example, I have a machine with 20GB and I would like to know which is the best ...
0
votes
1answer
9 views

shard key specification

I have a document like so: { "_id" : ObjectId("52f29f91ddbd75540aba6dae"), "teams" : [ { "name" : "Washington Bullets", "score" : 100, "home" : ...
0
votes
1answer
16 views

MongoDB specifying shard key and sharding strategy

What is the syntax to state that a collection is to be sharded as location based, while using the range based sharding strategy. I know you have to use the the sh.shardCollection() command, but what ...
2
votes
1answer
19 views

What is the meaning of “Monotonically Changing Shard Keys”?

I am writing my first pubblication concerning MongoDB and more specific MongoDB Sharded clusters. On the documentantion I found that the shard key in case of ranged sharding should be "Non-...
0
votes
0answers
17 views

Reading from two shards using ProxySQL

I have a table that has been sharded between two machines, and the same table on each machine holds different data (not replicated). With ProxySQL, how can I create a mysql_query_rule that allows a ...
0
votes
1answer
21 views

documents don't go into newly added shard

I have added a shard to an existing cluster, and when I do db.printShardingStatus() I see a distribution: mongos> db.printShardingStatus() --- Sharding Status --- sharding version: { "_id" :...
1
vote
0answers
32 views

adding documents to shard collection results in incorrect amount

I have added 2 shards to my cluster: sh.addShard("127.0.0.1:27023") { "shardAdded" : "shard0000", "ok" : 1, "operationTime" : Timestamp(1552115819, 4), "$clusterTime" : { "...
1
vote
1answer
17 views

mongodb sharding collection

Im trying to shard my collection. when I execute the command: sh.enableSharding("testdb") I get: { "ok" : 1, "operationTime" : Timestamp(1552083125, 2), "$clusterTime" : { "...
0
votes
0answers
23 views

Mongodb - add new node secondary in existing replica set shard

my mongodb configuration which i am trying to add. systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log storage: dbPath: /var/lib/mongo/mongo journal: enabled: ...
-3
votes
2answers
113 views

Replication and sharding combinations

In modern relational databases, what combinations of replication and sharding are possible? Actually, I'm interested in the following strategy. Is it possible to replicate a small table in order to ...
1
vote
1answer
31 views

MongoDB sharding and zone tag that has a range of 1 key value

Suppose I have a zone tag that has a range of exactly 1 key value, such as the letter A. Further, suppose I assign this zone to two shards. Will the data with a shard key of A be put on both shards? ...
0
votes
0answers
2 views

MongoDB, sharding with multi DB instance

I've a MongoDB instance with 40 DBs with same collections, every DB has a name like AWS1010, AWS1011 and so on. Now I've to activate sharding and I'd like to know if it is possible with that situation ...
1
vote
0answers
13 views

MongoDB ticket availability reaching zero

I recently came across intresting issue. I have a sharded cluster which has two replication sets with 2 secondaries. On my primaries, there were 300-400 connections at a point in time, shortly after ...
0
votes
3answers
52 views

MySQL insert into different databases tables based on PK value

Lets assume i have two databases, both having a table 'user_feedback', sharding data between the two databases. My goal is to insert all records where Primary key Id%2=0 goes to db1.user_feedback, and ...
0
votes
0answers
12 views

Syncing DDL accross Postgres shards

Our database writing needs exceed the capacity of a single Postgres cluster (since each cluster can only support 1 writer server). To combat this, we've sharded our data across 8 separate Postgres ...
0
votes
1answer
42 views

Managing 100s of similar databases

We are migrating to a model where every "customer" lives in its own dedicated schema (database per tenant). This will allow us to later move some accounts to different DB clusters or upgrade specific ...
1
vote
1answer
114 views

No shardIdentity document - MongoDB

Having a sharded cluster in production which has been running for a good year now without issues. Everything is on 4.0, classic 3-3-3(config) + 2 mongos. While revisiting our back-up strategy i ...
0
votes
0answers
37 views

[MongoDB]See Collections (and Files) created in a Shard on the Cluster

My structure: A Sharded Cluster with multiple Shards. Each Shard contains 1 Collection, which is pinned to it by tags. Every Shard is at the same time a ReplicaSet Secondary. It receives data from a ...
2
votes
0answers
40 views

MONGOS uselessly scanning indexed documents when counting on shards

I have a simple collection with 40M documents distributed over 2 shards. The documents contain a field named COUNTRY with a sparse index on it. db.contacts.count({ "COUNTRY": "US" }) I am trying to ...
1
vote
0answers
21 views

One of the mongo shards consuming too much disk space.

We have mongo cluster with 10 shards of each having 3 replicas. Recently we are facing an issue, exceptionally in a couple of shards, the disk space among all the 3 replicas has reached above 90%. ...
2
votes
1answer
29 views

Getting exception on running sh.status() command on MongoDB sharded cluster

I am running sh.status() command on my mongos instance of sharded Mongo cluster and I am getting below exception. mongos> sh.status() assert: command failed: { "ok" : 0, "errmsg" : "error ...
-1
votes
1answer
37 views

What is a beachhead? (replication / sharding)

I quote the following from this blog post from AWS: Because you’re running on MySQL, you can use one of the methods listed in the Aurora documentation to establish a beachhead in Aurora. What ...
0
votes
2answers
157 views

Migration of MariaDB/MySQL to another server without downtime

System : MariaDB (MySQL 5.5) Parameters : innodb_file_per_table=1 The database I have now is approaching its full capacity (94% full now, 6.4TB out of 6.9TB) I can't run OPTIMIZE TABLE to reclaim ...
1
vote
0answers
54 views

Choosing the correct shard key for MongoDB

I'm new to MongoDB. From the mongod documentation, I know that when choosing shard keys the key should always have high cardinality, so write operations are distributed to different shards (a kind of ...
1
vote
2answers
147 views

MySQL refuses UPDATE with (1071, 'Specified key was too long; max key length is 1024 bytes') sometimes

I'm using MySQL 5.7.2.23. I need to copy some columns between two tables (u -> m) and I am using the following update command: UPDATE m JOIN u ON u.id=m.user_id AND u.package_name=m....
1
vote
1answer
256 views

Is Postgres sharding available on Aurora (AWS)

I need to shard and/or partition my largeish Postgres db tables. The Postgres partitioning functionality seems crazy heavyweight (in terms of DDL). I see talk from <=2015 about pg_shard, but am ...
0
votes
1answer
210 views

Query sharded collection returns all the results from the primary shard AND from the other shards

I tried to set up a shard to test this functionality and maybe migrate some of our data into it. I followed the documentation guide https://docs.mongodb.com/v3.4/tutorial/convert-replica-set-to-...
1
vote
1answer
128 views

How to deal with unbounded growth in cassandra shards

Let's say we have a website like youtube where we have a comment session per e video. Users want to be able to comment and to see the comments made by other users by comment creation time. This is ...
1
vote
0answers
182 views

Sharding PostgreSQL with postgres_fdw (Foreign Data Wrapper)

We want to shard a single PostgreSQL 10.2 database by tenant (client id) to multiple servers. One of the easiest approach is to use Foreign Data Wrapper (postgres_fdw extension). It will looks like: ...
0
votes
0answers
113 views

Force remove a dead shard from mongodb

I have a cluster with 3 shards, and one of them died. There is no backup of the shard and I can live without the data. I'm trying to remove the shard from the cluster and restart with the 2 remaining ...
0
votes
1answer
76 views

what if mongos server goes down in shard cluster setup

I have followed the tuturial Deploy a Sharded Cluster I have setup config server replica set with 3 members (cfg1, cfg2, cfg3) [replica set name = configserverreplica] mongodb server replica set ...
1
vote
1answer
172 views

MongoDB sharding by monotonic id with zones for archiving

Situation: I have an ever-growing collection of documents which do have monotonically increasing unique id. All queries are direct lookup for documents with specified _id (no range queries nor ...
0
votes
1answer
32 views

MySQL- reducing memory footprint

Our InnoDB/Mysql5.6 database is growing at a fairly steady pace of doubling in size every year, currently at 22Gb. This is sales data; we have a number of merchants each generating millions of rows, ...
1
vote
1answer
46 views

MongoDB Shard distribution issue

I am facing an issue in the shard distribution. MongoDB Version: 3.0.15 I have a collection that is sharded. I have two shards. Output of db.getCollection('event').stats() { "sharded" : true, "...
1
vote
1answer
268 views

MongoDB Using String-Type Shard Key for Zone Ranges

Mongo lets any field type be used to set ranges for Shard Zones, but doesn't specify how analyses are conducted. My goal is to point queries and writes at the exact right server, WITHOUT using a ...
1
vote
1answer
35 views

MongoDB as a log storage. Choosing shard key

I'm designing a log storage system based on MongoDB. I want to shard a log collection to increase ingestion and capacity (distribute writes to several machines) while allow fast search. I should be ...
2
votes
1answer
392 views

MongoDB crashes with out-of-memory or is being killed by oom-killer

A two shards MongoDB database regularly crashes with out-of-memory error or is being killed by the oom-killer. The system runs on GCE Debian 9.4 with MongoDB v3.6.5, WiredTiger storage engine and ...
0
votes
1answer
632 views

Mongodb rs.add() new member in STARTUP state

I have been running a shell script that starts mongod instances on different machines, initializing them (the first machines) and adding them to a replica set (the others). The following is a summary ...
1
vote
1answer
134 views

Can mongorestore replay oplog via mongos?

I'm using MongoDB 3.6.2 and configured to sharded cluster. I used mongodump to dump oplog from every shard and then restored by mongorestore via mongos, but I got below error. Failed: no oplog file ...
0
votes
1answer
161 views

What's MongoDB's default behaviour when no sharded collection is specified

When we deploy a Sharded Cluster, without having a sharded collection in our database, what is the default behaviour of MongoDB? How does MongoDB, distributes the data across the shards?
0
votes
2answers
81 views

Mongodb chunksize issue

I have a collection which have size : 60 GB I want to shard this collection. Shard key is hash index and unique. I followed this document https://docs.mongodb.com/manual/reference/limits/#Sharding-...
2
votes
1answer
683 views

MongoDB - sharding an collection with data already in it

I'm using Mongo (shell version 2.6.12) in a clustered configuration. Right now I have a few collections sharded, and was looking to shard another collection. This collection already has data in it. ...
0
votes
1answer
111 views

Mongodb shard balancing failed

Data balancing fails on Mongo cluster with 1 mongos and 3 mongod instances. All the data for the collection fs.chunks stays in the same shard with the following error. Env: Ubuntu- AWS m3.large, ...
1
vote
1answer
108 views

MongoDB Zone Sharding without min & max key [closed]

I have a database structure that is similar to the following structure. User collection Sub collection0, with userId Sub collection1, with userId Sub collection2, with userId and so on Example: ...
0
votes
1answer
104 views

Handling Bus Error in Mongo Service

I have a Mongo service with which multiple hosts are interacting. The host on which the Mongo service is running is quite special -- it has 3TB of RAM. However, that host also throws intermittent Bus ...
1
vote
1answer
129 views

Mongo Zone Sharding Implementation Issue

As I am new with MongoDB. I have configured 2 config server, 2 shards and all are connected with mongod client. Also, One Mongos client is UP and Running. I am currenctly implementing Sharding ...
1
vote
1answer
43 views

Why have “unidirectional ‘mapping’” tables?

I've been reading about how Pinterest sharded their MySQL databases here. One of the things they talk about in that post is having “unidirectional ‘mapping’” tables such as CREATE TABLE ...
0
votes
2answers
1k views

Cannot connect to mongos

Following the MongoDB Sharding Documentation, I start a mongos using a configuration file with the specifics of my config server. It seems rather straightforward, yet I am running into an issue. I ...
0
votes
1answer
28 views

Mongo Shard Server Doubling as Another Shard's Replica Server

I'm going to be setting up a sharded + replicated mongo cluster, with application software servers co-located with mongod instances-- so that odds are a request is being processed on the same hardware ...
3
votes
3answers
7k views

MongoDB — Failed global initialization: Failed to open “/var/log/mongodb/mongod-config.log”

I'm trying to setup the config servers for mongodb sharding. I created a specific config file that is set to log to /var/log/mongodb/mongod-config.log. When I run mongod --config <path-to-config-...