Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [sharding]

Sharding is the practice of logically dividing or partitioning data, usually using a specific key (referred to as a shard key), and then placing that data on separate hosts (subsequently known as shards). This is generally done to scale horizontally (more hosts) as opposed to vertically (more powerful hosts) and can provide significant cost benefits, particularly with the now ubiquitous availability of cloud computing.

1
vote
0answers
6 views

No shardIdentity document - MongoDB

Having a sharded cluster in production which has been running for a good year now without issues. Everything is on 4.0, classic 3-3-3(config) + 2 mongos. While revisiting our back-up strategy i ...
0
votes
0answers
36 views

[MongoDB]See Collections (and Files) created in a Shard on the Cluster

My structure: A Sharded Cluster with multiple Shards. Each Shard contains 1 Collection, which is pinned to it by tags. Every Shard is at the same time a ReplicaSet Secondary. It receives data from a ...
2
votes
0answers
32 views

MONGOS uselessly scanning indexed documents when counting on shards

I have a simple collection with 40M documents distributed over 2 shards. The documents contain a field named COUNTRY with a sparse index on it. db.contacts.count({ "COUNTRY": "US" }) I am trying to ...
1
vote
0answers
16 views

One of the mongo shards consuming too much disk space.

We have mongo cluster with 10 shards of each having 3 replicas. Recently we are facing an issue, exceptionally in a couple of shards, the disk space among all the 3 replicas has reached above 90%. ...
2
votes
1answer
21 views

Getting exception on running sh.status() command on MongoDB sharded cluster

I am running sh.status() command on my mongos instance of sharded Mongo cluster and I am getting below exception. mongos> sh.status() assert: command failed: { "ok" : 0, "errmsg" : "error ...
-1
votes
1answer
28 views

What is a beachhead? (replication / sharding)

I quote the following from this blog post from AWS: Because you’re running on MySQL, you can use one of the methods listed in the Aurora documentation to establish a beachhead in Aurora. What ...
-1
votes
0answers
14 views

Query performance issue in Mysql

Table is completely normalized its in innodb. Size of data is more than 600 million records. Even after indexing and optimization queries are too slow. Size of records increases rapidly and it would ...
0
votes
2answers
106 views

Migration of MariaDB/MySQL to another server without downtime

System : MariaDB (MySQL 5.5) Parameters : innodb_file_per_table=1 The database I have now is approaching its full capacity (94% full now, 6.4TB out of 6.9TB) I can't run OPTIMIZE TABLE to reclaim ...
1
vote
0answers
33 views

Choosing the correct shard key for MongoDB

I'm new to MongoDB. From the mongod documentation, I know that when choosing shard keys the key should always have high cardinality, so write operations are distributed to different shards (a kind of ...
1
vote
0answers
26 views

Horizontally Scaling Database Guide [closed]

We want to horizontally scale our existing MongoDB database which is running on one server. Due to increased user base, we can't scale it vertically anymore. We need to scale it horizontally through ...
1
vote
2answers
95 views

MySQL refuses UPDATE with (1071, 'Specified key was too long; max key length is 1024 bytes') sometimes

I'm using MySQL 5.7.2.23. I need to copy some columns between two tables (u -> m) and I am using the following update command: UPDATE m JOIN u ON u.id=m.user_id AND u.package_name=m....
1
vote
1answer
133 views

Is Postgres sharding available on Aurora (AWS)

I need to shard and/or partition my largeish Postgres db tables. The Postgres partitioning functionality seems crazy heavyweight (in terms of DDL). I see talk from <=2015 about pg_shard, but am ...
0
votes
1answer
135 views

Query sharded collection returns all the results from the primary shard AND from the other shards

I tried to set up a shard to test this functionality and maybe migrate some of our data into it. I followed the documentation guide https://docs.mongodb.com/v3.4/tutorial/convert-replica-set-to-...
0
votes
0answers
20 views

Recursive search and sharding

Our application collect logs and writes them into PostgreSQL database. Currently each client has a separate database deployed on client's machines. But we want to make it "software as a service", ...
1
vote
1answer
94 views

How to deal with unbounded growth in cassandra shards

Let's say we have a website like youtube where we have a comment session per e video. Users want to be able to comment and to see the comments made by other users by comment creation time. This is ...
1
vote
0answers
141 views

Sharding PostgreSQL with postgres_fdw (Foreign Data Wrapper)

We want to shard a single PostgreSQL 10.2 database by tenant (client id) to multiple servers. One of the easiest approach is to use Foreign Data Wrapper (postgres_fdw extension). It will looks like: ...
0
votes
0answers
84 views

Force remove a dead shard from mongodb

I have a cluster with 3 shards, and one of them died. There is no backup of the shard and I can live without the data. I'm trying to remove the shard from the cluster and restart with the 2 remaining ...
0
votes
1answer
48 views

what if mongos server goes down in shard cluster setup

I have followed the tuturial Deploy a Sharded Cluster I have setup config server replica set with 3 members (cfg1, cfg2, cfg3) [replica set name = configserverreplica] mongodb server replica set ...
1
vote
1answer
135 views

MongoDB sharding by monotonic id with zones for archiving

Situation: I have an ever-growing collection of documents which do have monotonically increasing unique id. All queries are direct lookup for documents with specified _id (no range queries nor ...
0
votes
1answer
32 views

MySQL- reducing memory footprint

Our InnoDB/Mysql5.6 database is growing at a fairly steady pace of doubling in size every year, currently at 22Gb. This is sales data; we have a number of merchants each generating millions of rows, ...
1
vote
1answer
37 views

MongoDB Shard distribution issue

I am facing an issue in the shard distribution. MongoDB Version: 3.0.15 I have a collection that is sharded. I have two shards. Output of db.getCollection('event').stats() { "sharded" : true, "...
1
vote
1answer
205 views

MongoDB Using String-Type Shard Key for Zone Ranges

Mongo lets any field type be used to set ranges for Shard Zones, but doesn't specify how analyses are conducted. My goal is to point queries and writes at the exact right server, WITHOUT using a ...
1
vote
1answer
30 views

MongoDB as a log storage. Choosing shard key

I'm designing a log storage system based on MongoDB. I want to shard a log collection to increase ingestion and capacity (distribute writes to several machines) while allow fast search. I should be ...
2
votes
1answer
316 views

MongoDB crashes with out-of-memory or is being killed by oom-killer

A two shards MongoDB database regularly crashes with out-of-memory error or is being killed by the oom-killer. The system runs on GCE Debian 9.4 with MongoDB v3.6.5, WiredTiger storage engine and ...
0
votes
1answer
462 views

Mongodb rs.add() new member in STARTUP state

I have been running a shell script that starts mongod instances on different machines, initializing them (the first machines) and adding them to a replica set (the others). The following is a summary ...
1
vote
1answer
97 views

Can mongorestore replay oplog via mongos?

I'm using MongoDB 3.6.2 and configured to sharded cluster. I used mongodump to dump oplog from every shard and then restored by mongorestore via mongos, but I got below error. Failed: no oplog file ...
0
votes
1answer
121 views

What's MongoDB's default behaviour when no sharded collection is specified

When we deploy a Sharded Cluster, without having a sharded collection in our database, what is the default behaviour of MongoDB? How does MongoDB, distributes the data across the shards?
0
votes
2answers
59 views

Mongodb chunksize issue

I have a collection which have size : 60 GB I want to shard this collection. Shard key is hash index and unique. I followed this document https://docs.mongodb.com/manual/reference/limits/#Sharding-...
2
votes
1answer
429 views

MongoDB - sharding an collection with data already in it

I'm using Mongo (shell version 2.6.12) in a clustered configuration. Right now I have a few collections sharded, and was looking to shard another collection. This collection already has data in it. ...
0
votes
1answer
80 views

Mongodb shard balancing failed

Data balancing fails on Mongo cluster with 1 mongos and 3 mongod instances. All the data for the collection fs.chunks stays in the same shard with the following error. Env: Ubuntu- AWS m3.large, ...
1
vote
1answer
87 views

MongoDB Zone Sharding without min & max key [closed]

I have a database structure that is similar to the following structure. User collection Sub collection0, with userId Sub collection1, with userId Sub collection2, with userId and so on Example: ...
0
votes
1answer
93 views

Handling Bus Error in Mongo Service

I have a Mongo service with which multiple hosts are interacting. The host on which the Mongo service is running is quite special -- it has 3TB of RAM. However, that host also throws intermittent Bus ...
1
vote
1answer
110 views

Mongo Zone Sharding Implementation Issue

As I am new with MongoDB. I have configured 2 config server, 2 shards and all are connected with mongod client. Also, One Mongos client is UP and Running. I am currenctly implementing Sharding ...
1
vote
1answer
40 views

Why have “unidirectional ‘mapping’” tables?

I've been reading about how Pinterest sharded their MySQL databases here. One of the things they talk about in that post is having “unidirectional ‘mapping’” tables such as CREATE TABLE ...
0
votes
2answers
748 views

Cannot connect to mongos

Following the MongoDB Sharding Documentation, I start a mongos using a configuration file with the specifics of my config server. It seems rather straightforward, yet I am running into an issue. I ...
0
votes
1answer
26 views

Mongo Shard Server Doubling as Another Shard's Replica Server

I'm going to be setting up a sharded + replicated mongo cluster, with application software servers co-located with mongod instances-- so that odds are a request is being processed on the same hardware ...
3
votes
2answers
5k views

MongoDB — Failed global initialization: Failed to open “/var/log/mongodb/mongod-config.log”

I'm trying to setup the config servers for mongodb sharding. I created a specific config file that is set to log to /var/log/mongodb/mongod-config.log. When I run mongod --config <path-to-config-...
1
vote
2answers
106 views

Does shardCollection acquire lock in mongo db

We are planning to shard a collection that contains few hundred gigs of data. I am going to use sh.shardCollection api for this. This is supposed to take hours to complete. I couldn't find any ...
1
vote
1answer
267 views

MongoDB 3.6 create local Shard db without replica

This is what I want to do: Create a Config Server Create 2 Shard Create a Mongos Router I don't want to replicate my data using replica set, I want to simulate that each shard has only a part of ...
1
vote
1answer
81 views

Why findOne() hangs on a sharded collection?

A MongoDB v3.6.4 database with a large sharded collection and WiredTiger engine hangs when issuing a findOne() or find() command without any parameters. The collection has only the default index on ...
2
votes
1answer
64 views

Huge logs are generated in mongo db shard logs

We are using Mongo db Shard set up. What i see that on primary of each shard log file keep growing to GBs. Even in a day it reaches to 10GB for single day. This appears to be slowing down Mongo ...
0
votes
0answers
25 views

Issue in Shard Distribution

I am facing an issue in the shard distribution. MongoDB Version: 3.0.15 I have a collection that is sharded. I have two shards. Output of db.getCollection('event').stats() { "sharded" : true, "...
1
vote
1answer
84 views

MongoDB write lock every n seconds during data import

I am experiencing an issue with a data import in a sharded cluster and I'd like some help to investigate and maybe find where the bottleneck is in my cluster. So I'm hosting a sharded cluster (3 ...
0
votes
1answer
207 views

How to reconfig config server connection setting from shard cluster?

I have a mongo cluster which includes one "mongos" one "config" (rscfg/a.a.a.a:27019) two "shard" Here is the situation, I added a new rs on config server and remove the old one rs.add("b.b.b....
1
vote
1answer
160 views

How to properly use moveChunk when chunks of certain range needs to be moved?

A MongoDB 3.6.3 database has two shards and a mongodump file with partial data of one collection needs to be restored. The first shard is named "fast" while second "slow". The idea is to restore the ...
0
votes
2answers
443 views

How to speed up MongoDB chunk moving between shards

On a MongoDB 3.6 in a replica set and WiredTiger engine we have a large collection with 7.7TB of data. We started to see severe performance issues (extreme CPU usage and slowdown of data replication) ...
2
votes
1answer
216 views

Kafka Partitions vs Mongo Sharding which one is better throughput [closed]

I am using Mongo Sharding to register page views on my website. We have hashed shard key to evenly distribute data in multiple shards. Then our aggregation queries run over time range at interval to ...
0
votes
1answer
99 views

mongodump sharded collection size is smaller after restoring

I need to change shard index from range to hashed in a sharded collection. So I need to mongodump and restore the collection. I have 3 shards with 3 replica sets.MongoDB version is: 3.2.11 I started ...
0
votes
0answers
23 views

Mysql shard by database name

I need some advice about shard a single mysql db into multiple logic db, splitted for a single developer. I had a single database that i must replicate into a multiple copies like this: dbmain -> ...
1
vote
1answer
90 views

MySQL InnoDB or MySQL NDB Cluster which one to use

We have an application with 8000 users accessing in parallel(max). We are trying to bring in horizontal scaling. I came across MySQL NDB cluster. Should I go with it? Just for auto sharding and other ...