0

General

We generate about 10GB of records each day. Each record is composed of four columns: id,A,B,C.

The column id must be unique but will be never queried. Columns A,B,C must be indexed (or some other equivalent technique to reach them quickly) because they will undergone to very intensive queries every two or more hours.

We have a cluster composed of some ARM 64bit nodes. Max RAM available for each node is 8GB and a max of 1,5GHz of CPU computing power.

We tired to use MongoDB Shard, it was very funny using it, but we noticed that when we reached about 50GB of total storage over the cluster nodes, the entire MongoDB system becomes extremely slow in the inserting operation (but the queries over A,B,C remained very performant).

We don't know if it could be a MongoDB sharding misconfiguration issue or it is a limit of MongoDB. But overall we are still asking ourselves why the cluster becomes slower and slower as more data is ingested.

We tried to "play" with configuration varying the number of mongod-config, mongod-shard and mongos (the router), but each time, after a certain amount of data ingested, the system becomes very slow (only in the insert operation).

Our requisites are:

  1. capability of ingest continuously large amount of data daily (we have no problems to scalate horizontally the cluster)
  2. capability to perform instantaneous queries over A,B,C frequently

We are using Ubuntu 64-bit ARM version.

Nodes of the cluster communicates with each other via LAN ethernet.

The schema of the database

Data in all four columns are strings of now fixed length, but without whitespaces.

Input is mixed:

  1. many data comes from very large CSV files
  2. other data comes from API calls form other units (out of the cluster)

Our biggest problem is the data ingestion: after an amount of data, the systems becomes slow.

Mandatory: we cannot prune no data from the database.

Query

There is a single node that frequently performs some queries over A, B or C and is very important that these queries are fast as fast is the light.

Is very important that the system is capable to ingest a lot of data.

Side considerations

A. It will be a nice to have to keep the FS on SSD journaled, but we are conscious that it weight over the performance, so we are open to disable it.

B. The number of records inserted per second must be as fast as possible. This is the most important critical issue of my problem. So, thinking of the famous "5 Vs of Big Data", we can tailor it on our system as follows:

- Volume: YES
- Velocity: YES
- Variety: NO
- Veracity (quality): YES
- Value: NO

One more thing, and maybe the new one to be put at the top of the 5 Vs pyramid is:

- Voracity (capability of ingest large amount of data quickly): YES^2

The input file could be a CSV or sent via API as JSON such as two following examples:

id,A,B,C
qwertyuiop,asdfghjkl,zxcvbnm,plmokn
poiuytrewq,lkjhgfdsa,mnbvcxz,qazwsx

or via API (possible to send many records by one single API call):

[
  {
    "id": "qwertyuiop",
    "A": "asdfghjkl",
    "B": "zxcvbnm",
    "C": "plmokn"
  },
  {
    "id": "poiuytrewq",
    "A": "lkjhgfdsa",
    "B": "mnbvcxz",
    "C": "qazwsx"
  }
]

Question could be synthesized as: why MongoDB Shard becomes slower as more data is ingested? How can we improve this system?

1
  • Same questions: What is your sharding key? Can you provide more realistic data? How do you insert the CSV data? What are your indexes? Aug 15 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.