I am hoping some of you highly skilled DBAs can provide some insight as to the best DB to use for this dataset.

This is a relatively simple dataset with no relationship needed between the entries. This is a completely flat dataset.

6 fields: Name, username, email address, password, DOB, IP, address.

The problem is the size:

  • Number of entries: 5 billion
  • Approximate data size: 500gb

Current format of data: CSV - Approximately 3,000 different files.

We initially went with Elasticsearch as our search engine and have been working to import the data using JSON to convert it and dump it into ES.

The problem is ES keeps choking during the import process. I feel like we are trying to load too much data, and even so, given the hefty requirements to continue to run ES, I am no longer sure that is the best option.

Maybe the problem is limited funds for massive hardware. I am running everything on one server - 4CPU, 16gb ram. The ES index is on an SSD, which also happens to include the data that it is trying to import. In order to not get elasticsearch to chops, we had to write a special process in NodeJS to read/inject 10mb of data at a time. Some of these files can be GB in size, so it has to read 10mb at a time, inject, repeat. The process takes FOREVER. When working with a single 40gb file, the process took over 24 hours to complete.

I do not mind sacrificing search speed for ease of import and compatibility (as long as it is nothing crazy like a 60s search time).

What would you guys recommend? I have been looking into Hadoop as an alternative. What about MongoDB?

  • Sadly my initial answer (which would need to be improve for sure) has been removed and all the discussion we started as well. So I'll be happy to hear what are the blockers with elasticsearch. You can ask for questions on discuss.elastic.co where I'll be happy to help. – dadoonet Feb 11 '18 at 11:10
  • It appears that you may be working with data files much larger than your current deployment environment will efficiently support (16GB of total RAM to work with ~500GB of raw data to import). Even with compression, working with your data and indexes is likely to involve a lot paging of data between disk and memory depending on the size of your working set. Irrespective of the data storage approach you chose, you need to do some capacity planning to set realistic expectations on performance vs resources vs cost. Perhaps you can scale by sharding with some lower cost hardware or cloud instances. – Stennie Feb 12 '18 at 6:44

As MongoDB document sources Here MongoDB can certainly be considered a Big Data solution, it’s worth noting that it’s really a general-purpose platform, designed to replace or enhance existing RDBMS systems, giving it a healthy variety of use cases. As MongoDB BOL some of the documented features are Here

  1. MongoDB stores data in flexible, JSON-like documents, meaning fields can vary from document to document and data structure can be changed over time

  2. The document model maps to the objects in your application code, making data easy to work with

  3. Ad hoc queries, indexing, and real time aggregation provide powerful ways to access and analyze your data

  4. MongoDB is a distributed database at its core, so high availability, horizontal scaling, and geographic distribution are built in and easy to use

  5. MongoDB is free and open-source, published under the GNU Affero General Public License.

While Hadoop has a specific purpose, and is not meant as a replacement for transactional RDBMS systems, but rather as a supplement to them, as a replacement of archiving systems, or a handful of other use cases.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.