I am hoping some of you highly skilled DBAs can provide some insight as to the best DB to use for this dataset.
This is a relatively simple dataset with no relationship needed between the entries. This is a completely flat dataset.
6 fields: Name, username, email address, password, DOB, IP, address.
The problem is the size:
- Number of entries: 5 billion
- Approximate data size: 500gb
Current format of data: CSV - Approximately 3,000 different files.
We initially went with Elasticsearch as our search engine and have been working to import the data using JSON to convert it and dump it into ES.
The problem is ES keeps choking during the import process. I feel like we are trying to load too much data, and even so, given the hefty requirements to continue to run ES, I am no longer sure that is the best option.
Maybe the problem is limited funds for massive hardware. I am running everything on one server - 4CPU, 16gb ram. The ES index is on an SSD, which also happens to include the data that it is trying to import. In order to not get elasticsearch to chops, we had to write a special process in NodeJS to read/inject 10mb of data at a time. Some of these files can be GB in size, so it has to read 10mb at a time, inject, repeat. The process takes FOREVER. When working with a single 40gb file, the process took over 24 hours to complete.
I do not mind sacrificing search speed for ease of import and compatibility (as long as it is nothing crazy like a 60s search time).
What would you guys recommend? I have been looking into Hadoop as an alternative. What about MongoDB?