I need to upload daily CSV dumps to a blackbox of my choice, and it should be possible to run queries or create filters over the uploaded data. Daily dumps weight around 1TB and are basically id, datetime, latitude and longitude columns.

It's desirable to be able to store 1 year of info, so we're talking about 365TB here.

I believe that, provided enough bandwidth, I could upload the CSV to a cloud storage and import it to a Big data engine. For example: upload to Google Cloud Storage, then use it as source to import to Google Big Query, or upload to Amazon S3 and use it as source to import to Amazon Redshift.

At the beggining I thought hadoop was a third solution, but then I saw that hadoop can use Google BigQuery as its persistence layer, and that in front of hadoop there is yet another layer to pick (Apache Spark, Cloudera Impala).

So, at this point, I'm lost. In what use case should I go for hadoop? Is it safe to assume that for the most basic queries/filters I can rely on BigQuery alone, and leave open a further connection with hadoop only in case I need more elaborate Mapping/Reductions?

  • 3
    How much money have you got to throw at this? – Philᵀᴹ Jan 21 '15 at 19:44
  • I'd say I have 25K upfront to setup the layout, but the client that is generating these massive dumps (no pun intended) is willing to allocate around 1M for this project on a yearly basis. – ffflabs Jan 21 '15 at 19:55
  • Check Postgres-XL as well. – ypercubeᵀᴹ Oct 7 '16 at 8:41
  • whats the Use case? do you have experience and code already configured for Hadoop/spark? if so use Google Dataproc. I'd say pub/sub into GCP cloud storage, then use timeseries partition tables in BigQuery to query the data. Look into Lifecycle policies for the storage buckets based upon your usage. If you only access partitions of the data once a month, push it into nearline storage (performance is the same, but will be more expensive if you use more than once a month). You can also delete the data after a year or move it into Cold storage – DamagedGoods Jun 21 '19 at 10:51
  • 1
    This project was cancelled a few months after the initial question. The customer was a cellphone company wanting to make data mining or machine learning or whatever they thought they could do with many rows (but no meaningful data in them). Nowadays, if I wanted to perform queries over ridiculous amounts of data, I'd go for ElasticSearch. A couple of years ago I was working in a company where Postgres running on a x4-large-ish machine couldn't handle SELECT count(*) WHERE ... queries. We resorted to using estimates, then tried elasticsearch and it was amazing. Not cheap, of course. – ffflabs Jun 21 '19 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.