1

I am tasked with developing a system that stores massive quantities of machine generated data (syslogs) from a number of server farms and I am wondering what tools you fine folks use such instances. Our project will need to keep up to 10 TB of data (maybe more in the future) and serve queries fairly quickly. I have been reading about columnular rdms' as well as nosql options, which all seem promising.

4
  • Define 'fairly quickly',what kind of budget, and what kind of queries.
    – rfusca
    Nov 5, 2012 at 20:05
  • Fairly quickly = a couple seconds or less, as inexpensive as possible (already have hardware), fairly simple joins on two tables (SQL right now) Nov 5, 2012 at 20:10
  • 1
    What hardware do you have? Crunching 10TB requires non-trivial hardware and what you have will determine possible solutions if you don't intend to buy anything or buy extremely little.
    – rfusca
    Nov 6, 2012 at 13:31
  • Any possiblity of going to 3rd party softwares like SPLUNK ? splunk.com/view/machine-data/SP-CAAACDC Note: I dont work for SPLUNK, but heard that it is more geared for analyzing big data esp. machine generated like logs etc.
    – Kin Shah
    Jul 18, 2013 at 15:39

2 Answers 2

3

There's a few systems specifically tailored to what you're requesting. I don't know exactly what kind of logs you're dealing with (application specific logs, system logs, web-server logs, a combination, etc.), so you may need to research a bit further.

Please pardon the lack of inline links - I currently don't have the reputation to post more than 2 links (you can find the writeup with inline links here).

Commercially there's:

  • Splunk
  • Sumo Logic
  • Loggly
  • Papertrail

On the open-source side there's:

  • Graylog2 (meant as an open-source alternative to Splunk)
  • OpenTSDB
  • Roll your own with Apache Flume and elasticsearch
  • Logstash

Personally, I'd look into Graylog2, Logstash, and (depending upon the requirements) OpenTSDB.

One thing to also look at is the underlying database system and how well it works with the hardware that you've already bought (i.e. MongoDB may not work very well with the hardware you have and the size of data that you have, however Hadoop/Hbase or a SQL server may work out better).

1
  • Great resources. I will have to check these out. Jul 18, 2013 at 19:07
1

Storing and analyzing system logs seems like an ideal fit for a hadoop cluster. Here is the intro on the hadoop page.

The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.