Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I am tasked with developing a system that stores massive quantities of machine generated data (syslogs) from a number of server farms and I am wondering what tools you fine folks use such instances. Our project will need to keep up to 10 TB of data (maybe more in the future) and serve queries fairly quickly. I have been reading about columnular rdms' as well as nosql options, which all seem promising.

share|improve this question
    
Define 'fairly quickly',what kind of budget, and what kind of queries. –  rfusca Nov 5 '12 at 20:05
    
Fairly quickly = a couple seconds or less, as inexpensive as possible (already have hardware), fairly simple joins on two tables (SQL right now) –  somecallmemike Nov 5 '12 at 20:10
1  
What hardware do you have? Crunching 10TB requires non-trivial hardware and what you have will determine possible solutions if you don't intend to buy anything or buy extremely little. –  rfusca Nov 6 '12 at 13:31
    
Any possiblity of going to 3rd party softwares like SPLUNK ? splunk.com/view/machine-data/SP-CAAACDC Note: I dont work for SPLUNK, but heard that it is more geared for analyzing big data esp. machine generated like logs etc. –  Kin Jul 18 '13 at 15:39

2 Answers 2

There's a few systems specifically tailored to what you're requesting. I don't know exactly what kind of logs you're dealing with (application specific logs, system logs, web-server logs, a combination, etc.), so you may need to research a bit further.

Please pardon the lack of inline links - I currently don't have the reputation to post more than 2 links (you can find the writeup with inline links here).

Commercially there's:

  • Splunk
  • Sumo Logic
  • Loggly
  • Papertrail

On the open-source side there's:

  • Graylog2 (meant as an open-source alternative to Splunk)
  • OpenTSDB
  • Roll your own with Apache Flume and elasticsearch
  • Logstash

Personally, I'd look into Graylog2, Logstash, and (depending upon the requirements) OpenTSDB.

One thing to also look at is the underlying database system and how well it works with the hardware that you've already bought (i.e. MongoDB may not work very well with the hardware you have and the size of data that you have, however Hadoop/Hbase or a SQL server may work out better).

share|improve this answer
    
Great resources. I will have to check these out. –  StanleyJohns Jul 18 '13 at 19:07

Storing and analyzing system logs seems like an ideal fit for a hadoop cluster. Here is the intro on the hadoop page.

The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.