I am tasked with developing a system that stores massive quantities of machine generated data (syslogs) from a number of server farms and I am wondering what tools you fine folks use such instances. Our project will need to keep up to 10 TB of data (maybe more in the future) and serve queries fairly quickly. I have been reading about columnular rdms' as well as nosql options, which all seem promising.
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
There's a few systems specifically tailored to what you're requesting. I don't know exactly what kind of logs you're dealing with (application specific logs, system logs, web-server logs, a combination, etc.), so you may need to research a bit further.
Please pardon the lack of inline links - I currently don't have the reputation to post more than 2 links (you can find the writeup with inline links here).
On the open-source side there's:
Personally, I'd look into Graylog2, Logstash, and (depending upon the requirements) OpenTSDB.
One thing to also look at is the underlying database system and how well it works with the hardware that you've already bought (i.e. MongoDB may not work very well with the hardware you have and the size of data that you have, however Hadoop/Hbase or a SQL server may work out better).
Storing and analyzing system logs seems like an ideal fit for a hadoop cluster. Here is the intro on the hadoop page.