In my app I get data from two sources, from source 1 I get around 125000 records per seconds, say record type A, and from source 2 I get around 10000 records, per seconds ,say record type B.
Now I want to find the correlation of both record types in real time and store the final record say type C. That means to process billions of data per seconds. The record sizes are around 100 bytes each. I will be having a number of servers, (15) located at different sites for data collection but the final correlated record type C must be stored centrally.
Which could be the best way to achieve the processing time requirements. I have read about Hadoop, Hive, Cassandra, MongoDB, CouchDB, BigTable, MapReduce algorithm etc. I am not even sure whether I am going in the right direction or not.
Thanks in advance,