2

I am using Mongo Sharding to register page views on my website. We have hashed shard key to evenly distribute data in multiple shards. Then our aggregation queries run over time range at interval to aggregate this data and provide trends on site.
We came across Kafka for write distribution for heavy load and this kind of streaming.
I compared both systems and both provide distribution on partitions in a topic with leader follower approach. Kafka does it using multiple partition on different brokers with partition replication and Mongo does it with multiple shards which have replica sets.
As aggregation query will always be on time range than it will go to multiple shards/ partitions always.
My question is how we can compare which will provide better through put and run time scalability in case of heavy load as i understand both use same mechanism adding new partitions in case of Kafka or adding new shards in case of Mongo.
Please provide suggestions.

closed as primarily opinion-based by Stennie, hot2use, Marco, RDFozz, LowlyDBA Mar 26 '18 at 16:33

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Since you are comparing two very different solutions and implementations which depend on your requirements, this sounds like something you should evaluate in your own environment. – Stennie Mar 26 '18 at 3:47
0

My question is how we can compare which will provide better through put and run time scalability in case of heavy load as i understand both use same mechanism adding new partitions in case of Kafka or adding new shards in case of Mongo.

Definitely each and every technology they have their own pros and cons. In case on MongoDB and Kafka, it's depends on your types of data like.

MongoDB does do data distribution and balancing by itself, but it is only for Document data.

Streaming like Kafka or MapR Streams can handle streaming data on HDFS/MapRFS.

I am writing down some of pros and cons of MongoDB as well as Kafka as mention below. May be it will help out to you.

MongoDB

MongoDB stores data in JSON-like documents that can vary in structure, offering a dynamic, flexible schema. MongoDB was also designed for high availability and scalability, with built-in replication and auto-sharding.

Pros

  • Document-oriented storage

  • No sql

  • Ease of use

  • Fast

  • High performance

  • Free

  • Open source

  • Flexible

  • Replication & high availability

  • Easy to maintain

Cons

Very slowly for connected models that require joins

Kafka

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

Pros

  • High-throughput

  • Distributed

  • Scalable

  • High-Performance

  • Durable

  • Publish-Subscribe

  • Simple-to-use

  • Written in Scala and java. Runs on JVM

  • Open source

  • Message broker + Streaming system

Cons

  • Needs Zookeeper

  • Non-Java clients are second-class citizens

I would say, it all depends on your use cases.

If you only have one type of data, like Document data, and you do not need to process other data types, then yes, MongoDB alone is the best bet as of now. However in modern big data environment, different data sources and data types may require different processing engines.

For further your ref here, here and here

Not the answer you're looking for? Browse other questions tagged or ask your own question.