I was going through Database storage engines and found out about LSM trees. I had also read about Kafka architecture and know that Kafka internally is a commit log. I want to know if Kafka internally uses LSM data structure for append-only store or uses some other data structure for storing data?
No Kafka , does not use LSM trees or any tree based data structure for data representation. It relies upon Sequential IO so that it is not relying upon JVM to maintain cache which would then be presenting as an overhead.
Kafka generally implements a queue based structure rather than a tree based structure (memory + merge sort on disk , something like sstables as disk store format).
Hope this helps a bit