MuleESB / Apache Kafka Interview questions

• Producer is an application that sends message/data to the Kafka.
• Consumer is also an application that receives data from Kakfa,
• Broker is nothing but the KAFKA Server,
• Cluster, group of computer nodes sharing workload,
• Topic, Kafka stream
• Partitions, a portion of Topics
• Offset, an unique id for a message within a partition,
• and Consumer Groups, group of consumers acting as a single logical unit.

Topic Name, Partition Number and Offset id identifies a message.

Apache Kafka is a publish-subscribe open source message broker application.

It is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. It is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant and fast.

Zookeeper builds coordination between different nodes in a cluster. Apache Kafka uses Zookeeper to recover from previously committed offset if any node fails because it works as periodically commit offset.

Apache ZooKeeper is an open-source server which enables highly reliable distributed coordination.

ZooKeeper acts as a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Apache Kafka is a community distributed event streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being created and open-sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform.

Confluent Platform improves Kafka with additional community and commercial features designed to enhance the streaming experience of both operators and developers in production, at a massive scale.

Kafka’s zero-copy principle is an optimization technique that enables the operating system to transfer data directly from the disk (page cache) to the network socket, bypassing the application (JVM) buffer entirely. This reduces context switches between user/kernel mode and eliminates unnecessary CPU-intensive memory copies, dramatically improving throughput and reducing latency.

Kafka uses sequential I/O as a primary design choice to achieve its high throughput and performance. By treating data as an append-only log, Kafka can leverage the optimal performance characteristics of both traditional hard disks (HDDs) and Solid State Drives (SSDs), avoiding the significant latency penalties associated with random disk access.

• High Throughput: Sequential writes enable Kafka to handle massive volumes of data (millions of messages per second) with very low latency.
• Cost-Effectiveness: It allows Kafka to use cheaper, high-capacity HDDs effectively, rather than requiring expensive, high-performance storage systems like specialized random-access databases might.
• Simplified Caching Logic: By offloading much of the data management to the OS page cache, Kafka avoids complex in-application caching logic, which can lead to high garbage collection overhead in the Java Virtual Machine (JVM).