The Past and Present of Stream Processing (Part 13): Kafka Streams — A Lean and Agile King’s Army

Introduction

For a product to achieve success, there are generally two paths it can take:

The first is to cultivate deeply within its own domain, developing itself and achieving excellence in all areas within that field. We call this “involution” — essentially competing with yourself to the extreme.

The second is to venture beyond its own domain and expand into related or unrelated fields, adding new product capabilities. We call this “evolution” — essentially competing with others.

More often, products need to pursue both approaches simultaneously — competing in all directions.

Apache Kafka, as the leader in stream data storage, found limited room for internal competition because being unrivaled can be quite lonely. So it began external expansion, setting its sights on stream data processing. Thus, Kafka Streams was born.

Background

Kafka Streams was first released in May 2016 with Kafka version 0.10.0.0 as a client library for building streaming applications that process data stored in Kafka. Created by the Confluent/Apache Kafka team, it was designed to fill a critical gap in the stream processing ecosystem: the need for a lightweight, Kafka-native stream processing solution without requiring separate cluster management.

Before Kafka Streams emerged, organizations faced the following choices: either use heavyweight frameworks like Apache Spark or Flink that require separate clusters, or build custom stream processing logic from scratch using Kafka’s consumer/producer APIs.

The emergence of Kafka Streams filled this gap. Kafka Streams applications don’t run inside brokers, but rather in independent JVM instances or completely independent clusters, making stream processing as simple as adding a library dependency to your application. Kafka Streams is built on top of Kafka’s producer/consumer APIs, leveraging Kafka’s own capabilities for data parallel processing, distributed coordination, and fault recovery.

Design Architecture

Processor Topology

The processor topology defines the stream processing computation logic of the application — how input data is transformed into output data. The topology is a graph composed of stream processors (nodes) connected by streams (edges) or shared state stores.

The topology is essentially the same concept as the DAG (Directed Acyclic Graph) we’ve seen in other stream processing tools — flowing from data source to data destination nodes.

There are three types of processors:

1. Source Processor: A special processor with no upstream processors that generates input streams by consuming records from one or more Kafka topics and forwarding them to downstream processors.
2. Stream Processor: Intermediate nodes that receive records, apply operations, and produce output.
3. Sink Processor: A special processor with no downstream processors that sends received records to specified Kafka topics.

Source — Processor — Sink are also old friends in stream processing.

The topology is the logical definition of a stream data job.

Stream Tasks

Stream Tasks are the physical definition of stream data jobs.

• A Stream Task executes the topology
• 1 partition == 1 stream task — A stream task always works on one Kafka partition
• Since Kafka topics are partitioned, the unit of parallelism in Kafka Streams is the partition. The process of executing stream processing logic on that partition is called a stream task
• There is zero sharing between two stream tasks — they are independent

The share-nothing architecture between stream tasks means each task independently processes its assigned partitions. This design achieves:

• Lock-free parallel processing
• Simplified fault isolation
• Easier horizontal scaling

Stream tasks are CPU-intensive because they require deserializing, processing, and re-serializing large amounts of data.

A single CPU core cannot truly process two partitions in parallel, so we need… multithreading!

Stream Threads

Stream Threads are threads that run stream tasks.

• One stream thread can run multiple stream tasks
• One thread owns a single consumer and producer client. Stream tasks within the thread share these Kafka clients
• For optimal performance, you want at least one thread per CPU core
• In cases where stream tasks are IO-bound — you want more threads than cores (e.g., when calling external APIs)

Kafka was created because its data couldn’t simply be placed on a single machine, requiring data sharding into partitions. This should also apply to Kafka Streams, as processing is much more complex than simply storing data on disk.

Let’s create a distributed system:

Stream App (Node) — An instance of the KafkaStreams() class. In practice, you run one of these on each node (virtual machine). These applications coordinate their work through Kafka.

The hierarchical relationship of Kafka Streams resources is as follows:

Stream App Instance
└── Stream Thread [shares Kafka clients]
    └── Stream Task [processes a single partition]

Any distributed system faces countless challenges:

• What happens if a stream node goes down? Who takes over the tasks and how?
• What happens if a new stream node comes online? Which tasks does it take over, from whom, and how?
• How do you warm up another node (build the same state) for future failover?
• How do you smoothly handle upgrades that change the coordination protocol schema?
• How is partition progress persisted across nodes? If a new stream node takes over another node’s work, how does it know from which offset to start?

There’s a solution you may already be familiar with — Kafka’s consumer group protocol.

Each consumer in a Streams application uses the same consumer group ID, together forming the overall Kafka Streams consumer group.

During a group rebalance, stream tasks (partitions) are assigned to stream threads (consumer instances). This is how a distributed Kafka Streams application with many nodes and threads distributes work across the entire system.

Handling Distributed System Challenges

1. Node Failure Handling

• Problem: Stream node goes down
• Solution: Consumer group automatically detects failed consumers
• Mechanism:
• Triggers rebalance
• Reassigns failed node’s partitions to healthy nodes
• New nodes continue processing from the last committed offset

2. Dynamic Node Scaling

• Problem: New stream node comes online
• Solution: Consumer group protocol automatically triggers rebalance
• Mechanism:
• Detects new consumer joining the group
• Redistributes partitions to balance load
• New node takes over some tasks from existing nodes

3. State Warming and Failover

• Problem: Need to warm up backup nodes
• Solution:
• Backup nodes can join as group members
• Restore state through changelog topics
• Ready to take over work immediately when prepared

In summary, by leveraging Kafka’s native APIs, Kafka Streams can quickly and simply build distributed stream data processing applications.

State Management

Local State Storage: For stateful operations, Kafka Streams uses local state stores made fault-tolerant through associated changelog topics stored in Kafka. RocksDB is used as the default storage to maintain local state on compute nodes.

RocksDB Integration: RocksDB is an embedded key-value store that runs in the same process as the Kafka Streams application, eliminating network calls during reads and writes. This reduces latency and ensures high throughput for stateful operations.

State Recovery Process: When Kafka Streams restarts after a failure, it reads the last committed offset from checkpoint files to identify the latest state persisted to RocksDB, then replays state changes from the changelog topic starting from that offset.

Developer Experience

Kafka Streams provides two user interfaces:

• DSL: Provides operations for common stream processing patterns, mapping inputs and outputs to create streaming applications with complex topologies
• Processor API: Low-level API for fine-grained control over processing logic and state management

Here’s a simple DSL example. The same logic using the Processor API might require 200 lines of code, which I won’t demonstrate here:

StreamsBuilder builder = new StreamsBuilder();
// Read data stream from input topic
KStream<String, String> clicks = builder.stream("user-clicks");
// Processing logic
clicks
    // Parse JSON and extract userId
    .selectKey((key, value) -> extractUserId(value))
    
    // Define 5-minute time window (tumbling window)
    .groupByKey()
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
    
    // Count aggregation
    .count(Materialized.as("user-clicks-store"))
    
    // Convert to stream for further processing
    .toStream()
    
    // Filter users with click count > 10
    .filter((windowedKey, count) -> count > 10)
    
    // Format output: extract user ID and count
    .map((windowedKey, count) -> {
        String userId = windowedKey.key();
        String output = String.format("{\"userId\":\"%s\",\"clickCount\":%d}", 
            userId, count);
        return KeyValue.pair(userId, output);
    })
    
    // Write to output topic
    .to("active-users");
// Start Streams application
KafkaStreams streams = new KafkaStreams(builder.build(), props);
// Graceful shutdown hook
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
streams.start();

Advantages and Disadvantages Analysis

Advantages of Kafka Streams

Simplicity and Low Barrier to Entry

• No separate cluster management required
• Standard Java/Scala application deployment
• Seamless integration with existing microservice architectures

Kafka Native Integration

• Deep integration with Kafka’s server-side cluster technology
• Leverages Kafka’s reliability, durability, and exactly-once guarantees
• Supports complex data pipelines through Kafka Connect and Kafka Streams API

Fault Tolerance and Reliability

• State stores are made fault-tolerant through associated changelog topics. Kafka Streams makes RocksDB fault-tolerant by replicating state store data to Kafka topics

Stateful Processing

• Kafka Streams can leverage stateful processing, considering duration through windows, and considering state by converting streams to tables and back to streams

Performance

• Low latency from embedded state storage
• High throughput leveraging Kafka capabilities

Disadvantages

Limited Language Support

• Kafka Streams has limited support for non-JVM programming languages, only supporting Java and Scala

Kafka Dependency

• Tight coupling with Kafka may be a limitation for systems that don’t widely use Kafka (dependency on Kafka is both an advantage and disadvantage, depending on your application context)

Operational Complexity

• Tuning RocksDB is very difficult, especially when there are multiple RocksDB instances on a single node
• Complex rebalance protocol for stateful applications
• State recovery can cause delays during failures

Missing Advanced Features

• Compared to comprehensive stream processing solutions like Flink, Kafka Streams may lack some advanced features, especially in complex event processing and handling large-scale data streams

Memory Management Challenges

• When multiple RocksDB instances run on a single node, they don’t share cache or write buffers by default. Physical memory limitations mean buffers are tuned relatively small or flushed before filling up

Conclusion

Kafka Streams represents a new paradigm and approach to stream processing — transforming it from an infrastructure problem to a software client problem. Its embedded nature, Kafka-native design, and exactly-once guarantees make it very suitable for event-driven microservices and real-time analytics. However, organizations must carefully consider language limitations, operational complexity in RocksDB tuning, and whether a Kafka-centric architecture aligns with their broader infrastructure strategy.

In my exchanges with some developers, their feedback on Kafka Streams has been quite positive. Developers really appreciate this simple, lightweight stream processing tool.

Here I must mention another stream storage system, Pulsar, Kafka’s main competitor, which launched Apache Pulsar Functions around the same time period.

Pulsar Functions is also bound to Pulsar, providing simple data format transformation, filtering, and routing capabilities. Although its features are far from Kafka Streams, if you do a feature checklist analysis, it can also check some boxes.

What peers have, I must have too — in short, just keep competing.

Additional Note: There’s a project on GitHub called streamiz-kafka-net that is a .NET stream processing library for Apache Kafka inspired by Kafka Streams. If you’re in the Microsoft development stack and looking for a stream processing tool, you might consider it.

Thanks for reading! I’m the co-founder and CTO of Timeplus. Proton https://github.com/timeplus-io/proton is our open-source, high-performance streaming SQL engine built in C++. It enables you to process real-time data streams using standard SQL, perfect for real-time security and monitoring, real-time machine learning pipelines, IoT analytics, and more.

Feel free to star us on GitHub or join our Slack community at https://timeplus.com/slack to discuss the future of streaming data together!

Originally Published in: Wen Shu Qi Wu (闻数起舞)