Kinesis Data Stream Consumers
A consumer is a custom application that is built to read and process data from a Kinesis Data Stream. We can build a custom consumer application using the Kinesis Client Library.
Kinesis Client Library
KCL is a Java library which is designed to take care of many complex task required for reading and processing data from a Kinesis data stream. This task involves load balancing across multiple consumer application instances.
Currently, you can use one of the following supported versions of KCL to build your custom consumer applications:
- KCL 1.x - With KCL 1.x, we can build consumers that use shared throughout.
- KCL 2.x - With KCL 2.x, we can either build consumers that use shared throughout or enhanced fan-out.
Default Throughput (Shared Throughput)
A shard provides a fixed read throughput of 2 MiB/sec by default. When a consumer application uses default throughput of shards, each shard provides a fixed read throughput of 2 MiB/sec. When multiple consumers reads data from the same Data Stream, the throughput gets shared across all the consumers. Message propagation delay using default throughput is an average of around 200 ms per comsumer. If you use five consumers, the message propagation delay goes up to 1000 ms.
Enhanced Fan-Out (Dedicated Throughput)
A consumer application that uses enhanced fan-out, gets its own read throughput of 2 MiB/sec. This enables the capacity of using multiple consumers to read data from the same Data Stream in parallel. Each consumer gets its own read throughput of 2 MiB/sec to read data independently of other consumers. Message propagation delay using enhanced fan-out is 70 ms whether you use one consumer or 5 consumers.