Kinesis Data Stream Producer

A producer is an application that is used to write data to Amazon Kinesis Data Streams. The process of writing data into a Kinesis data stream is called data ingestion .

Developing Producers

A producer application can be developed using one the following:

  • Kinesis Producer Library (KPL)
  • AWS SDK for Java

Kinesis Producer Library (KPL)

Kinesis Producer Library (KPL) is a pre-built binaries of C++ that are packaged as a part of Java .jar files for use in different platforms such as MacOS, Windows, Linux and other Operating Systems.

The Kinesis Producer Library (KPL) makes the development of a Producer application very simple.

AWS SDK for Java

You can develop a Producer application using Amazon Kinesis Data Streams API with the AWS SDK for Java.

Prefer using the Kinesis Producer Library (KPL) as it support more use cases than AWS SDK for Java.

Send Data to Kinesis Data Streams using Kinesis Agent

Kinesis Agent is a standalone Java Software that is designed to collect data by monitoring files and send new available data to Kinesis Data Streams. The agent application is capable of retrying upon failures, file rotating, checkpointing for fault tolerance so as to restart from the point in case of failures.

The data are parsed based on newline character \n from each file by default.

The agent application can also be configured to parse multi-line data.

The Agent application can be installed on Linux-based web servers, database servers or log servers.