Read Data from a Kafka Topic in Go programming language Example

In this tutorial, you will learn how to read data from a Kafka topic in Go programming language.

To read data from a Kafka topic, we will use Confluent Kafka Golang library for Apache Kafka. It provides a high level Producer, and Consumer with support for balanced consumer groups of Apache Kafka 0.9 and above.

An application that reads data from a Kafka topic is called a Kafka Consumer Application.

You can also learn how to write data to a Kafka topic here.

Follow the steps below to install Confluent Kafka Golang Client library and create a Kafka consumer application to send data to a Kafka topic in Go programming language:

Installing Confluent Kafka Golang Library

Install the latest version of Confluent Kafka Golang Library:

go get -u gopkg.in/confluentinc/confluent-kafka-go.v1/kafka

Creating the Kafka Consumer

Consumer code to read data from a Kafka topic:

package main

import (
	"fmt"

	"github.com/confluentinc/confluent-kafka-go/kafka"
)

func consume() {
	fmt.Printf("Starting consumer...")
	c, err := kafka.NewConsumer(&kafka.ConfigMap{
		"bootstrap.servers": "localhost:9092",
		"group.id":          "myGroup",
		"auto.offset.reset": "earliest",
	})

	if err != nil {
		panic(err)
	}

	c.SubscribeTopics([]string{"myTopic"}, nil)

	for {
		msg, err := c.ReadMessage(-1)
		if err == nil {
			data := string(msg.Value)
			fmt.Printf("Message on %s: %s\n", msg.TopicPartition, data)
		} else {
			// The client will automatically try to recover from all errors.
			fmt.Printf("Consumer error: %v (%v)\n", err, msg)
		}
	}

	c.Close()
}

// Running consumer
func main() {
	consume()
}

Kafka Consumer application is ready. Next, run the consumer application. Send data to the Kafka topic using the Producer application. As soon as there is data in the Kafka topic, it will be read by the consumer.

If you do not have Apache Kafka server already setup on your local machine then learn how to do it here for Windows and here for Mac/Ubuntu/Linux.