Read Data from a Kafka Topic in Go programming language Example

This tutorial will teach you how to read data from a Kafka topic using the Go programming language. You can also learn how to write data to a Kafka topic here.

To read data from a Kafka topic, we will use Confluent Kafka Golang library for Apache Kafka. It provides a high level Producer, and Consumer with support for balanced consumer groups of Apache Kafka 0.9 and above. An application that reads data from a Kafka topic is called a Consumer Application.

Follow these steps to create a Go consumer application for reading data from a Kafka topic using the Confluent Kafka Golang Library:

1. Adding Dependencies:

go get -u gopkg.in/confluentinc/confluent-kafka-go.v1/kafka

If you encounter the error 'C compiler 'gcc' not found: exec: 'gcc': executable file not found in %PATH%' when installing the confluent-kafka-go library on Windows, install TDM-GCC, which provides the required C compiler.

2. Creating the Kafka Consumer:

package main

import (
	"fmt"

	"github.com/confluentinc/confluent-kafka-go/kafka"
)

func consume() {
	fmt.Printf("Starting consumer...")
	c, err := kafka.NewConsumer(&kafka.ConfigMap{
		"bootstrap.servers": "localhost:9092",
		"group.id":          "myGroup",
		"auto.offset.reset": "earliest",
	})

	if err != nil {
		panic(err)
	}

	c.SubscribeTopics([]string{"myTopic"}, nil)

	for {
		msg, err := c.ReadMessage(-1)
		if err == nil {
			data := string(msg.Value)
			fmt.Printf("Message on %s: %s\n", msg.TopicPartition, data)
		} else {
			// The client will automatically try to recover from all errors.
			fmt.Printf("Consumer error: %v (%v)\n", err, msg)
		}
	}

	c.Close()
}

// Running consumer
func main() {
	consume()
}

The Consumer application is ready. Run the Consumer application, and then send data to the Kafka topic using the Producer application. Once data is available in the Kafka topic, the Consumer will read it.