How to produce records to a topic in Redpanda from a shell script?

Introduction

Redpanda is a Kafka-compatible event stream platform written in C++.

It comes with a simple command-line interface called rpk that lets you interact with the platform.

In this tutorial, we will be using the rpk command-line interface to produce events to a topic in Redpanda from a shell script.

Setup

You can follow the instructions from the Redpanda documentation to install Redpanda and the rpk command-line interface:

• Getting Started with Redpanda

Once you have installed Redpanda and the rpk command-line interface you would also have to install the jq command-line utility:

• How to work with JSON in BASH using jq

Create a topic

To create a topic you need to run the following command:

rpk topic create my-topic

Produce a record

As stated in the Redpanda documentation the producing records reads from STDIN. You can produce a record to a topic by running the following command:

rpk record produce my-topic

// Then type the record that you want to produce e.g.:
// '{"key": "value"}'

Shell script

The above command produces a record to a topic ad hoc, however, you can also produce a record to a topic using a shell script too.

Here are a few examples of how to produce a record to a topic:

🤩 Our Amazing Sponsors 👇

View Website

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

View Website

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

View Website

A Laravel Starter Kit that includes Authentication, User Dashboard, Edit Profile, and a set of UI Components. Learn more about the DevDojo sponsorship program and see your logo here to get your brand in front of thousands of developers.

Produce a record to a topic using jq

First, start by creating your message in JSON format:

JSON_STRING=$( jq -n --arg id "$ID" --arg name "user_$ID" '{id: $id, name: $name}' ) ;

Then you can produce a record to a topic using the rpk command-line interface:

echo "$JSON_STRING" | rpk record produce my-topic

To put this into a demo script that generates random user IDs and names in a loop you can use the following:

#!/bin/bash

# Create Redpanda topic
rpk topic create demo

# Counter for the id
ID=1

# Infinite loop to produce data
while true ; do
    JSON_STRING=$( jq -n --arg id "$ID" --arg name "user_$ID" '{id: $id, name: $name}' );
    # Send the message
    echo ${JSON_STRING} ; done | rpk topic produce demo

    # Increment the id
    ID=$((id+1))
done

Produce a record to a topic from a file

If you already have a file with the message you want to produce you can use the rpk command-line interface to produce a record to a topic:

cat messages.json | rpk record produce my-topic

Conclusion

For more information about Redpanda and the rpk command-line interface, you can visit the Redpanda documentation.

To see Redpanda in action, make sure to check out this demo here:

• Redpanda + Materialize Demo

Happy coding!