Getting Kafka Up and Running with Vagrant

In this article, we will see how to use a Kafka Vagrant box to quickly create a development environment for a Ruby project. We will create a gem and work on Ruby messaging app.

Step 1

Create a Vagrant file in the project folder. We will use an existing Kafka Vagrant box that has Kafka, Zookeeper and all the required dependencies including Java.

Vagrant.configure("2") do |config|  
  config.vm.box = "fscm/kafka"
  config.vm.provider "virtualbox" do |v|
    v.memory = 1024
    v.cpus = 1
  end
    config.vm.synced_folder ".", "/src/messenger"
end

According to the home page for the fscm Kafka Vagrant box, the miniinimum specs are:

memory: 1GB
cpu: 1

We have specified these values in the Vagrantfile.

Step 2

Start Vagrant.

vagrant up

The output that is interesting are the folders that are shared on the host and the VM.

 default: /shared => /Users/bparanj/projects/kgem
 default: /src/messenger => /Users/bparanj/projects/kgem

Step 3

We can ssh into the Vagrant instance.

vagrant ssh

Find the directory where the shell scripts are located:

find / -type d -name "kafka"

The Kafka is installed in /srv/kafka folder. Start Zookeeper and Kafka:

$ /srv/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties &
$ /srv/kafka/bin/kafka-server-start.sh config/server.properties &

The shared folder is created by Vagrant automatically. You can ssh as Vagrant and see:

pollywog@kafka:/src/messenger

Step 4

We can check the Linux distribution of the Kafka Vagrant box:

cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Install Ruby on Debian.

sudo apt update
sudo apt install ruby-full
ruby -v

We now have: ruby 2.3.3p222 (2016-11-21) [x86_64-linux-gnu] installed. We can view the messages in a topic using the command:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic "inline_batch_data" --from-beginning

Step 5

Install bundler.

vagrant ssh
gem install bundler

Create a gem folder structure.

bundle gem franz

Run tests:

rake test

This will throw an error message:

Bundler::GemspecError: 
[!] There was an error while loading `franz.gemspec`: No such file or directory - git. Bundler cannot continue.

Resolution:

git init

Change the test file to require the `test_helper':

require_relative "test_helper"
require "minitest/autorun"

class FranzTest < Minitest::Test
  def test_that_it_has_a_version_number
    refute_nil ::Franz::VERSION
  end

  def test_it_does_something_useful
    assert true
  end
end

We can run the tests with red and green color output:

ruby -rminitest/pride test/franz_test.rb

We can read any messages from a topic by running bundle console and:

kafka = Kafka.new(["localhost:9092"], client_id: "test-application")
kafka.each_message(topic: "test-topic") do |message|
  puts message.value
end

This will output the messages in the test-topic topic.

References

Configuring Memory in Vagrant
Kafka Vagrant Box
Synched Folders
Install Ruby on Debian
Kafka Quick Start


Related Articles


Ace the Technical Interview

  • Easily find the gaps in your knowledge
  • Get customized lessons based on where you are
  • Take consistent action everyday
  • Builtin accountability to keep you on track
  • You will solve bigger problems over time
  • Get the job of your dreams

Take the 30 Day Coding Skills Challenge

Gain confidence to attend the interview

No spam ever. Unsubscribe anytime.