Whilst working with golang, my first project was to create a CQRS framework with event sourcing using kafka.

Then, my first project to use this needed to be running 24/7 collecting statistical data from a stream of data using web sockets.  I did not want to be running this on my laptop, nor my desktop – I wanted a permanent home for it – so I bought another raspberry PI which are great for things like this.

The First thing I wanted on it was kafka, and there are a few articles out there on installing a cluster on many raspberry PI’s – I wanted something simpler – just the one for now until I start scaling.

Here is how I did it

Step 1 – Download and Install Kafka

Most articles I have read start with installing zookeeper, but I discovered that kafka has what they call a ‘quick and dirty setup’ including zookeeper.  I also wanted to stay as close to the instructions provided by the apache project and only change things if they need changing.  So, the first thing I did was to follow the quick start guide at https://kafka.apache.org/quickstart (stop at step 5 though)

When I got to step 2, starting the kafka server – it failed as there wasnt enough memory.  The raspberry PI only has 1 GB and I guess kafka wants a lot to start with, so I remembered an article (https://medium.com/@oliver_hu/set-up-a-kafka-cluster-with-raspberry-pi-2859005a9bed) that described how to change this.  This is the part of interest

2. Update the bin/kafka-server-start.sh file:

export JMX_PORT=${JMX_PORT:-9999}
export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M" # Otherwise, JVM would complain not able to allocate the specified memory.

3. Update bin/kafka-run-class.sh

KAFKA_JVM_PERFORMANCE_OPTS="-client -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true" # change -server to -client

This does 2 things, first it sets the JVM memory allocation to run within the limits of the raspberry PI 3 and secondly it uses ‘client’ instead of ‘server’ for the JVM performance options.

Now start the kafka server as described originally in the quick start and away you go

Running Zookeeper and Kafka

Long term I want to setup a supervisor script or something, but for now I start it manually using the following commands

In terminal window 1

sudo /opt/kafka_2.11-1.0.1/bin/zookeeper-server-start.sh config/zookeeper.properties

In terminal window 2

sudo /opt/kafka_2.11-1.0.1/bin/kafka-server-start.sh config/server.properties