Let’s talk about Kafka operations! How do we know if a Kafka broker or a Zookeeper node is healthy or not? How about CPU, disk space and memory usage? Good news, we can have all this useful information in a very fancy dashboard using open-source tools!
In part 1 of this series, I covered how to automate Zookeeper and deploy it as a service, in part 2, I discussed how we can deploy Apache Kafka on AWS in a fully automated and self-healing manner. Now let’s talk about operation! Now that we have a fully automated streaming platform, we need to make sure that our Kafka brokers and Zookeeper nodes are healthy and that we have enough of them in our cluster.
With Control Center, one of the Confluent Platform Enterprise components, you get a dashboard for monitoring and managing Kafka, Zookeeper, and other Confluent components. However, the Control Center is not free! Fortunately, there are alternatives. In this post, I will demonstrate how you can create a monitoring dashboard using open source tools and libraries.
Here is a list of the tools and libraries we’ll use:
Let’s dive into each tool and explore how to set them up.
For automating the process of installation, we can use this Chef cookbook, however, for simplicity, I will demonstrate how to install using shell commands. Before we do that, we need an EC2 instance with Ubuntu 16.04. Create one and ssh into it. Now you are ready to install and configure Prometheus:
From the Prometheus download page, get the latest stable version:
$ wget https://github.com/prometheus/prometheus/releases/download/v2.11.1/prometheus-2.11.1.linux-amd64.tar.gz
# compare the checksum
$ sha256sum prometheus-2.11.1.linux-amd64.tar.gz
# untar
$ tar xvf prometheus-2.11.1.linux-amd64.tar.gz
# create prometheus user
$ sudo useradd --no-create-home --shell /bin/false prometheus
# create the necessary directories with proper ownership
$ sudo mkdir /etc/prometheus
$ sudo chown prometheus:prometheus /etc/prometheus
$ sudo mkdir /var/lib/prometheus
$ sudo chown prometheus:prometheus /var/lib/prometheus
# copy the binary files to the bin folder
$ sudo cp prometheus-2.11.1.linux-amd64/prometheus /usr/local/bin/
$ sudo cp prometheus-2.11.1.linux-amd64/promtool /usr/local/bin/
# change the ownership to prometheus user
$ sudo chown prometheus:prometheus /usr/local/bin/prometheus
$ sudo chown prometheus:prometheus /usr/local/bin/promtool
# set consoles and console_libraries
$ sudo cp -r prometheus-2.11.1.linux-amd64/consoles /etc/prometheus
$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles
$ sudo cp -r prometheus-2.11.1.linux-amd64/console_libraries /etc/prometheus
$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
# cleanup
rm -rf rm -rf prometheus-2.11.1.linux-amd64*
Create prometheus.yml file in the folder /etc/prometheus and copy this content:
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
Change the ownership of the file to the newly created user prometheus:
Exploring the Complexity of Visualizing COVID-19 Case Data
Over the last four+ months, the world has logged on daily to track case counts...
Microsoft recently reached a major milestone in the development of the open-source cross-platform library ML.NET...
Apache Kafka Automation, Part 1: Zookeeper as a Service
Zookeeper is an orchestration service, typically associated with distributed systems (think Hadoop or Kafka). Managing...