Build and Configure a Kafka Cluster on AWS EC2: A DevOps Guide


Apache Kafka is a powerful distributed streaming platform for building real-time data pipelines and streaming applications. Deploying Kafka on AWS EC2 offers flexibility and control, making it a preferred choice for DevOps professionals who want to manage the infrastructure directly. This guide walks you through the end-to-end setup process.

1. Prerequisites

Before diving in, ensure the following:

  • An AWS account with permission to launch EC2 instances and configure networking components.

  • Basic knowledge of SSH, Linux commands, and Kafka architecture.

  • Security groups, VPCs, and subnets are set up (or permission to create them).

2. Architecture Overview

Your Kafka cluster will consist of the following components:

  • Zookeeper nodes: 3 nodes (odd number for quorum).

  • Kafka brokers: At least 3 EC2 instances.

  • Clients: Producer/consumer applications or test utilities.

  • Optional monitoring tools: e.g., Prometheus + Grafana.

3. Step-by-Step Setup

Step 1: Launch EC2 Instances

  • Choose Amazon Linux 2 or Ubuntu.

  • Select instance types (e.g., t3.medium or higher).

  • Configure network: Use the same VPC and availability zone for all nodes.

  • Open required ports in the security group:

    • Kafka: 9092

    • Zookeeper: 2181

    • SSH: 22

    • (Optional: Prometheus, Grafana ports)

Step 2: Install Java

Kafka requires Java. On each EC2 instance, run:


sudo yum install java-1.8.0-openjdk -y

java -version


Step 3: Download and Install Kafka

On each instance:


wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz

tar -xzf kafka_2.13-3.7.0.tgz

cd kafka_2.13-3.7.0


Step 4: Configure Zookeeper Cluster

Edit config/zookeeper.properties to define each Zookeeper node:


tickTime=2000

dataDir=/tmp/zookeeper

clientPort=2181

initLimit=5

syncLimit=2

server.1=<zk1_private_ip>:2888:3888

server.2=<zk2_private_ip>:2888:3888

server.3=<zk3_private_ip>:2888:3888


Create myid file on each Zookeeper node in /tmp/zookeeper/ with a unique ID (1, 2, or 3).

Start Zookeeper:


bin/zookeeper-server-start.sh -daemon config/zookeeper.properties


Step 5: Configure Kafka Brokers

Edit config/server.properties:


broker.id=1  # unique per broker

listeners=PLAINTEXT://<broker_private_ip>:9092

log.dirs=/tmp/kafka-logs

zookeeper.connect=<zk1_ip>:2181,<zk2_ip>:2181,<zk3_ip>:2181


Start Kafka:


bin/kafka-server-start.sh -daemon config/server.properties


Step 6: Create a Kafka Topic

From one of the broker nodes:


bin/kafka-topics.sh --create --topic test-topic --bootstrap-server <broker_ip>:9092 --replication-factor 3 --partitions 3


Step 7: Test Kafka Setup

Producer:


bin/kafka-console-producer.sh --topic test-topic --bootstrap-server <broker_ip>:9092


Consumer:


bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server <broker_ip>:9092


4. Hardening and Optimization

  • Use systemd services for managing Kafka/Zookeeper.

  • Mount EBS volumes for log storage durability.

  • Enable monitoring via JMX and integrate with Prometheus.

  • Tune JVM and broker configurations for production load.

  • Add security: Use TLS, SASL, and IAM for access control.

5. Automation with Ansible or Terraform

For large-scale deployments, automate instance provisioning and Kafka configuration using:

  • Terraform is used to provision VPC, subnets, EC2, and security groups.

  • Ansible for installing Java, Kafka, and configuring systemd units.

6. Monitoring and Logging

  • Use Prometheus JMX Exporter to collect metrics from Kafka and Zookeeper.

  • Visualize metrics with Grafana.

  • Forward logs to CloudWatch or an ELK stack.

Conclusion

Deploying Kafka on AWS EC2 gives you complete control over the environment and tuning capabilities. While it demands more operational effort than managed solutions like MSK (Amazon Managed Streaming for Apache Kafka), it’s ideal for custom DevOps pipelines and cost-sensitive deployments.


Comments

Post a Comment

Popular posts from this blog

ECS Deployment Best Practices: Blue/Green with CodePipeline and CodeDeploy

Image
DevOps engineers and AWS cloud architects looking to improve their container deployment processes will benefit from implementing blue/green deployments with Amazon ECS. This guide walks through setting up reliable, zero-downtime deployments using AWS CodePipeline and CodeDeploy for your containerized applications. We’ll cover how to configure your ECS environment properly, create automated deployment pipelines, and implement blue/green deployment strategies that minimize risk during updates. Understanding ECS Deployment Strategies What is Amazon ECS and why it matters Amazon Elastic Container Service (ECS) isn’t just another tool in AWS’s massive catalog—it’s the backbone of modern containerized applications. At its core, ECS is a fully managed container orchestration service that handles all the complex tasks of running, stopping, and managing Docker containers. Think of ECS as the conductor of an orchestra where each container is an instrument. Without proper coordination, you’d just...
Read more

HTTP Basic vs API Key Auth: Best Practices for Secure API Development

Image
Picture this: You’ve spent weeks building your API, and now your security approach boils down to a coin flip between HTTP Basic and API Keys. Choose wrong, and your data’s basically wearing a “hack me” sign. Every developer faces this exact decision, yet most guides leave you with more questions than answers. When implementing authentication for your API, the choice between HTTP Basic Authentication and API Key Authentication can significantly impact your security posture and user experience. So what makes one better than the other? When should you use HTTP Basic over API Keys? Is there ever a scenario where the “simpler” option is actually more secure? The answers might surprise you – and they definitely aren’t what most Stack Overflow threads would have you believe. Understanding API Authentication Fundamentals Why API Security Matters in Modern Development API security isn’t just some technical checkbox—it’s the fortress protecting your digital kingdom. With businesses exposing crit...
Read more

Creating BI Solutions: AI/BI Genie Space Authoring Best Practices in Databricks

Image
Looking to build powerful business intelligence solutions in Databricks? This guide is for data analysts, BI developers, and data engineers who want to master the AI/BI Genie Space. We’ll cover essential authoring techniques that combine the best of AI and BI capabilities in Databricks. Learn about setting up your ideal authoring environment, preparing data for optimal performance, and designing effective visualizations that drive insights. Plus, discover how to implement proper security measures while enabling smooth team collaboration. Understanding the AI/BI Genie Space in Databricks Key features and capabilities of AI/BI Genie Space The AI/BI Genie Space isn’t just another BI tool—it’s Databricks’ answer to making data visualization and analytics dead simple while keeping all the power under the hood. Think of it as your command center for creating dashboards and reports without jumping between platforms. What makes it special? First off, real-time query processing. You ask a quest...
Read more

YouTube Channel