Sending Amazon EKS Logs to AWS OpenSearch Using Fluent-Bit


Introduction

Amazon Elastic Kubernetes Service (EKS) generates logs that provide valuable insights into cluster health, application performance, and security. Sending these logs to AWS OpenSearch Service (formerly Elasticsearch) enables real-time monitoring and analysis. This guide outlines the step-by-step process to configure Fluent-Bit for efficient log forwarding from Amazon EKS to AWS OpenSearch.

Prerequisites

Before setting up Fluent-Bit, ensure the following:

  • An active AWS account with Amazon EKS and AWS OpenSearch Service configured.

  • A running EKS cluster.

  • The Kubernetes command-line tool (kubectl) installed and configured.

  • AWS CLI installed and authenticated.

Step 1: Deploy Fluent-Bit on Amazon EKS

Fluent-Bit is a lightweight and highly efficient log processor that helps forward logs from Amazon EKS to AWS OpenSearch.

  1. Create a Kubernetes namespace for logging:

kubectl create namespace logging

  1. Create a Fluent-Bit ConfigMap:


apiVersion: v1

kind: ConfigMap

metadata:

  name: fluent-bit-config

  namespace: logging

data:

  fluent-bit.conf: |

    [SERVICE]

        Flush         5

        Log_Level     info

    

    [INPUT]

        Name         tail

        Path         /var/log/containers/*.log

        Parser       docker

        Tag         kube.*


    [OUTPUT]

        Name        opensearch

        Match       *

        Host        <opensearch-domain>

        Port        443

        TLS         On

        AWS_Auth    On

       Region      <aws-region>

  1. Apply this configuration:

kubectl apply -f fluent-bit-config.yaml

  1. Deploy Fluent-Bit as a DaemonSet:


apiVersion: apps/v1

kind: DaemonSet

metadata:

  name: fluent-bit

  namespace: logging

spec:

  selector:

    matchLabels:

      name: fluent-bit

  template:

    metadata:

      labels:

        name: fluent-bit

    spec:

      serviceAccountName: fluent-bit

      containers:

      - name: fluent-bit

        image: fluent/fluent-bit:latest

        volumeMounts:

        - name: varlog

          mountPath: /var/log

        - name: config-volume

          mountPath: /fluent-bit/etc

      volumes:

      - name: varlog

        hostPath:

          path: /var/log

      - name: config-volume

        configMap:

         name: fluent-bit-config

  1. Apply the DaemonSet configuration:

Kubectl apply -f fluent-bit-daemonset.yaml

Step 2: Grant Fluent-Bit IAM Permissions

To allow Fluent-Bit to send logs securely to AWS OpenSearch, attach the necessary IAM permissions.

  1. Create an IAM policy:

aws iam create-policy --policy-name FluentBitOpenSearchPolicy --policy-document file://fluent-bit-policy.json

  1. Attach the policy to a new IAM role:

aws iam create-role --role-name FluentBitRole --assume-role-policy-document file://trust-policy.json


aws iam attach-role-policy --role-name FluentBitRole --policy-arn arn:aws:iam::<account-id>:policy/FluentBitOpenSearchPolicy

  1. Annotate the Kubernetes service account to use this IAM role:

kubectl annotate serviceaccount fluent-bit -n logging eks.amazonaws.com/role-arn=arn:aws:iam::<account-id>:role/FluentBitRole



Step 3: Verify Logs in AWS OpenSearch

Once Fluent-Bit is running, logs should begin streaming into AWS OpenSearch. To verify:

  1. Log into the AWS Management Console.

  2. Navigate to AWS OpenSearch Service.

  3. Open Kibana and search for logs using:

{ "query": { "match_all": {} } }

Conclusion

By following these steps, Amazon EKS logs can be efficiently collected and stored in AWS OpenSearch using Fluent-Bit. This setup enhances observability, enabling proactive troubleshooting and real-time analysis of Kubernetes workloads.

Comments

Post a Comment

Popular posts from this blog

ECS Deployment Best Practices: Blue/Green with CodePipeline and CodeDeploy

Image
DevOps engineers and AWS cloud architects looking to improve their container deployment processes will benefit from implementing blue/green deployments with Amazon ECS. This guide walks through setting up reliable, zero-downtime deployments using AWS CodePipeline and CodeDeploy for your containerized applications. We’ll cover how to configure your ECS environment properly, create automated deployment pipelines, and implement blue/green deployment strategies that minimize risk during updates. Understanding ECS Deployment Strategies What is Amazon ECS and why it matters Amazon Elastic Container Service (ECS) isn’t just another tool in AWS’s massive catalog—it’s the backbone of modern containerized applications. At its core, ECS is a fully managed container orchestration service that handles all the complex tasks of running, stopping, and managing Docker containers. Think of ECS as the conductor of an orchestra where each container is an instrument. Without proper coordination, you’d just...
Read more

Creating BI Solutions: AI/BI Genie Space Authoring Best Practices in Databricks

Image
Looking to build powerful business intelligence solutions in Databricks? This guide is for data analysts, BI developers, and data engineers who want to master the AI/BI Genie Space. We’ll cover essential authoring techniques that combine the best of AI and BI capabilities in Databricks. Learn about setting up your ideal authoring environment, preparing data for optimal performance, and designing effective visualizations that drive insights. Plus, discover how to implement proper security measures while enabling smooth team collaboration. Understanding the AI/BI Genie Space in Databricks Key features and capabilities of AI/BI Genie Space The AI/BI Genie Space isn’t just another BI tool—it’s Databricks’ answer to making data visualization and analytics dead simple while keeping all the power under the hood. Think of it as your command center for creating dashboards and reports without jumping between platforms. What makes it special? First off, real-time query processing. You ask a quest...
Read more

AWS Console Not Loading? Here’s How to Fix It Fast

Image
When the AWS Management Console refuses to load, it can be incredibly frustrating, especially when trying to deploy, debug, or manage production infrastructure. Fortunately, quick and effective fixes get you back on track. This guide outlines common causes and step-by-step solutions to resolve the issue quickly. 1. Check AWS Service Status Start by verifying if AWS itself is experiencing issues: Visit the AWS Health Dashboard to check for regional outages. Look for service-specific issues affecting IAM , EC2 , S3 , or the Console itself . Tip: Subscribe to AWS status updates on Twitter for real-time alerts. 2. Clear Browser Cache and Cookies Corrupted cache or cookies often prevent pages from rendering correctly. Steps: Open browser settings. Clear cache , cookies , and site data for aws.amazon.com . Reload the page. Try logging in using Incognito/Private Mode to rule out browser extensions interfering. 3. Try a Different Browser or Device Sometimes, browser-specific quirks b...
Read more

YouTube Channel