Kafka - Section 0: Resources 🗃️

Table of Contents 📜

• 🏠 home
• 🗃️ resources
  • 🔗 links
    • 📄 kafka documentation

Description 👀

Resources for the Kafka Zero to Pro Guide. Has images, pdfs, and links to other resources.

Helpful Content 📌

This is a collection of helpful content for the Kafka Zero to Pro Guide. It is not a part of Kafka, but it is helpful for learning Kafka.

Kafka Setup

Kafkas own quick start guide for setting up Kafka

• kafka quick start

If your lazy, or don’t want to leave these docs, you can use the quick start I made below.

1. get the java runtime environment (jre)

   sudo apt install default-jre
   

2. get kafka

   • download kafka

      wget https://dlcdn.apache.org/kafka/3.3.1/kafka_2.13-3.3.1.tgz
     

   • extract the tar file

      tar -xzf kafka_2.13-3.3.1.tgz
     

   • move the extracted folder to your home directory

      mv kafka_2.13-3.3.1 ~/
     

3. set up your environment variables

   • go to you .bashrc or .profile file and add the following lines

        export KAFKA_BIN_ROOT="$HOME/kafka_2.13-3.3.1/bin"
        export KAFKA_CONFIG_ROOT="$HOME/kafka_2.13-3.3.1/config"
        PATH="$PATH:$KAFKA_BIN_ROOT:$KAFKA_CONFIG_ROOT"
     

   • test your environment variables in a new terminal

      echo $KAFKA_BIN_ROOT
      echo $KAFKA_CONFIG_ROOT
      echo $PATH
     

   • test to see if you can run kafka commands from anywhere

      kafka-topics.sh
     

4. start up Kafka (using KRaft)

   • generate a cluster UUID

      KAFKA_CLUSTER_ID="$(kafka-storage.sh random-uuid)"
     

   • format the log directories

      kafka-storage.sh format -t "$KAFKA_CLUSTER_ID" -c "$KAFKA_CONFIG_ROOT/kraft/server.properties"
     

   • start the server

      kafka-server-start.sh  $KAFKA_CONFIG_ROOT/kraft/server.properties
     

   • Kafka is now running, keep this terminal open
   • logs are located in /tmp/kraft-combined-logs, and can be configured in $KAFKA_CONFIG_ROOT/kraft/server.properties

Kafka Troubleshooting

• when running a command from the CLI and you get the following error

  Warn [AdminCLient clientId=adminclient-1] Connection to node -1
  

  • this is a WSL2 networking issue
    • stop the Kafka server and either KRaft or Zookeeper (depending on which you are using)

    • then run the following commands

      sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
      

      sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
      

    • then go to the Kafka config file server.properites and change the listeners to PLAINTEXT://localhost:9092

      cd $KAFKA_CONFIG_ROOT
      

      nano server.properties
      

      listeners=PLAINTEXT://localhost:9092
      

    • then start the Kafka server and either KRaft or Zookeeper (depending on which you are using)