Connect Conduktor Platform with Kafka Container on Docker

We will spin up a Kafka Container on Docker Desktop and connect to it using Conduktor Platform Application.

Setup Kafka Container

Create a yaml file named docker-compose-only-kafka.yaml with the following:

version: "3"
services:
  zookeeper:
    container_name: zookeeper
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    container_name: kafka
    image: docker.io/bitnami/kafka:3.3
    ports:
      - "9092:9092"
      - "9094:9094"
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_LISTENERS=INTERNAL://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094
      - KAFKA_ADVERTISED_LISTENERS=INTERNAL://kafka:9092,OUTSIDE://localhost:9094
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL      
    depends_on:
      - zookeeper
volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise, they’ll try to connect to the internal host address—and if that’s not reachable, then problems ensue.

https://www.confluent.io/blog/kafka-listeners-explained/

We export port 9092 and 9094 and create an advertised.listener so that Conduktor can connect to our Docker Kafka Cluster.

Using docker-compose we will now run the Kafka Container on Docker Desktop.

docker-compose should point to the yaml file.

user@SEGOTW4PXY2J3 JAVA_8/cygdrive/c/conduktor_demo
$ ls -ltr
total 4
-rwxrwx---+ 1 a406604 Domain Users 915 Jan 24 21:54 docker-compose-only-kafka.yaml


If you want to start up the Kafka Container:

user@SEGOTW4PXY2J3 JAVA_8/cygdrive/c/conduktor_demo

$ docker-compose -f docker-compose-only-kafka.yaml up -d
Starting zookeeper ... done
Starting kafka     ... done



If you want to stop the Kafka Container without deleting:

$ docker-compose -f docker-compose-only-kafka.yaml stop

Stopping kafka     ... done
Stopping zookeeper ... done

Setup Conduktor Platform

Download the Conduktor Platform and install.

You will also need to register for a free account if you want to play around with it. The Conduktor Platform will allow you to link a Web Authentication Provider (Gmail for example) and you will have a trial to test the product (Option 1)

After you have installed Conduktor, you can now login using the previously linked Web Authentication Provider.

If you successfully succeed in logging in, it will state that. However if you face issues in logging it, it could be that you have a VPN running. In this case, stop the VPN service.

Previously we have started up a Kafka Container, and have exposed external connections towards port 9094. We should now connect to this docker kafka cluster using Conduktor.

Click on the created Kafka Cluster on Conduktor and verify that you have access to the Docker Kafka Cluster:

Considerations:

• If you have a newer Docker Desktop, you might need to modify Settings –> File Sharing to allow access for the folder you mount volumes to.
• Kafka Container + ZooKeeper must both be running in docker without any exceptions to be able to connect using Conduktor. Check the container logs.
• You must configure advertised.listeners