Setting Up Zookeeper, Kafka, and Schema Registry

NOTE: Windows is currently not a supported platform for running Confluent Kafka, ensure that your OS is on the list of supported operating systems:
https://docs.confluent.io/4.0.0/installation/installing_cp.html#system-requirements

 

Before proceeding, you must install and setup the following prerequisites:

  • Java JDK 64-bit, version 1.7 or later
  • System Environment variable JAVA_HOME set to the Java JDK 64-bit

 

To set up:

 

  1. Download one of the Confluent Kafka archives from http://confluent.io/download.
  2. Extract the contents of the archive to a new location.
  3. Below are the top-level folders of the archive:

    confluent-3.1.1/bin/        # Driver scripts for starting/stopping services
    confluent-3.1.1/etc/        # Configuration files
    confluent-3.1.1/share/java/ # Jars

    Start the ZooKeeper, Kafka and Schema Registry processes in the correct order. Make sure the previous process has been started before continuing to the next one.

    • Start ZooKeeper

      $ ./bin/zookeeper-server-start ./etc/kafka/zookeeper.properties

    • Start the Kafka broker

      $ ./bin/kafka-server-start ./etc/kafka/server.properties

    • Start Schema Registry

      $ ./bin/schema-registry-start ./etc/schema-registry/schema-registry.properties

  4. When these three processes have been started, you can now connect Panopticon Streams to your local Kafka cluster to execute and deploy your applications.

    For more details, refer to the Confluent Kafka Installation-and Quick Start guides:

     

 

NOTE: When connecting to a Kafka broker on a separate machine, exposing different IP addresses internally and externally, you need to configure KAFKA_ADVERTISED_LISTENERS. This is typically the case when running Kafka in a Docker container.

 

The symptoms of the missing configuration are:

 

  • Panopticon Streams can connect to ZooKeeper and the Kafka Broker
  • No data is written to topics

 

In [Kafka]/etc/kafka/server.properties, uncomment advertised.listeners and replace “your.host.name” with the externally exposed host name or IP address.

 

# Hostname and port the broker will advertise to producers and consumers. If not
set,
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://your.host.name:9092

 

When using the Confluent Docker image, you can pass the KAFKA_ADVERTISED_LISTENERS
as a parameter:

 

docker run -d --restart=always \

 

--net=confluent \
--name=kafka \
-p 9092:9092 \
-e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://your.host.name:9092 \
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
confluentinc/cp-kafka:5.1.0

 

 

 

 

 

(c) 2013-2024 Altair Engineering Inc. All Rights Reserved.

Intellectual Property Rights Notice | Technical Support