Deploy it in a separate Container

First of all, Kafka relies on ZooKeeper, and even if it’s standalone, it has to be configured as a cluster…

So, first download two official images:

docker pull confluentinc/cp-zookeeper
docker pull confluentinc/cp-kafka
Copy the code

Then create a compose (thank linghu for your help) :

version: '2'services: zookeeper: image: confluentinc/cp-zookeeper container_name: zookeeper mem_limit: 1024M environment: ZOOKEEPER_CLIENT_PORT: 2181 kafka: image: confluentinc/cp-kafka container_name: kafka mem_limit: 1024M depends_on: - Zookeeper environment: KAFKA_BROKER_NO: 1 KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT: / / 127.0.0.1:9092 KAFKA_ZOOKEEPER_CONNECT: zookeeper: 2181 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_HEAP_OPTS:"-Xmx512M -Xms16M"
Copy the code

Because this test is performed directly on the server, the ports are not bound for security reasons.

Compose:

docker-compose up -d
Copy the code

Now open two new terminal Windows and log in to the Container using the following command:

docker exec -it kafka /bin/bash
Copy the code

Create a topic in one of the Windows and run Producer:

kafka-topics --zookeeper zookeeper:2181 --create --replication-factor 1 --partitions 1 --topic kafkatest
kafka-console-producer --broker-list localhost:9092 --topic kafkatest
Copy the code

Run Consumer in another window:

kafka-console-consumer --bootstrap-server localhost:9092 --topic kafkatest --from-beginning
Copy the code

Now, anything entered into the Producer is received in the Consumer.

Deployment across Containers

The preceding configuration can only be used in a single Container. This is because Kafka Advertised is configured on localhost.

To access the container, you need to access the docker network. Modify this configuration:

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    container_name: zookeeper
    mem_limit: 1024M
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka:
    image: confluentinc/cp-kafka
    container_name: kafka
    mem_limit: 1024M
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_NO: 1
      KAFKA_ADVERTISED_HOST_NAME: domain_name  # modified
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://domain_name:9092  # modified
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_HEAP_OPTS: "-Xmx512M -Xms16M"
Copy the code

Docker-compose is restarted after the modification. In addition, topic will not be persisted after ZooKeeper restarts, so topic needs to be re-created after the restart.

Then start two new Containers to simulate network access:

docker run -it --rm --link kafka:domain_name --network kafka_default --name consumer confluentinc/cp-kafka /bin/bash
docker run -it --rm --link kafka:domain_name --network kafka_default --name producer confluentinc/cp-kafka /bin/bash
Copy the code

Note that the docker network needs to be specified as kafka_default, which is the default network used by the official image.

Then test in consumer container and Producer Container respectively:

kafka-console-consumer --bootstrap-server domain_name:9092 --topic kafkatest --from-beginning
kafka-console-producer --broker-list domain_name:9092 --topic kafkatest
Copy the code

The effect is the same as that of a single container.

Deployment from outside the Docker network

If you need access from outside the Docker network, you need to map the port to the host.

You also need to modify the configuration to add network mapping.

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    container_name: zookeeper
    mem_limit: 1024M
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka:
    image: confluentinc/cp-kafka
    container_name: kafka
    mem_limit: 1024M
    depends_on:
      - zookeeper
    ports:  # increase
      - 9092:9092  # increase
    environment:
      KAFKA_BROKER_NO: 1
      KAFKA_ADVERTISED_HOST_NAME: domain_name
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://domain_name:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_HEAP_OPTS: "-Xmx512M -Xms16M"
Copy the code

Then start two new Containers to simulate external network access:

Docker run-it --rm --add-host=domain_name:172.17.0.1 --name Consumer confluentInc /cp-kafka /bin/bash docker run-it --rm --add-host=domain_name:172.17.0.1 --name producer confluentInc /cp-kafka /bin/bashCopy the code

172.17.0.1 is the IP address of the docker host in the default docker network (not kafka_default).

ip route
Copy the code

Then test in consumer container and Producer Container respectively:

kafka-console-consumer --bootstrap-server domain_name:9092 --topic kafkatest --from-beginning
kafka-console-producer --broker-list domain_name:9092 --topic kafkatest
Copy the code

There is a pit to note here:

If the host has a firewall, you need to add a rule to allow the Docker network to access the host port, otherwise the connection will fail. Such as:

Get the line number
iptables -L INPUT --line-num
# xx is the line number of the last DROP line and is inserted before it
iptables -I INPUT xx -p tcp -m tcp -s 172.17.0.0/16 --dport 9092 -j ACCEPT
Copy the code

The effect is the same as in the previous two examples.