Kafka cluster re-election fails after shutdown all brokers and re-start the not-coordinators brokers

Question

I have a Kafka cluster with 3 brokers and when I shutdown all the brokers and re-start only the brokers that weren't coordinator, I expected that Kafka re-elect the coordinator between them, but this never happens and the cluster doesn't work until I re-start the coordinator broker.

Is this a correct behaviour?

I have a docker-compose with the next configuration

  provision:
    image: ${DOCKER_ORG:-mailu}/${DOCKER_PREFIX:-}admin:${MAILU_VERSION:-1.7}
    env_file: mailu.env
    volumes:
      - "/mailu/data:/data"
      - "/mailu/dkim:/dkim"
    command: > 
      bash -c "sleep 20 & curl cmak:9000/clusters --data 
      'name=KafkaCluster&zkHosts=zookeeper%3A2181&kafkaVersion=2.4.0&jmxEnabled=true&jmxUser=&jmxPass=&pollConsumers=true&activeOffsetCacheEnabled=true&tuning.brokerViewUpdatePeriodSeconds=30&tuning.clusterManagerThreadPoolSize=2&tuning.clusterManagerThreadPoolQueueSize=100&tuning.kafkaCommandThreadPoolSize=2&tuning.kafkaCommandThreadPoolQueueSize=100&tuning.logkafkaCommandThreadPoolSize=2&tuning.logkafkaCommandThreadPoolQueueSize=100&tuning.logkafkaUpdatePeriodSeconds=30&tuning.partitionOffsetCacheTimeoutSecs=5&tuning.brokerViewThreadPoolSize=4&tuning.brokerViewThreadPoolQueueSize=1000&tuning.offsetCacheThreadPoolSize=4&tuning.offsetCacheThreadPoolQueueSize=1000&tuning.kafkaAdminClientThreadPoolSize=4&tuning.kafkaAdminClientThreadPoolQueueSize=1000&tuning.kafkaManagedOffsetMetadataCheckMillis=30000&tuning.kafkaManagedOffsetGroupCacheSize=1000000&tuning.kafkaManagedOffsetGroupExpireDays=7&securityProtocol=PLAINTEXT&saslMechanism=DEFAULT' 
      -X POST"
    depends_on:
      - admin

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    restart: always
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker0:
    image: confluentinc/cp-kafka:latest
    restart: always
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_HOST_NAME: 172.16.0.207
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://172.16.0.207:29092,PLAINTEXT_HOST://localhost:9092'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      
  broker1:
    image: confluentinc/cp-kafka:latest
    restart: always
    depends_on:
      - zookeeper
    ports:
      - "29093:29093"
      - "9093:9093"
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_HOST_NAME: 172.16.0.207      
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.16.0.207:29093,PLAINTEXT_HOST://localhost:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0  
        
  broker2:
    image: confluentinc/cp-kafka:latest
    restart: always
    depends_on:
      - zookeeper
    ports:
      - "29094:29094"
      - "9094:9094"
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_HOST_NAME: 172.16.0.207      
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.16.0.207:29094,PLAINTEXT_HOST://localhost:9094
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0       
  
  cmak:
    image: kafkamanager/kafka-manager:latest
    depends_on:
      - zookeeper
    ports:
      - "9999:9000"
    environment:
      ZK_HOSTS: 'zookeeper:2181'

question from:https://stackoverflow.com/questions/65900893/kafka-cluster-re-election-fails-after-shutdown-all-brokers-and-re-start-the-not

Answer 1

Waitting for answers