Kafka connect docker image - Failed to find any class that implements Connector and which name matches ElasticsearchSinkConnector

Question

I have been using the kafka-connect image confluentinc/cp-kafka-connect for some time now. According to the Confluent documentation, this docker image comes with pre-installed connector plugins including Elastic.

I have previously been using version 5.4.1-ccs which works great, I can add Elastic sink connnector configs and they work just fine. However I have tried to update confluentinc/cp-kafka-connect to latest v6.0.1 and now I get the error.

ConnectException: Failed to find any class that implements Connector and which name matches ElasticsearchSinkConnector

I have read a bunch of documentation on Confluent website but it's a bit sporadic. I understand that the issue is the plugins are not mounted due to either them being removed from the new docker image or a wrong path (unsure which one it is).

How can I fix this issue? (note: I have written my own java plugin also so both need to work)

Here is my docker-compose file currently (again, this works with version 5.4.1-ccs)

kafka-connect-node-1:
  image: confluentinc/cp-kafka-connect:5.4.1 #using old version because of breaking change
  hostname: kafka-connect-node-1
  ports:
    - '8083:8083'
  environment:
    CONNECT_BOOTSTRAP_SERVERS: [MY_SERVER]
    CONNECT_REST_PORT: 8083
    CONNECT_GROUP_ID: compose-connect-group
    CONNECT_CONFIG_STORAGE_TOPIC: connect-configs
    CONNECT_OFFSET_STORAGE_TOPIC: connect-offsets
    CONNECT_STATUS_STORAGE_TOPIC: connect-status
    CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
    CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: 'http://kafka-schema-registry:8084'
    CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
    CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://kafka-schema-registry:8084'
    CONNECT_INTERNAL_KEY_CONVERTER: 'org.apache.kafka.connect.json.JsonConverter'
    CONNECT_INTERNAL_VALUE_CONVERTER: 'org.apache.kafka.connect.json.JsonConverter'
    CONNECT_REST_ADVERTISED_HOST_NAME: 'kafka-connect-node-1'
    CONNECT_LOG4J_ROOT_LOGLEVEL: 'INFO'
    CONNECT_LOG4J_LOGGERS: 'org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR'
    CONNECT_PLUGIN_PATH: '/usr/share/java,/etc/kafka-connect/jars'
    CONNECT_ZOOKEEPER_CONNECT: [MY_ZOOKEEPER]
  volumes:
    - /efs/connector:/etc/kafka-connect/jars/ # mounting my custom JAR
  depends_on:
    - kafka-schema-registry
    - kafka-rest-proxy

question from:https://stackoverflow.com/questions/65901864/kafka-connect-docker-image-failed-to-find-any-class-that-implements-connector

Answer 1

Since Confluent Platform 6.0 connectors are no longer bundled, and need to be installed separately.

You can build your own image based on cp-kafka-connect-base, or you can install connectors at runtime by overriding the image command thus:

  kafka-connect:
    image: confluentinc/cp-kafka-connect-base:6.0.0
    container_name: kafka-connect
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: "broker:29092"
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: compose-connect-group
      CONNECT_CONFIG_STORAGE_TOPIC: _kafka-connect-group-01-configs
      CONNECT_OFFSET_STORAGE_TOPIC: _kafka-connect-group-01-offsets
      CONNECT_STATUS_STORAGE_TOPIC: _kafka-connect-group-01-status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
      CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: 'http://schema-registry:8081'
      CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect"
      CONNECT_LOG4J_ROOT_LOGLEVEL: "INFO"
      CONNECT_LOG4J_LOGGERS: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR"
      CONNECT_LOG4J_APPENDER_STDOUT_LAYOUT_CONVERSIONPATTERN: "[%d] %p %X{connector.context}%m (%c:%L)%n"
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"
      CONNECT_PLUGIN_PATH: '/usr/share/java,/connectors,/usr/share/confluent-hub-components/'
    depends_on:
      - broker
      - schema-registry
    volumes:
      - $PWD/data/connectors/:/connectors/
    command:
      - bash
      - -c
      - |
        echo "Installing Connector"
        confluent-hub install --no-prompt confluentinc/kafka-connect-elasticsearch:10.0.1
        #
        echo "Launching Kafka Connect worker"
        /etc/confluent/docker/run &
        #
        sleep infinity