docker - Zipkin is not connecting to RabbitMQ

Question

Please, notice that this question can seem a duplicate of this one, but it's not the case. Below I include my rational

I'm trying to add Sleuth/Zipkin trace to my project. For this I have followed this tutorial.

My project is already using RabbitMQ for communication among the different microservices working fine.

My problem is that I'm able to get the traces fine when I use the web connection, but I get an unable to connect error when I try to communicate using RabbitMQ.

product-composite_1  | 2021-01-28 18:11:46.799  WARN [product-composite,,] 1 --- [           main] o.s.c.s.a.z.ZipkinAutoConfiguration      : 
Check result of the [RabbitMQSender{addresses=[rabbitmq:5672], queue=zipkin}] contains an error 
[CheckResult{ok=false, error=java.lang.RuntimeException: Unable to establish connection to RabbitMQ server}]

As commented in the first line, my problem does not seem to be related to rabbitmq host itself, because it is up and running, and providing service to my microservices.

Sure that I missing something in the configuration, but I cannot find it (I have also checked this post, this and this.

My files (I have removed not relevant part):

docker-compose.yml

version: '3'

services:
  product-composite:
    build: microservices/product-composite-service
    mem_limit: 350m
    environment:
      - SPRING_PROFILES_ACTIVE=docker
      - CONFIG_SERVER_USR=${CONFIG_SERVER_USR}
      - CONFIG_SERVER_PWD=${CONFIG_SERVER_PWD}
    depends_on:
      - "rabbitmq"
      - "eureka"

  eureka:
    build: spring-cloud/eureka-server
    mem_limit: 350m
    environment:
      - SPRING_PROFILES_ACTIVE=docker
      - CONFIG_SERVER_USR=${CONFIG_SERVER_USR}
      - CONFIG_SERVER_PWD=${CONFIG_SERVER_PWD}
    depends_on:
      - "config"
    
  rabbitmq:  
    image: rabbitmq:3.7.8-management  
    mem_limit: 350m  
    ports:    
      - 5672:5672    
      - 15672:15672  
    healthcheck:    
      test: ["CMD", "rabbitmqctl", "status"]    
      interval: 10s    
      timeout: 5s    
      retries: 10

  # https://hub.docker.com/r/openzipkin/zipkin
  zipkin:
    image: openzipkin/zipkin-slim
    mem_limit: 512m
    expose:
      - "9411"
    ports:
      - "9411:9411"
    environment:
    - RABBIT_ADDRESSES=rabbitmq
    - STORAGE_TYPE=mem
    depends_on:
      - "rabbitmq"

build.gradle

plugins {
    id 'org.springframework.boot' version '2.4.2'
    id 'io.spring.dependency-management' version '1.0.11.RELEASE'
    id 'java'
}

group = 'com.b-thinking.microservices.composite.product'
version = '1.0.0-SNAPSHOT'
sourceCompatibility = '1.8'

repositories {
    mavenCentral()
    maven {
        url 'https://repo.spring.io/milestone'
    }
}

ext {
   resilience4jVersion = "1.6.1"
}

dependencies {
    // Local projects depedencies
    // implementation files('../../api/build/libs/api.1.0.0-SNAPSHOT.jar')
    // implementation files('../../util/build/libs/util.1.0.0-SNAPSHOT.jar')    
    implementation project(':api')
    implementation project(':util') 

    // Testing: Use JSR330 injection
    // implementation 'javax.inject:javax.inject:1'

    // Implementations dependencies
    // Standard (actuator - for monitoring and Health)  
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    // WebFlux (asynchronous Web)
    implementation 'org.springframework.boot:spring-boot-starter-webflux'

    // SpringFox dependencies
    implementation "io.springfox:springfox-boot-starter:3+"
    implementation('io.springfox:springfox-spring-webflux:3+')

    // Implementation: Spring cloud
    implementation('org.springframework.cloud:spring-cloud-starter-config')
    implementation('org.springframework.cloud:spring-cloud-starter-stream-rabbit')
    implementation('org.springframework.cloud:spring-cloud-starter-stream-kafka')

    // Eureka: Service discovery client
    implementation('org.springframework.cloud:spring-cloud-starter-netflix-eureka-client')

    // Security
    implementation('org.springframework.boot:spring-boot-starter-security')
    implementation('org.springframework.security:spring-security-oauth2-resource-server')
    implementation('org.springframework.security:spring-security-oauth2-jose')

    // Spring Cloud Configuration through config server
    // https://stackoverflow.com/questions/30016868/spring-cloud-config-client-not-loading-configuration-from-config-server
    implementation('org.springframework.cloud:spring-cloud-starter-config')
    implementation('org.springframework.cloud:spring-cloud-starter-bootstrap')
    implementation('org.springframework.retry:spring-retry')
    implementation('org.springframework.boot:spring-boot-starter-aop')

    // CircuitBreaker with Resilience4J
    // From Spring Cloud (abstraction layer):  https://spring.io/projects/spring-cloud-circuitbreaker
    // implementation('org.springframework.cloud:spring-cloud-starter-circuitbreaker-reactor-resilience4j')
    // Native from github
    implementation("io.github.resilience4j:resilience4j-spring-boot2:${resilience4jVersion}")
    implementation("io.github.resilience4j:resilience4j-reactor:${resilience4jVersion}")

    // Implementation: Tracing
    implementation('org.springframework.cloud:spring-cloud-starter-sleuth') 
    // implementation('org.springframework.cloud:spring-cloud-starter-zipkin:3.0.0-M4')
  implementation "org.springframework.cloud:spring-cloud-sleuth-zipkin" 

    // Test dependencies
    testImplementation('org.springframework.boot:spring-boot-starter-test') {
        exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
    }
    testImplementation 'io.projectreactor:reactor-test'
    testImplementation('org.springframework.cloud:spring-cloud-stream-test-support')

}


dependencyManagement {
    imports {
        // mavenBom 'org.springframework.cloud:spring-cloud-dependencies:2020.0.0-M5'
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:2020.0.0"
    }
}

test {
    useJUnitPlatform()
}

application.yml

Notice that is started with profile 'docker', so host for rabbit is rabbitmq

app:
  eureka-username: u
  eureka-password: p
  eureka-server: localhost
  config-server: localhost
  zipkin-server: localhost

# Eureka & Ribbon client & instance
eureka:
  client:
    serviceUrl:
      defaultZone: "http://${app.eureka-username}:${app.eureka-password}@${app.eureka-server}:8761/eureka/"
    initialInstanceInfoReplicationIntervalSeconds: 5
    registryFetchIntervalSeconds: 5
  instance:
    leaseRenewalIntervalInSeconds: 5
    leaseExpirationDurationInSeconds: 5

ribbon:
  ServerListRefreshInterval: 5000
  NFLoadBalancerPingInterval: 5    

spring.cloud.stream:
  defaultBinder: rabbit
  default.contentType: application/json
  rabbit.bindings:
    input-products.consumer:
      autoBindDlq: true
      republishToDlq: true
    input-recommendations.consumer:
      autoBindDlq: true
      republishToDlq: true
    input-reviews.consumer:
      autoBindDlq: true
      republishToDlq: true    
  bindings:
    input-products:
      destination: products
      group: productsGroup
      consumer:
        maxAttempts: 3
        backOffInitialInterval: 500
        backOffMaxInterval: 1000
        backOffMultiplier: 2.0            
    input-recommendations:
      destination: recommendations
      group: recommendationsGroup
      consumer:
        maxAttempts: 3
        backOffInitialInterval: 500
        backOffMaxInterval: 1000
        backOffMultiplier: 2.0            
    input-reviews:
      destination: reviews
      group: reviewsGroup
      consumer:
        maxAttempts: 3
        backOffInitialInterval: 500
        backOffMaxInterval: 1000
        backOffMultiplier: 2.0            
    output-products:
      destination: products
      producer:
        required-groups: auditGroup
    output-recommendations:
      destination: recommendations
      producer:
        required-groups: auditGroup
    output-reviews:
      destination: reviews
      producer:
        required-groups: auditGroup
    sleuth-stream-save:
      destination: sleuth-stream-save
      group: sleuth-stream
      content-type: application/json

#endregion: Microservices streams    

spring.rabbitmq:
  host: 127.0.0.1
  port: 5672
  username: guest
  password: guest

# Tracing
spring.zipkin.enabled: true
# spring.zipkin.sender.type: web
spring.zipkin.sender.type: rabbit
spring.zipkin.base-url: http://localhost:9411
spring.zipkin.rabbitmq.addresses: localhost:5672
# // Trace all (normally is only 10% with 0.1)
spring.sleuth.sampler.probability: 1.0
# spring.sleuth.sampler.percentage: 1.0

# WARNING: Exposing all management endpoints over http should only be used during development, must be locked down in production!


logging:
  level:
    root: DEBUG


---
spring.config.activate.on-profile: docker
app:
  eureka-server: eureka
  config-server: config
  zipkin-server: zipkin
  

spring.rabbitmq.host: rabbitmq
spring.zipkin.base-url: http://zipkin:9411
spring.zipkin.rabbitmq.addresses: rabbitmq:5672

question from:https://stackoverflow.com/questions/65943408/zipkin-is-not-connecting-to-rabbitmq

Answer 1

I was not able to reproduce your issue with the spring-cloud-sleuth-sample-zipkin app (it worked to me), here's what I did:

1. Added org.springframework.amqp:spring-rabbit to the pom.xml
2. Added spring.zipkin.sender.type: rabbit to the application.yml
3. Started RabbitMQ using this docker-compose.yml
4. Manually created a queue (named zipkin) on the Rabbit Management UI
5. Started the app and hit it with a request
6. Manually get the messages out of the queue and checked if they have the right payload (they did)

A few pointers to troubleshoot this:

1. You should see this log event at startup

Created new connection: rabbitConnectionFactory#21917b6f:0/SimpleConnection@40803682 [delegate=amqp://[email protected]:5672/, localPort= 60265]

2. The connection is created by AbstractConnectionFactory, you can debug it
3. Telnet (telnet localhost 5672) can help to troubleshoot connectivity issues

Try to make it work using the sample and try to bring the working example closer to your app (by adding dependencies) and see what is the difference between the two and where will it break. If you can create a minimal sample app (e.g.: based on the zipkin sample) that reproduces the issue, please feel free to create an issue on GH: https://github.com/spring-cloud/spring-cloud-sleuth and tag me (@jonatan-ivanov), I can take a look.