docker - wildfly-logstash does not send logs to logstash

Question

I am using jboss/keycloak:11.0.2 and wildfly-logstash (https://github.com/kifj/wildfly-logstash) for transferring logs from wildfly to logstash. I met a problem when logstash up earlier than wildfly in this case logs do not send to logstash. Sometimes I see log like this myHost: Name or service not known. Logger does not try to reconnect to myHost. An only restart helps. Wildfly startup script example:

embed-server --server-config=standalone-ha.xml --std-out=echo
module add --name=x1.wildfly-logstash --dependencies=org.jboss.logmanager,org.jboss.logging,javax.json.api --resources=/tmp/wildfly-logstash.jar

/subsystem=logging/custom-formatter=LOGSTASH-PATTERN:add(
  class=net.logstash.logging.formatter.LogstashUtilFormatter,
  module=x1.wildfly-logstash)

/subsystem=logging/custom-handler=LOGSTASH-SOCKET:add(level=DEBUG, class=net.logstash.logging.handler.SocketHandler,module=x1.wildfly-logstash,named-formatter=LOGSTASH-PATTERN,properties={hostname=myHost, port=${env.port}})

/subsystem=logging/async-handler=LOGSTASH-ASYNC:add(queue-length=512, subhandlers=[LOGSTASH-SOCKET])

/subsystem=logging/root-logger=ROOT:add-handler(name=LOGSTASH-ASYNC)

stop-embedded-server

Does anybody have an idea how to fix it?

Updated

08:42:15,759 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final
08:42:15,770 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
08:42:15,913 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: Keycloak 11.0.2 (WildFly Core 12.0.3.Final) starting
08:42:16,080 INFO  [org.jboss.vfs] (MSC service thread 1-2) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
08:42:17,124 INFO  [org.wildfly.security] (ServerService Thread Pool -- 19) ELY00001: WildFly Elytron version 1.12.1.Final
08:42:17,948 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
08:42:18,055 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
08:42:18,205 INFO  [org.jboss.as.patching] (MSC service thread 1-7) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
08:42:18,219 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost
08:42:18,330 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
08:42:18,335 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 11.0.2 (WildFly Core 12.0.3.Final) started in 2661ms - Started 56 of 79 services (32 services are lazy, passive or on-demand)
The batch executed successfully
08:42:18,661 INFO  [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: Keycloak 11.0.2 (WildFly Core 12.0.3.Final) stopped in 15ms
08:42:20,127 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.10.1.Final
08:42:20,197 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final
08:42:20,206 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
08:42:20,337 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 11.0.2 (WildFly Core 12.0.3.Final) starting
08:42:20,428 INFO  [org.jboss.vfs] (MSC service thread 1-2) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
08:42:21,238 INFO  [org.wildfly.security] (ServerService Thread Pool -- 21) ELY00001: WildFly Elytron version 1.12.1.Final
08:42:22,373 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
08:42:22,526 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
08:42:22,671 INFO  [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
08:42:22,685 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-8) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost
08:42:22,772 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
08:42:22,773 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 11.0.2 (WildFly Core 12.0.3.Final) started in 2640ms - Started 56 of 86 services (39 services are lazy, passive or on-demand)
The batch executed successfully
08:42:22,943 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 11.0.2 (WildFly Core 12.0.3.Final) stopped in 18ms
Executing cli script: /opt/jboss/startup-scripts/logstash-logging.cli
08:42:24,182 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 1.10.1.Final
08:42:24,233 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.11.Final
08:42:24,243 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.3.3.Final
08:42:24,360 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 11.0.2 (WildFly Core 12.0.3.Final) starting
08:42:24,449 INFO  [org.jboss.vfs] (MSC service thread 1-6) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
08:42:25,020 INFO  [org.wildfly.security] (ServerService Thread Pool -- 20) ELY00001: WildFly Elytron version 1.12.1.Final
08:42:25,928 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
08:42:26,032 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
08:42:26,172 INFO  [org.jboss.as.patching] (MSC service thread 1-8) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
08:42:26,182 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-5) WFLYDM0111: Keystore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost
08:42:26,266 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
08:42:26,268 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 11.0.2 (WildFly Core 12.0.3.Final) started in 2081ms - Started 56 of 86 services (39 services are lazy, passive or on-demand)
{"outcome" => "success"}
{"outcome" => "success"}
{"outcome" => "success"}
{"outcome" => "success"}
{"outcome" => "success"}
JAVA_OPTS already set in environment; overriding default settings with values:  -Dkeycloak.profile.feature.authz_drools_policy=enabled -Dkeycloak.profile.feature.upload_scripts=enabled -Djava.net.preferIPv4Stack=true
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/jboss/keycloak

  JAVA: java

  JAVA_OPTS:  -server  -Dkeycloak.profile.feature.authz_drools_policy=enabled -Dkeycloak.profile.feature.upload_scripts=enabled -Djava.net.preferIPv4Stack=true  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED

=========================================================================

08:42:27,098 INFO  [org.jboss.modules] (main) JBoss Modules version 1.10.1.Final
08:42:27,570 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.11.Final
08:42:27,577 INFO  [org.jboss.threads] (main) JBoss Threads version 2.3.3.Final
08:42:27,661 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 11.0.2 (WildFly Core 12.0.3.Final) starting
08:42:27,755 INFO  [org.jboss.vfs] (MSC service thread 1-8) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
08:42:28,424 INFO  [org.wildfly.security] (ServerService Thread Pool -- 20) ELY00001: WildFly Elytron version 1.12.1.Final
08:42:29,278 INFO  [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/core-service=management/management-interface=http-interface' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
08:42:29,367 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 28) WFLYCTL0028: Attribute 'security-realm' in the resource at address '/subsystem=undertow/server=default-server/https-listener=https' is deprecated, and may be removed in a future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
08:42:29,403 INFO  [org.jboss.as.repository] (ServerService Thread Pool -- 23) WFLYDR0001: Content added at location /opt/jboss/keycloak/standalone/data/content/2f/209df99aaed727ee0550f516d84eeb80382895/content
08:42:29,646 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
08:42:29,667 INFO  [org.xnio] (MSC service thread 1-5) XNIO version 3.8.1.Final
08:42:29,677 INFO  [org.xnio.nio] (MSC service thread 1-5) XNIO NIO Implementation Version 3.8.1.Final
08:42:29,739 INFO  [org.jboss.remoting] (MSC service thread 1-8) JBoss Remoting version 5.0.18.Final
08:42:29,772 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 40) WFLYIO001: Worker 'default' has auto-configured to 16 IO threads with 128 max task threads based on your 8 available processors
08:42:29,786 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 43) WFLYCLJG0001: Activating JGroup

Answer 1

You actually don't need to install that module to achieve this. WildFly has a built-in json-formatter and socket-handler. In fact a lot of the code looks like it was taken from the JBoss Log Manager :) The one difference is the fields. With the below example I'll just show you an option you can use.

embed-server --server-config=standalone-ha.xml --std-out=echo

/subsystem=logging/json-formatter=LOG-STASH:add(key-overrides={timestamp=@timestamp,message=@message,logger-name=@source,host-name=@source_host}, exception-output-type=formatted)

/socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=log-stash:add(host=localhost, port=${env.port})
/subsystem=logging/socket-handler=LOGSTASH-SOCKET:add(named-formatter=LOG-STASH, outbound-socket-binding-ref=log-stash, level=DEBUG)

/subsystem=logging/async-handler=LOGSTASH-ASYNC:add(queue-length=512, subhandlers=[LOGSTASH-SOCKET])

/subsystem=logging/root-logger=ROOT:add-handler(name=LOGSTASH-ASYNC)

stop-embedded-server

The socket-handler will attempt to reconnect if a connection fails. You can use the block-on-reconnect=true attribute if you want to block until the connection can be reset.