kubernetes - Logging presto queries on grafana dashboard using event-listener and fluent-bit

Question

I have a presto instance running on a namespace and it has a custom plugin and event-listener configured that gets all the logs and forwards them to 24224 port.

I can see the logs if they are written as log.info("any string here") but fluency.emit(tag, map<>) is not working at all and i dont see any logs or event object being sent to fluentd instance.

can you please help in configuring how it can work or how i can debug this when on AWS EKS?

When I check the logs of daemonset fluentbit, I can see it is listening to TCP port 24224

[input:forward:forward.0] listening on 0.0.0.0:24224

Here is the configuration of fluent-bit:

fluent-bit.conf: |-
    [SERVICE]
        HTTP_Server    On
        HTTP_Listen    0.0.0.0
        HTTP_PORT      2020
        Flush          10
        Daemon         Off
        Log_Level      trace
  
    [INPUT]
        Name           forward
        Port           24224
        Listen         0.0.0.0
    
    [Output]
        Name grafana-loki
        Match *
        Url <MY LOKI URL HERE>
        TenantID ""
        BatchWait 1
        BatchSize 1048576
        Labels {job="fluent-bit"}
        RemoveKeys kubernetes,stream
        AutoKubernetesLabels false
        LineFormat json
        LogLevel warn
  labelmap.json: |-
    {
      "kubernetes": {
        "container_name": "container",
        "host": "node",
        "labels": {
          "app": "app",
          "release": "release"
        },
        "namespace_name": "namespace",
        "pod_name": "instance"
      },
      "stream": "stream"
    }

Here is my event-listener.properties file contents from presto:

event-listener.name=presto-fluent-logger
event-listener.fluentd-host=127.0.0.1
event-listener.fluentd-port=24224
event-listener.fluentd-tag=presto.query

and I have followed all the steps from this link: https://github.com/komamitsu/fluency

I am getting error in plugin ConnectionException: Connection Refused

I am sure that the presto plugin has got registered successfully and the error shows that it is not able to connect to my fluentbit instance may be. Not sure what host:port I should keep in plugin or event-listener.properties file

But I get no logs when I execute a query from presto cli. Can anyone please help? I am not very sure what all codes do I need to place here

question from:https://stackoverflow.com/questions/66061794/logging-presto-queries-on-grafana-dashboard-using-event-listener-and-fluent-bit

Answer 1

Waitting for answers