logging - Why is Loki's Docker Driver Client stopping to log after some time?

Question

I want to send logs of my Docker containers to Grafana Loki. Therefore, I installed Loki's Docker Driver Client and started my containers with it. First I can see logs, but after some time I see no more logs.

Installation

I installed Loki's Docker Driver Client as a Docker plugin on my Docker Engine (version 20.10.2):

$ docker plugin install grafana/loki-docker-driver:master-54d1d3b --alias loki --grant-all-permissions

I didn't use the tag lastest, because of the bug Unable to connect to logging plugin in Swarm

Configuration

I started my Docker containers with Loki's Docker Driver Client as log driver:

$ docker container run
  --log-driver=loki
  --log-opt loki-url="$LOKI_URL"
  --log-opt loki-retries=5
  --log-opt loki-batch-size=400
  --log-opt max-size="10m"
  --log-opt max-file=5
  --detach
  --name $CONTAINER_NAME
  --restart unless-stopped
  $IMAGE:$TAG

I also added json-log driver's max-size and max-file to limit disk space, see Configuring the Docker Driver.

Problem

First I could see logs in Grafana and in command line with docker container logs, but after some time no more logs were shown. If I tried to look into the logs on Docker host and I saw an error:

$ docker container logs 75d4b13eb3e8
error from daemon in stream: Error grabbing logs: error getting log reader: LogDriver.ReadLogs: logger does not exist for 75d4b13eb3e8203b9247ecdeb41fdf495cc8fea7dcfc4775fd8261263b1dcd32

Research

I looked into the directories of the containers (see Where is a log file with logs from a container?), but I couldn't see any log files:

$ sudo ls /var/lib/docker/containers/75d4b13eb3e8203b9247ecdeb41fdf495cc8fea7dcfc4775fd8261263b1dcd32
checkpoints  config.v2.json  hostconfig.json  hostname  hosts  mounts  resolv.conf  resolv.conf.hash

I also checked the log path (see Get an instance’s log path), but it was empty:

$ docker inspect --format='{{.LogPath}}' 75d4b13eb3e8

I found container's logs in plugin's directory (see Loki log driver not storing logs as files on disk, even with keep-file: true), but the log files don't change anymore:

$ sudo ls -la /var/lib/docker/plugins/eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288/rootfs/var/log/docker/75d4b13eb3e8203b9247ecdeb41fdf495cc8fea7dcfc4775fd8261263b1dcd32
total 912
drwxr-xr-x  2 root root   4096 Jan 22 12:59 .
drwxr-xr-x 17 root root   4096 Jan 22 15:46 ..
-rw-r-----  1 root root 923177 Jan 22 13:34 json.log

I looked into Docker daemon's logs (see Read the logs) and found errors and a warning (at the same time logging stopped):

$ sudo journalctl -u docker.service | grep eac33cc9913c
[...]
[...]level=error msg="panic: send on closed channel" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="goroutine 153 [running]:" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="main.(*loki).Log(0xc0000c5e00, 0xc0001d81c0, 0xc0000c5e80, 0x0)" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="/src/loki/cmd/docker-driver/loki.go:69 +0x2fb" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="main.consumeLog(0xc0002c0480)" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="/src/loki/cmd/docker-driver/driver.go:165 +0x4c2" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="created by main.(*driver).StartLogging" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=error msg="/src/loki/cmd/docker-driver/driver.go:116 +0xa75" plugin=eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288
[...]level=warning msg="Unable to connect to plugin: /run/docker/plugins/eac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288/loki.sock/LogDriver.StopLogging: Post http://%2Frun%2Fdocker%2Fplugins%2Feac33cc9913ca962a189904392e516dd495d6fd52391fb5af4a34af46b281288%2Floki.sock/LogDriver.StopLogging: EOF, retrying in 1s"
[...]

What did I do wrong?

question from:https://stackoverflow.com/questions/65857938/why-is-lokis-docker-driver-client-stopping-to-log-after-some-time

Answer 1

Waitting for answers