docker - Problem with mount path using the volumes on Kubernetes

Question

I have a 3 node cluster (1 master and 2 worker nodes) I have a deployment running with pod image as nginx on one of workers node. The following is its manifest definition:-

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: nginx
    spec:
      volumes:
      - name: logs
        emptyDir: {}
      containers:
      - image: nginx
        name: nginx
        resources: {}
        volumeMounts:
          - name: logs
            mountPath: /var/log/nginx

If I tail the nginx logs, I can see logs are getting generated in the location as /var/log/nginx:-

vagrant@mykubemaster:~/my-k8s/sidecar$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-5bb7d5c6dd-hjxnr   1/1     Running   0          8m8s
vagrant@mykubemaster:~/my-k8s/sidecar$ kubectl exec nginx-5bb7d5c6dd-hjxnr -- tail -f /var/log/nginx/access.log
10.32.0.1 - - [06/Jan/2021:02:43:11 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.58.0" "-"
10.32.0.1 - - [06/Jan/2021:02:43:56 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.58.0" "-"
10.32.0.1 - - [06/Jan/2021:02:46:11 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.58.0" "-"
10.32.0.1 - - [06/Jan/2021:02:46:17 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.58.0" "-"
10.32.0.1 - - [06/Jan/2021:02:48:29 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.58.0" "-"

However, the problem is if I ssh into the worker nodes (both) and even the master node, I do not see any folder as nginx created under /var/log then where this file (access.log) is getting stored which I can stream very well using the -f and --tail command?

My understanding was when we do a volume mount then Pod uses the VM's storage location where the pod got provisioned. If on node02, then at some location as specified in the manifest on Node02.

I would really appreciate if you can help me to understand on this and find the file path.

Answer 1

You're storing the logs in an emptyDir volume. That's not intended to be persistent, or to be accessed from outside the pod; it doesn't have a fixed location on the host system, and the content there will be lost as soon as the pod is deleted.

In the specific case of image: nginx, the Docker Hub nginx image is configured by default to write its access logs to the container's stdout. If you delete the volumes: and volumeMounts:, then kubectl logs deployment/nginx will show you the access logs. A cluster administrator can then deploy a log collector to forward these logs to somewhere else; see e.g. Using a node logging agent in the Kubernetes documentation.

This same advice can apply to most other processes: set logging to go to the process's standard output and not a file, and Kubernetes will collect it, and you can deploy a forwarder to send the logs somewhere else if you need to review them later.