This error
[FATAL] plugin/loop: Seen "HINFO IN 6900627972087569316.7905576541070882081." more than twice, loop detected
is caused when CoreDNS detects a loop in the resolve configuration, and it is the intended behavior. You are hitting this issue:
https://github.com/kubernetes/kubeadm/issues/1162
https://github.com/coredns/coredns/issues/2087
Hacky solution: Disable the CoreDNS loop detection
Edit the CoreDNS configmap:
kubectl -n kube-system edit configmap coredns
Remove or comment out the line with loop
, save and exit.
Then remove the CoreDNS pods, so new ones can be created with new config:
kubectl -n kube-system delete pod -l k8s-app=kube-dns
All should be fine after that.
Preferred Solution: Remove the loop in the DNS configuration
First, check if you are using systemd-resolved
. If you are running Ubuntu 18.04, it is probably the case.
systemctl list-unit-files | grep enabled | grep systemd-resolved
If it is, check which resolv.conf
file your cluster is using as reference:
ps auxww | grep kubelet
You might see a line like:
/usr/bin/kubelet ... --resolv-conf=/run/systemd/resolve/resolv.conf
The important part is --resolv-conf
- we figure out if systemd resolv.conf is used, or not.
If it is the resolv.conf
of systemd
, do the following:
Check the content of /run/systemd/resolve/resolv.conf
to see if there is a record like:
nameserver 127.0.0.1
If there is 127.0.0.1
, it is the one causing the loop.
To get rid of it, you should not edit that file, but check other places to make it properly generated.
Check all files under /etc/systemd/network
and if you find a record like
DNS=127.0.0.1
delete that record. Also check /etc/systemd/resolved.conf
and do the same if needed. Make sure you have at least one or two DNS servers configured, such as
DNS=1.1.1.1 1.0.0.1
After doing all that, restart the systemd services to put your changes into effect:
systemctl restart systemd-networkd systemd-resolved
After that, verify that DNS=127.0.0.1
is no more in the resolv.conf
file:
cat /run/systemd/resolve/resolv.conf
Finally, trigger re-creation of the DNS pods
kubectl -n kube-system delete pod -l k8s-app=kube-dns
Summary: The solution involves getting rid of what looks like a DNS lookup loop from the host DNS configuration. Steps vary between different resolv.conf managers/implementations.