Debugging Kubernetes DNS
Categories: tech
Tags: kubernetes dns
ArgoCD was timing out against the Gitea instance in the same cluster on the same node! After checking if the node could access Gitea directly it was time to put on my network hat.
Basic Kubernetes advice is start with DNS.
Since this issue appeared to be host centric I modified the pod manifest with a nodeSelector with arkhal. Using
dig google.com was speedy with a return from the Cluster DNS service IP. When using something like
dig gitea.prod.workshop.meschbach.org I was receiving the message:
;; reply from unexpected source: 172.31.0.112#53, expected 10.96.0.10#53
;; reply from unexpected source: 172.31.0.112#53, expected 10.96.0.10#53
nslookup returned similar results. cat /etc/resolv.conf produced reasonable results. Range 172.31.0.0 is the pod
subnet for arkhal meaning the response is going directly from pod to pod instead of through the service.
StackOverflow to the rescue,
as I do not appear to be the first to encounter such an issue! A quick verification with lsmod| grpe br_netfilter on
the node confirmed this was the problem.
A recent power outage knocked the entire system offline last week. I assume I have either been manually loading the module or I missed change log note in the huge tech stack of moving parts to make this all work.