When DNS Fails: Troubleshooting the Silent Killer of Distributed Systems

If you're staring at 'Name or service not known' errors or inexplicable connection timeouts, you're likely grappling with DNS resolution problems. This article walks through diagnosing these issues, which frequently manifest as application slowness or outright failures across distributed systems. We'll cover the usual suspects: local resolver configuration mistakes, various levels of DNS caching gone awry, network path obstructions, and problems originating from upstream authoritative name servers. We'll also touch on how these issues present in containerized environments and provide practical, production-tested approaches for tracking them down before your pager really starts screaming.

Alright, so you're seeing those 'Host not found' or 'Temporary failure in name resolution' messages again. It's 2 AM, your coffee's cold, and the dashboard is a sea of amber warnings, but everything else looks fine. Database CPU is chill, Kafka lags are stable, and your application containers haven't OOMed. Yet, nothing can talk to anything outside its own IP. Classic DNS. It's always DNS. The unsung hero, until it inevitably becomes the silent killer, taking down services in the most subtle, insidious ways. It's less a system and more a series of deeply layered assumptions, each one capable of turning a perfectly good request into a 'connection refused' error five hops down the line.

The 'etc/resolv.conf' Minefield and Client-Side Resolvers

Your journey into DNS debugging usually starts with '/etc/resolv.conf'. It's the sacred text for any Linux-based client trying to translate a hostname. This file dictates which DNS servers your system should ask and how it should ask them. And it's a trap. A glorious, subtle trap.

Let's talk about 'options ndots:5' and 'timeout:2' and 'attempts:3'. These might seem innocuous. A default 'ndots:5' means if a hostname doesn't contain at least five dots, your system will first try appending every 'search' domain listed in '/etc/resolv.conf' before trying the hostname as-is. In a busy microservices environment where internal service names often have zero or one dot (e.g., 'auth-service', 'user-api.dev'), this means every single DNS query for an internal service will first generate multiple failed queries against your search domains before finally resolving the correct internal one. If your 'search' list contains unresponsive or slow DNS servers, or if you have many search domains, this immediately adds significant latency to every single DNS lookup. Multiply that by thousands of requests per second, and your application's p99 latency just jumped from 50ms to 500ms, not because your service is slow, but because it's spending half a second per request waiting for DNS to give up on searching for 'auth-service.us-east-1.svc.cluster.local.mycompany.com' before finally asking for 'auth-service.svc.cluster.local'.

Then there are 'timeout' and 'attempts'. The default 'timeout:5' and 'attempts:2' can mean a single lookup takes 10 seconds (5s timeout * 2 attempts) to fail per nameserver if the first nameserver is unreachable. If you have two nameservers listed, that's potentially 20 seconds. Some client libraries might handle this better, but many just blindly defer to the system resolver. If your application connection timeout is 10 seconds, and DNS takes 10 seconds to fail, you're getting a connection timeout, not a DNS error, which makes debugging even more fun.

Internal Link Suggestion: Application Connection Timeouts

Local Caching: Your Friend, Until It's Not

Beyond '/etc/resolv.conf', your system probably has local DNS caching. 'systemd-resolved', 'nscd', or even application-level caches (Java's networkaddress.cache.ttl, Go's lack of a default cache) all play a role. When a DNS entry changes – say, a service's IP shifts due to a redeployment, or a Canary rollout updates DNS records – these caches can hold onto stale data. Your application connects to the old IP, gets a 'connection refused' or 'connection reset', and blames the network. But the network's fine; it's just trying to connect to a server that no longer exists at that address.

Debugging this often means clearing local caches (sudo systemctl restart systemd-resolved, sudo nscd -i hosts, or restarting the application entirely if it's got an internal cache) and re-testing. It's frustrating because it's so transient. You restart, it works, you pat yourself on the back, and then two days later, another service update brings the ghost IP back to haunt you.

Recursive Resolvers and the Upstream Chain

Your client-side resolver doesn't know the world's IPs. It asks a recursive resolver (like your corporate DNS server, your ISP's server, or public ones like 8.8.8.8 or 1.1.1.1) to find the answer. This recursive resolver then walks the DNS tree, starting from the root servers, down to the TLD servers (.com, .org), and finally to the authoritative nameservers for the specific domain you're looking for.

Failures here can be:

• Recursive resolver overloaded/unresponsive: Too many queries, a DoS, or just under-provisioned. This results in client timeouts.
• Network path to recursive resolver blocked/slow: Firewalls, routing issues.
• Upstream authoritative server issues: The server actually owning the domain you're querying might be down, misconfigured, or experiencing its own network issues. This often results in SERVFAIL responses, meaning the recursive resolver tried but couldn't get an answer.

Cache Stampede vs. Cache Avalanche

While related, a cache stampede usually refers to many clients simultaneously trying to fetch data from a backend after a cache invalidation, whereas a DNS cache avalanche refers to a slightly different scenario where a single, critical DNS record (e.g., for a frequently accessed dependency) expires at the same time across many recursive resolvers. When this happens, all those resolvers simultaneously hit the authoritative server for that record, creating a massive, concentrated spike in queries that can overwhelm the authoritative server. Both lead to bad times, but the "avalanche" often implies a synchronized expiry causing the problem.

The Infamous Kubernetes DNS

Kubernetes environments add another layer of delightful complexity, typically with CoreDNS (or kube-dns historically) running inside the cluster. Each pod has its '/etc/resolv.conf' generated by the kubelet, often pointing to the CoreDNS service IP. The ndots:5 option is a particularly common culprit here, causing headaches for internal service resolution (e.g., 'my-service' within the same namespace). If your service names are short, you might try to set ndots:1 or ndots:2 in your Pod's dnsConfig to reduce query amplification.

Another common pattern is CoreDNS itself struggling. It might be CPU-bound, memory-starved, or its upstream resolvers (often the VPC's default DNS server) are themselves having issues. Monitoring CoreDNS pods (CPU, memory, query latency, error rates) is crucial. A single CoreDNS pod struggling can affect all pods routed through it, causing intermittent NXDOMAIN or timeouts that look like application issues.

Internal Link Suggestion: Kubernetes Network Policy Troubleshooting

Debugging Tools and a Realistic Scenario

Let's get to the nitty-gritty. When your application, 'ProductCatalogService', starts logging 'java.net.UnknownHostException: inventory-api.internal', here's how you'd typically proceed:

1. Check client-side '/etc/resolv.conf':

• On the affected instance/pod, cat /etc/resolv.conf. Note 'nameserver' IPs, 'search' domains, 'options'.
• Are the nameservers correct? Are there too many search domains? Is ndots reasonable for your internal service names?

1. Basic Queries from the Client:

• Use dig (my personal preference) or nslookup (if you're feeling nostalgic) from the problematic host.
• dig inventory-api.internal: Does it resolve? What's the response time? Do you get NXDOMAIN, SERVFAIL, or a timeout?
• dig @<nameserver_IP_from_resolv.conf> inventory-api.internal: Bypasses your local resolver chain and directly queries the configured nameserver. This tells you if the nameserver itself is the problem, or if the issue is reaching the nameserver.
• dig +trace inventory-api.internal: Shows the full delegation path from root servers, which is invaluable for identifying where in the chain a query might be failing (e.g., wrong NS records at the TLD, authoritative server unresponsive).
• dig +short txt chaos VERSION.BIND @<nameserver_IP>: If your nameserver is BIND, this can tell you its version, confirming you're talking to a live BIND instance. Other resolvers might support similar 'CHAOS' queries.

Here's a quick comparison of popular DNS tools:

Feature	'dig'	'nslookup'	'host'
Detail Level	Very detailed, canonical for diagnostics	Moderate, simpler query/response	Simple, quick lookup of A/AAAA/MX records
Output Format	Standard DNS message format, verbose	More human-readable, less raw data	Concise, designed for quick answers
Recursion Control	Fine-grained (+norecurse, +trace)	Limited	None
Query Types	All record types (A, AAAA, MX, NS, SOA, TXT)	Common record types	A, AAAA, MX
Usage	dig example.com A	nslookup example.com	host example.com
Preferred for	Deep troubleshooting, scripting	Quick checks, legacy scripts	Simplicity, scripting simple lookups

1. Network Path:

• traceroute <nameserver_IP>: Is there a routing issue preventing your host from reaching the DNS server?\n * ping <nameserver_IP>: Is the nameserver reachable at all? (Though ICMP isn't always indicative of UDP/TCP reachability for DNS).
• tcpdump -i any port 53 -n: Watch DNS traffic live. This is often the silver bullet. You'll see queries leaving your host, and critically, what responses are coming back (or not coming back). Look for retransmissions, NXDOMAIN vs SERVFAIL, and latency between query and response. If you see queries going out but no responses, it's a network issue or the nameserver is truly dead. If you see SERVFAIL, the nameserver is alive but can't resolve it upstream.

Production Scenario: The Cascading Timeout

Let's say 'ProductCatalogService' (running in a Kubernetes pod in us-east-1) needs to call 'inventory-api.internal'. This 'inventory-api.internal' is an ExternalName service pointing to an AWS ALB in a different VPC, accessible via a VPC Peering connection. Your CoreDNS pods are configured to forward queries for *.internal to your corporate DNS resolver at 10.100.0.10 (the VPC DNS resolver).

Suddenly, 'ProductCatalogService' logs are filling with ConnectTimeoutException.

Steps:

1. Initial check: kubectl exec -it <product-catalog-pod> -- cat /etc/resolv.conf. You find nameserver 10.96.0.10 (the CoreDNS service IP) and search default.svc.cluster.local svc.cluster.local cluster.local us-east-1.mycorp.internal. options ndots:5. This is already suspicious.
2. Verify resolution from pod: kubectl exec -it <product-catalog-pod> -- dig inventory-api.internal. The output shows a 5-second timeout, then an NXDOMAIN. This is bad. The query timed out, then CoreDNS likely returned NXDOMAIN after its own retries.
3. Check CoreDNS: kubectl get pods -n kube-system | grep coredns. Check logs of a CoreDNS pod: kubectl logs <coredns-pod> -n kube-system. You see errors like 'upstream timeout' for queries to 10.100.0.10.
4. Is CoreDNS overloaded? kubectl top pod -n kube-system | grep coredns. High CPU usage? Check CoreDNS metrics (if exposed, e.g., via Prometheus). Maybe too many pods hammering it, or too many ndots:5 queries are stressing it.
5. Test corporate resolver: From the CoreDNS pod itself (or any host that can reach 10.100.0.10), dig @10.100.0.10 inventory-api.internal. It times out too.
6. Network to corporate resolver: From the CoreDNS pod: traceroute 10.100.0.10. You notice the traceroute stalls at a router within your VPC peering connection, indicating a routing issue or firewall blocking UDP 53 traffic between your Kubernetes VPC and the corporate DNS resolver's VPC. Maybe a security group was updated, or a network ACL.

In this scenario, 'ProductCatalogService' is experiencing connection timeouts, but the root cause is a network-level blockage preventing CoreDNS from reaching its upstream resolver, which then causes CoreDNS to timeout and eventually return NXDOMAIN or just silently drop queries, leading to application timeouts. This cascades quickly: 'ProductCatalogService' retries, hammering CoreDNS more, exhausting its connection pool trying to hit 'inventory-api', and potentially triggering circuit breakers or making downstream services appear slow.

What if It's Not a Timeout? (NXDOMAIN vs SERVFAIL vs REFUSED)\n\nUnderstanding DNS response codes helps pinpoint the issue:

• NXDOMAIN (Non-Existent Domain): The DNS server explicitly told you the domain doesn't exist. This usually means a typo in the hostname, the record was never created, or it was deleted. Less commonly, it could be a search domain issue (as discussed with ndots).
• SERVFAIL (Server Failure): The DNS server you queried couldn't complete the request. It's not saying the domain doesn't exist; it's saying it failed to resolve it. This often points to issues with the recursive resolver itself (e.g., overloaded, internal error) or an authoritative server further up the chain being unreachable or misconfigured. SERVFAIL is often more critical than NXDOMAIN because it indicates a fundamental problem with the resolution infrastructure rather than just a non-existent name.
• REFUSED: The DNS server actively denied your query. This is typically a security measure – your client isn't allowed to query that server, or the server is configured to refuse recursive queries from your IP. Common in firewalled environments or when trying to use an internal DNS server from outside its intended network segment.

Observability for DNS

Real-time monitoring of DNS is crucial. CoreDNS exposes Prometheus metrics, which you should be scraping. Look at query rates, response codes (especially nxdomain_count_total, servfail_count_total), and upstream latency. Similarly, if you're running a custom resolver, ensure it exposes metrics. On your application hosts, monitor nscd or systemd-resolved health if they're active. Beyond specific DNS metrics, keep an eye on application connection establishment rates, service latencies, and error rates, as these are often the first symptoms of underlying DNS woes.

DNS is one of those deeply fundamental abstractions that's easy to take for granted until it decides to spontaneously combust. Debugging it usually means peeling back layers of caching, network routes, and resolver configurations, often finding that the problem isn't with your application code but with a five-line text file or a single firewall rule buried deep in the infrastructure. It's a game of careful observation and methodical elimination, played under pressure, with the entire system potentially hanging in the balance. Good luck, you'll need it.