This post originally appeared on the Packet Pushers Ignition site on December 10, 2019.
DNS is sometimes used to optimize traffic between client and server. That is, a client needs to connect to a server. Resolving the IP address of the server’s hostname is the first thing the client must do before making the connection to the server. Different clients might be returned different IP addresses based on an intelligent algorithm designed to distribute server load or optimize user experience.
Traditionally, this was often used for Global Server Load Balancing (GSLB). You’ve probably heard of GSLB. It’s hardly new.
Sending Clients To Servers When You Don’t Know What You’re Talking About
An intermediate DNS server is often involved in the name resolution chain. A client could conceivably use any open DNS resolver to get an answer to their query. Herein lies a possible problem. In default DNS behavior, the authoritative DNS server won’t know the IP of the requesting client when an intermediate resolver is used. The authoritative server is just going to know the IP of the intermediate resolver.
Let’s assume our DNS scenario is one where the authoritative server uses an intelligent algorithm designed to return an IP address that optimizes user experience. If the authoritative name server only knows the IP of the intermediate resolver, how will it know what IP address to return? In this situation, the authoritative name server assumes that the intermediate resolver is close the querying client, and runs the algorithm against the IP of the intermediate resolver.
You got it. The authoritative name server says, “Close enough.” But is it close enough? Many times, yes, the resolving name server is awfully close to the requesting client. The “close enough” answer is exactly that, despite the domain name server not really knowing what it’s talking about.
However, there are scenarios where the intermediate name server & requesting client really aren’t the same distance from the authoritative name server. The assumption is bad, and you know what that means. This can happen on the Internet in a number of scenarios, but it can also happen enterprises with applications spread over multiple regions in one or more public clouds.
Sending a client to the wrong server could means tens of milliseconds of unnecessary latency, directly affecting user experience. You know. This app is slow. Why is it so slow? I’m going to take a smoke break while I wait for the next screen to come up. I hate my job. I’m stealing every paper clip in this place and not feeling guilty about it.
Introducing EDNS Client Subnet (ECS)
One possible solution to this situation is outlined in RFC 7871, Client Subnet In DNS Queries.
In RFC 7871, the EDNS0 field is used to carry a CSUBNET value pair. In plain English, that means sticking the subnet of the requesting user into the DNS query. Just so we’re clear, the EDNS0 field could carry lots of different things. The purpose of the field is to provide a way to extend DNS functionality.
For our purposes, we’re investigating RFC 7871’s particular scenario of using an EDNS option to carry the client subnet, commonly known as EDNS Client Subnet, or ECS.
“This document defines an EDNS0 [RFC6891] option to convey network information that is relevant to the DNS message. It will carry sufficient network information about the originator for the Authoritative Nameserver to tailor responses. It will also provide for the Authoritative Nameserver to indicate the scope of network addresses for which the tailored answer is intended. This EDNS0 option is intended for those Recursive Resolvers and Authoritative Nameservers that would benefit from the extension and not for general purpose deployment. This is completely optional and can safely be ignored by servers that choose not to implement or enable it.”
IETF RFC 7871, section 1.
ECS might remind you of the functionality provided by the X-Forwarded-For (XFF) header in HTTP. Different implementation, but the same idea. Know who, at least approximately, asked the question to begin with despite obfuscation by proxies or intermediate resolvers.
Seeing The DNS Client Subnet In Action
Put your propeller beanie on. We’re now taking a look at DNS queries with and without the CSUBNET field populated.
The DNS tool dig (more recent versions of it, anyway) gives us the ability to add the client subnet to the query via the +subnet CLI parameter. Below, we’re displaying three different queries. The goal of these is to illustrate how a DNS query and response looks as we use two different client subnets and no client subnet at all.
We’ll look at output from a CLI using dig, and then follow it with a Wireshark analysis of the outbound (client-side) query packet.
Query 1
In query 1, we ask the Google name server 8.8.8.8 to resolve google.com, and we include 165.166.167.0/24 as the client subnet.
Query 2
In query 2, we ask the Google name server 8.8.8.8 to resolve google.com, and we include 65.66.67.0/24 as the client subnet. Notice that instead of the group of IPs we got back in query 1, we get back a single IP from a different range. In other words, the client subnet field was acted upon.
Query 3
In query 3, we don’t populate the CSUBNET optional field at all. Ergo, this is just a normal DNS query. We get back a standard response. This doesn’t mean that the name servers authoritative for google.com didn’t apply an intelligent algorithm to the response. Rather, it means that the algorithm was not additionally informed by the inclusion of the client subnet metadata in the DNS query.
What DNS Servers Support EDNS Client Subnet (ECS)?
Public DNS servers supporting the EDNS Client Subnet field include Google name servers, such as 8.8.8.8 used in the examples above. Quad9 supports ECS. AWS Route 53 also supports it. F5 BIG-IP DNS supports ECS as of v14.0. I’m sure there are more.
Yeah, Cool. What About Privacy, Though?
I don’t want to belabor the issue of DNS privacy, as it’s a stridently discussed problem among the DoH community and elsewhere. But yes, there is some data leakage happening with ECS. There’s no way around it. The IETF recognizes this concern.
“We recommend that the feature be turned off by default in all nameserver software, and that operators only enable it explicitly in those circumstances where it provides a clear benefit for their clients. We also encourage the deployment of means to allow users to make use of the opt-out provided. Finally, we recommend that others avoid techniques that may introduce additional metadata in future work, as it may damage user trust.”
IETF RFC 7871, section 2.
Turn ECS Off? Turn It On? Just Tell Me What To Do!
In IT, the running joke is that it’s always DNS, no matter what the problem is. Getting DNS right is becoming ever more crucial for enterprises with apps scattered hither and yon across public cloud, colocation facilities, and on the workstation under some hapless developer’s desk. You know the guy. He’s also got that Linksys box he brought from home that was serving up DHCP on 192.168.0.0/24 that Friday afternoon you were out of the office having a decent lunch with friends for once. That guy. But I digress.
ECS could be a tool in your arsenal to help get DNS right in your environment. And yet, I suspect the feature is not well known to most enterprise IT engineers, because ECS was really meant to solve problems for Internet content delivery networks (CDNs). I’d never heard of ECS until a vendor friend of mine deep in the bowels of load balancers and service meshes brought it up over dinner at a conference one night. The other guy with us hadn’t heard of ECS either.
Maybe it was just the two of us living under a rock. But on the chance you’re also under the same rock, ECS seemed like it was worth sharing.
Does this mean you’re supposed to figure out what your ECS options are in your environment and turn it on? No, not necessarily. This article is meant simply to arm you with more DNS knowledge. If you’re fighting a problem of inappropriate IPs being returned to clients, maybe ECS is a way to handle it.
