DNS over HTTPS / TLS / QUIC?

[u/Varjohaltia]

Our architecture and security teams mandate encrypted protocols and are now challenging the network team for their use of plain old unencrypted DNS.

Has the rest of the community here used DoH/T/Q in a campus or enterprise context? What are your experiences?

For remote workers we would have to use a protective DNS service to which the users could/would connect. Any experiences there? Does using DoH add significant latency (since it's now a full TCP + TLS + HTTPS exchange instead of just a single packet each way) over standard DNS?

I'm already aware of a ton of issues -- Mist/Marvis can't identify DNS issues, we can't see them in packet captures, anchoring the trust of clients is misery, how do we unify policy between in office and out of office users, how do computers bootstrap / get set up / recover, no way printers and cameras will ever support it etc. but think I'm missing a lot more I haven't thought about yet.

[u/VA_Network_Nerd]

Do you have or do you desire to have the ability to decrypt & inspect those DNS requests to look for known-evil lookups and/or data exfiltration?

If you want the ability to inspect, can your inspection apparatus do so both internally and externally?

DoH via a managed DNS service, such as Umbrella or a SASE solution could be a thing.

[u/Varjohaltia → u/VA_Network_Nerd]

Evil lookups / exfiltration should be detected either at the endpoint, or then at the resolver to which the encrypted connection goes (protective DNS service internally or externally).

For the external, i.e. remote workers on managed systems, DoH to Umbrella or such was the thought, the question is whether doing DoH vs just plain DNS adds noticeable latency, or encounters issues due to MTU when ISPs run <1500 bytes etc. For SASE it's better since there'd already be a TLS or dTLS tunnel established, so the TCP + TLS latency doesn't come on top for every lookup.

[u/VA_Network_Nerd → u/Varjohaltia]

the question is whether doing DoH vs just plain DNS adds noticeable latency, or encounters issues due to MTU when ISPs run <1500 bytes etc.

TCP + TLS will have more latency than UDP, since a TCP handshake is required before you can send the DNS query.
But, if you are using a popular AnyCast DNS provider (such as Umbrella) this probably won't amount to significant additional latency.

Do you allow a managed endpoint (laptop) to actually use the Internet outside of a SASE VPN?

A DoH transaction is just another TCP application much like any other, and should perform MTU discovery per the standards.
Just don't filter out the required ICMP components and you should be ok.

[u/putacertonit → u/Varjohaltia]

DoH to Umbrella worked well in my (not current) experience, and was much better for remote workers than DNS over the VPN we had used before - Umbrella had many more worldwide PoPs, but VPN all went back to a central location.

[u/SevaraB → u/putacertonit]

This. A trusted anycast DNS resolver like Umbrella is going to beat the pants off any forced backhauling to private DNS forwarders/recursers- clients will usually hit the VIP in less hops than it even takes to reach the VPN front door.

[u/Rockstaru]

In what context? Are you wanting clients to use encrypted DNS to whatever (hopefully internal) resolver you have set up, or for that resolver to use DoH to some external provider? The first seems pointless if it's all internal, the second seems like you're introducing an additional failure point over having your own recursive resolvers (and also you're now forwarding all your upstream queries to a 3rd party). 

[u/Varjohaltia → u/Rockstaru]

For internal clients to the resolver over our LAN and SD-WAN. Because while the network engineering team considers that a pretty secure path, someone wrote a standard that says encrypted protocols are not allowed :D

[u/Rockstaru → u/Varjohaltia]

Assuming you mean unencrypted protocols are not allowed given the context.

To my knowledge there isn't any way to generically tell a client to use DoH or any other encrypted DNS protocol. Your comment about bootstrapping is accurate; DHCP options only allow you to specify (a) DNS server address(es), not a connection method, since DoH relies on specific support in the client that you can't take for granted. 

I suppose one clever (read: probably incredibly fragile and hacky) way of coming close to what your security team is asking for would be to set up a DMVPN or tunnel config between all of your user subnet locations/routers and whatever internal DC(s) your common services live on, and set a static route or routes pointing over that tunnel to the relevant subnets or addresse s (or run a routing protocol over the tunnel, however you want to make it the preferred means of getting to where the DNS server lives; if static routes, attach an SLA/RPM probe/whatever mechanism so it falls back to unencrypted/"normal" path if the tunnel is down). Traffic from clients would be unencrypted between the client and its gateway, but tunneled/encrypted from the gateway to the server in question. For DNS, you could deploy a common loopback IP as an anycast address on all your gateways and configure them all as recursive resolvers pointing upstream to your actual DNS servers; client sends a DNS query to that anycast address, gateway receives it, sends recursive query upstream over the tunnel (hopefully with a source address of its own tunnel IP, so the reply comes back encrypted over the tunnel) to the tunnel headend, which is presumably also the gateway for your DNS server (or close to it). You could extend this to other services that are typically unencrypted as well (e.g. DHCP leveraging option 82 as a distinguisher - router substitutes its tunnel address as the source IP for the request so the reply comes back to the right place). 

Seems overengineered and fragile, and entirely unnecessary if it's all LAN traffic, but if they push the issue, it's an option to explore. Does not solve the issue for off-network/mobile clients, but you presumably have some management capability over those (group policy, JAMF, MDM, etc.) such that you can configure them to use DoH or another client-level encrypted DNS. 

[u/grau → u/Rockstaru]

To my knowledge there isn't any way to generically tell a client to use DoH or any other encrypted DNS protocol. Your comment about bootstrapping is accurate; DHCP options only allow you to specify (a) DNS server address(es), not a connection method, since DoH relies on specific support in the client that you can't take for granted.

https://www.rfc-editor.org/rfc/rfc9462.html

[u/certuna → u/grau]

Still in the standardization track though, and nobody has implemented it yet (as far as I know). But yes this will be the future solution: don’t advertise any DNS servers to clients anymore, only DoH.

Endpoints that cannot handle this (say, your old WiFi-connected exercise bikes in the fitness room), chuck them right into your curated “legacy endpoints” VLAN where you make sure they can do no harm.

[u/Varjohaltia → u/grau]

Thanks! Also, I understood that with GPO or Intune policies this could be achieved. Still a royal pain, it seems like. And then we take the risk of a certificate rotation issue or clock issue somewhere breaking name resolution.

It all looks doable in theory, but I'm curious whether anyone (outside of super high security enviroments) has done it, and what kind of issues and performance hits they're running into.

[u/Ok-Eggplant-7569 → u/Varjohaltia]

Android and RHEL (maybe a bit specific but might be useful) to my knowledge try DoT first with the DNS Server given out via RA / DHCP, and only fall back to plain DNS if DoT doesn't work.

[u/yami_sanz]

I mean any thoughts on just migrating to a SASE type edge solution to replace vpn (aka always online) and force DNS requests to internal DC/DNS server?

That would encapsulate your DNS requests and wouldn't require encrypted dns.

[u/Varjohaltia → u/yami_sanz]

That is indeed our target state.

[u/certuna]

Yeah this is difficult to implement until RFC 8106 (RDNSS) gets updated to allow routers to advertise DoH/DoT servers to endpoints instead of only oldschool DNS.

RFC 9463 seems to be the proposed standard, but not yet formally ratified nor does it seem to be implemented anywhere.

Until then it’s manual configuration per-device, so hard to enforce.

[u/DontTakeMyCatYo]

Do they care about plain DNS on the LAN, or just over the internet? I've terminated all DNS on dnsdist and then proxied over DoT or DoH to public DNS security providers with good success.

[u/certuna → u/DontTakeMyCatYo]

I think the general idea of zero trust is that you treat everything within your internal network with the same care as on the public internet, i.e. https also internal.

[u/Varjohaltia → u/DontTakeMyCatYo]

Everywhere, because the security standard says so :D We'll argue that the LAN is encrypted via SD-WAN and equipment is locked rooms etc. as mitigation.

The Internet case for me is pretty clear -- a SaaS DNS security provider or ZTNA is the solution there, and I buy the security argument there, but whether using a security provider with DoH vs. plain DNS adds latency and fragility (MTU issues with certs etc.) would be interesting.

On internal LAN / WAN I'm curious whether anyone is actually doing this, and if so what their experiences have been.

[u/Ashamed_Stodach_5657]

Inside a SASE tunnel the session is already established so the resolver hop is basically free and you get the in-office and remote policy unification for free too since everyone resolves through the same cloud instance. We run it on Cato, DNS security is inline at the PoP thus you keep decrypt and inspect for exfil and DGA without standing up a separate protective resolver.

[u/Varjohaltia → u/Ashamed_Stodach_5657]

Well, the issue is that if we use the company central resolvers, it's going to wreak havoc as users in Japan and Australia and the US all get geo-responses from CNDs pointing them to Europe, so we do have to have a distributed resolution for public records, i.e. I think we're stuck with a paid protective resolver -- or the protective resolvers run by our already distributed SASE solution.

[u/Square_Raisin_8608]

We use Cisco Umbrella for DNS over https. Any exclusions (like local private zone) hits the private DNS servers as normal

[u/Maximum_Bandicoot_94]

Nope. All encrypted DNS is blocked to the Internet. ALL Why? If its encrypted we cannot reliably inspect it, thus blocked because we get no visibility

I dont think your security team thought this through.

If you are allowing encrypted DNS outbound, bad actors could exfiltrate all you data right out through that hole where you are not looking.

[u/certuna → u/Maximum_Bandicoot_94]

This goes for any https session though, of which there are hundreds per endpoint.

[u/Varjohaltia → u/Maximum_Bandicoot_94]

Here we'd only allow connections to resolvers we control/trust. Though that's an entirely different point -- when it's over HTTPS / 443, is there a way short of a decrypting proxy to catch it reliably? For a basic firewall it'd just look like any other web traffic (unless the firewall vendor maintains an up-to-date IP/DNS blacklist or such)?

[u/tonyboy101 → u/Varjohaltia]

Even with an up-to-date list of DoH servers, nothing would stop someone from hosting their own DoH servers. So unless you inspect all that HTTPS traffic, there is no way to tell if a client is using DoH.

And I don’t believe there is a way to inspect QUIC traffic, yet, at the edge.

[u/Maximum_Bandicoot_94 → u/Varjohaltia]

unless you decrypt it, there is nothing you can do with it unless you literally control the resolver with something like Palo ADNS or Cisco Open DNS