Resolving DNS Asynchronously

well done peteris.

so we can optimise the dns query in terms of: 1. the user's code and 2. the server's response time.

but i think the real slowdown with dns is that not all servers are configured the same. as a result we often have to do more lookups (more recursion) than is really necessary.

this is i think what slows down so many programs that rely on DNS lookups.

i played around with trying to write my own simplistic recursion script using only standard UNIX tools and a simple 'host'-type lookup tool. e.g., the adns libs come with one called adnshost.

what i found is that ideally it should never take more than 3 queries to work from root to tld to host ip (A record). but people use gimmicks like CNAME or they configure their dns records in a number of different ways that force us to make extra queries, e.g. i counted 7 in one instance (i've read about people counting many more)- all this just to find a single ip.

it is very inefficient.

so here's an idea for you. this may get some of your readers agitated because in reaches back to the future.

compare the speed of async lookups over the internet versus the speed of having the ip's already in your hosts file, mounted in ram via a symlink. no disk reads.

in other words, you prefetch all the ip's using bulk queries (sounds like the chrome approach, eh?) and drop them in /etc/hosts to make them available to your DNS-requiring programs. the question is: is this faster than the 'real-time' way (DNS lookups, piecemeal, on the fly)?
google chrome takes this prefetch approach.

for something a bit more 'sophisticated' than a hosts file approach (i.e. something that could potentially scale up), here's a similar setup i'm experimenting with at the moment:

we run 'pdns_recursor' on a local ip, non-standard port, not port 53. then we point our 'host' lookup tool at that port, for our bulk queries- i.e., the 'prefetch' stage. pdns_r send out the queries and caches the reponses in memory. we dump the cache from pdns to a very simple, single zone file (our own simple 'root zone'), sort it and prune it. this zone file is then compiled into a binary .db format.
on a local ip, port 53, we run 'nsd', a fast authoritative only nameserver, serving up records that we prefetched, to our DNS-requiring programs. it simply reads the .db file. that's all it does. no slaves or zone xfers (although it can do those things). in our use, it serves only the local user programs that require DNS.

in other words we separate the lookups over the internet (the slow step) from the lookups done by our DNS-requiring programs. all lookups sent to port 53 by our DNS-requiring programs are local and never recursive. at that stage we have 'eliminated' the variablity introduced into DNS by varying nameserver configurations. local lookups. same effect as making use of a large hosts file.

as stated above, the bulk querying over the internet uses pdns_recursor. a few hundred lines of c++. it works pretty good.

i gave up trying to write my own recursion script. i concluded the reason it's difficult is server 'misconfigurations' (unnecessary complexity, imo). in other words, a very simple script will handle the majority of nameservers, which are configured in a straightforward manner and don't use forwarding gimmicks like CNAME.