Subject: AXFR and TCP

From: Edward Lewis <[email-masked]>
Date: Mon, 4 Feb 2008 15:59:30 -0500

I am going over the AXFR on TCP part of the -07 draft.  Reading some 
comments and contrasting with some on this list I wanted to find out 
what's really happening now as opposed to what could be.

My vision of an AXFR session is, client opens a connection, asks, 
gets, closes connection, session over.  That seems quite trivial, 
something I can do with dig, etc.

It's possible to chain and multiplex zone transfers on the connection 
if the TCP connection management is independent of the AXFR session 
management.  I know this can be done.  But does any code do this now?

With the cheapness of just doing non-AXFR queries over UDP, why would 
a client go out of its way to use an open TCP connection (if its 
there) or go out of its way to open and sustain an TCP connection 
instead of just sending via UDP?

It seems too much work for me to have to keep a list of machines with 
which I have an open TCP session so I know if I shoulf shunt a query 
to the UDP or TCP transport managers.

So - does any code today multiplex zone transfers (between a pair of 
processes) or use the TCP connection for more than AXFR (and TC 
induced retries)?  Does any code maintain open TCP connections with 
other servers when "idle?"
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis                                                +1-571-434-5468
NeuStar

Mail archives, backups.  Sometimes I think the true beneficiaries of
standards work are the suppliers of disk drives.

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/namedroppers/>

---

From: Paul Vixie <[email-masked]>
Date: Tue, 05 Feb 2008 04:43:16 +0000

> > ...
> > the spec does not prohibit parallelism.  a conforming implementation
> > could spew requests down a TCP session, suck up the responses that come
> > back (in any order), demux based on query-ID, and get "query windowing";
> > however, that implementation would not be widely interoperable at this
> > time, and the spec should probably be updated to prohibit parallelism
> > and demultiplexing.
> 
> 	Why?  The spec will out live any broken implementations.

then we should update the spec to say, this might not work, here's what it
could look like when you try it, here's what you should do when it fails.

> 	A server that doesn't support multiplexed responses will
> 	just serialise the response.  If a client doesn't want to
> 	handle multipleplexed responses will just wait until the
> 	final soa to ask the query.  Each side is independently
> 	capable of enforcing what it supports without any extensions.

if a server can answer in order A-C-B when asked questions in order A-B-C,
then a client who does not look at the query-ID since it "knows" that the
order will be respected, will fail, and might have no backoff capability.
(granted, such a client would be unlikely to send request B before getting
response A, but the spec is a minefield that could make that look safe.)

---

From: Edward Lewis <[email-masked]>
Date: Mon, 4 Feb 2008 23:24:53 -0500

At 11:46 +1100 2/5/08, Mark Andrews wrote:

>	Handling one request at a time just serialises the response.
>	I presume that what you mean by "single threaded on a TCP
>	connection".

No, I meant that it is conceivable that a server might assign a 
connection (=BSD socket) to a thread and that thread isn't always 
reading data.

I'm just thinking of possible ways a server might be handling the 
connection in ways that might make multiplexing not work.  Or worse, 
a multiplexing client might confuse and kill the server process. 
Dunno, just thinking now.
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis                                                +1-571-434-5468
NeuStar

---

From: Mark Andrews <[email-masked]>
Date: Tue, 05 Feb 2008 10:53:27 +1100

> > It's possible to chain and multiplex zone transfers on the connection if
> > the TCP connection management is independent of the AXFR session
> it's common to open a TCP session, ask the SOA query, and then, using
> the same TCP session, ask for an AXFR.  i know of implementations like
> that.  i do not know of any current implementation that reuses the TCP
> session again after an AXFR, but the spec allows it.
> 
> it's also possible, using a widely popular stub resolver implementation,
> to open a TCP session and use it for a multiple queries.
> 
> none of this is parallelized in any implementation i'm aware of.  so,
> the query ID of the response isn't being used to demultiplex against
> multiple outstanding requests.  one of the reasons we don't do that is
> that some DNS TCP implementations do not put the query ID into every
> message of an AXFR response.
> 

	Why?  The spec will out live any broken implementations.
	Within a couple of years of a definitive spec being out
	there the vast majority of servers will have been fixed if
	they have not already been fixed.   For the few remaining
	ones you will have a flag that says don't do it for this
	machine.

	If this had got out in 2002 we would already be able to use
	it by now.

	How many people really need to set "transfer-format one-answer;"
	anymore?

> > With the cheapness of just doing non-AXFR queries over UDP, why would a
> > client go out of its way to use an open TCP connection (if its there) or go
> > out of its way to open and sustain an TCP connection instead of just
> > sending via UDP?
> 
> long story, many possible answers.  there are valid reasons to do this.
> 
> > It seems too much work for me to have to keep a list of machines with which
> > I have an open TCP session so I know if I shoulf shunt a query to the UDP
> > or TCP transport managers.
> 
> then when you are acting as an implementor, you should not do it that way.
> 
> > processes) or use the TCP connection for more than AXFR (and TC induced
> > retries)?  Does any code maintain open TCP connections with other servers
> > when "idle?"
> 
> see above.  it's possible that a number of things not prohibited by the
> current spec need to get prohibited since they would not actually work.
> 
> --
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Wouter Wijngaards <[email-masked]>
Date: Tue, 05 Feb 2008 14:11:21 +0100

Hi,

Quick notes about what NSD does.

|> My vision of an AXFR session is, client opens a connection, asks,
|> gets, closes connection, session over.  That seems quite trivial,
|> something I can do with dig, etc.

This is what NSD does on the AXFR-TCP on the query side. open, transfer,
close.

|> It's possible to chain and multiplex zone transfers on the connection
|> if the TCP connection management is independent of the AXFR session
|> management.  I know this can be done.  But does any code do this now?

NSD does not. Not even the SOA query (uses UDP, so as to not tie up TCP
resources).

|> With the cheapness of just doing non-AXFR queries over UDP, why would
|> a client go out of its way to use an open TCP connection (if its
|> there) or go out of its way to open and sustain an TCP connection
|> instead of just sending via UDP?

NSD prefers IXFR over UDP but some situations, as Paul alludes to,
require AXFR (and thus TCP).

|> It seems too much work for me to have to keep a list of machines with
|> which I have an open TCP session so I know if I shoulf shunt a query
|> to the UDP or TCP transport managers.
|
| 	A nameserver usually keeps this list as a by product of
| 	having the TCP session open.  There is no extra work to
| 	keep the list, just a litte extra code to use it instead of
| 	opening a new session.

Check, NSD also keeps this (sort of) list as a by product of having the
TCP session open.

| 	We definitely have transfers that queue awaiting TCP sockets
| 	being able to use a existing TCP connection will help.

Same here, long queues can form with transfers waiting for TCP sockets.
Operators complained about the long waiting times that it causes,
together with the overhead of startup and teardown of the TCP sessions.
Still, this is an issue in corner case situations only.

Best regards,
~   Wouter

---

From: Mark Andrews <[email-masked]>
Date: Tue, 05 Feb 2008 15:42:08 +1100

> connection (=BSD socket) to a thread and that thread isn't always 

	Well if it is not reading it won't respond to the next query
	until it decides to read the socket again.  Today servers
	have to ensure that reply messages are written in a atomic
	manner even if there are multiple readers.  It also has to
	ensure questions are read in a atomic manner.

	Think multiple PTR queries down a TCP socket from netstat.

	Mark

---

From: Edward Lewis <[email-masked]>
Date: Mon, 4 Feb 2008 19:15:39 -0500

>	Why?  The spec will out live any broken implementations.

Especially if it keeps getting derailed...

>	ones you will have a flag that says don't do it for this

My real motivation for asking what I asked is that my hope is to 
write a strawman approach which makes this new definition backwards 
compatible with the old.

(We can skip new client - new server and old client - old server and 
skip to the mixed cases.)

A new server will have no problem dealing with an old client in this 
case as this is basically a case of the client getting more 
aggressive.  An old client will be a degenerate form of a new client.

How will a new client know it can get away with the new behavior? 
Alternatively the question can be how will it know if it has to 
behave the old way?

I'm trying to run through my mind a scenario in which a new client 
sends two AXFR requests down the pipe, and/or a bunch of other 
requests, etc.  I think the only serious problem is the multiple 
concurrent responses without query IDs or question sections.

>	capable of enforcing what it supports without any extensions.

I can imagine that an old server might have a different failure mode 
or two, such as being single threaded on a TCP connection or being 
taught to panic if two requests come in.  That's why I'm asking.
-- 
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Edward Lewis                                                +1-571-434-5468
NeuStar

---

From: Paul Vixie <[email-masked]>
Date: Mon, 04 Feb 2008 22:18:27 +0000

---

From: Tony Finch <[email-masked]>
Date: Tue, 5 Feb 2008 12:41:32 +0000

On Mon, 4 Feb 2008, Paul Vixie wrote:
> multiple outstanding requests.

ADNS can have multiple outstanding queries on a TCP connection.

> Edward Lewis <[email-masked]> wrote:
> >

ADNS keeps the TCP connection open so it is ready to handle fallback
queries for truncated replies.

> time,

It has been doing this for over 10 years and there have been no bug
reports about its TCP behaviour.

ADNS doesn't do AXFR, and because it only uses TCP for TC fallback, it
doesn't usually depend heavily on concurrent TCP queries.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
TYNE DOGGER: SOUTHWEST VEERING NORTHWEST 5 TO 7, OCCASIONALLY GALE 8 AT FIRST
IN DOGGER. MODERATE OR ROUGH. SHOWERS. GOOD.

---

From: Mark Andrews <[email-masked]>
Date: Tue, 05 Feb 2008 11:46:33 +1100

> taught to panic if two requests come in.  That's why I'm asking.

	A client that panics with two requests is already broken and
	is ripe for a DoS attack.

	Mark

---

From: Mark Andrews <[email-masked]>
Date: Tue, 05 Feb 2008 15:54:35 +1100

> response A, but the spec is a minefield that could make that look safe.)

	netstat did just that.  If it timed out waiting for the
	server it would ask for C even if it hadn't a reply for B.
	This all works for ordinary queries.  Even BIND 4 did this
	right for ordinary queries.

	The only time when things get messy is when you start
	throwing in AXFR responses that don't follow the basic
	requirements of DNS (response id = query id).

	Mark

---

From: Mark Andrews <[email-masked]>
Date: Tue, 05 Feb 2008 10:31:59 +1100

> something I can do with dig, etc.

	You should be asking what nameservers need.  dig's requirements
	are trivial compared to a nameserver's requirements.
 
> management.  I know this can be done.  But does any code do this now?

	Not that I am aware of.  It's been one of my plans for a long
	time.  It should provide benifits for nameservers that slave
	lots of zones from the same master.
 
> instead of just sending via UDP?

	Because sending a IXFR query over a open TCP session always
	succeeds.  IXFR over UDP may tell you to use TCP.  Once the
	connection is available there is almost zero cost to use it.

	UDP queries put all the retry logic in the server.
	IXFR+EDNS+UDP requires a lot retry logic to handle packet
	loss and broken firewalls.

> to the UDP or TCP transport managers.

	A nameserver usually keeps this list as a by product of
	having the TCP session open.  There is no extra work to
	keep the list, just a litte extra code to use it instead of
	opening a new session.

> other servers when "idle?"

	We definitely have transfers that queue awaiting TCP sockets
	being able to use a existing TCP connection will help.

	Mark

---

From: Ian Jackson <[email-masked]>
Date: Tue, 5 Feb 2008 19:46:26 +0000

[ Moderators note: Post was moderated, either because it was posted by
   a non-subscriber, or because it was over 20K.  
   With the massive amount of spam, it is easy to miss and therefore 
   delete relevant posts by non-subscribers. 
   Please fix your subscription addresses. ]

> message of an AXFR response.

As Tony Finch has pointed out, the GNU adns stub resolver (of which
I'm the author and maintainer) has been doing multiple outstanding
queries on its TCP connections ever since I wrote it.  No-one has ever
reported any bugs or other problems from this.

Normally adns only uses TCP to retry on truncated replies, but
applications are free to pass a flag asking it to always use TCP.

> back (in any order), demux based on query-ID, and get "query windowing";

This is exactly what adns does.

> time, and the spec should probably be updated to prohibit parallelism

It seems interoperable from where I'm sitting.  If there is any doubt
in anyone's mind that this is permitted, the spec should be clarified.

Ian.

---

From: Ian Jackson <[email-masked]>
Date: Tue, 5 Feb 2008 19:47:22 +0000

> 	requirements of DNS (response id = query id).

Quite so.  adns doesn't do axfr so that isn't an issue; if I were to
add axfr as a feature (which is unlikely) it would use a separate
connection.

Ian.

---

From: Florian Weimer <[email-masked]>
Date: Fri, 08 Feb 2008 13:34:35 +0100

* Edward Lewis:

> connection in ways that might make multiplexing not work.

This pattern might be fairly common:

  while 1:
    buffer = socket.read( 65538)
    plen = int16at(buffer, 0)
    packet = buffer[2:2 + plen]
    process(packet):

It only works with half-duplex connections because the read call might
gnaw into the subsequent packet if pipelining/full duplex is used.

-- 
Florian Weimer                <[email-masked]>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99

---

From: Florian Weimer <[email-masked]>
Date: Fri, 08 Feb 2008 13:31:31 +0100

> 	Why?  The spec will out live any broken implementations.

A spec that gives implementors a chance to end up with an
interoperable implementation without too much tweaking is a worthwhile
goal, I guess.  For classic DNS, the days of prescriptive protocol
work are gone.

> 	they have not already been fixed.

Not if it is a frivolous protocol change, without any operational
advantage.

---

From: Florian Weimer <[email-masked]>
Date: Fri, 08 Feb 2008 13:38:04 +0100

> something I can do with dig, etc.

My gut feeling is that this is fine, with a small extension: ordinary
queries should be allowed on the connection before the AXFR session
starts.

---

From: Mark Andrews <[email-masked]>
Date: Sat, 09 Feb 2008 00:52:03 +1100

>     buffer =3D socket.read( 65538)
>     plen =3D int16at(buffer, 0)
>     packet =3D buffer[2:2 + plen]
>     process(packet):
> 

	RFC 1034 does not preclude the client sending multiple
	queries without waiting for a answer.  Such code is already
	broken as there are clients which don't always wait for a
	answer before send the next query.

	netstat could make libresolv have multiple outstanding queries.
	I first saw them 15+ years ago.

> --=20
> Kriegsstra=DFe 100              tel: +49-721-96201-1