Subject: Re: [dnsext] New Version Notification for draft-ah-dnsext-rfc1995bis-ixfr-02

From: Ondřej Surý <[email-masked]>
Date: Thu, 18 Aug 2011 17:53:33 +0200

On 21. 6. 2011, at 5:10, Masataka Ohta wrote:
> Shane Kerr wrote:
> 
>> The reason this came up is that two operators (Ondrej and me) both
>> encountered the problem in the real world and it adversely affected
>> our operations.
> 
> Are your examples essentially different from Brian Dickson's?

Much simpler.  The DNSSEC signed zone is huge and with anycasts scattered
around the world downloading the zone from hidden master(s) I just don't
want the AXFR happen if it can be prevented.

So if I had a knob which would turn the AXFR fallback off I could easily
setup an infrastructure which will buzz me if the IXFR cannot be done because
something has happened on the master and I could f.e. allow to sync one slave
at the time (or few of them).  And there's a lot of stuff which can happen
on the master to forget the differences.

The problem is that the 500MB zone * number of anycast slaves (25ish) is
10 GB of data to transfer.  Sure, I can just buy a really fat pipe, but
I am no friend of "throw more money" there solutions when we can have
simple protocol solution.

The IXFR-only is not a mechanism which would be used by small DNS operators,
but it can make a difference for huge zone owners like the average DNSSEC
signed TLDs.

O.
--
 Ondřej Surý
 vedoucí výzkumu/Head of R&D department
 -------------------------------------------
 CZ.NIC, z.s.p.o.    --    Laboratoře CZ.NIC
 Americka 23, 120 00 Praha 2, Czech Republic
 mailto:[email-masked]    http://nic.cz/
 tel:+420.222745110       fax:+420.222745112
 -------------------------------------------

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 19 Aug 2011 06:05:02 +0900

> want the AXFR happen if it can be prevented.

You fail to explain why the AXFR happens.

> And there's a lot of stuff which can happen

I can see none.

> simple protocol solution.

The simplest solution is not to forget the differences.

						Masataka Ohta

---

From: Ondřej Surý <[email-masked]>
Date: Fri, 19 Aug 2011 09:28:59 +0200

>> on the master to forget the differences.

From an operational experience, I see a lot.  You can run out of memory,
out of disk space, on disk journal can get corrupt (yes this happens),
DNS software can (and will) have other bugs, people can make mistakes.
Yes, all of this happens in the real world.

> The simplest solution is not to forget the differences.

From the pure theoretical view point, yes.  From the operational view,
I cannot agree with you.  You seem to operate on the grounds that
the run environment is perfect, the software is bug free, the people
don't make mistakes.  The operational experience don't concur with
that view.

Simply put, I see this as an design flaw in the original IXFR.
The decision on whether to do an AXFR-style IXFR or not should
be the decision of the slave (the client) and not the master server.
It should not be the decision of the server whether to send a few
megabytes of differences (IXFR) or hundreds of megabytes (AXFR-style).

The IXFR-only is a simple solution to remedy this problem.  It is
backwards compatible with existing servers and clients, nobody is
forced to use it, yet you are strongly opposed claiming that it's
not needed.  Even though at least mine and Shane's experience with
the operations of big-to-huge zones differ.

O.

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 19 Aug 2011 16:49:10 +0900

> Yes, all of this happens in the real world.

You are saying people can't properly operate a simple system.

Then, the solution can not be something to make the system more
complex.

 > From the pure theoretical view point, yes.  From the operational view,

I'm talking from the operational view point.

 > The IXFR-only is a simple solution to remedy this problem.

It is not.

							Masataka Ohta

---

From: Ondřej Surý <[email-masked]>
Date: Fri, 19 Aug 2011 16:58:47 +0200

> > The IXFR-only is a simple solution to remedy this problem.

Then I would say that we cannot come to agreement and it seems to
be pointless to continue the discussion.

The standard IETF mechanisms (consensus of the WG) would decide
whether the WG wants to have IXFR-ONLY or not.  I am not going
to push this forward if the WG doesn't want to adopt this mechanism.

O.

---

From: Masataka Ohta <[email-masked]>
Date: Sat, 20 Aug 2011 07:50:45 +0900

> be pointless to continue the discussion.

It is simply impossible to make protocols fool proof against
severe operational errors such as disk journal corruption.

Attempts to do so will result in complicated protocols with
a lot more operational problems.

If you can't admit so, there can be no agreement, of course.

						Masataka Ohta

---

From: Mark Andrews <[email-masked]>
Date: Mon, 22 Aug 2011 05:47:04 +1000

In message <[email-masked]>, =?utf-8?Q?Ond=C5=99ej
> megabytes of differences (IXFR) or hundreds of megabytes (AXFR-style).

Or one could say the it is a design flaw in how the TLD operators
generate the deltas.  Compression of delta is optional and the
consequences of doing so were well documented.  The TLD operators
wrote software that only compressed deltas then configured the
servers exactly as they were warned not to do if they compressed
deltas.

If one ignores changes to the SOA record, how may bytes do TLD
operators actually save by compressing deltas?  (2 SOA records cost
2 * (2 + 2 + 2 + 4 + 2 + 2 + 2 + 5 * 4) = 72 bytes as compression
pointers can almost always be used).  I realise there is extra work
to extract all the deltas rather than just the compressed delta
from the database.

If one saves the last hour (day?) of serials (so you have a precomputed
list of recent change sets to extract from the database) how much
extra work is it really to extract uncompressed deltas?  How far
behind do the distribution masters typically get (time and deltas)?

> _______________________________________________
> dnsext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsext

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Masataka Ohta <[email-masked]>
Date: Mon, 22 Aug 2011 09:41:17 +0900

> servers exactly as they were warned not to do if they compressed

The consequences can be avoided, if the result of compression is
consistent between servers. For example, if the compression is done
only on the primary server and only for deltas between two
consecutive XFRs.

But, as journal corruption is mentioned, there should be a lot
more severe operational errors involved, I guess.

						Masataka Ohta

---

From: Shane Kerr <[email-masked]>
Date: Mon, 22 Aug 2011 14:24:48 +0200

Mark,

> servers exactly as they were warned not to do if they compressed

There are other circumstance which can result in unwanted full zone
transfer.

Consider an environment where one has 2 distribution servers at a node,
which act as slaves to off-site server. These distribution servers act
as masters to any number of local slaves on site.

If the node suffers a power failure, then it is quite possible that both
distribution servers will need a full AXFR when they restart. If zone
updates are frequent, there is a good chance that they will not have the
exact same starting serial.

All of the local slaves will need to do a full zone transfer when they
start or soon after, getting the data from the first distribution server
to complete the AXFR. At the first zone update, then on average half of
them will need to get the entire zone contents again, because they will
try a distribution master that does not have the same starting serial.

For small zones this is not a big problem, but for large zones this can
result in several minutes of delay before zones get updated - at a
difficult time operationally. ("Yes, the node is up - it'll be a few
more minutes before we can put it back online...")

One solution is to have a single distribution server. However, this
introduces a single point of failure into the design. It may also be
possible to use some sort of high-availability setup to avoid this
problem. A better solution would be for something like IXFR-only.

There are probably other failure scenarios. My point is that there are
cases other than compressed deltas that would benefit from IXFR-only.

Forced fallback to full zone transfer is not a crucial operational
problem or it would have been solved already. But it is a problem - or
as Ondrej calls it a design flaw - and not too difficult to fix.

--
Shane

---

From: Masataka Ohta <[email-masked]>
Date: Mon, 22 Aug 2011 22:07:02 +0900

> Consider an environment where one has 2 distribution servers at a node,

Only 2 distribution servers? It should be a toy zone.

And, do you mean "zone", not "node"?

> distribution servers will need a full AXFR when they restart.

Are you seriously saying that it is quite possible that two servers
simultaneously loss checkpointing information because of power
failure?

Even then, there is no point of IXFR-only because, with your example,
neither server can offer IXFR.

> If zone

So?

> One solution is to have a single distribution server.

It's no solution. I'm afraid you completely misunderstand DNS.

							Masataka Ohta

---

From: Mark Andrews <[email-masked]>
Date: Tue, 23 Aug 2011 05:49:50 +1000

> simultaneously loss checkpointing information because of power

No.  I think Shane means that they have been out of contact long
enough that the AXFR style response is smaller than the IXFR style
response and the upstream supplied a AXFR style response.

> neither server can offer IXFR.

You are missing Shanes point, which is that after the initial AXFR
style response that the slaves that transferred from the DM that
transfered first, and hence had a lower serial, will not be able
to get a delta style response from the other DM.

> > exact same starting serial.

See above.
 
> It's no solution. I'm afraid you completely misunderstand DNS.

Shane isn't saying that it is a desirable solution.
 
The other solution is to prevent the axfr's out from the DM's until they
both have the same serials available ixfr from.

The is not to say that I'm apposed to ixfr only.

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 23 Aug 2011 09:34:45 +0900

> response and the upstream supplied a AXFR style response.

So, it's not two distribution servers but their clients, which
fail?

It's not a problem, as long as the distribution servers are
synchronized.

But, as Shane said "both distribution servers will need a full
AXFR when they restart", he seemingly assumes failures of
distribution servers, I'm afraid.

Anyway,

> You are missing Shanes point,

What?

> which is that after the initial AXFR

Are you saying two distribution servers (never introduce new
terminology such as DM only to amplify the confusion) do not
have the same history of a zone?

How can it happen without severe operational errors?

Note that you don't assume failure of distribution servers.

> both have the same serials available ixfr from.

I'm afraid both Shane and you have confused ideas on the configuration
of DNS servers, partially because of improper terminologies.

						Masataka Ohta