Subject: Re: [dnsext] Obsoleting SPF RRTYPE

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 3 May 2013 07:40:40 -0400

Responding to multiple arguments in the thread.

1) The majority of participants in the SPF group never intended to
transition to the SPF record. It was only ever proposed to get the spec
past IETF last call over objections from DNSEXT. I can't see anything has
changed since.

The DNSEXT participants did not exactly help their cause by telling
everyone that the Microsoft DNS server supported unknown record types long
after Microsoft had denied that it did and showed the source code to prove
that the server could not save or read unknown record types.

2) The issues involved in SPF, DKIM and DANE are different. DANE has a
dependency on DNSSEC and thus a requirement for new RR type support, DKIM
and SPF do not. DKIM did not require wildcard records (for signing mail at
least) and so use of prefix records was practical. SPF required wildcard
support because of its function and required that it did not have a DNSSEC
dependency.

3) +1 to Paul Hoffman on the not answering the question point. DNSSEC would
have deployed ten years ago with the ATLAS rollout had this WG accepted one
simple change to the specification for the sake of deployment. Not that the
fault is entirely in the WG, the chair and the AD bear most of the blame as
did the IESG for allowing an AD in one area to chair a WG in another.

4) SPF existed and had a significant deployed base before the WG was
started. The most that the WG could have done was to have persuaded some
users of SPF to use the new record instead of TXT. Ending use of the TXT
record was never an achievable goal and that is the only outcome that would
have had any value.

But anyway, what is done is done.

If people think that there is a value to TXT record like things and that
the new RR mechanism works then why not address the problem of SPF
polluting the TXT namespace (which I never liked) by creating a new TXT
record or so?

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 3 May 2013 07:42:20 -0400

It is not just stupid to further overload TXT, it is impossible.

The SPF group knew when they were doing the draft that they were
essentially making any other use of the TXT record infeasible in the
future. Or at least I pointed that out.

On Mon, Apr 29, 2013 at 11:19 AM, Jim Reid <[email-masked]> wrote:

> On 29 Apr 2013, at 15:49, Edward Lewis <[email-masked]> wrote:
>
> > The only concern is that pushing more data into TXT records is that, if
> more than one application does this and involves the same name, we are
> increasing the number of large RRsets pushed around.
>
> Nope. ALL the applications looking up TXT records will have to wade
> through a pile of them to find and make sense of the one(s) that are of
> specific interest. Good luck if > 1 of these TXT records have similar RDATA
> but are meant for use by different things. Though perhaps that's just a
> concern for dnsop or developers rather than this WG.
>
> Frankly, it's beyond stupid to overload TXT records now a unique RRtype
> code is simple to get (ha!) or some prefix convention could be applied:
> like adding _spf (say) to the QNAME so a big bunch of TXT records don't all
> have to sit at the same node in the tree.
>
> _______________________________________________
> dnsext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsext
>

-- 
Website: http://hallambaker.com/

---

From: Nicholas Weaver <[email-masked]>
Date: Fri, 3 May 2013 06:18:19 -0700

On May 3, 2013, at 4:42 AM, Phillip Hallam-Baker <[email-masked]> wrote:

> The SPF group knew when they were doing the draft that they were essentially making any other use of the TXT record infeasible in the future. Or at least I pointed that out.

No its not, its trivial to further overload the text record.  All you have to do is make sure YOUR strings don't start with the same magic string, and/or are located in a different convention in the DNS hierarchy, as SPF records.

And given the silly resistance amongst some in this group to tolerate allocating new RR types (look at the debate over the entirely sensible EUI48 and EUI64 RTYPEs), and the annoyance of getting authority software to allow one to provision new RTYPEs in an easily readable form [1], it makes perfect sense for developers who want to shove something into DNS to skip the whole IETF crap, create a convention, and shove things either in A-records (like the RBLs have done) or TXT records.

[1] Yes, you can specify it as:

>       The special token \# (a backslash immediately followed by a hash
>       sign), which identifies the RDATA as having the generic encoding
>       defined herein rather than a traditional type-specific encoding.
> 
>       An unsigned decimal integer specifying the RDATA length in octets.
> 
>       Zero or more words of hexadecimal data encoding the actual RDATA
>       field, each containing an even number of hexadecimal digits.

But lets face it, thats a PITA compared to going "MYTYPE: this is my record" as one string in a TXT record.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 03 May 2013 23:53:08 +1000

In message <[email-masked]>, Nicholas Weaver writes:
> > The SPF group knew when they were doing the draft that they were essentially making any other use of the 
> > TXT record infeasible in the future. Or at least I pointed that out.
> 
> No its not, its trivial to further overload the text record.  All you have to do is make sure YOUR strings 
> don't start with the same magic string, and/or are located in a different convention in the DNS hierarchy, 
> as SPF records.

As well as whatever anyone else is doing with TXT records.

> And given the silly resistance amongst some in this group to tolerate allocating new RR types (look at the 
> debate over the entirely sensible EUI48 and EUI64 RTYPEs), and the annoyance of getting authority software 
> to allow one to provision new RTYPEs in an easily readable form [1],

Yet there was code shipped that supported EUI48 and EUI64 before the first
word of complaint was raised.

> it makes perfect sense for developers 
> who want to shove something into DNS to skip the whole IETF crap, create a convention, and shove things eit
> her in A-records (like the RBLs have done) or TXT records.
>

Or you could write a tool that uses the existing resolver libraries
to dynamically load the records.  You don't have to wait for
nameserver developer to update nameservers or registrars to update
web forms to use new types.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 3 May 2013 09:57:53 -0400

DNS has an effective hard limit on the total size of all the records in a
RR set of 500 bytes. It is theoretically possible to go over that limit but
bad things start to happen and TCP fallback does not work any more reliably
than new RR types do.

SPF was already a deployed standard before the WG started. The IETF wrote
the specification after it was too late to change.

Now what could make the whole process a lot easier would be to allocate a
band of DNS RR codes for records that would all have TXT syntax. That would
allow BIND etc. to make one change to support the new syntax. Alternatively
we could extend the handling of unknown RR syntax so that there was a
string presentation option.

For example, imagine that all RRs from 1024 to 2048 are defined to be TXT
syntax. Anyone wanting any other syntax has to use a different RR code. So
a record would be:

example.com TXT1024 "Some stuff here"
example.com TXT1025 "A different project"

There would still be a need for prefixing but in general a DNS record
should be always prefixed or never prefixed.

On Fri, May 3, 2013 at 9:18 AM, Nicholas Weaver
<[email-masked]>wrote:

> No its not, its trivial to further overload the text record.  All you have
> to do is make sure YOUR strings don't start with the same magic string,
> and/or are located in a different convention in the DNS hierarchy, as SPF
> records.
>
> And given the silly resistance amongst some in this group to tolerate
> allocating new RR types (look at the debate over the entirely sensible
> EUI48 and EUI64 RTYPEs), and the annoyance of getting authority software to
> allow one to provision new RTYPEs in an easily readable form [1], it makes
> perfect sense for developers who want to shove something into DNS to skip
> the whole IETF crap, create a convention, and shove things either in
> A-records (like the RBLs have done) or TXT records.
>
>
>

---

From: Ted Lemon <[email-masked]>
Date: Fri, 3 May 2013 14:03:26 +0000

On May 3, 2013, at 9:57 AM, Phillip Hallam-Baker <[email-masked]<mailto:[email-masked]>> wrote:
Now what could make the whole process a lot easier would be to allocate a band of DNS RR codes for records that would all have TXT syntax. That would allow BIND etc. to make one change to support the new syntax. Alternatively we could extend the handling of unknown RR syntax so that there was a string presentation option.

I see some benefit in that if in fact TXT is the right format for storing the things that would get stored in these records.   However, the point has been made with respect to SPF that it would have been better had it not been just a text record.   I don't know whether this argument is correct or not, but if it is correct, making it easy to put in TXT records and hard to put in anything else is probably going to make things worse, not better.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 3 May 2013 10:22:49 -0400

The syntax of the record is an editor war.

There are arguments to be made in favor of a compact representation but the
IETF has traditionally favored verbose text headers over compact
representations for anything above the application layer.

If the WG wants to influence design choices then they have to provide an
infrastructure that supports the requirements that the applications area
people assert. Instead the interaction with this group has been that apps
area people say that they need X and this group tells them that they are
wrong. And that is not a productive way to go forward.

What it really comes down to is what the DNS server maintainers are willing
to support. Adding new RR types to my code is easy as I generate all the
code for parsing/emitting the RR data. The same is not true for BIND and
the other servers.

But for most applications, packing/unpacking binary DNS records really does
not help at all unless there is 1024-2048 bits worth of opaque binary data
involved. Which means that the only records for which binary format really
makes a difference tend to be records that involve PKI in some way.

On Fri, May 3, 2013 at 10:03 AM, Ted Lemon <[email-masked]> wrote:

>  On May 3, 2013, at 9:57 AM, Phillip Hallam-Baker <[email-masked]
> > wrote:
>
> I see some benefit in that if in fact TXT is the right format for storing
> the things that would get stored in these records.   However, the point has
> been made with respect to SPF that it would have been better had it not
> been just a text record.   I don't know whether this argument is correct or
> not, but if it is correct, making it easy to put in TXT records and hard to
> put in anything else is probably going to make things worse, not better.
>
>

---

From: Joe Abley <[email-masked]>
Date: Fri, 3 May 2013 11:46:21 -0400

On 2013-05-03, at 09:57, Phillip Hallam-Baker <[email-masked]> wrote:

> DNS has an effective hard limit on the total size of all the records in a RR set of 500 bytes.
> 
> It is theoretically possible to go over that limit but bad things start to happen and TCP fallback does not work any more reliably than new RR types do.

I don't think we necessarily see widespread problems with large responses.

I realise that large responses are a reasonable concern, and there is much scope for actual measurement to better characterise the landscape with facts and science, but I don't think it's sensible to work from the basis that the non-EDNS0-capable UDP-only DNS client sets an effective hard limit.

This is a very general comment with no science behind it, of course, so it doesn't really say anything beyond "here be dragons, or maybe not, who knows".

> Now what could make the whole process a lot easier would be to allocate a band of DNS RR codes for records that would all have TXT syntax. That would allow BIND etc. to make one change to support t…

Recent experience suggests it's possible to apply for a new RRType and have code-points assigned in less than a week.

It also seems possible (with only the effort involved in opening a couple of trouble tickets to make the feature request) to get the code-points implemented in multiple major authoritative servers in less than three weeks.

Most deployed DNS authority servers (I have no numbers, but given the purported market share of BIND9 I'll say "most") follow RFC3597 and hence are able to understand (for example)

  krill.hopcount.ca. 86400 IN TYPE108 \# 6 7c d1 c3 e8 10 2f

even if they can't understand

  krill.hopcount.ca. 86400 IN EUI48 7c-d1-c3-e8-10-2f

The fact that EUI48 has a conveniently-hexadecimal RDATA payload makes this an over-simple example, but it's not that hard to convert RRs with textual data into the same format for provisioning purposes if your nameserver doesn't understand the new RRType's presentation format ("not hard" used here to mean "possible in less than 20 lines of awk").

Given all of this, I don't really understand the benefits of pre-assigning TXT-like RRTypes for future use. The barrier to entry (today!) does not seem high, and having RRTypes defined with meaningful canonical representations and stable references for use seems beneficial.

Joe

---

From: David Conrad <[email-masked]>
Date: Fri, 3 May 2013 08:59:00 -0700

Nicholas,

On May 3, 2013, at 6:18 AM, Nicholas Weaver <[email-masked]> wrote:
> No its not, its trivial to further overload the text record.  

Not really. The ABNF of SPF does not take into account the order of RRs within an RRset is not guaranteed. The "v=spf1" version discriminator does not prefix each "term", it only prefixes a "record" and SPF terms can be split over multiple TXT records.

> And given the silly resistance amongst some in this group to tolerate allocating new RR types (look at the debate over the entirely sensible EUI48 and EUI64 RTYPEs),

The EUI48 and EUI64 RR types were allocated.  The 'silly resistance' came later.

> and the annoyance of getting authority software to allow one to provision new RTYPEs in an easily readable form [1],

Yep, that's a challenge, and hence the underscore label convention.  

> it makes perfect sense for developers who want to shove something into DNS to skip the whole IETF crap, create a convention, and shove things either in A-records (like the RBLs have done) or TXT records.

Yes, and the SPF community has greatly complicated (or removed) that capability for domains used in "MAIL FROM/HELO".

Regards,
-drc

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 3 May 2013 12:17:09 -0400

My point on TXT records and the 500 byte limit is that when you make a
query for an unprefixed TXT record you are going to get ALL the records for
all hacks that are overloading TXT with additional semantics. So the
approach is not scalable which I think you agree with.

Getting a RR assignment isn't the pain point. The only reason for getting
an assignment is to make it more likely we will get decent support in BIND
etc.

I know all about the unknown RR coding. But that is second best support.
What I am proposing is that we cut off a chunk of RR space that allows it
to be used in a useful fashion without waiting for new deployments of BIND
etc.

Having to update BIND is second best in any case. The fewer updates and the
fewer code paths to be verified, the better. Adding a new RR type to BIND
only takes me a few hours. But writing the regression tests and the support
infrastructure would be a lot more work and that is what it would be
necessary to consider the code to be 'production'.

On Fri, May 3, 2013 at 11:46 AM, Joe Abley <[email-masked]> wrote:

> > It is theoretically possible to go over that limit but bad things start
> to happen and TCP fallback does not work any more reliably than new RR
> types do.
>
> I realise that large responses are a reasonable concern, and there is much
> scope for actual measurement to better characterise the landscape with
> facts and science, but I don't think it's sensible to work from the basis
> that the non-EDNS0-capable UDP-only DNS client sets an effective hard limit.
>
> This is a very general comment with no science behind it, of course, so it
> doesn't really say anything beyond "here be dragons, or maybe not, who
> knows".
>
> > Now what could make the whole process a lot easier would be to allocate
> a band of DNS RR codes for records that would all have TXT syntax. That
> would allow BIND etc. to make one change to support the new syntax.
> Alternatively we could extend the handling of unknown RR syntax so that
> there was a string presentation option.
>
> Recent experience suggests it's possible to apply for a new RRType and
> have code-points assigned in less than a week.
>
> It also seems possible (with only the effort involved in opening a couple
> of trouble tickets to make the feature request) to get the code-points
> implemented in multiple major authoritative servers in less than three
> weeks.
>
> Most deployed DNS authority servers (I have no numbers, but given the
> purported market share of BIND9 I'll say "most") follow RFC3597 and hence
> are able to understand (for example)
>
> The fact that EUI48 has a conveniently-hexadecimal RDATA payload makes
> this an over-simple example, but it's not that hard to convert RRs with
> textual data into the same format for provisioning purposes if your
> nameserver doesn't understand the new RRType's presentation format ("not
> hard" used here to mean "possible in less than 20 lines of awk").
>
> Given all of this, I don't really understand the benefits of pre-assigning
> TXT-like RRTypes for future use. The barrier to entry (today!) does not
> seem high, and having RRTypes defined with meaningful canonical
> representations and stable references for use seems beneficial.
>
>
> Joe

---

From: "John Levine" <[email-masked]>
Date: 3 May 2013 17:18:43 -0000

> However, the point has been made with respect to SPF
>that it would have been better had it not been just a text record.

Uh, no.  SPF (the protocol) is overcomplex, with much of the
complexity ending up in the DNS records.  Decoding SPF records uses a
fairly complex parser; storing a tokenized version wouldn't help and
would probably be bulkier.

A decade ago there were a bunch of designated sender schemes kicking
around of which SPF was by far the most complicated and kludgy, but
it's the one that survived?  Why?  Partly it's because the proponents
of SPF were much better at PR than the rest of us, but mostly it's
because they correctly recognized that publishing the sender policies
in the DNS was the most difficult step, so SPF made it really easy to
do, even at the cost of making the rest of SPF much messier.  You go
to the SPF web wizard, check a few boxes to decribe your sending
setup, and it gives you a TXT record you can paste into your DNS and
you're done.  If stuff changes, e.g., the address of a mail host, or
the set of hosts your ISP uses, it's not your problem, your own SPF
record doesn't have to change.

SPF requires that mail recipients interpret a complex and fragile
description language to come up with a set of IP addresses for each
incoming message.  The other approaches had the sender publish the
addresses directly, rather than a rule for finding them.  But that
meant that senders had to do the equivalent work before publishing the
IPs, and had to update their DNS when the set changed.  That was and
is too hard for the small senders who are the bulk of sending domains
if not the bulk of mail, so SPF won.  (Don't argue, I'm describing the
actual history.)

SPF requires a ridiculous amount of work by recipients.  The source
for libspf2 is about 17,000 lines of code, and interpreting SPF
records requires more DNS queries than any other DNS application I
know.  But all this is to work around the chokepoint of publishing DNS
records, since everything else, looking them up, and calling libraries
from mail servers, is vastly easier.

Phill is right that 4408 added the type 99 record at the last minute
since it was otherwise blocked by the DNS theorists.  Nobody I know
ever thought there was a realistic chance of existing SPF users
switching.  There is a protocol bug in 4408 due to the last minute
insertion of type 99.  Apparently none of the type 99 enthusiasts
cared enough to review the changed spec and notice the bug, but of
course in the long run it didn't matter.

I agree that it's certainly somewhat easier to get an RRTYPE than it
was a decade ago, so long as you understand that you just ignore all
the nitpickery, but reasonable people still avoid doing so because
using a prefixed TXT or other existing type is so much easier.  It's
not just because they're all stupid.

R's,
John

---

From: Nicholas Weaver <[email-masked]>
Date: Fri, 3 May 2013 10:29:15 -0700

On May 3, 2013, at 9:17 AM, Phillip Hallam-Baker <[email-masked]> wrote:

> My point on TXT records and the 500 byte limit is that when you make a query for an unprefixed TXT record you are going to get ALL the records for all hacks that are overloading TXT with additional semantics. So the approach is not scalable which I think you agree with. 
> 
> 
> Getting a RR assignment isn't the pain point. The only reason for getting an assignment is to make it more likely we will get decent support in BIND etc.
> 
> I know all about the unknown RR coding. But that is second best support. What I am proposing is that we cut off a chunk of RR space that allows it to be used in a useful fashion without waiting for new deployments of BIND etc. 
> 
> Having to update BIND is second best in any case. The fewer updates and the fewer code paths to be verified, the better. Adding a new RR type to BIND only takes me a few hours. But writing the regression tests and the support infrastructure would be a lot more work and that is what it would be necessary to consider the code to be 'production'.

Why not instead just add a SECOND "unknown RTYPE encoding": a series of strings thats encoded like it is a TXT record.

---

From: David Conrad <[email-masked]>
Date: Fri, 3 May 2013 10:42:57 -0700

> Why not instead just add a SECOND "unknown RTYPE encoding": a series of strings thats encoded like it is a TXT record.

And we can use typecode 99!  :)

Regards,
-drc

---

From: David Conrad <[email-masked]>
Date: Fri, 3 May 2013 14:01:10 -0700

Phil,

On May 3, 2013, at 1:39 PM, Phil Pennock <[email-masked]> wrote:
> That is not my understanding as a reader of RFC4408 and as someone who
> worked with the coder (and documented the results) for the handling of
> TXT records in a widespread MTA to be as flexible as possible and to
> support SPF-style lookups.

Last sentence of RFC 4408, section 3.1.3:

"  SPF or TXT records containing multiple strings are useful in
   constructing records that would exceed the 255-byte maximum length of
   a string within a single TXT or SPF RR record."

Sure sounds to me like 4408 anticipates multiple TXT RRs.

Regards,
-drc

---

From: Doug Barton <[email-masked]>
Date: Fri, 03 May 2013 14:10:11 -0700

> Sure sounds to me like 4408 anticipates multiple TXT RRs.

Yeah, this is not theoretical. They show up in practice for 
organizations with larger mail infrastructures. IIRC the source for 
libspf2 has some info on this as well.

Doug

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 3 May 2013 14:57:20 -0700

> Sure sounds to me like 4408 anticipates multiple TXT RRs.

No, that section is all about having a single TXT RR whose complete content
doesn't fit in 255 characters.  It illustrates how to achieve this within a
single RR in zone file format.

---

From: Doug Barton <[email-masked]>
Date: Fri, 03 May 2013 15:30:43 -0700

On 05/03/2013 02:57 PM, Murray S. Kucherawy wrote:
> On Fri, May 3, 2013 at 2:01 PM, David Conrad <[email-masked]
> <mailto:[email-masked]>> wrote:
>
> No, that section is all about having a single TXT RR whose complete
> content doesn't fit in 255 characters.  It illustrates how to achieve
> this within a single RR in zone file format.

... which doesn't prevent people from splitting them across multiple 
records.

Doug

---

From: Phil Pennock <[email-masked]>
Date: Fri, 03 May 2013 23:29:55 +0000

If folks can read this message, then it appears that my posting problems are
caused by the IETF mail-system silently discarding mail with a BATV/PRVS
sender.  Reposting with that disabled.

On 2013-05-03 at 08:59 -0700, David Conrad wrote:
> Not really. The ABNF of SPF does not take into account the order of
> RRs within an RRset is not guaranteed. The "v=spf1" version
> discriminator does not prefix each "term", it only prefixes a "record"
> and SPF terms can be split over multiple TXT records.

That is not my understanding as a reader of RFC4408 and as someone who
worked with the coder (and documented the results) for the handling of
TXT records in a widespread MTA to be as flexible as possible and to
support SPF-style lookups.  (Exim, ${dnsdb...} stuff, used in some
minimal setups when libspf2 is not used.)

SPF can be split over multiple strings within a single TXT record but
can not be split over multiple RR of type TXT.

RFC 4408:
----------------------------8< cut here >8------------------------------
3.  SPF Records

   An SPF record is a DNS Resource Record (RR) that declares which hosts
   are, and are not, authorized to use a domain name for the "HELO" and
   "MAIL FROM" identities.  Loosely, the record partitions all hosts
   into permitted and not-permitted sets (though some hosts might fall
   into neither category).
[...]
3.1.2.  Multiple DNS Records

   A domain name MUST NOT have multiple records that would cause an
   authorization check to select more than one record.  See Section 4.5
   for the selection rules.

3.1.3.  Multiple Strings in a Single DNS record
[...]
4.5.  Selecting Records

   Records begin with a version section:

   record           = version terms *SP
   version          = "v=spf1"

   Starting with the set of records that were returned by the lookup,
   record selection proceeds in two steps:

   1. Records that do not begin with a version section of exactly
      "v=spf1" are discarded.  Note that the version section is
      terminated either by an SP character or the end of the record.  A
      record with a version section of "v=spf10" does not match and must
      be discarded.

   2. If any records of type SPF are in the set, then all records of
      type TXT are discarded.

   After the above steps, there should be exactly one record remaining
   and evaluation can proceed.  If there are two or more records
   remaining, then check_host() exits immediately with the result of
   "PermError".

   If no matching records are returned, an SPF client MUST assume that
   the domain makes no SPF declarations.  SPF processing MUST stop and
   return "None".
----------------------------8< cut here >8------------------------------

---

From: Phil Pennock <[email-masked]>
Date: Fri, 3 May 2013 18:00:31 -0400

> Sure sounds to me like 4408 anticipates multiple TXT RRs.

One TXT RR contains one or more strings, each of maximum length 255.

To rip an example I wrote for Exim docs:
   
      foo.example.  IN TXT "a" "b" "c"
      foo.example.  IN TXT "d" "e" "f"

      ${lookup dnsdb{>/ txt=foo.example}}   -> "a/d"
      ${lookup dnsdb{>/; txt=foo.example}}  -> "def/abc"
      ${lookup dnsdb{>/,+ txt=foo.example}} -> "a+b+c/d+e+f"

That's two RRs, each with three strings.

You can also look up spftest$N.test.globnix.net for N in 1..6 for more
examples.  N 2, 3 and 6 are most relevant.  (Both SPF=99 and TXT records
are present.)

spftest6 is fun, in that it breaks into multiple strings in the middle
of "v=spf1", which is absolutely allowed by the SPF specification.

-Phil

---

From: "John Levine" <[email-masked]>
Date: 4 May 2013 01:48:25 -0000

>... which doesn't prevent people from splitting them across multiple 

Section 3.1.2 of RFC 4408 forbids an SPF checker from looking at
multiple records at the same name.  Section 3.2 of the current draft
has the same language.  SPF has to forbid multiple records at the same
name, since their semantics are obviously impossible to define.

A single TXT record can, of course, contain any number of strings and
can be arbitrarily long.  See RFC 1035 sec 3.3.14.

Is there some reason it's preferable to guess rather than reading the
spec?

R's,
John

---

From: David Miller <[email-masked]>
Date: Fri, 03 May 2013 23:57:00 -0400

> can be arbitrarily long.  See RFC 1035 sec 3.3.14.

TXT records don't have a specified maximum length, but they cannot be
arbitrarily long.  They have to fit into an RR.  RDLENGTH, which
specifies the length of the RDATA field, is an unsigned 16 bit integer
(max 65535).  See RFC 1035 sec 3.2.1.

64k is very long, but not arbitrarily long.

> Is there some reason it's preferable to guess rather than reading the

-DMM

---

From: "John Levine" <[email-masked]>
Date: 4 May 2013 04:30:00 -0000

>64k is very long, but not arbitrarily long.

Well, yes, you're right, the RR's total length is a 16 bit value.  I'd
be pretty horrified if that were ever an issue for SPF.

R's,
John

---

From: [email-masked]
Date: Sat, 4 May 2013 13:33:12 +0000

On Fri, May 03, 2013 at 05:18:43PM -0000, John Levine wrote:
> ... and interpreting SPF
> know.  

	So what you are saying is that SPF is a carefully crafted DNS
	DDoS attack because it was too hard to do the work inside your
	own protocol?

	What ever happened to "Be Conservitive in What you Send..."

/bill

---

From: "John R Levine" <[email-masked]>
Date: 4 May 2013 11:16:38 -0400

>> ... and interpreting SPF records requires more DNS queries than any 
>> other DNS application I know.

> 	DDoS attack because it was too hard to do the work inside your

Yup, just like CNAME.

In the decade that SPF has been around, the putative DDoS has never been 
observed in the wild, ever, despite Doug Otis warning us about it every 15 
minutes since 4408 was a draft, and a few experiments I did with stunt DNS 
servers that returned giant trees of SPF records very slowly.  It turns 
out everyone does loop breaking, just like for CNAME.  It's a sloppy 
design from a decade ago that succeeded because it made an end run around 
the DNS provisioning problems of "better" alternatives.

> 	What ever happened to "Be Conservative in What you Send..."

It lost out to Stuff That Actually Exists Works Better than Stuff That 
Doesn't.

A decade ago, SPF was far from my favorite authentication design, but now 
it exists, it's more widely used than most standards track protocols, and 
it would be silly to pretend otherwise.  Hence the spfbis charter to 
standardize existing practice.

R's,
John

---

From: Douglas Otis <[email-masked]>
Date: Sat, 4 May 2013 18:01:02 -0700

On May 4, 2013, at 8:16 AM, John R Levine <[email-masked]> wrote:

>>> ... and interpreting SPF records requires more DNS queries than any other DNS application I know.
> 
> In the decade that SPF has been around, the putative DDoS has never been observed in the wild, ever, despite Doug Otis warning us about it every 15 minutes since 4408 was a draft, and a few experiments I did with stunt DNS servers that returned giant trees of SPF records very slowly.  It turns out everyone does loop breaking, just like for CNAME.  It's a sloppy design from a decade ago that succeeded because it made an end run around the DNS provisioning problems of "better" alternatives.
> 
> It lost out to Stuff That Actually Exists Works Better than Stuff That Doesn't.
> 
> A decade ago, SPF was far from my favorite authentication design, but now it exists, it's more widely used than most standards track protocols, and it would be silly to pretend otherwise.  Hence the spfbis charter to standardize existing practice.

Dear John and Bill,

I have heard this canard several times as if it offers some level of assurance.  How does one know whether SPF sourced any particular DDoS related query?  These represent reverse PTR, TXT, A, AAAA record types at any location.

It is also wrong to suggest SPF specification represents existing practice. Not all providers implemented SPF macro components.  The decision to depreciate SPF (99) RRs happened at roughly the same level of local-part macro expansion use.  Use of local-part macros requires other mechanisms to prevent unlimited spoofing when SPF pass becomes independent of the sending client.  The impact that such clever mechanisms may have is unknown if they ever become popular. 

Thank you Bill for clearly expressing this opinion.

Regards,
Douglas Otis

---

From: [email-masked]
Date: Sun, 5 May 2013 01:22:16 +0000

> >	DDoS attack because it was too hard to do the work inside your

	excuse me,  how do you reconcile your first statement; "more DNS queries
	than -any- other DNS application I know"  with "just like CNAME"

	CNAME semantics and behaviour are well known and studied.  You get -ONE-
	redirect.   Other DNS tricks have been DNS-abusive and have been abandon
	(BITSTRING) or redesigned (KEY/SIG).

> the DNS provisioning problems of "better" alternatives.

	care to publish the experiment and its results?
	I'd like to replicate it.

> It lost out to Stuff That Actually Exists Works Better than Stuff That 

	actually, not so much - there is certainly a whole lot of parasitic 
	behaviour in this decades work - there appears to be evidence that 
	the SPF RR type exists and works.

> standardize existing practice.

	Now that I have a hard time believing... "more widely used that most
	standards track protocols"  is a mightly big brush.  Perhaps you want
	to focus on SMTP authentication - then I would have an easier time 
	believing you.

---

From: Douglas Otis <[email-masked]>
Date: Sat, 4 May 2013 18:30:29 -0700

> 	to focus on SMTP authentication - then I would have an easier time 

Dear Bill and John,

May I also add, SPF verification of the Mail From parameter provides authorization for Non-Delivery Notifications.  It does not provide any form of Authentication.

Regards,
Douglas Otis

---

From: [email-masked]
Date: Sun, 5 May 2013 08:53:48 +0000

On Sun, May 05, 2013 at 12:51:52AM -0400, John R Levine wrote:
> Re CNAME and SPF looping, whatever you call them, the loop breaking issues 
> are the same.  This really isn't a complicated idea.  SPF is slightly 
> uglier since a single SPF record can have multiple includes, but the 
> solution, a fixed limit on the iterations, is the same.

	one vs. many ...  and the number of includes in SPF is bounded
	at the upper limit of...?

> > Please enumerate "all of the large systems" that use SPF. etc, etc
> 
> Sorry, it's not my job to do your homework.  You can get a pretty good 
> idea of whether a system uses SPF by seeing it if publishes SPF records 
> (the TXT records mail systems actually use, not the type 99 they don't.) 
> I expect you can do that better than I can.  I'll be pretty surprised if 
> you find a large system that doesn't other than Yahoo, who doesn't for 
> internal political reasons, but we know they check SPF since they do 
> queries for the records when you send them mail.

	Not my job to prove your unfounded assertions.

> >	John, your lack of crispness and accuracy suggest that you are 
> >	conflating
> >	any number of concepts.  While the number of messages surrounding 
> >	this
> >	topic is large, I am confident that your estimates are off by several
> >	orders of magnitude.
> 
> If you're saying that the author of RFC 6686 is lying about the data he 
> presents, including 400,000 domains that publish TXT SPF records and the 
> few thousand that publish type 99, I'll pass the message along.

	I'm saying that your claim of millions of messages is flawed.
	No as to the claims for RFC 6686, I'll be happy to take those numbers
	at face value. (but, yeah, pass my concerns along)
	402,000 domains using SPF is barely statistically relevent, considering
	there are over 350 million domains in existance.  just over 1%.

> >   If the SPF WG had actually read the DNS RFCs, they
> >   would have known about the formal advice regarding overloading TXT.
> >   Why did the WG reject this RFC?
> 
> As you would know if you had read the recent traffic in dnsxet and spfbis, 
> or looked at the spfbis charter, the WG is updating RFC 4408 and 
> documenting existing practice.  Nobody claims that it was a swell idea to 
> use unprefixed TXT records a decade ago, but unless you can provide us 
> with a time machine, there's nothing to be done about it now.

	apparently no one in the spfbis wg bothered to read http://tools.ietf.org/html/rfc5507
	and there is no time limit to the choice of a good idea v. a bad idea.
	a bad idea can and should be questioned at any time - there is no
	"its too late" arguemnt that should hold, particularly when there is
	roughly 1% penetration of the service against the number of existing domains.

> Despite what the fantasists in dnsext imagine, there is no chance 
> whatsoever of getting those hundreds of thousands of existing mail systems 
> to change the way they publish and check SPF data, particularly a change 
> that has less than no operational benefit.  (Don't argue unless you know 
> more people who run large mail systems than I do, and I meet pretty much 
> all of them at MAAWG meetings.)  The only recent changes I can think of 
> are that Yahoo used to check both TXT and type 99 and now only checks TXT, 
> and Micsosoft mail properties including Hotmail gave up on Sender ID and 
> just check SPF.

	my what a pessemistic/fatalistic attitude you have there.
	and again with your unsupported assertions.  

> >	To be clear - at this point I'd really like to see you document any
> >	formal study of the impact of SPF on the DNS.  From any credible 
> >	source.
> >	Anything.
> 
> The closest thing to a study is what's described in RFC 6686, and it 
> wasn't looking for effects on the DNS other than to observe in passing the 
> *fact* that in 2012, when it was published, a lot of DNS servers didn't 
> respond to type 99 requests at all (no NODATA, no NXDOMAIN, no nothing), 
> despite that being obviously, painfully wrong.  Again, if you don't 
> believe it, I'll be happy to let the author know you think he's lying.
> 
> Beyond that there is none.  No study, no effect.  Feel free to prove 
> otherwise.

	"No effect"???  You've just completely flipped from your original
	assertion back on May 3rd, "interpreting SPF records requires more 
	DNS queries than any other DNS application I know."  Which is it?

	I don't need to prove your assertions, I want some proof that your 
	assertions about the impact on the DNS are based on actual facts.

/bill

---

From: Mark Andrews <[email-masked]>
Date: Sun, 05 May 2013 21:06:34 +1000

I looked a 25579 unique domains that have sent me email
	over the last 20 odd years.

	737 (2.88%) failed to resolve when queried with SPF
	of those 737, 178 subequently succeeded with A TXT query

	853 (spf query) + 102 (txt query) of those return a non
	empty answer section (138 + 4 cnames). 

	As far as I can see  success difference between SPF and TXT
	is at the noise level.

	Mark

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 5 May 2013 17:34:00 -0700

>         there are over 350 million domains in existance.  just over 1%.

Isn't "domains that appear to be sending mail" a more useful universe from
which to sample than "registered domains"?

        apparently no one in the spfbis wg bothered to read
> http://tools.ietf.org/html/rfc5507

I'm an existence proof that your claim is false.  I've read RFC5507 and I'm
familiar with its contents.  I've already said that, were we writing this
anew, I think we'd likely be taking a different path here, one that would
make the members of dnsext much happier.  But since the former is false,
and there's a substantial deployed base much of which is unlikely to change
its behaviour for various reasons, we have to look at this a different way.

>         and again with your unsupported assertions.

His pessimism is founded in reality.  I have similar contact with the same
people, and I reach the same conclusion.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 5 May 2013 17:36:13 -0700

>         over the last 20 odd years.

That's a far more constrained sample size than the RFC6686 surveys used,
and I have some vague thoughts about likely bias of mail going to isc.org.

-MSK

---

From: Mark Andrews <[email-masked]>
Date: Mon, 06 May 2013 11:12:36 +1000

In message <CAL0qLwa-fWyB2NtVdMu02-iz8ZWnYo3+PJ4qFtxYeWe=[email-masked]>
> and I have some vague thoughts about likely bias of mail going to isc.org.

That list of domains includes personal as well as business
correspondence, spam sources, mail from various mailing lists.

And RFC6686 is biased as it use the Alexa top X which is known to
use more load balancers which are often not RFC 103[45] compliant
name servers.  They don't do negative answers properly.  Fixing one
set of nameservers in the Alexa top X can drastically change the
numbers as many domains Alexa top X are served by identical sets
of name servers.

The vast majority of name servers (from sites sending email or not)
respond to both TXT and SPF queries.  Of those that don't most are
broken for both TXT and SPF (and AAAA and NS and SOA).

Mark

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 6 May 2013 01:31:16 -0700

> numbers as many domains Alexa top X are served by identical sets

1) I think you're supporting RFC6686's conclusions there.

2) There was more than just the Alexa survey in RFC6686.

-MSK

---

From: Doug Barton <[email-masked]>
Date: Mon, 06 May 2013 01:38:05 -0700

On 05/06/2013 01:31 AM, Murray S. Kucherawy wrote:
> On Sun, May 5, 2013 at 6:12 PM, Mark Andrews <[email-masked]
> <mailto:[email-masked]>> wrote:
>

6686 asked the wrong questions, and then used the data to come to the 
wrong conclusions. It's totally irrelevant to the question, "How should 
SPF be done properly going forward?"

Doug

---

From: [email-masked]
Date: Mon, 6 May 2013 12:58:23 +0000

> which to sample than "registered domains"?

	Probably - but I was following Johns usage.

> its behaviour for various reasons, we have to look at this a different way.

	there is this wonderful thing called "O'Dells Law" which, paraphrased
	is:  "The installed based doesn't matter".   However, there is nothing
	preventing the SPF community from using TXT to store thier particularly
	unique stuff.  But there can be zero whining when other folks use TXT for
	their own purposes and confuse the heck out of SPF processors which get 
	(for thier purposes) malformed SPF data...

> people, and I reach the same conclusion.

	Sounds very much like the folks who hijacked net 1.0.0.0/8 for their
	wireless/NAT space.  (Or pretty much anyone who has "borrowed" IP space
	because it was too hard to get/justify the space properly.)  They have no
	incentive to change their installed base...  except for interoperability
	problems.

/bill

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 6 May 2013 07:04:16 -0700

>         (for thier purposes) malformed SPF data...

Numerous such cases exist (I gave ut.edu as an example) and nobody is doing
any of the aforementioned whining.  Establishing a loop across a set of
strings looking for the one that starts "v=spf1" is hardly rocket science.
If that's the primary concern, I think we're good to go from here.

-MSK

---

From: Mark Andrews <[email-masked]>
Date: Wed, 08 May 2013 00:44:16 +1000

In message <CAL0qLwaiL64XLxyKX2i94NAfAvMOqJwfdL3R9oB01FxJ=[email-masked]>, "Murray S. Kucherawy" writes:
> --047d7b86de323cd4af04dc088666
> Content-Type: text/plain; charset=ISO-8859-1
> 

As far as I can see there is nothing in it with respect to failure
rates and spf sites using type SPF are approaching 8% which is a
significant increase (2x) since the RFC6686 surveys.

TXT: 22157 NOERROR, 2777 NXDOMAIN, 645 SERVFAIL, 9289 v=spf1
SPF: 22068 NOERROR, 2774 NXDOMAIN, 737 SERVFAIL,  730 v=spf1

                                   CASE1 CASE2 CASE3
8614	txt only    33.7%   92.2%  96.8% 96.6% 95.2%
55	spf only     0.2%    0.6%
675  	txt + spf    2.6%    7.2%
        have spf             7.8%   3.2%  3.4%  4.8%

Mark

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 8 May 2013 00:14:46 +0200

>         (for thier purposes) malformed SPF data...
O'Dells Law only applies AFTER you have reached critical mass and growth is
automatic.

If you are in a situation where the installed base meets the requirements
just fine then the new proposal doesn't matter and will actually shrink
over time as a percentage of the installed base as people continue to use
the legacy system.

If the numer of domains with feature X is growing at a significantly faster
rate than the Internet then it will become ~100% in due course.

If the numer of domains with feature X is growing at a significantly slower
rate than the Internet then it will become ~0% in due course.

About one year after an RFC has been published there is sometimes a sudden
shock of realization that maybe deployment did matter after all. Catching
an existing system is very hard. Apart from POP vs IMAP and WWW vs Gopher,
I can't remember any examples offhand.

---

From: Mark Andrews <[email-masked]>
Date: Wed, 08 May 2013 09:13:55 +1000

In message <[email-masked]>
> I can't remember any examples offhand.

Both SPF queries and SPF records are growing w.r.t. TXT records
and queries so the arguement is moot.

Mark

---

From: Florian Weimer <[email-masked]>
Date: Sat, 11 May 2013 22:36:41 +0200

> 64k is very long, but not arbitrarily long.

It's also the overall packet length limit, so an entire RRset cannot
be larger.