Subject: Re: [spfbis] Obsoleting SPF RRTYPE

From: Stuart Gathman <[email-masked]>
Date: Sun, 19 May 2013 23:20:00 -0400

New observation:  claims that spf libraries didn't support SPF RR are
likely correct.

Long ago, Nostradamus foresaw that on 04/26/2013 01:39 PM, Dave Crocker
would write:
>
> Even then, retention is a bad idea.  There is no current basis for
> seeing any likelihood of utility for the RR.  None.

After seeing a complaint that spf libraries didn't support SPF RR, and
protesting that pyspf did, I went and checked, and lo and behold, SPF
RRs were disabled in pyspf at some point - and I hadn't noticed.  I
turned them back on, and discovered that we were missing important SPF
policies for business partners due to not checking SPF RRs!  I have not
yet run into any of the braindead servers (that timeout on unknown RRs)
that caused the initial difficulties with using the SPF RR.

Interestingly, I find that all the (rare) SPF RRs in the field I've
encountered so far publish *only* SPF RR.  Probably a wise strategy. 
The "publish both" recommendation in rfc4408 is problematic.

---

From: Andrew Sullivan <[email-masked]>
Date: Sun, 19 May 2013 23:53:30 -0400

Chair hat: this is a real question to the WG.

> The "publish both" recommendation in rfc4408 is problematic.

Given our charter, what conclusion for our work product do you think
should produce?

A

-- 
Andrew Sullivan
[email-masked]

---

From: Mark Andrews <[email-masked]>
Date: Mon, 20 May 2013 14:54:18 +1000

> Given our charter, what conclusion for our work product do you think

Well given removal of features that are in use is prohibited and
that SPF records are both looked up (libspf2) and published (8% of
domains that publish SPF record publish a type 99 SPF record) and
there are not significantly different failure rates.  I respectfully
suggest that we require that type 99 records be looked up first and
that type TXT records be looked up on SERVFAIL or a NODATA response.

Note type 99 use has doubled since the RFC 6686 surveys were done.

Mark

% grep status: domainlist.{spf,txt,mx,a,aaaa} | awk '{print $1, $6}' | sed 's/:;;//' | sort | uniq -c
22287 domainlist.a NOERROR,
2765 domainlist.a NXDOMAIN,
 524 domainlist.a SERVFAIL,
22149 domainlist.aaaa NOERROR,
2766 domainlist.aaaa NXDOMAIN,
 661 domainlist.aaaa SERVFAIL,
22240 domainlist.mx NOERROR,
2768 domainlist.mx NXDOMAIN,
 568 domainlist.mx SERVFAIL,
22068 domainlist.spf NOERROR,
2774 domainlist.spf NXDOMAIN,
 737 domainlist.spf SERVFAIL,
22157 domainlist.txt NOERROR,
2777 domainlist.txt NXDOMAIN,
 645 domainlist.txt SERVFAIL,
% grep v=spf1 domainlist.spf | wc 
     730    6923   74024
% grep v=spf1 domainlist.txt | wc
    9289   91287  984437
% grep v=spf1 domainlist.txt | awk '{print $1}' > txt.list
% grep v=spf1 domainlist.spf | awk '{print $1}' > spf.list
% comm -23 txt.list spf.list | wc
    9256    9256  132941
% comm -13 txt.list spf.list | wc
     697     697    9832
% comm -12 txt.list spf.list | wc
      33      33     439
% 

> A
> 
> -- 
> Andrew Sullivan
> [email-masked]
> _______________________________________________
> spfbis mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/spfbis
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 20 May 2013 01:11:40 -0400

> The "publish both" recommendation in rfc4408 is problematic.

It was turned off because I turned it off after it caused problems (Stuart and I 
are co-maintainers of pyspf).  This was several years ago, so either I 
misremember discussing the change with him when I did it or he's forgotten the 
discussion.

Until it's reliable enough to use, it shouldn't be used and, of course, no one 
will care if it's reliable if it's not used.

All of which is irrelevant to the rationale for removing it.

Scott K

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 20 May 2013 01:13:29 -0400

> that type TXT records be looked up on SERVFAIL or a NODATA response.

That would be a ridiculous design.

Scott K

---

From: Commerco WebMaster <[email-masked]>
Date: Mon, 20 May 2013 02:52:45 -0600

> Given our charter, what conclusion for our work product do you think

Andrew,

As I recall, the charter may not allow additional features to SPF1 
syntax, so this may not even be an answer.  Even so...

It would be great to add something into the SPF TXT record to serve as 
an SPF type 99 preference indicator, that simply redirects the DNS 
requester to the type 99 SPF RR, much in the way that a redirect can 
take place to point to a different TXT record via the "redirect" syntax 
in SPF1 today.

To that end I propose type99:[0|1] for both TXT and SPF (99) RRs.

For example:
TXT "v=spf1 type99:1 ip4:192.168.123.123 ip4:10.10.123.123 -all"
could mean for existing libraries that this domain only sends email via 
the two IP4 addresses presented (as it always would have), but if you 
happen to support type99 syntax, the domain is also published using the 
SPF RR, so chose that for now and don't bother doing TXT request in 
future SPF requests for this domain that may be required for this SPF 
transaction, as since now you know we publish in both, you can limit 
your requests to SPF 99 to stop doing double work from your side and 
double taxing our DNS servers (and heck, you might even go so far as to 
cache that knowledge for the TTL supplied in the response).

This would mean that a publisher MUST then have an identical spf1 record 
like this:
SPF "v=spf1 type99:1 ip4:192.168.123.123 ip4:10.10.123.123 -all"

Alternately:
TXT "v=spf1 type99:0 ip4:192.168.123.123 ip4:10.10.123.123 -all"
could mean we are aware of type 99, but for whatever reasons (e.g., our 
DNS or device does not support 99), we are only publishing a TXT record 
for SPF, so you must not even use type99.

Perhaps one could also allow the type99:[0|1] syntax in the SPF type 99 
record, to serve as a contextual switch back mechanism when redirecting 
over to a domain known to only publish type TXT to avoid causing any 
harm (unintended or otherwise).

Again, I'm pretty sure the above goes beyond charter, but it might 
present an easy way to help libs going forward understand that the 
preferred method of handing SPF for a given domain is via SPF type 99 
records without breaking the original TXT only SPF environments for 
records existing in the time preceding MARID and the approval and 
introduction of Type99 SPF RRs at or about that time.

Since TXT was the original RR spec for SPF1, it is reasonable to keep 
the TXT as the first DNS lookup choice for SPF1, but if a domain's dns 
host/zone file indicates its preference for SPF type 99, perhaps the 
above might allow for using the SPF type 99 and start the process of 
coaching people to implement type 99 RR records for any future SPF 
versions, where it becomes the primary RR.

Since it was the work product of this group that uncovered the technical 
problem with having the SPF type 99 RR published coterminous with the 
TXT RR and consequentially suggesting use of SPF type 99 be depreciated, 
perhaps it might not be beyond the bounds of reason to push the limits 
of the charter by instead supplying a fix in the SPFbis to address the 
technical problem with a possible solution rather than simply depreciating.

Best,

Alan M.

---

From: "John Levine" <[email-masked]>
Date: 20 May 2013 12:00:33 -0000

>requester to the type 99 SPF RR

By the time it saw the flag, an SPF checker would already have fetched
the TXT record, which would presumably be in the local DNS cache for
subsequent lookups.  Other than gratuitously increased DNS traffic,
and decreased reliability when the type 99 lookup fails or the SPF
record isn't consistent with the TXT record, what would this
accomplish?

R's,
John

---

From: Dan Schlitt <[email-masked]>
Date: Mon, 20 May 2013 11:47:37 -0400

+1

-- 

Dan Schlitt
[email-masked]

---

From: Olafur Gudmundsson <[email-masked]>
Date: Mon, 20 May 2013 13:01:06 -0400

> That would be a ridiculous design.

Why ? 
This is the right way to build fallback in DNS given the installed base, ask for one type and if it is not available 
try the fallback. 
If your client can do queries in parallel then you can do both and use one of the ones that returns an positive answer, but in that case
there is a race condition if both return and the two types differ, that  can be addressed by preferred type rule. 
Mark's suggestion is SPF is the preferred type, and TXT is compatibility type in sequence. 
You may disagree with that order …. 
 
	Olafur

---

From: "John Levine" <[email-masked]>
Date: 20 May 2013 17:08:32 -0000

>You may disagree with that order …. 

The issue of sequential lookups was debated at great length on this
list.  Could you explain why you disagree with the reasoning that led
to our conclusion?

---

From: Olafur Gudmundsson <[email-masked]>
Date: Mon, 20 May 2013 13:23:19 -0400

On May 20, 2013, at 1:08 PM, "John Levine" <[email-masked]> wrote:

>> You may disagree with that order �. 
> 

Thanks your selective copying the main reason I was forced to reply is lost:
Scott Kitterman said about Mark's proposal: 
    "That would be a ridiculous design."

I was trying to point out that Mark's proposal is sound and the only point of contention is the possible order of 
queries, without saying anything about what my position is. 

I do not have a bone in this fight other than the FUD that some people spew out about DNS and
DNS software. 

In the prior message that I read on this thread Scott said the following
"
It was turned off because I turned it off after it caused problems (Stuart and I 
are co-maintainers of pyspf).  This was several years ago, so either I 
misremember discussing the change with him when I did it or he's forgotten the 
discussion.

"

Which can be taken to mean he is partially responsible for the difficulties that some people have in getting SPF records 
from the DNS when present, thus has skewed the measurements used to justify the removal SPF type. 

	Olafur

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 20 May 2013 13:39:29 -0400

On Monday, May 20, 2013 01:01:06 PM Olafur Gudmundsson wrote:
> This is the right way to build fallback in DNS given the installed base, ask
> for one type and if it is not available try the fallback.
> If your client can do queries in parallel then you can do both and use one
> of the ones that returns an positive answer, but in that case there is a
> race condition if both return and the two types differ, that  can be
> addressed by preferred type rule. Mark's suggestion is SPF is the preferred
> type, and TXT is compatibility type in sequence. You may disagree with that
> order ….

There's no value proposition for receivers.  We've been through all this in 
the working group before, but I'll give at least a summary ...

In RFC 4408 there is an interoperability defect.  Senders can publish TXT or 
SPF types and meet the requirements in 4408.  Receivers can publish TXT or SPF 
and similarly meet the requirements of 4408.  We concluded that to be 
interoperable there had to be a common format that senders MUST publish and 
receivers MUST check, otherwise they could end up completely failing to 
communicate.

We considered to which type to make the MUST format and concluded that given 
the overwhelming deployed based for TXT only versus SPF only (publishers that 
publish both SPF and TXT can be ignored for this analysis), making TXT the 
mandatory format was the only thing that made sense.

Given that, checking type TXT first is the only sensible approach.  Then you 
have to consider the value proposition of a follow-on type SPF check.  Once 
the TXT lookup is complete, you have ~98% of the SPF records you are going to 
get.  Offsetting the value of trying to get those few additional records are 
not only the costs of doing the queries (virtually none of which will return 
useful information), but also the delays caused by DNS servers that timeout 
for TYpe SPF (but not TXT) that can slow down busy mail streams.

As a receiver, I don't see the value in getting those last few records that's 
worth not only the added resources, but the additional complexity in code.  I 
get the protocol purity argument, but in the real world, I don't think people 
care.

Scott K

---

From: Dave Crocker <[email-masked]>
Date: Mon, 20 May 2013 10:49:46 -0700

> from the DNS when present, thus has skewed the measurements used to justify the removal SPF type.

This is the essential problem with trying to re-argue a topic that was 
already discussed extensively and exhaustively and was fully resolved 
very, very long ago:  it imposes the burden on defenders of the decision 
to be just as thorough as they/we were long ago.  Forgive me if I think 
that that is placing the cost on the wrong people.

It also causes challengers to think that they are being reasonable in 
re-raising an old topic and claiming that it was a bad decision.

So here's what it normally considered reasonable in the IETF:

    Working Group Guidelines and Procedures
    RFC 2418:

      It is occasionally appropriate to revisit a topic, to re-evaluate
    alternatives or to improve the group's understanding of a relevant
    decision.  However, unnecessary repeated discussions on issues can be
    avoided if the Chair makes sure that the main arguments in the
    discussion (and the outcome) are summarized and archived after a
    discussion has come to conclusion.

We've got a lengthy archive on this topic.  This particular item was not 
an easy or quick decision.  The odds that there's some key point that 
was missed are quite small.

So what I'd appreciate is for folks who want to challenge or criticize 
an existing working group decision to do their f'ing homework with the 
archive and point to specific errors in the sequence or its content.

Please do not put a burden on the working group until that homework has 
been done.

But of course, that's just my personal preference.  No doubt, I'm the 
one being unreasonable.

d/
-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Scott Kitterman <[email-masked]>
Date: Mon, 20 May 2013 13:50:41 -0400

On Monday, May 20, 2013 01:23:19 PM Olafur Gudmundsson wrote:
> >> Mark's suggestion is SPF is the preferred type, and TXT is compatibility
> >> type in sequence. You may disagree with that order �.
> > 
> I was trying to point out that Mark's proposal is sound and the only point
> of contention is the possible order of queries, without saying anything
> about what my position is.
> 
> I do not have a bone in this fight other than the FUD that some people spew
> out about DNS and DNS software.
> 
> It was turned off because I turned it off after it caused problems (Stuart
> and I are co-maintainers of pyspf).  This was several years ago, so either
> I misremember discussing the change with him when I did it or he's
> forgotten the discussion.
> 
> Until it's reliable enough to use, it shouldn't be used and, of course, no
> one will care if it's reliable if it's not used.
> 
> Which can be taken to mean he is partially responsible for the difficulties
> that some people have in getting SPF records from the DNS when present,
> thus has skewed the measurements used to justify the removal SPF type.

I turned it off because it wasn't reliable.  It's still not reliable.  As a 
software developer I've got no obligation to deliver inferior performance to 
users because someone has a theory it might some day be useful.

Scott K

---

From: Dotzero <[email-masked]>
Date: Mon, 20 May 2013 14:02:16 -0400

On Mon, May 20, 2013 at 1:49 PM, Dave Crocker <[email-masked]> wrote:
>> Which can be taken to mean he is partially responsible for the
>> difficulties that some people have in getting SPF records
> already discussed extensively and exhaustively and was fully resolved very,
> very long ago:  it imposes the burden on defenders of the decision to be
> just as thorough as they/we were long ago.  Forgive me if I think that that
> is placing the cost on the wrong people.
>
> We've got a lengthy archive on this topic.  This particular item was not an
> easy or quick decision.  The odds that there's some key point that was
> missed are quite small.
>
> So what I'd appreciate is for folks who want to challenge or criticize an
> existing working group decision to do their f'ing homework with the archive
> and point to specific errors in the sequence or its content.
>
> Please do not put a burden on the working group until that homework has been
> done.
>
> But of course, that's just my personal preference.  No doubt, I'm the one
> being unreasonable.
>
> d/
> --
> Dave Crocker

While there are times I have found Dave to be unreasonable, this is
not one of those times. As he points out, this has been discussed
extensively and exhaustuvely. Unless there is something truly new
being brought to the table or a glaring error in the previous
discussions is pointed out (other than "I don't like the prior
decision of the group" then I personally think additional rehashing of
the same discussions does not advance the WG in a meaningful manner.

We made a design error in RFC4408. I was ambivalent about potential
resolution when this first came up as a discussion. I was ultimately
swayed that dropping type-99 makes the best sense  from a practical
standpoint given where things stand today. Like Dave, I appreciate the
desire for a "pure" solution but I truly believe that from a practical
and operational perspective, the approach the WG came up with makes
the most sense.

Mike

Mike

---

From: Mark Andrews <[email-masked]>
Date: Tue, 21 May 2013 08:02:52 +1000

In message <[email-masked]>, Olafur Gudmundsson writes:
Input error
> This is the right way to build fallback in DNS given the installed base,
> ask for one type and if it is not available
> try the fallback.
> If your client can do queries in parallel then you can do both and use
> one of the ones that returns an positive answer, but in that case

If two type are returned and they are different you are in transition.
You pick one.  It does not matter which because there is no way to
tell which is the newest.  If you want a explict order then ignore
the TXT record.  SPF records don't hace a sequence number and TTL
doesn't help.  This is no different to querying two or more recusive
servers for TXT records after the zone has been updated and getting
different results.  You cannot tell which of the results is current.

Note nameserver vendors are free to add code to keep SPF and TXT
records in sync already.  The simplest would be to document that
if both SPF and TXT types are present and differ then the TXT record
will be updated to match the SPF record.  Alternatively they could
add a "xorig=<type>" and the code would use its presence to select
which record is the orginal and which is the copy.  Both ideas are
permisible by RFC 4408.

It took me a couple of minutes to add such code to dnssec-signzone
the other day to do "xorig=<type>".  Adding it to the update and
the post zone loading codes in named would not take much longer.
Having a RFC that said it MAY/SHOULD be done would make it easier
for anyone where the web front end still only supports TXT records.

> type in sequence.

---

From: Mark Andrews <[email-masked]>
Date: Tue, 21 May 2013 08:23:03 +1000

In message <2559081.7IgQ8Mc0EM@scott-latitude-e6320>, Scott Kitterman writes:
> one
> > order .
> There's no value proposition for receivers.  We've been through all this 
> in the working group before, but I'll give at least a summary ...
> 
> In RFC 4408 there is an interoperability defect.  Senders can publish TXT
> or SPF types and meet the requirements in 4408.  Receivers can publish
> TXT or SPF and similarly meet the requirements of 4408.  We concluded
> that to be interoperable there had to be a common format that senders
> MUST publish and receivers MUST check, otherwise they could end up
> completely failing to communicate.

And the SHOULD publish both records could be encoded into nameservers
but unless you ask they are unlikely to do it.

Note I do not believe this is a real defect.  The final state should
be SPF queries and records only which RFC 4408 permitted the same
as it permitted TXT queries and records only to make the pre RFC
4408 situation compatible with RFC 4408.

> We considered to which type to make the MUST format and concluded that
> given the overwhelming deployed based for TXT only versus SPF only
> (publishers that publish both SPF and TXT can be ignored for this
> analysis), making TXT the mandatory format was the only thing that made
> sense.

And the number SPF record has gone from 4% to 8% in the last two years
and will only continue to increase.
 
> Given that, checking type TXT first is the only sensible approach.

No it isn't the only sensible approach.  There wouldn't be this arguement
if it was.

> Then
> you have to consider the value proposition of a follow-on type SPF check.
> Once the TXT lookup is complete, you have ~98% of the SPF records you
> are going to get.  Offsetting the value of trying to get those few
> additional records are not only the costs of doing the queries (virtually
> none of which will return useful information), but also the delays caused
> by DNS servers that timeout for TYpe SPF (but not TXT) that can slow down
> busy mail streams.
>
> As a receiver, I don't see the value in getting those last few records
> that's worth not only the added resources, but the additional complexity
> in code.  I get the protocol purity argument, but in the real world, I
> don't think people care.

If you look as SPF in isolation you come to that conculsion if you
look with a broader perspective you come to different conclusions.

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 20 May 2013 19:02:51 -0400

Chair hat on.

> look with a broader perspective you come to different conclusions.

As Dave Crocker pointed out elsewhere in this thread, there was
already a considerable number of electrons spilled on this topic.
The above claim, however, appears to be hinting at a new argument.
Unfortunately, instead of actually making the argument, it merely
hints that there is one.  If this is the one about polluting TXT
space, I think you will find that argument _was_ considered in the
previous discussions.  I urge those with a temptation to weigh in on
this topic to please review the history of the topic in the WG before
saying more.

If that's _not_ the argument, then what is the argument, please?

Best,

A

-- 
Andrew Sullivan
[email-masked]