Subject: Re: [Dbound] multiple PSL-equivalents exist (was: algorithmic background material for BoF

From: Jothan Frakes <[email-masked]>
Date: Mon, 17 Nov 2014 08:56:07 -0800

As Jeremy pointed out, the IANA TLD list is the main "PSL".

 I have raised this point before about the Mozilla maintained PSL being one
of the "Suffix Lists", the more notorious of them being the IANA PSL.

It gets down to splitting hairs over one's definition of "public".  Does
public mean "publicly available string resource", or does it mean "a list
of strings that are publicly accessible"?  If the former, then the IANA TLD
list is a PSL.

Even if the definition is the former, it does still qualify, but as a
subset of larger more detailed lists.  The IANA TLD list is really somewhat
of patient zero upon which most other lists are derived.

There is a demograph mix of use on these public suffix lists beyond
the browsers and certificate authorities.

Purely anecdotally, there is a '"TLD" or not' logic that developers and
integrators have created over the years, which the IANA PSL
gets incorporated into due to the efficiencies of not making network calls
(and having to depend upon them) to validate.

Later, as they notice there are strings that need more elegance to that
"TLD or not" logic, there is a branch out to either an internal proprietary
list or the list at publicsuffix.org once that developer or integrator
comes to the realization that it is necessary to also solve for the stub
zones at registries such as co.uk or com.au.

Historically, there have been other public lists, but many have fallen away
over time or have gone proprietary / private domain.

Yes, browsers use these, and yes, certificate authorities use these, and
those are slightly different use cases.  There are more.

We will see some more examples of use cases as we proceed, but I encourage
that the group wisdom on this dbound effort might keep in mind the
overarching use case of "TLD or not" logic, and benefit from the elements
of whatever PSL that can be leveraged and carried forward.

I have some asks out to some of the integrators and developers who have
reached out to me over the course of the past 8 or so years of my volunteer
time with Mozilla about their PSL effort to also describe use cases.

-jothan

On Monday, November 17, 2014, Casey Deccio <[email-masked]> wrote:

> On Mon, Nov 17, 2014 at 10:01 AM, Jeremy.Rowley <
> [email-masked]
> <javascript:_e(%7B%7D,'cvml',[email-masked]');>> wrote:
>
>>  There's also this one maintained by IANA:
>>
>> http://data.iana.org/TLD/tlds-alpha-by-domain.txt
>>
>
> Thanks, Jeremy.  I note that this particular listing doesn't tell us
> anything more than what's in the root zone (it's exactly the list of TLDs
> delegated in the root zone), so I might hesitate to call it PSL-like in
> that sense.
>
> Best regards,
> Casey
>

-- 

Jothan Frakes
Tel: +1.206-355-0230

---

From: Casey Deccio <[email-masked]>
Date: Mon, 17 Nov 2014 12:49:52 -0500

Hi Jothan,

On Mon, Nov 17, 2014 at 11:56 AM, Jothan Frakes <[email-masked]> wrote:

>  I have raised this point before about the Mozilla maintained PSL being
> one of the "Suffix Lists", the more notorious of them being the IANA PSL.
>

The IANA list of TLDs is just that--a complete list of TLDs delegated from
the IANA root.  It can be generated from the root zone with the following:

awk '$4 == "NS" { print $1 }' root.zone | sed 's/\.$//' | sort -u > tlds.txt

If folks are using that list, it tells me several things:

1. They are interested in only top-level domains.
2. Unless their definition of "public" is limited to TLDs, then they are
getting an incomplete list.
3. Unless their definition of "public" considers that some TLDs might not
actually be public (not just speaking about HTTP cookies), their list might
include too much (the Mozilla PSL suffers from this as well).
4. This list can be derived from the root zone, either completely (e.g., by
the command above) or selectively (e.g., by issuing simple one-label
queries to the root servers).

That doesn't mean the list isn't useful to some, but simply that I don't
see it adding anything new to the root zone, either in terms of maintenance
or accessibility.  However, I'm interested to learn if I've missed
something and/or if folks are in fact using this list as some sort of PSL
equivalent/replacement.

> of patient zero upon which most other lists are derived.

Let me be more clear about what I am asking.  DBOUND is working on a
problem statement, which, in part, should reflect demand and use cases for
learning relationships (or boundaries) between different domains--if it is
to be useful.  As there is some precedent, embodied in the Mozilla Public
Suffix List (PSL), it is useful to understand PSL use cases--not as the
only problem or solution, but as concrete examples of real use, both in
browsers and other applications.  The Mozilla PSL is not merely a list of
TLDs extracted from the root zone, but maintains nearly 7,000 entries,
delineating a "public" (for some definition of public) boundary in the DNS
namespace.  While it can likely be agreed that it is not a perfect or
comprehensive solution, nor does it address the general problem of domain
relationships--only boundaries--it is well used, and represents a
non-trivial compilation and effort.

Now my question: it has been suggested that there are other lists being
maintained and consumed in *like* manner.  Can some lists and use cases be
positively identified?

> the browsers and certificate authorities.

That's what we're looking to understand.

> (and having to depend upon them) to validate.

TLD (i.e., existing single-label DNS name) or not can be done online or
offline, using only the root zone as the ultimate source.

> ...
>
> Historically, there have been other public lists, but many have fallen
> away over time or have gone proprietary / private domain.
>

Lists and use cases - that's what we're looking for.

> ...

Thanks!

Best regards,
Casey

---

From: "John Levine" <[email-masked]>
Date: 17 Nov 2014 19:04:47 -0000

>It gets down to splitting hairs over one's definition of "public". ...

At our meeting in Honolulu, it seemed pretty clear to me that the
distinction between public and private isn't useful here.  What is
useful is the boundaries between one authority (for a suitable and
probably painful to tease out definition of "authority") and another.

R's,
John

---

From: Casey Deccio <[email-masked]>
Date: Mon, 17 Nov 2014 14:27:42 -0500

Hi John,

> probably painful to tease out definition of "authority") and another.
I realize that there were different opinions on this, but that the
distinction is not useful was not clear to me at all in the meeting.
Whether or not public and private are well defined and whether or not they
are even the right terms for use in that particular boundary might be in
question (both were raised during the side meeting in Honolulu).  But I
disagree on your assessment of the utility of the boundary, and dismissing
them early on as part of the background and problem statement is
premature.  As far as I have observed, no one considers such boundaries or
the related solutions (e.g., the PSL) comprehensive, but it obviously has
been useful in solving some subset of the problem with existing means.

That being said, I had proposed that the problem of identifying domain
relationships and boundaries was broken up into several areas, one of which
was "cross-scope" (i.e., public/private), which I believe is the part
directly addressed by the Mozilla PSL.  Independent of whether the public
boundaries are drawn accurately by the Mozilla (and I'm not suggesting one
way or the other), I am curious what the demand for identifing cross-scope
vs. other relationships is in application use.  I've attached a problem
breakdown treemap, where aggregate area represents demand/use cases.  In my
opinion, making the problem demand tree map more proportional to the demand
and use cases can really help us understand what we're trying to solve.  We
have lots of conjecture (including my own), but we need real data points.
Otherwise we will get caught up in either justifying proposed solutions or
arguing philosophy.

Best regards,
Casey

---

From: "John R Levine" <[email-masked]>
Date: 17 Nov 2014 16:50:35 -0500

> was "cross-scope" ...

Honestly, I saw your chart, but I didn't understand it.

Every application I've seen for the PSL asks one of these three questions:

* Are names A and B under the same authority? (the cookie question)

* Are all the names covered by this wildcard under the same authority? (the cert question)

* What's the shortest* name under the same authority as this one? (the DMARC question)

None of those answers seem to me to make any sort of public/private 
distinction.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail.

* - shortest by number of components, e.g., foo.org is shorter than bar.foo.org

---

From: Jothan Frakes <[email-masked]>
Date: Mon, 17 Nov 2014 18:03:38 -0800

Hi Casey-

> the IANA root.  It can be generated from the root zone with the following:
Right, to us it does.  We live and breathe domain names - I am saying that
to a developer breaking ground on solving something like bozo filtering
invalid domains would use something in a caveman simple manner.  A
hypothetical of this would be using it to filter the rightmost element of
the FQDN in form validation of an email field, or URL entry field to solve
a form validation problem.    Example:   user enters [email-masked] ->
validation fails that because xh wasn't in IANA PSL

Often, a developer is working to solve tactical problems, like in rapid
development or scrum, these are chopped up into stories.  Often, they are
going to look for the fastest way through the maze to the cheese on solving
a tactical need.  This is just like a coder in the 70's or 80's not knowing
better may have used a two digit year field before y2k, or another example
I use frequently is where a coder used a numeric integer field or
constrained numeric only 5 character fields for zip codes and then
face-palmed when they realized much of the rest of the planet uses alpha
numeric six characters for postal codes.

When the coder sees they needed to do something with a little more finesse,
they then do it, but it takes awareness of it.  We have that awareness -
not everyone does.

> awk '$4 == "NS" { print $1 }' root.zone | sed 's/\.$//' | sort -u >
Thank you for that shell scripting lesson, which I will take as intended as
offered in sincere kindness without awareness my background with your
employer.    :)

> 1. They are interested in only top-level domains.

Well I also find that when I am asked about an approach that helps solve a
coding challenge, often they don't know what they don't know about what
nuances are present underneath ... which is also common.
Like legacy RFC 1480 _stuff_ under .US

or the RFC 6761 .TEST etc _stuff_ that would still be invalid but not be
present in any PSL while .ARPA might

And then drafts like this happen here in IETF:
http://tools.ietf.org/html/draft-grothoff-iesg-special-use-p2p-names-02
(since expired)

> getting an incomplete list.

Amen, but still that quick-pass invalid filter hypothetical would still
function for their needs

> include too much (the Mozilla PSL suffers from this as well).

I agree and that is an important point to not just be speaking about
cookies, which has a distinct need from that example I just gave

> 4. This list can be derived from the root zone, either completely (e.g.,
> by the command above) or selectively (e.g., by issuing simple one-label

No doubt about that.

> something and/or if folks are in fact using this list as some sort of PSL

Right, so what happens is that people start with the IANA TLD List "PSL" or
the awk sed sort root zone as a starting point.

> maintained and consumed in *like* manner.  Can some lists and use cases be

http://en.wikipedia.org/wiki/List_of_Internet_top-level_domains

Looking for a few - many / most are now in the realm of internal /
proprietary and are not shared, the others like these examples have fallen
into atrophy, though on Greasemonkey you can see the wiki history:
http://www.neuhaus.com/domaincheck/domain_list.htm
http://www.tld-resource.com/tldinfo.php
http://wiki.greasespot.net/index.php?title=Magic_TLD

>> time with Mozilla about their PSL effort to also describe use cases.
There is a section from a support document in another group you and I are
participating in that I put some effort into accumulating, I will get that
together with some responses from folks once they come in from LEA,
Developers, Whois services and others and do it as a fresh email so its not
buried in a thread.  Will take a few weeks to put together.

-J

---

From: Casey Deccio <[email-masked]>
Date: Tue, 18 Nov 2014 10:16:13 -0500

Hi John,

> Honestly, I saw your chart, but I didn't understand it.

The tree map represents a breakdown of comparison characteristics for
domain names, for the purposes of determining whether the two can be
positively associated (i.e., for some application).  Think of it like a
decision tree, where the area of each subdivision in the map represents the
likelihood that the branch will be followed.  The tree looks something like
this (note that this is considered a work-in-progress and not a definitive
model):

cross scope?
     |\
     |  yes -> "no positive association, by definition of public"
    no
(intrascope)
     |
ancestrally related (i.e., one domain is an ancestor of the other)?
     | \
     |   yes
     |     \
     |       \   public-in-public
     |                      |\
     |                      |  yes -> ??
    no                   no
(cross-domain)  (private-in-private)
     |                      |
   ??                    ??

Why is it useful to understand this breakdown to understand the problem?
Because in many cases it might be *sufficient* to simply know if there is a
boundary and if the two names are on different sides.  In other cases
(either because of the nature of the names or the application) it is
important to more fully understand the relationships in their entirety.
The Mozilla PSL allows a client to answer (only) the very top question.

An analogy that might help is setting up firewall rules for machines on a
campus.  Rules can be set on the border of the campus border or on
individual hosts.  Suppose 90% of the machines on campus require no
inbound-initiated network connectivity.  What would happen if the campus
didn't distinguish between the 90% and the 10%, but rather allowed all
clients to handle their own firewall configurations?  Certainly it could
work, but there are concerns about consistency and complexity.  When you
have one primary rule to weed out the majority of the issues (in this case,
blocking *all* inbound traffic for the 90% of hosts), that simplifies the
problem space.  For finer-grained tuning, the 10% are configured with
policy at their respective hosts.

Suppose we learn from application use case and demand that learning a
boundary is indeed sufficient for some high number of cases.  To me it
would behoove the group to consider that as part of the solution(s).
Additional considerations are the portability and scalability
characteristics of this solution(s), the flexibility of the solution, the
comparison of lookup time with freshness time (i.e., comparing the current
Mozilla PSL with a more dynamic, e.g., DNS-based, approach), and others.

> Every application I've seen for the PSL asks one of these three questions:

Note that these are application-specific questions under the more general
question of domain relationships.

> * Are names A and B under the same authority? (the cookie question)

I would think the question here is: is A allowed to set cookies for B (or
vice-versa)?  One way of solving that is asking whether A and B are under
the same authority, but then how do you define authority?  For example,
foo.uk.com resides within the uk.com zone.  The Mozilla PSL tells us that
uk.com is "public" and foo.uk.com is not.  Does that mean they are under
the same authority?  However, the Mozilla PSL (and, in this case, the
decision tree above) tell us that there is a public/private boundary
between foo.com and foo.uk.com, so they are not "positively associated"
(i.e., for policy).

> * Are all the names covered by this wildcard under the same authority?
(the cert question)

This is an example of a (good) question that cannot be fully answered by
simply answering the top question in the decision tree.  While a
relationship can be ruled out if the name resides within a first-level
public boundary, more fine-grained control is necessary to understand the
problem at hand.

> * What's the shortest* name under the same authority as this one? (the
DMARC question)

It seemed to me that the DMARC question was also about general
relationships between domains, and determining the shortest name under some
authority was more about applying the current PSL to the problem space.

distinction.

But all of these can make *use* of a known public boundary, if one exists.
We know that from application (sometimes over-application) of the Mozilla
PSL.

Best regards,
Casey

---

From: Casey Deccio <[email-masked]>
Date: Tue, 18 Nov 2014 12:31:52 -0500

On Tue, Nov 18, 2014 at 12:05 PM, Kurt Andersen <[email-masked]> wrote:

>
> On Tue, Nov 18, 2014 at 7:16 AM, Casey Deccio <[email-masked]> wrote:
>>
> No, the PSL was used as a workable short term strategy to accomplish
> something much like your campus firewall approach: look for a policy on the
> actual from domain and if not found, fall back to the "campus" policy (the
> "organizational domain" in DMARC terms). The "shortest name" was adopted
> from an ancestral descent model.
>

I think that is what I was trying to say: lacking a full-blown mechanism
for specifying relationships between arbitrary domains, the PSL was applied
to the problem, identifying the highest ancestor below the public boundary
as the organizational domain.  Does that sound more accurate?

I am not suggesting that using the PSL was the optimal approach, only that
it seemed to me that the terminology used reflected the use of the PSL as
the interim approach.

Best regards,
Casey

---

From: Kurt Andersen <[email-masked]>
Date: Tue, 18 Nov 2014 09:05:02 -0800

> policy at their respective hosts.

<elided>

No, the PSL was used as a workable short term strategy to accomplish
something much like your campus firewall approach: look for a policy on the
actual from domain and if not found, fall back to the "campus" policy (the
"organizational domain" in DMARC terms). The "shortest name" was adopted
from an ancestral descent model.  Except for the lookup overhead, SOPA
offered something closer to what DMARC would really like to have available.

--Kurt Andersen

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 24 Nov 2014 14:53:09 -0800

> None of those answers seem to me to make any sort of public/private

Aren't the first and third questions really the same one?  Or am I showing
a lot of email bias by asking that?

-MSK

---

From: "John R Levine" <[email-masked]>
Date: 24 Nov 2014 17:57:22 -0500

> a lot of email bias by asking that?

Only if you rule out cross tree links, e.g. youtube.com is under the same 
authority as google.com.  Someone (Kurt?) said that it would be useful to 
do that for DMARC, although the current PSL implementation can't.

Regards,