Subject: [weirds] one protocol to rule them all

From: Andy Newton <[email-masked]>
Date: Tue, 14 Feb 2012 19:50:05 -0500

As the charter discussion ebbs and flows between "numbers now, names later" and "numbers now, names now", I couldn't help but get all teary eyed thinking of the days when we use to do technical work in the IETF. Ah, those were the days!

So I thought I'd spend a couple of minutes trying to answer the question of just what a names protocol would look like using the same approach that the RIRs have put forward in rir-query and rir-json. The question of just how compatible the efforts would be would seem to rest on just how amenable the names proponents are to following the example and direction set by the RIRs. Sorry to be blunt about that, but I'm too tired to find a more diplomatic way of putting it.

Using Steve Sheng's and Franciso Aria's dnrd draft as a basis, the following queries would be removed:
  http://whois.test/conact/CID-4005/ and  http://whois.test/host/ns1.example.test/

Internet users lookup domain names. They have no idea which registry to contact if they saw a naked contact ID or NS name in the wild. rir-query takes the same approach. It only supports queries for information that users would find in other contexts.

But relative queries would be supported, such as
  http://whois.test/example.test/operator and http://whois.test/example.test/operator/contacts/abuse

Similarly, the response(s) would have to align with the query structure. The following is an example of a JSON response for http://whois.test/example.test/  (where "entity" and "contacts" would follow the entity object from rir-json):

{
    "domain": {
        "handle": "9690-TEST",
        "name": "example.com",
        "status": [
            "clientHold",
            "clientRenewProhibited",
            "clientUpdateProhibited"
        ],
        "registrationDate": "1992-07-26T09:10:56Z",
        "expirationDate": "2019-01-21T10:11:18Z",
        "remarks": [
            "she sells seas shells",
            "down by the seashore"
        ],
        "uris": [
            {
                "type": "source",
                "uri": "http://whois.registry.example/domain/xxxx"
            },
            … 
        ],
        "nameServers": [
            "ns1.example.com",
            "ns2.example.com"
        ],
        "delegationKeys": [{
            "algorithm": 7,
            "digest": "E68C017BD813B9AE2F4DD28E61AD014F859ED44C",
            "digestType": 1,
            "keyTag": 53814
        }]
    },
    "registrant": {
        "entity": { … },
        "contacts": {
            "tech": [ … ],
            "admin": [ … ]
        }
    }
}

And that's basically it. Seems fairly straight forward to me.

Thoughts? 
Would a names proponent be willing to write this up more formally? 
Would anybody be willing to prototype it?

-andy

---

From: Francisco Obispo <[email-masked]>
Date: Tue, 14 Feb 2012 16:55:23 -0800

I personally think JSON is the wrong approach for this protocol.

It lacks support for namespaces, (although some do it by adding a key with the namespace). And although it is generally easy to parse, it does not have the automatic formatting options (XSL, Stylesheets) that XML does.

I would much rather see JSON as an optional requirement, but instead use XML as the main protocol encoding option.

Francisco

On Feb 14, 2012, at 4:50 PM, Andy Newton wrote:

> _______________________________________________
> weirds mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/weirds

---

From: "John Levine" <[email-masked]>
Date: 15 Feb 2012 01:48:51 -0000

>It lacks support for namespaces, (although some do it by adding a key with the
>namespace). And although it is generally easy to parse, it does not have the
>automatic formatting options (XSL, Stylesheets) that XML does.

Are we designing a computer to computer data retrieval protocol, or a
computer to human display system?  All of the WHOIS problems I need to
solve are computer to computer, with any display to people happening
several layers up.  My key needs are that my client can mechanically
create the query to ask for the information it needs, and mechanically
parse the answer it gets back.  JSON and XML can both do that, but
JSON is somewhat easier.

If you want to build a proxy to reformat the answers and make them
beautiful, that's a project you can do, but not one that I would want
to put in the path with other stuff.

Also, I don't understand the desire for name spaces unless you
anticipate clients that forget what they asked for between the time
they send the request and the time they get the answer back.  Can you
give some examples where it would be a problem?

R's,
John

---

From: Andy Newton <[email-masked]>
Date: Tue, 14 Feb 2012 21:16:13 -0500

> parse the answer it gets back.  JSON and XML can both do that, but

JSON is easier and far more common in REST. And apps that run in web browsers have an easier time understanding it, oddly enough.

I should also point out that there is nothing in the proposal or in the rir-query, rir-json drafts that precludes a registry from serving up XML or even HTML. If a user puts one of those URLs into a browser, it would be perfectly acceptable to respond with something that looks pretty to humans. But as John points out computer-to-human and computer-to-computer are two different problems.

> give some examples where it would be a problem?

There is, btw, a JSON namespaces draft out there somewhere, along with a JSON schema draft. I've referenced neither by design.

-andy

---

From: Francisco Obispo <[email-masked]>
Date: Tue, 14 Feb 2012 18:32:54 -0800

XML is computer-to-computer, but it happens to have wide support and adoption. It might not be as trivial to process as JSON, although, depending on the use and the language, sometimes its just as easy.

IMHO namespaces will be provided to identify the various 'objects' to be represented within the response format. This will allow for extensibility in a well-documented way, leaving off the 'guess' work.

XML is already used in EPP which is used in both RIRs and in domain registries, and browsers fully support them without JS support or external plugins.

XML has mechanisms not only to parse, but also to 'validate' its content. This feature is currently unavailable in JSON, where all the validation would have to be done by the programmer reading the content.

XML represents a Data-exchange format, while JSON is more of a computer representation of a data structure.

JSON is assumed to be encoded in UTF-8, while XML allows you to specify the encoding.

Don't get me wrong, I'm a big fan of JSON, but I also believe that we should use the right tool for the job.

regards,

Francisco

---

From: Byron Ellacott <[email-masked]>
Date: Wed, 15 Feb 2012 17:11:41 +1000

> Would anybody be willing to prototype it?

https://github.com/Byron-at-APNIC/Weirds-Experimental-Implementation-for-Redirecting-Data-Output

-- 
bje

---

From: Andy Newton <[email-masked]>
Date: Wed, 15 Feb 2012 05:07:14 -0500

> https://github.com/Byron-at-APNIC/Weirds-Experimental-Implementation-for-Redirecting-Data-Output

django.wsgi: sys.path.append('/usr/local/django/weirdo')

I guess we should have seen that coming. :)

We have a spec and we have an implementation. All we need now is an RFC number.

-andy

---

From: Dave Piscitello <[email-masked]>
Date: Wed, 15 Feb 2012 06:05:08 -0800

If I understand Andy correctly, this was a JSON example.

I don't read anything in his mail suggesting that JSON is on the table and XML is off. What is required and what (how many) options is a matter for the working group to decide. However, I take your point that if the working group chose JSON, some issues are created for namespaces.

Personally, I think that choice is good. I'd like a client to be able to ask for JSON, XML, other metalanguages, text and the provider can decide what to serve up.  Granted, normalization/transformation work would be necessary, but providing what the user wants sounds right to me.

---

From: Carlos Martinez-Cagnazzo <[email-masked]>
Date: Wed, 15 Feb 2012 12:21:55 -0200

A +1 for supporting multiple output formats, at least JSON and XML 
should be on the plate. Our reference implementation already does both 
of them plus human-oriented HTML as well, based on HTTP headers. ARIN's 
also supports DOS-like extensions like (.xml, .json). This is a point 
that needs standardizing (one can do both, it's not much of an effort)

In fact the data model agreed should IMO be format-agnostic. I am not 
sure what would be the best way to write down such a spec, maybe we can 
spec an XML format and then have a JSON model that follows it.

regards

Carlos

---

From: Andy Newton <[email-masked]>
Date: Wed, 15 Feb 2012 09:25:00 -0500

> If I understand Andy correctly, this was a JSON example.

Actually, it is a proposal by example.

> I don't read anything in his mail suggesting that JSON is on the table and XML is off. What is required and what (how many) options is a matter for the working group to decide. However, I take your…

Not that XML is off the table, but the suggestion to use JSON is sincere.

With regards to namespaces, if the names solution requires an 
interleaving multi-verse of namespaces, then it is a much different 
problem than the numbers solutions and therefore might warrant a 
separate protocol. (btw, I don't think the names solution requires an 
interleaving multi-verse of namespaces.)

Modulo nit-picking of element names, etc... would the approach I have 
just outlined work for the names solution?

-andy

---

From: Gavin Brown <[email-masked]>
Date: Wed, 15 Feb 2012 14:43:33 +0000

> interleaving multi-verse of namespaces.)

I can see how it might, if we want to reuse RFC 5731-5733. Having a data
model that's common to both EPP and the new protocol is a clear advantage.

EPP's data model isn't sufficient here, but can be complemented by using
an additional namespace. For example, a query for a domain name might
return the equivalent of an EPP <domain:info> response, but with
namespaced attributes containing hyperlinks for contact, registrar and
host objects, eg:

<domain:registrant
  weirds:href="https://whois.example.com/rest/contact/jd1234
    jd1234
</domain:registrant>

<domain:hostObj
  weirds:href="https://whois.example.com/rest/host/ns1.example.com"
    ns1.example.com
</domain:hostObj>

<domain:host
  weirds:href="https://whois.example.com/rest/host/ns1.example.com"
    ns1.example.com
</domain:host>

<domain:clID
  weirds:href="https://whois.example.com/rest/registrar/ClientX"
    ClientX
</domain:clID>

G.

-- 
Gavin Brown
Chief Technology Officer
CentralNic Ltd
Innovative, Reliable and Flexible Registry Services
for ccTLD, gTLD and private domain name registries
https://www.centralnic.com/

CentralNic Ltd is a company registered in England and Wales with company
number 4985780. Registered Offices: 35-39 Moorgate, London, EC2R 6AR.

---

From: Andy Newton <[email-masked]>
Date: Wed, 15 Feb 2012 10:47:09 -0500

> model that's common to both EPP and the new protocol is a clear advantage.

It isn't clear to me. Not all domain registries use EPP. Same for the 
RIRs. As a user of the Internet, needing to understand EPP to do a whois 
query seems obnoxious. EPP is relevant in the context of 
registry-registrar transactions and is not needed to query for meta-data 
about a domain name.

If EPP is necessary for the names solutions, then it is a much different 
beast than the numbers solution.

-andy

---

From: Francisco Obispo <[email-masked]>
Date: Wed, 15 Feb 2012 07:56:34 -0800

On Feb 15, 2012, at 7:47 AM, Andy Newton wrote:

> It isn't clear to me. Not all domain registries use EPP. Same for the RIRs. As a user of the Internet, needing to understand EPP to do a whois query seems obnoxious. EPP is relevant in the context of registry-registrar transactions and is not needed to query for meta-data about a domain name.
> 

You don't need to understand EPP, the only thing that's necessary is for the client to understand how to parse it and display it, in which case, it would be no different than JSON, with the advantages described in my other email.

I believe what Gavin is suggesting is to re-use some of the data model proposed in EPP to represent whois data. 

We (ISC) are already working on a proposal that looks like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<whois xmlns="http://www.isc.org/xml/ns/whois-1.0">
  <response>
    <domain:object xmlns:domain="http//www.isc.org/xml/ns/domain-1.0">
      <domain:name>example.tld</domain:name>
      <domain:status s="ok"/>
      <domain:registrant>fo1234</domain:registrant>
      <domain:contact type="tech">fo1234</domain:contact>
      <domain:contact type="admin">fo1234</domain:contact>
      <domain:clID>registrar_a</domain:clID>
      <domain:crDate>2010-09-03T10:47:46.6Z</domain:crDate>
      <domain:exDate>2020-09-03T10:47:46.6Z</domain:exDate>
      <domain:upDate>2010-09-04T05:03:30.0Z</domain:upDate>
      <domain:ns>
        <domain:hostObj>ns1.example.net</domain:hostObj>
        <domain:hostObj>ns2.example.net</domain:hostObj>
      </domain:ns>
    </domain:object>
  </response>
  <additional>
    <contact:object xmlns:contact="http//www.isc.org/xml/ns/contact-1.0">
      <contact:id>fo1234</contact:id>
      <contact:status s="clientDeleteProhibited"/>
      <contact:postalInfo type="int">
        <contact:name>John Doe</contact:name>
        <contact:org>Example Inc.</contact:org>
        <contact:addr>
          <contact:street>123 Example Dr.</contact:street>
          <contact:street>Suite 100</contact:street>
          <contact:city>Dulles</contact:city>
          <contact:sp>VA</contact:sp>
          <contact:pc>20166-6503</contact:pc>
          <contact:cc>US</contact:cc>
        </contact:addr>
      </contact:postalInfo>
      <contact:voice x="1234">+1.7035555555</contact:voice>
      <contact:fax>+1.7035555556</contact:fax>
      <contact:email>[email-masked]</contact:email>
      <contact:clID>ClientY</contact:clID>
      <contact:crID>ClientX</contact:crID>
      <contact:crDate>1999-04-03T22:00:00.0Z</contact:crDate>
      <contact:upID>ClientX</contact:upID>
      <contact:upDate>1999-12-03T09:00:00.0Z</contact:upDate>
    </contact:object>
    <host:object xmlns:host="http//www.isc.org/xml/ns/host-1.0">
      <host:name>ns1.example.net</host:name>
      <host:status s="clientUpdateProhibited"/>
      <host:addr ip="v4">192.11.12.13</host:addr>
      <host:addr ip="v4">192.14.15.16</host:addr>
      <host:addr ip="v6">2001::A:B:C:1</host:addr>
    </host:object>
    <host:object xmlns:host="http//www.isc.org/xml/ns/host-1.0">
      <host:name>ns2.example.com</host:name>
      <host:status s="clientUpdateProhibited"/>
      <host:addr ip="v4">192.1.2.3</host:addr>
      <host:addr ip="v4">192.4.5.6</host:addr>
      <host:addr ip="v6">2001::C:D:E:1</host:addr>
    </host:object>
    <registrar:object xmlns:registrar="http//www.isc.org/xml/ns/registrar-1.0">
      <registrar:id>registrar_a</registrar:id>
      <registrar:name>Example Registrar Inc.</registrar:name>
    </registrar:object>
  </additional>
</whois>

As you can see, we are using a format that allows us to provide a response, and some additional objects as necessary to complement it.

Perhaps we could use this as a starting point?

Francisco

> If EPP is necessary for the names solutions, then it is a much different beast than the numbers solution.
> 
> -andy

---

From: Keith Gaughan <[email-masked]>
Date: Wed, 15 Feb 2012 15:59:20 +0000

On Wed, Feb 15, 2012 at 10:47:09AM -0500, Andy Newton wrote:

> It isn't clear to me. Not all domain registries use EPP. Same for
> the RIRs. As a user of the Internet, needing to understand EPP to do
> a whois query seems obnoxious. EPP is relevant in the context of
> registry-registrar transactions and is not needed to query for
> meta-data about a domain name.

It's hardly obnoxious, nor does it require an understanding of EPP. What Gavin's
suggesting is that it's worth reusing the schemata for the various EPP object
mappings, specifically those segments relating to responses to <obj:info>
requests. This is actually very reasonable and very sensible and not at all
difficult or complex to understand.

> If EPP is necessary for the names solutions, then it is a much
> different beast than the numbers solution.

To be clear, Gavin isn't suggesting EPP, but the reuse of a number of object
mappings for EPP. This is orthogonal to that discussion. RFC 5733, for instance,
has a very reasonable way marking up contact information, and is thus specific
not neither problem.

-- 
Keith Gaughan, Senior Developer
PGP/GPG key ID: 82AC3634
Blacknight Internet Solutions Ltd. <http://blacknight.com/>
12A Barrowside Business Park, Carlow, Ireland
Registered in Ireland, Company No.: 370845

---

From: Francisco Arias <[email-masked]>
Date: Wed, 15 Feb 2012 08:01:35 -0800

On 2/15/12 7:47 AM, "Andy Newton" <[email-masked]> wrote:

>query seems obnoxious.

Why?, it's just an XML schema as good or bas as any other.

For the common user the registry could add a CSS so it can have
human-friendly HTML. But as others have mentioned, that is likely outside
the protocol.

>EPP is relevant in the context of

EPP schemas are the low-hanging fruit to use in the data profile. I don't
feel anywhere near strong about using them, just noting they are there to
use if we wanted.

>beast than the numbers solution.

I miss to see the problem. The data schema/model is relevant in the
context of the data profile, not the base protocol. I don't see how this
affect the solution for the number registries.

__

Francisco.

---

From: Francisco Obispo <[email-masked]>
Date: Wed, 15 Feb 2012 08:02:09 -0800

As you can see from my previous email, our implementation uses those same identifiers as EPP, but does not reference the same XMLNS as EPP, this is due to the fact that some of them may not apply to a whois answer.

Francisco

---

From: "John Levine" <[email-masked]>
Date: 15 Feb 2012 16:03:47 -0000

In article <[email-masked]> you write:
>should be on the plate.

A -99 for them.  Unless I'm seriously misunderstanding something, the
goal for most of us here is to define a spec so that clients and
servers can interoperate when retrieving the kind of data that we get
via RIR WHOIS now, not a wish list for every possible bell and whistle
anyone might want.  The way you design a standard that works is to
constrain it to the smallest set of features that will get the job
done, which in this case is one query format, and one result format.

If you want to provide beautifully formatted web pages for clients
that don't ask for the standard result format (preferably JSON, but I
could live with XML), you're welcome to do so, but there's nothing to
interoperate there so it doesn't go in a spec.

I can't imagine the IESG accepting a plan for open-ended formats,
so I won't waste further time arguing against it.

R's,
John

---

From: Gavin Brown <[email-masked]>
Date: Wed, 15 Feb 2012 16:05:53 +0000

> RIRs. As a user of the Internet, needing to understand EPP to do a whois

I don't see why an end user would ever need to see the XML or JSON sent
between client and server. It had always seemed clear to me that the
client would render the server response into something more human
readable. Or do you browse the web in "view source" mode?

> registry-registrar transactions and is not needed to query for meta-data

EPP provides a data model for domain names, the only one I'm aware of
that has seen even moderate adoption. Reusing the <info> response
formats from RFC 5731-5733 seems sensible if we want to avoid
reinventing the wheel. It should at least be a starting point.

G.

---

From: Carlos Martinez-Cagnazzo <[email-masked]>
Date: Wed, 15 Feb 2012 14:22:49 -0200

My +1 was for the *possibility* of supporting multiple formats. I don't 
mean that every conceivable possibility has to fall in scope of the WG, 
and I certainly believe that human-oriented HTML should be definitely out.

At least one format (XML preferably, due to the possibility of 
validating document structure),  should be the authoritative one, and 
maybe the only one worth the time of this WG. JSON evidently has a 
strong following as well, so maybe it would also be a good idea to 
specify it.

I agree with you that not every bell and whistle has to be included, and 
that KISS usually helps.

But, as it stands now, every implementation will support multiple 
formats, so I think that yes, the *signalling* for indicating formats 
should be in scope.

Since it all points out that we will be using REST, well, HTTP already 
does that for us, so maybe this boils down to a single paragraph saying 
"responses will be in the format specified by the Accept: header or will 
default to X"

Carlos

On 2/15/12 2:03 PM, John Levine wrote:
> In article<[email-masked]>  you write:

---

From: Andy Newton <[email-masked]>
Date: Wed, 15 Feb 2012 11:44:25 -0500

On 02/15/2012 11:05 AM, Gavin Brown wrote:
> that has seen even moderate adoption. Reusing the<info>  response

Given it took me all of 15 minutes to re-invent the wheel spec-wise and 
Byron not much longer to implement what I specified, I can't see how 
involving EPP helps in any way. Why create a dependency when none is needed?

However my response seems to have uncovered a hidden requirement to use 
EPP from many names proponents, a requirement that departs from the 
approach taken from the rir drafts. It is also interesting to note that 
not a single response gave credence to my proposal. From this I cannot 
help but conclude that the names and numbers problems are separate and 
should be separate efforts.

-andy

---

From: Francisco Arias <[email-masked]>
Date: Wed, 15 Feb 2012 09:27:04 -0800

>should be separate efforts.

Hi Andy,

Feeling too fast with the gun today? :-)

I don't see such thing as a hidden requirement to use EPP. I for one,
doesn't believe it. We based our data model schema on EPP's because it was
there, but don't feel anywhere near strong about using it.

By the way I see no problem in using the URI structure you propose, in
principle. The URI structure used in our testbed was the simple one we
came up with, no particular reason to have it. I'm happy to discuss your
proposal.

For the data profile, I do believe names and numbers would have each their
own. But again, that's not in the base protocol, so no big deal.

__

Francisco.

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 15 Feb 2012 18:51:21 +0100

> reinventing the wheel. It should at least be a starting point.

Perhaps, EPP is much more than a starting point.  It seems to me that
if we modify RFC 5730 so as to allow *anonymous EPP*, it would be
possible to use it instead of whois right away --mission accomplished.

What am I missing?

---

From: Arturo Servin <[email-masked]>
Date: Wed, 15 Feb 2012 16:00:40 -0200

Then you may have 4 verbs:

<check> ,<info, <poll> ,<transfer>

	It is unclear to me why we need 4 (at least for numbers) when it seems that we can have a simpler approach:

http://tools.ietf.org/html/draft-newton-et-al-weirds-rir-query-00

Regards,
.as

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 15 Feb 2012 20:45:21 +0100

On 15/Feb/12 19:00, Arturo Servin wrote:
> <check> ,<info>, <poll>, <transfer>

Anonymous EPP would probably allow only <info>.  And it would verify
any <contact:disclose> elements to make sure it is not providing any
more information than whois to anonymous clients.

> http://tools.ietf.org/html/draft-newton-et-al-weirds-rir-query-00

The rir-query approach is only simpler for number resources, assuming
that RIRs don't use EPP.  Not reinventing the wheel is a deal.  IOW,
would you have developed rir-query if you had had EPP already up and
running?

Admittedly, these considerations don't go much toward a common
solution.  But I'm not clear on what a common "base protocol" would
consist of, anyway.

---

From: Carlos Martinez-Cagnazzo <[email-masked]>
Date: Wed, 15 Feb 2012 17:52:37 -0200

I thought all mentions to EPP referred only to re-using the data model 
(something I could agree to). Now that mentions of protocol operations 
appear I'm getting scared :-)

IMO one of the greatest strengths of the REST approach is that users do 
not need specialized clients. Any browser, wget, curl, urllib2 will do. 
You can write a specialized client for some purpose, but you don't need to.

If the proposal is to re-use EPP as a whole, then this is no longer the 
case.

cheers

Carlos

On 2/15/12 5:45 PM, Alessandro Vesely wrote:
>> <check>  ,<info>,<poll>,<transfer>
> Anonymous EPP would probably allow only<info>.  And it would verify
> any<contact:disclose>  elements to make sure it is not providing any

---

From: Arturo Servin <[email-masked]>
Date: Wed, 15 Feb 2012 17:56:05 -0200

Alessandro,

	LACNIC does use EPP for interactions with the NIRs databases, we use some EPP extensions (bottom of the link attached). We also have a service call SARA to interact with ISPs with several assignments.

http://lacnic.net/en/sara/

	The data model for the EPP extensions I think is fine for a bi-direccional service where you need to do more operations (create, transfer, info, etc. However I find it a bit complex just for queries as we need in restwhois.

	The answer to your question is: Yes.

Regards,
as

---

From: Dave Piscitello <[email-masked]>
Date: Wed, 15 Feb 2012 12:55:18 -0800

> should be separate efforts.

My bad. Please don't hate me for being old and slow All these brackets and such look the same after a while.:-)

I don't have an issue with your proposal. I'd like to leave the issue of JSON, XML or support for both to people who are consuming the data. So long as it's clearly distinguished in signaling, I think it's up to the provider to decide what its customers want and then decide how much they can accommodate of what customers want.

---

From: SM <[email-masked]>
Date: Wed, 15 Feb 2012 13:39:58 -0800

Hi Dave,
At 12:55 15-02-2012, Dave Piscitello wrote:
>I don't have an issue with your proposal. I'd like to leave the 
>issue of JSON, XML or support for both to people who are consuming 
>the data. So long as it's clearly distinguished in signaling, I 
>think it's up to the provider to decide what its customers want and 
>then decide how much they can accommodate of what customers want.

I don't have an opinion about whether to pick JSON or XML.  I am ok 
with the provider deciding on what to serve their customers.  If 
there are several data formats, the consumer will have to implement 
each of them.  John Levine commented [1] on open-ended formats.

The producers here are:

  (a) Number resources registries (and others)

  (b) ccTLDs

  (c) gTLDs

  (d) ICANN

Would specifying one mandatory format be something ICANN can work out 
with the gTLDs?

The alternative is for a future working group to determine 
that.  Given the diverse constituencies, it might end up being a 
problematic decision.

Regards,
-sm

1. http://www.ietf.org/mail-archive/web/weirds/current/msg00558.html

---

From: Keith Gaughan <[email-masked]>
Date: Thu, 16 Feb 2012 10:50:04 +0000

> possible to use it instead of whois right away --mission accomplished.

That could work, though it does go against some of the stated aims of WEIRDS
(RESTful WHOIS, that is). As the provreg list is currently active again, and I'm
sure nobody there would mind terribly if further useful object mappings were
added. It's at least an avenue worth exploring.

---

From: Keith Gaughan <[email-masked]>
Date: Thu, 16 Feb 2012 11:01:42 +0000

On Wed, Feb 15, 2012 at 05:52:37PM -0200, Carlos Martinez-Cagnazzo wrote:

> I thought all mentions to EPP referred only to re-using the data
> model (something I could agree to). Now that mentions of protocol
> operations appear I'm getting scared :-)
> 
> IMO one of the greatest strengths of the REST approach is that users
> do not need specialized clients. Any browser, wget, curl, urllib2
> will do. You can write a specialized client for some purpose, but
> you don't need to.

I wouldn't be too scared. EPP is hardly rocket science. The default TCP
transport is simple, and building <info> requests and parsing their results
isn't difficult either.

One of the big problems with IRIS/DCHK/DREG was that the whole agglomeration
was so very complex, from the RFC 4993 UDP transport, right up to the top of
the stack. EPP isn't anywhere near as much of a pain to implement. Moreover,
the problem set we're exploring does fit into the scope of EPP, specifically:

    [A] [...] protocol for the provisioning and management of objects stored
    in a shared central repository.

Which can and does include querying for those objects within its scope.

Now, I'm *not* saying we should use EPP, but at the very least it's an idea
worth exploring.

> If the proposal is to re-use EPP as a whole, then this is no longer
> the case.

K.

---

From: Dave Piscitello <[email-masked]>
Date: Thu, 16 Feb 2012 05:09:04 -0800

> each of them.  John Levine commented [1] on open-ended formats.

It's the web. We have lots of data formats and we haven't seen the last of them. I'm not suggesting that every dat format is in play *and* mandatory. I'm OK with choosing a single mandatory/default and I'm even happy to let others make a reasoned decision as to what format this is.

If the protocol identifies a mandatory/default data format and accommodates optional formats in signaling then it would look quite a bit like what ARIN has in production today, no?

I'm merely pointing out that data format choice and selection aren't solely technical decisions when you already have multiple APIs and data formats. Is there is harm in identifying a standard signaling mechanism for data formats in the protocol? 

> Would specifying one mandatory format be something ICANN can work out 

I think so. A possible path forward is that technical folks in a group like SSAC would develop and propose the set of data that satisfy what Jim Galvin calls a "domain name life cycle" and possibly recommend or define a data format. That data set and format would be submitted for community review. If this effects contracts then it falls into a policy development process but the contracts could then oblige contracted parties to use the mandatory format.

ccTLDs that participate in ICANN's CCNSO could agree to adopt the same mandatory format through their own processes. 

> that.  Given the diverse constituencies, it might end up being a 

Replace "might" with "would" and I agree.

---

From: Dave Piscitello <[email-masked]>
Date: Thu, 16 Feb 2012 05:18:37 -0800

On Feb 16, 2012, at 6:01 AM, Keith Gaughan wrote:

>> model (something I could agree to).

This was my understanding, too.

>> Now that mentions of protocol

I think this takes us off track.

>> will do. You can write a specialized client for some purpose, but

At least for name resources, this is an important strength. Registrars, for example, are contractually obliged to provide web access. The user community is accustomed to this interface. 

> transport is simple, and building <info> requests and parsing their results

It really isn't strictly a matter of how easy or difficult it is to write a new client. 

For name resources at least, I'd prefer to take a thoughtful look at the several domain name registration data models and definitions that exist today (EPP  the RAA, escrow…), consider internationalization,  decide what data format is desired as mandatory, and then produce the data profile.

---

From: SM <[email-masked]>
Date: Thu, 16 Feb 2012 09:34:55 -0800

Hi Dave,
At 05:09 16-02-2012, Dave Piscitello wrote:
>If the protocol identifies a mandatory/default data format and 
>accommodates optional formats in signaling then it would look quite 
>a bit like what ARIN has in production today, no?

I'll leave it to a better informed party to comment on the above.

>I'm merely pointing out that data format choice and selection aren't 
>solely technical decisions when you already have multiple APIs and 
>data formats. Is there is harm in

It's rarely an easy choice.

>  identifying a standard signaling mechanism for data formats in the protocol?

There is generally someone in the IETF which will find something 
harmful. :-)  If  one of more data formats are to be supported, the 
protocol would need a signaling mechanism.

>I think so. A possible path forward is that technical folks in a 
>group like SSAC would develop and propose the set of data that 
>satisfy what Jim Galvin calls a "domain name life cycle" and 
>possibly recommend or define a data format. That data set and format 
>would be submitted for community review. If this effects contracts 
>then it falls into a policy development process but the contracts 
>could then oblige contracted parties to use the mandatory format.

The data set and data format could end up as out of scope then.

>ccTLDs that participate in ICANN's CCNSO could agree to adopt the 
>same mandatory format through their own processes.

Ok.

Such points end up as "would" (re: last comment in your reply). It 
helps when "paths forward" are stated in a charter.  The groups do 
not have to be explicitly mentioned unless there is a coordination 
mechanism in place.

Regards,
-sm

---

From: Francisco Arias <[email-masked]>
Date: Thu, 16 Feb 2012 09:55:54 -0800

Hi SM,

>The data set and data format could end up as out of scope then.

The data set (which fields are required) is a policy matter for each
registry's policy-maker body to decide.

However, the format (e.g., the tag (if there is one), attributes, the
syntax) of the possible fields is something that should be standardized
and as such be in scope. If there is a field that could be used in the
protocol, we should define its format.

In other words, we should standardize the "dictionary" of possible fields
(for some definition of possible), but what fields are served in a server
response is outside the protocol.

Regards,

__

Francisco.

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 16 Feb 2012 20:38:24 +0100

> added. It's at least an avenue worth exploring.

PROVREG had a BoF in Dec 2000 and terminated in Jan 2004.  That seems
to be just before CRISP started, but that's not enough to explain why
those protocols ignored one another.  I participated in neither, so I
change the subject to catch the attention of someone who participated
in both, e.g. Scott.

Provisioning implies updating a DB.  Browsing that data doesn't seem
to have ever been an issue for EPP.  That protocol is designed to be
extensible with respect to the data model, not the access model.

However, from a software POV, it is usually convenient to use the same
API and shared chunks of code for both updating and browsing.
So, it is natural to think that someone who is familiar with an EPP
implementation could easily derive an utility for querying that data
from it.  I'd feel uncomfortable with pursuing WEIRDS for names
without having checked out why that's not true.

CRISP's charter explicitly excluded provisioning.  Although it
addresses the same registry data, it didn't seem to want to reuse the
same data models.  Interaction between IRIS and EPP was asked in 2008
http://www.ietf.org/mail-archive/web/crisp/current/msg00496.html
but not answered.

---

From: Francisco Obispo <[email-masked]>
Date: Thu, 16 Feb 2012 11:48:54 -0800

> extensible with respect to the data model, not the access model.

Well, the access model is actually not part of EPP.. the widely used 
EPP Transport over TCP is a totally different RFC [1].

I personally don't feel like EPP would fit the requirements of a whois 
protocol. Although there is functionality that is fairly similar, there 
is a whole bunch of separate issues that include access control, policy,
and transactions that would not apply to whois systems.

> without having checked out why that's not true.

If someone is capable of understanding EPP, and its data structures, it
would not have any trouble understanding an XML document with a response.

And if we go the RESTful route, it would be even easier, since there are
tons of libraries out there to query HTTP/HTTPS servers

[1] http://tools.ietf.org/html/rfc5734

Francisco Obispo 
email: [email-masked]
Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
PGP KeyID = B38DB1BE

---

From: Andy Newton <[email-masked]>
Date: Thu, 16 Feb 2012 19:50:24 +0000

> If the protocol identifies a mandatory/default data format and accommodates optional formats in signaling then it would look quite a bit like what ARIN has in production today, no?

Yes, ARIN's RESTful Whois service hands back multiple formats.

As I see it, there are two large issues here: computer to computer data marshaling, and browsability (for lack of a better name).  They each have differing needs.

For computer to computer data interchange, there needs to be a mandatory format otherwise we make the job of the client implementers much harder. There are a few qualities we should look for in that single format: 1) it shouldn't be something we have to invent, 2) it should be easy to find a parser for it; the more client environments it targets, the better, 3) it should be simple, and 4) our resulting use of it should not be awkward or invoke any unnecessary features.

For browsability, we want to be able to use a RESTful URL in a browser and have data displayed that doesn't require a programmer to interpret. There are many ways of doing, the most common being HTML. XML has other advantages, but they quickly become cumbersome with every new "feature" added to the dataset.

-andy

---

From: Andy Newton <[email-masked]>
Date: Thu, 16 Feb 2012 19:59:34 +0000

> It really isn't strictly a matter of how easy or difficult it is to write a new client. 

The ease of writing a new client or of using an existing one such as a web browser or wget should be a large consideration.

The notion that re-using EPP is easy is from the point of view people who already have EPP servers and does not consider implementers writing clients for the Internet at large.

I'd prefer we place words in the charter that restrict this working group to RESTful web services. Such words would put EPP out of scope as well as the sure-to-come proposals for SOAP, ssh-channels, SNMP, etc…

-andy

---

From: Francisco Obispo <[email-masked]>
Date: Thu, 16 Feb 2012 12:01:24 -0800

On Feb 16, 2012, at 11:50 AM, Andy Newton wrote:

> XML has other advantages, but they quickly become cumbersome with every new "feature" added to the dataset.

The most frequent problem I find with XML processing is when programmers don't follow the rules to parse XML. I've even seen people using sed/awk scripts to parse them…

Like I said in my previous email, I think XML is the right tool for the job here.

Francisco Obispo 
email: [email-masked]

---

From: Carlos Martinez-Cagnazzo <[email-masked]>
Date: Thu, 16 Feb 2012 18:06:57 -0200

I agree that XML is the right tool. JSON support is also *very*
desirable. However, IMO, plain access using a non-specialized client is
a MUST in my opinion. The good thing is that this is easy to achieve as
long as we stick to HTTP transport, either by using XSLT sheets or other
mechanisms.

I don't think we can ask our communities to do XML programming in order
to access the registry database. The good thing is that we have all the
tools in our hands to produce something that can cater both the folks
doing machine processing of WHOIS data and also the browser-access crowd.

regards

Carlos

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 16 Feb 2012 12:08:11 -0800

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Andy Newton
> Sent: Thursday, February 16, 2012 12:00 PM
> To: Dave Piscitello
> Cc: [email-masked]
> Subject: Re: [weirds] one protocol to rule them all
> 
> The ease of writing a new client or of using an existing one such as a
> web browser or wget should be a large consideration.
> 
> The notion that re-using EPP is easy is from the point of view people
> who already have EPP servers and does not consider implementers writing
> clients for the Internet at large.
> 
> I'd prefer we place words in the charter that restrict this working
> group to RESTful web services. Such words would put EPP out of scope as
> well as the sure-to-come proposals for SOAP, ssh-channels, SNMP, etc...

EPP's not a term with which I'm familiar, so I just read the Wikipedia page on it.  Assuming it's accurate, it looks like it's meant for use for communicating between registries and registrars.  I hadn't really considered that application before; I thought we were talking only about servicing arbitrary queries from end users (or their agents).  That means EPP and its ilk aren't within the scope I thought we were trying to cover.

So I'd support Andy's suggestion.  Since CRISP (RFC3707) did consider EPP-like issues, we probably should make it clear if we want to avoid that for this round.

-MSK

---

From: "John Levine" <[email-masked]>
Date: 16 Feb 2012 20:29:27 -0000

In article <[email-masked]> you write:
>desirable.

I prefer JSON, but whatever.

> However, IMO, plain access using a non-specialized client is
>a MUST in my opinion. 

-99, for reasons that I'd think would be obvious to anyone who's ever
done standards development.  SMTP isn't browsable.  POP isn't
browsable.  BGP isn't browsable.  TELNET isn't browsable.  SIP isn't
browsable.  They're accesed via clients that understand the formats.
If you want to build debugging tools to look at the data, that's a
fine idea, but don't demand them as part of the spec.

>The good thing is that this is easy to achieve as long as we stick to
>HTTP transport, either by using XSLT sheets or other mechanisms.

I don't speak Chinese.  Do Chinese RIRs have to provide everything in
English?  Conversely, do North American RIRs have to provide
everything in Chinese?  Being able to browse something isn't worth much
if I can't understand the results.

If the answer is, it doesn't matter if you can't understand them, you
can see what the data is, why doesn't that apply equally to JSON or
XML results, which are 100% text that any browser can display?

>to access the registry database.

I still prefer JSON, but XML libraries are cheap and plentiful.  If we
choose XML, generating and parsing it is an insignificant part of the
programming effort.

R's,
John

---

From: SM <[email-masked]>
Date: Thu, 16 Feb 2012 12:20:22 -0800

Hi Francisco,
>response is outside the protocol.

I am ok with the above.

I suggest putting what's out of scope into the charter.

Regards,
-sm

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 16 Feb 2012 16:24:03 -0500

On Thu, Feb 16, 2012 at 07:59:34PM +0000, Andy Newton wrote:

> group to RESTful web services.

That was the goal of this text:

      The
    overall effort will be broadly aligned with the Cross Registry
    Internet Service Protocol (CRISP) Requirements (RFC 3707), but
    with the explicit additional goal of producing a simple,
    easy-to-implement protocol that is to be deployed via REST

It's not a restriction, but I think it offers a pretty strong
suggestion.  

Besides, I don't know why we're talking about EPP when the very idea
that EPP is any sort of basic standard that everyone adheres to is
wrong on its face.  Many name registries don't use it (partly because
of disagreements about the data model), some number resource
registries don't use it, and many name registries use an alternative
mapping from the "standard" one because they also don't like the data
model.

I have no objection to looking at EPP for inspiration.  But the idea
that the EPP mappings are going to be the basis of work strikes me as
wrong.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Thu, 16 Feb 2012 23:59:20 +0000

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Murray S. Kucherawy
> Sent: Thursday, February 16, 2012 3:08 PM
> To: [email-masked]

[snip]

> EPP's not a term with which I'm familiar, so I just read the Wikipedia
> page on it.  Assuming it's accurate, it looks like it's meant for use
> for communicating between registries and registrars.  I hadn't really
> considered that application before; I thought we were talking only
> about servicing arbitrary queries from end users (or their agents).
> That means EPP and its ilk aren't within the scope I thought we were
> trying to cover.
> 
> So I'd support Andy's suggestion.  Since CRISP (RFC3707) did consider
> EPP-like issues, we probably should make it clear if we want to avoid
> that for this round.

I don't disagree with your conclusion (in fact, I agree), but I think you were misled on your path to reaching it.

RFC 5730 would be a better read than the Wikipedia article. As noted in both places, the core EPP protocol was *not* designed for exclusive use by domain name registries and registrars. Their needs provided the inspiration for the work and I'm not aware of use outside that environment, but it's entirely possible to develop mappings for other provisioning needs.

Having said that, I must note that the protocol has been typecast.

Scott

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Fri, 17 Feb 2012 00:17:20 +0000

> Behalf Of Alessandro Vesely
> Sent: Thursday, February 16, 2012 2:38 PM
> To: [email-masked]
> Subject: [weirds] Anonymous EPP, was one protocol to rule them all
> 

Actually, I didn't follow CRISP all that closely because Andy was doing most of the driving for the Verisign Labs team. I don't remember talking to him about this topic, but he must have convinced me that it wasn't a big deal since I never raised any concerns. Is your memory any better than mine, Andy?

Scott

---

From: Andy Newton <[email-masked]>
Date: Thu, 16 Feb 2012 20:24:18 -0500

> Actually, I didn't follow CRISP all that closely because Andy was doing most of the driving for the Verisign Labs team. I don't remember talking to him about this topic, but he must have convinced…

EPP was discussed briefly in CRISP, but the CRISP WG aimed for a transport that could support SASL which EPP does not. And at the time, BCP 56 was enforced so HTTP was not viable. These are different times.

-andy

---

From: Carlos Martinez-Cagnazzo <[email-masked]>
Date: Thu, 16 Feb 2012 23:31:27 -0200

A few answers in between lines

> I prefer JSON, but whatever.
HTTP + REST and a clean data model can make both of us happy.
> fine idea, but don't demand them as part of the spec.
I haven't been involved in standards development for a long time, but I
have been on the pain-suffering side of WHOIS users for longer than I
can remember.

The examples you provide have little to do with this problem space,
Users, goals and expectations are completely different. And even so,
SMTP, POP, HTTP and TELNET can be pretty much used with a generic tool
like netcat. We cannot ignore the fact that in the WHOIS problem space
there are as many automatic information processing needs as human
consumption needs.

And again, you can get as formal as you want with your XML schema,
define your ontologies, your validation DTDs, Schemas and even so with
some clever XSLT transforms (that obviously *do not need* to be
specified in this WG) you can provide for all needs.

I really don't get the negative attitude towards using signaling that
*is already part of HTTP*.
> XML results, which are 100% text that any browser can display?
I'm not sure I get your point here. You don't want multiple formats, I
get it, fine. We can agree to disagree and let the group decide.
> choose XML, generating and parsing it is an insignificant part of the
Then I'm not sure what we are arguing about. Would you disapprove if an
implementer of whatever WEIRDS standardizes writes XSLT transforms to
present it for human users and accepts a header for signaling desired
output ? No one is asking the WG to work on these XSLTs, my original
point was that the *signaling* should be agreed on.

It seems to me that you are getting excessively concerned about using
standard HTTP headers in a commonly-agreed way... As you say, I haven't
done a lot of standards development, but perhaps I've felt the pain of
the badly broken WHOIS system for a long time.

Warm regards,

Carlos

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 17 Feb 2012 11:57:25 +0100

> EPP Transport over TCP is a totally different RFC [1].

A registry could use (a pool of) local TCP connections to marshal
their EPP server's data to a web server.

> and transactions that would not apply to whois systems.

http://www.verisigninc.com/assets/whois-info-extension.pdf

> would not have any trouble understanding an XML document with a response.

I think the problem is not to understand the response, but to provide
homogeneous responses from all registries.  AFAICS, that's the reason
why number registries are considered "easy".

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 17 Feb 2012 12:07:16 +0100

> It's not a restriction, but I think it offers a pretty strong

An EPP server could be plugged into Apache using a local, read-only,
"anonymous" EPP client.  Would you consider that a fulfillment of the
suggestion above?

> mapping from the "standard" one because they also don't like the data

For number registries, or name registries that don't use EPP because
they don't need it, developing a new service may well be the best
solution.

For name registries where the data model is the problem, why do we
think we'll get a better success than EPP?

> that the EPP mappings are going to be the basis of work strikes me as

For RIRs, I fully agree.

---

From: Keith Gaughan <[email-masked]>
Date: Fri, 17 Feb 2012 12:13:01 +0000

> browsable.  They're accesed via clients that understand the formats.

I'm playing Devil's Advocate here, but in fairness, for their day, SMTP and POP
were at least human readable and writable and it was possible for a human being
to sit down with a general-purpose client (a TELNET client at the time) and
interact with the server using the protocol without any special purpose tools.
Some systems administrators occasionally do just that when diagnosing problems
with mailservers.

> fine idea, but don't demand them as part of the spec.

IIRC, SMTP, POP3, FTP, HTTP, and a good number of protocols were designed with
that ability originally by using TELNET as their basis.

Now that I'm done with being Devil's Advocate, so long as requests can be
constructed and responses read by somebody with access to a general-purpose
client (be it a HTTP browser, TELNET, or even netcat) who is at least half-way
competent and understands the application protocol itself, then all that matters
is that the format used is human readable. Both XML and JSON fulfil that goal
perfectly well as metaformats. Arguing over which one ought to be used is
basically bikeshedding.

In fact, I don't get why people are advocating multiple machine-readable
response formats. At the end of the day, the machine doesn't give a whit what
format they receive the information in. The *only* reason this debate is
happening is because we're arguing over something that only becomes an issue
once a human is looking at the response. Personally, while I might prefer JSON
for its readability to due being less cluttered, XML at least has the advantages
that (a) you can slap stylesheet on it and (b) it's extensible through
namespaces. That puts me very weakly in the position of advocating the use of
XML, but at the end of the day, it really doesn't matter a jot. To paraphrase a
certain former US president, it's the data model, stupid.

[Sidebar: So long as there's at least one mandatory response type, any client
that allows the JSON and XML serialisation can provide Accept headers to the
client for them, say application/json+weirds and application/xml+weirds. If the
server supports one or the other or both, all well and fine.]

> choose XML, generating and parsing it is an insignificant part of the

And not difficult either. If a developer is going to play fast and loose with
the protocol, they're going to do that regardless of whether the data format the
request and response information is presented in.

---

From: Keith Gaughan <[email-masked]>
Date: Fri, 17 Feb 2012 12:19:36 +0000

On Thu, Feb 16, 2012 at 04:24:03PM -0500, Andrew Sullivan wrote:

> and many name registries use an alternative

Some significant examples of exactly that, such as EURid and dns.be, are moving
towards a model closer to the EPP model over time, however.

K.

---

From: Carlos Martinez-Cagnazzo <[email-masked]>
Date: Fri, 17 Feb 2012 11:34:20 -0200

Just to clarify mi opinion, for when consensus evaluation comes:

- As Keith stated in a more eloquent manner than I, the ability to use
non-specialized clients is important

- I agree we look into the EPP data model for inspiration and maybe
re-use (see below for my comment on XML)

- XML is a very good tool, it can be validated and formally specified,
which are nice properties in the quest for interoperability and enabling
machine processing

- JSON is highly desirable as well, it *could* be spec'ed as a by
product of the XML format through an XSLT transformation (in this case
*yes* I advocate for standardizing this transform, so the formalization
properties of the XML format are inherited by the JSON spec)

- Signaling for other formats also important (again, HTTP already does
it for us, we just need to agree on which header, if other than plain
"Accept: "). And here, *no* I don't think the WG has to spec the
representations themselves, just the *signaling*. I offer to document
the way it's currently done among the RIRs implementations.

regards

Carlos

---

From: Eric Brunner-Williams <[email-masked]>
Date: Fri, 17 Feb 2012 10:47:46 -0500

> their EPP server's data to a web server.

Like Scott, I paid little attention to the CRISP WG, though probably
for different reasons.

I want to point out a layer 9 complication to using EPP "anonymously".

There exists a set of authorized dbms accessors, aka "ICANN accredited
registrars", which historically have viewed the service boundary
between registries and registrars as both materially important, and
defined by enforceable contract.

IMHO, it is conceivable that one or more of these access providers
will object to a contractual counter-party offering a thread to a
non-contractual party, independent of the operation originating from
the EPP client to the EPP server.

Historically, access to the drop pool motivated the accumulation of
registrar accreditations by parties pursuing the secondary market in
expired names. Had there been a means of providing secondary market
actors with substantive access (threads of execution, CIDR ranges)
other than through execution of the then-current RAA, it is possible
that the number of "ICANN accredited registrars" might number around
10^2 rather than 10^3.

There are, as others have pointed out, other reasons not to just take
EPP and run with it. As this one hadn't been mentioned, I mentioning it.

Cheers,
Eric

---

From: "Michael Young" <[email-masked]>
Date: Fri, 17 Feb 2012 11:01:46 -0500

Whatever we decide on, authentication has to be considered, the days of
blind IP source based rate-limiting - due IPv6 - are coming to an end.
Well, they should anyways,.........

I don't see any value in recreating the problem of having to answer every
query because you don't know whats legitimate or not (DNS) if we are
rebuilding Whois.  IMHO, the overhead of building up and tearing down
sessions is preferable to blindly having to answer Whois DDOS attacks by
data-miners with large IPv6 allocations.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Eric Brunner-Williams
Sent: February-17-12 10:48 AM
To: [email-masked]
Subject: Re: [weirds] Anonymous EPP, was one protocol to rule them all

> their EPP server's data to a web server.

Like Scott, I paid little attention to the CRISP WG, though probably for
different reasons.

There exists a set of authorized dbms accessors, aka "ICANN accredited
registrars", which historically have viewed the service boundary between
registries and registrars as both materially important, and defined by
enforceable contract.

IMHO, it is conceivable that one or more of these access providers will
object to a contractual counter-party offering a thread to a non-contractual
party, independent of the operation originating from the EPP client to the
EPP server.

Historically, access to the drop pool motivated the accumulation of
registrar accreditations by parties pursuing the secondary market in expired
names. Had there been a means of providing secondary market actors with
substantive access (threads of execution, CIDR ranges) other than through
execution of the then-current RAA, it is possible that the number of "ICANN
accredited registrars" might number around
10^2 rather than 10^3.

There are, as others have pointed out, other reasons not to just take EPP
and run with it. As this one hadn't been mentioned, I mentioning it.

Cheers,
Eric

---

From: Andy Newton <[email-masked]>
Date: Fri, 17 Feb 2012 11:59:07 -0500

> The most frequent problem I find with XML processing is when programmers don't follow the rules to parse XML. I've even seen people using sed/awk scripts to parse them…

Probably because they're use to doing something simpler with JSON such as using jsawk (https://github.com/micha/jsawk). By contrast the XML equivalent is a rather abusive exercise in writing an XSL stylesheet and running it through the xsltrpoc.

Mindshare has shifted away from XML to JSON in the programming world. When you look around at the RESTful services, JSON is the clear winner.

"So what's the way forward? I think the Web community has spoken, and it's clear that what it wants is HTML5, JavaScript and JSON. XML isn't going away but I see it being less and less a Web technology; it won't be something that you send over the wire on the public Web, but just one of many technologies that are used on the server to manage and generate what you do send over the wire."
-James Clark of W3C fame
http://blog.jclark.com/2010/11/xml-vs-web_24.html

"It looks like the debate between XML and JSON may be coming to a close with JSON poised to take the title of preferred format for web applications."
http://devcentral.f5.com/weblogs/macvittie/archive/2011/04/27/the-stealthy-ascendancy-of-json.aspx

As James Clark notes, Microsoft was the big player in the push for XML. So it should say something that Microsoft were one of the biggest proponents of the IETF's new JSON security objects efforts. And the other newer IETF efforts are looking at JSON rather than XML when the subject comes up.

JSON also has something to offer this group beyond the arguments of the shape of the brackets, and that is a fresh start. With XML we've already seen on this list the propensity to pull in unnecessary cruft. Moving to JSON will let people focus on the real work instead of bashing each other about who's schema is better and which XML schema language is best, do we use Xlink, what about Xpointer, just a little XPath here and there, etc…

-andy

---

From: Eric Brunner-Williams <[email-masked]>
Date: Fri, 17 Feb 2012 12:02:33 -0500

> Well, they should anyways,.........

Agree. However, scaling-based observation(s) have not, thus far,
motivated changes to two layer 9 registration data consumer use cases
-- "law enforcement" and "intellectual property rights enforcement" --
to differentiate their basic access model from that of entities for
which blind IP source base rate-limiting (and other management tools)
have been developed to prevent large-scale data mining, equivalent to
(registrar, or registry, or both) bulk access.

> data-miners with large IPv6 allocations.

Agree. Sessions wins. Now we just have to convince the LEOs of the
world that they really should cease acting like criminals.

Eric

---

From: Andy Newton <[email-masked]>
Date: Fri, 17 Feb 2012 12:09:41 -0500

On Feb 17, 2012, at 7:13 AM, Keith Gaughan wrote:

> Personally, while I might prefer JSON
> namespaces.

XSLT programmers have a low life expectancy. Been there, done that, have the big red marks on my forehead to prove it.

As for Namespaces in XML making XML extensible, that is false. The X in XML means "extensible" which predates Namespaces in XML. And it shows. Ever wonder why namespace qualification is different for XML attributes than it is for XML elements? Because Namespaces in XML was an afterthought.

Not only that, but the use of XML Schema and even the better XML schema languages such as Relax NG actually limit extensibility over what you get with plain XML.

-andy

---

From: Francisco Arias <[email-masked]>
Date: Fri, 17 Feb 2012 09:20:10 -0800

>world that they really should cease acting like criminals.

I think we should build a protocol that enables this feature, but let the
respective policy-makers decide whether to activate or not for each
registry/registrar deployment as they see fit.

__

Francisco.

---

From: Eric Brunner-Williams <[email-masked]>
Date: Fri, 17 Feb 2012 12:47:16 -0500

> registry/registrar deployment as they see fit.

could you restate this as a statement on a scaling property of the
protocol?

-e

---

From: "John Levine" <[email-masked]>
Date: 17 Feb 2012 17:48:37 -0000

>response formats.

Assuming we agree that the goal is to design a standard way for clients
and servers to interoperate, me neither.

>server supports one or the other or both, all well and fine.]

Right.  Debugging either is relatively simple.  You can use a web
browser or command line HTTP client to poke at a JSON or XML server,
even though that server doesn't have a "web page mode", much as you
can use a Telnet client to poke at an SMTP or POP or FTP server, even
though they don't have a Telnet mode.

R's,
John

---

From: Keith Gaughan <[email-masked]>
Date: Fri, 17 Feb 2012 17:54:12 +0000

On Fri, Feb 17, 2012 at 12:09:41PM -0500, Andy Newton wrote:
> > Personally, while I might prefer JSON for its readability to due being less
> > cluttered, XML at least has the advantages that (a) you can slap stylesheet
> > on it and (b) it's extensible through namespaces.
> 
> XSLT programmers have a low life expectancy. Been there, done that, have the
> big red marks on my forehead to prove it.
> 
> As for Namespaces in XML making XML extensible, that is false. The X in XML
> means "extensible" which predates Namespaces in XML. And it shows. Ever wonder
> why namespace qualification is different for XML attributes than it is for XML
> elements? Because Namespaces in XML was an afterthought.
> 
> Not only that, but the use of XML Schema and even the better XML schema
> languages such as Relax NG actually limit extensibility over what you get with
> plain XML.

You'll be glad to know that you're preaching to the choir, though I have to say
I've only ever had problems with XSLT when my knowledge of XPath gets fuzzy, or
when dealing with multiple schemata in the same document.

And I'm well old enough to remember pre-namespaces XML and what a hacky bodge
those were. I never said that XML's extensibility story was good, just that it
had one, unlike JSON currently does at the moment. Whether said extensibility is
needed or not in the scope of WEIRDS can be questioned, but I think it likely
is. The less said about XML Schema the better, frankly, though I don't recall
bringing schema definition languages up myself.

---

From: Andy Newton <[email-masked]>
Date: Fri, 17 Feb 2012 13:13:44 -0500

On Feb 17, 2012, at 12:54 PM, Keith Gaughan wrote:

> I never said that XML's extensibility story was good, just that it
> is.

It has an extensibility similar to plain old well-formed XML.

As nobody has pointed out a deficiency in Section 3 of rir-json, perhaps nothing complicated is needed.

> The less said about XML Schema the better, frankly, though I don't recall

Head down the XML route and that's the next topic.

-andy

---

From: Keith Gaughan <[email-masked]>
Date: Fri, 17 Feb 2012 18:49:45 +0000

On Fri, Feb 17, 2012 at 01:13:44PM -0500, Andy Newton wrote:
> > I never said that XML's extensibility story was good, just that it had one,
> > unlike JSON currently does at the moment. Whether said extensibility is
> > needed or not in the scope of WEIRDS can be questioned, but I think it
> > likely is.
> 

As I said, my preference for XML was weak at best. I don't care strongly enough
to argue in favour of either all that vehemently. So long as one
machine-readable format that all servers produce and all clients can parse
exists, I'm happy.

---

From: Dave Piscitello <[email-masked]>
Date: Fri, 17 Feb 2012 14:12:07 -0800

We agree. You stated this more thoroughly and elegantly than I did.

---

From: Dave Piscitello <[email-masked]>
Date: Fri, 17 Feb 2012 14:14:01 -0800

Again, we agree.

Your clip takes my statement a bit out of context. I was saying (perhaps again, not elegantly) that client development is one consideration, not the only one.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Fri, 17 Feb 2012 15:19:14 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Michael Young
> Sent: Friday, February 17, 2012 8:02 AM
> To: [email-masked]; [email-masked]

I think REST over HTTP solves this nicely, given the abundant HTTP infrastructure out there that has support for authentication.

-MSK

---

From: Avri Doria <[email-masked]>
Date: Sat, 18 Feb 2012 13:13:58 -0500

> bringing schema definition languages up myself.

I think that in general data models need to be extensible and that is a default requirement.  
Perhaps there are occasions when extensibility is not needed or is a luxury that can't be afforded, but in those cases, I beleive a case for non-extensibility must be made.

avri

---

From: Eric Brunner-Williams <[email-masked]>
Date: Sun, 19 Feb 2012 11:18:50 -0500

> I think that in general data models need to be extensible and that is a default requirement.  

Extensibility is both abstract and concrete. The concreteness is the
(possibly implicit) model, which, in theory, could be useful in the
application problem domain. The abstractness may make implementation
an interesting problem in computer science.

As others have pointed out, life before XML Schema was iffy, and life
after XML Schema has disproportionate complexity.

As others have pointed out in related venues e.g., PROVREG, and
DOMAINREP, implementation of extension of a protocol syntactically
expressed in XML can be difficult, in theory and in practice.

As a choice of working idioms, semantics via a decade old subset of a
text markup language, for which validation tools are recent, and
limited to text markup use, or semantics via a subset of C, for which
syntax validation tools (compilers, editor modes, etc) have existed
for a generation, and common to a vast array of programming idioms and
uses, is a reasonable software architecture and engineering choice.

My view is that the gang of N (Scott, Jordyn, Ross, ... self) erred in
2002 in selecting XML syntax to specify the semantics of a replacement
to the then prevalent provisioning protocol specified in key-value
pairs. See http://tools.ietf.org/html/draft-ietf-provreg-grrp-reqs-06,
where we'd not yet taken the step of committing to a syntax for
command payload.

So my two beads are for JSON, or rather, ease of implementation of a
known application domain, and therefore, ease of extension within the
implied and express constraints, that is, for C programming as the
idiom rather than text and its markup.

Eric

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Mon, 20 Feb 2012 00:17:49 +0000

> Behalf Of Eric Brunner-Williams
> Sent: Sunday, February 19, 2012 11:19 AM
> To: [email-masked]
> Subject: Re: [weirds] data formats (was one protocol to rule them all)

[snip]

> where we'd not yet taken the step of committing to a syntax for

Given the options available *at the time* I still think we made the right choice. With additional options available today it makes sense to consider alternatives.

Scott

---

From: Byron Ellacott <[email-masked]>
Date: Mon, 20 Feb 2012 10:19:49 +1000

> Just to clarify mi opinion, for when consensus evaluation comes:

Good plan!

> non-specialized clients is important

I'd further this that a wider interest in the protocol than just registries means there's a lot more base level support out there.  Using REST/HTTP lets future client developers take advantage of all the other REST/HTTP work out there, and gives them a fair chance of already understanding the mechanisms.  Using a protocol specific to registry data introduces a learning curve for every new developer.

> re-use (see below for my comment on XML)

The data mode prototyped by Francisco (Message-Id <[email-masked]>) shows that there's some easy ground to be made with this approach.

> the way it's currently done among the RIRs implementations.

I agree that signalling is very important, and is another thing HTTP gives us "free".  I think the representations should be specced by the WG, and I think there should be one mandated representation format (per registry context, preferably with significant overlap on entities and contact information), and as the WG is willing to work on it, other optional formats.  This would match up just fine with your preference to have both XML and a transform of the XML into JSON documented.

And to merge replies:

> I think REST over HTTP solves this nicely, given the abundant HTTP infrastructure out there that has support for authentication.

I'd want to hear very strong arguments against REST/HTTP before we throw away all that support.  There's authentication, channel security, language, data format, compression signalling, redirections if that's a suitable way to do referrals, caching, and a lot of work put into scalability of servers.

  Byron

---

From: Francisco Arias <[email-masked]>
Date: Mon, 20 Feb 2012 09:44:22 -0800

>could you restate this as a statement on a scaling property of the

I think RFC 3707, section 3.1.4.  Level of Access already provides good
language for this requirement.

__

Francisco.

---

From: Francisco Arias <[email-masked]>
Date: Mon, 20 Feb 2012 09:44:29 -0800

On 2/19/12 4:19 PM, "Byron Ellacott" <[email-masked]> wrote:

>I agree that signalling is very important, and is another thing HTTP
>gives us "free".  I think the representations should be specced by the
>WG, and I think there should be one mandated representation format (per
>registry context, preferably with significant overlap on entities and
>contact information), and as the WG is willing to work on it, other
>optional formats.  This would match up just fine with your preference to
>have both XML and a transform of the XML into JSON documented.

+1

>>I think REST over HTTP solves this nicely, given the abundant HTTP
>>infrastructure out there that has support for authentication.
>
>I'd want to hear very strong arguments against REST/HTTP before we throw
>away all that support.  There's authentication, channel security,
>language, data format, compression signalling, redirections if that's a
>suitable way to do referrals, caching, and a lot of work put into
>scalability of servers.

+1 

__

Francisco.

---

From: Keith Gaughan <[email-masked]>
Date: Mon, 20 Feb 2012 18:35:39 +0000

On Mon, Feb 20, 2012 at 10:19:49AM +1000, Byron Ellacott wrote:

> > - Signaling for other formats also important (again, HTTP already does it
> > for us, we just need to agree on which header, if other than plain "Accept:
> > "). And here, *no* I don't think the WG has to spec the representations
> > themselves, just the *signaling*. I offer to document the way it's currently
> > done among the RIRs implementations.
> 
> I agree that signalling is very important, and is another thing HTTP gives us
> "free".  I think the representations should be specced by the WG, and I think
> there should be one mandated representation format (per registry context,
> preferably with significant overlap on entities and contact information), and
> as the WG is willing to work on it, other optional formats.  This would match
> up just fine with your preference to have both XML and a transform of the XML
> into JSON documented.

Agreed. In addition, I'd prefer if the WG did *not* mandate any particular URI
schema. I'd prefer to see us building on the work of the W3C's URI WG on URI
templates instead, even if that means we stick to a minimal, easily implemented
subset of their work. My reasons for asking for this are:

 (a) Mandating a particular URI schema places implementation limits on server
     implementors.

 (b) By mandating fixed URI schemata, we introduce a degree of brittleness into
     the system: if a future revision needs to revise how resources are
	 referenced, fixed schemata nix this ability.

 (c) Fixd URI schemata remove server's ability to decide how it divides up its
     own URI space by preventing them from putting certain resources on
	 separate hosts.

At the end of the day, the URIs themselves are an implementation detail. It's
the resources and their representations that matter more than anything.

---

From: Eric Brunner-Williams <[email-masked]>
Date: Mon, 20 Feb 2012 16:27:51 -0500

> language for this requirement.

There's a little more to 3.1.4 than just "MUST provide an anonymous
access mechanism", and if there are advocates for that particular
access use case contributing to this specification, they have the
opportunity to make the necessity and utility claims for their
preferred use case(s).

To restate Michael Young's earlier point: The attack universe consists
of an effectively infinite reservoir of assets. Anonymous access means
the "mumble" protocol scales to very large numbers of communicating
endpoints. Do you have a solution to this in mind?

Eric

---

From: "John Levine" <[email-masked]>
Date: 20 Feb 2012 21:48:27 -0000

> Agreed. In addition, I'd prefer if the WG did *not* mandate any
> particular URI schema.

I couldn't disagree more.  The point of this spec is to allow clients
and servers to interoperate, clients can mechanically ask questions to
which servers can return answers that can be mechanically parsed and
interpreted.

If there's no URI schema, how does the client what URI to use to
ask a particular question?

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: "John Levine" <[email-masked]>
Date: 20 Feb 2012 21:54:27 -0000

> (a) Mandating a particular URI schema places implementation limits on server

Yes, that's a feature.

>	 referenced, fixed schemata nix this ability.

if you really need to do that, it means you add version numbers.  A
competent standards effort will do enough test implementations (the
"running code" part) that we'll believe that what we have actually
works before publishing the spec.

>     own URI space by preventing them from putting certain resources on

I really don't think it'd be a good idea to add extra complication to
a design for the benefit of servers that don't know how to do load
balancing.  ARIN has a fairly complete prototype, and if they want
features like this, they haven't said anything about it.

---

From: Francisco Arias <[email-masked]>
Date: Mon, 20 Feb 2012 16:17:47 -0800

>endpoints. Do you have a solution to this in mind?

"The protocol MUST NOT prohibit an operator from granularly assigning
   multiple types of access to data according to the policies of the
   operator.  The protocol MUST provide an authentication mechanism and
   MUST NOT prohibit an operator from granting types of access based on
authentication."

"Server operators will offer varying degrees of access depending on
policy and need."

__

Francisco.

---

From: Andy Newton <[email-masked]>
Date: Tue, 21 Feb 2012 15:02:40 +0000

> features like this, they haven't said anything about it.

I'm not sure how URI templating solves load balancing issues. And certainly load balancing HTTP is well-trodden territory, so I do not think that should be a great consideration anyway.

Though I have not looked at the W3C URI templating work, I have looked at the URI template draft in the IETF. It seems like a lot of work on the client just to follow a URI. Perhaps things would be different if URI templating libraries were as common as HTTP and JSON libraries, but they are not and therefore a WEIRDS client implementer would be doing a lot of work.

-andy

---

From: Eric Brunner-Williams <[email-masked]>
Date: Tue, 21 Feb 2012 10:21:55 -0500

there seems to be a disconnect.

this exchange began with two comments to the effect that
authentication was necessary, for scaling reasons.

you replied that authentication should be optional, discretionary to
the endpoint identifier space, or i misunderstood you completely.

i asked how what i understood to be your relaxation of a
authentication requirement could be expressed in terms of scaling.

your reply referred to 3.1.4 of rfc3707.

i pointed out that rfc3707's 3.1.4 mentioned anonymous access, among
other access requirements, without a reference to scaling and asked if
you'd a solution to very large numbers of anonymous access attempting
endpoints.

you reply cites language from 3.1.4 of rfc3707 which appears to me to
require subsequent protocol specification authors to allow endpoint
identifier space allocators to assign arbitrary policy to data.

assuming i'm anywhere close to understanding your position, and
ignoring the implicit mandatory-to-implement burden on protocol
specification authors to accommodate arbitrary metadata implementing
arbitrary access policies of all endpoint identifier space allocators,
i fail to see a response to the problem of scaling.

in the off chance that a concrete example may help, i wrote a server
that listens to port 43/tcp and makes a likely response. during a
short period of time it responded to several orders of magnitude more
queries than the number of records in the database this server
published. not having intended to write robust code i was pleased that
my little loop ran at a significant portion of wirespeed, for queries
within the implicit design range, as well as for queries exceeding the
design range, without failing. i discarded the log files but i've the
impression that the size of the asset used to mine, or fail, this
particular loop and its data, was modest, less than a thousand source
endpoints.

as assemblages of endpoints as assets available to mine or fail
services exist, one or more orders of magnitude greater than reflected
in the data i accidentally gathered, and casually discarded for its
lack of novelty, the absence of a non-trivial authentication
requirement for endpoints creates an issue of scale.

overlooking the possible claim that the CRISP working group's work
product is controlling on all endpoint identifier space work attempted
by all future IETF working groups, where in the CRISP text is there a
means to ensure that "WEIRDS" (and subsequent) implementations,
whatever "WEIRDS" (and subsequent) eventually specifies, are robust,
accessible, and capable of inter-operation, under known conditions?

well, this has gone on longer than i'd planned, and at this point i've
only noticed one other contributor for whom anonymous access implies a
scaling issue.

-e

---

From: "John R. Levine" <[email-masked]>
Date: 21 Feb 2012 08:20:42 -0800

> Though I have not looked at the W3C URI templating work, I have looked at the URI template draft in the IETF. It seems like a lot of work on the client just to follow a URI. Perhaps things would be…

We looked at it in another group.  If you have an existing set of services 
with random URLs, it's a fairly clever way to put a layer of order over 
chaos, vaguely reminiscent of NAPTR.  In a situation like this, where we 
expect that servers will be built to the spec, it's extreme overkill.

Also, keep in mind that even without templating, server features like 
Apache's mod_rewrite provide server operators with a great deal of 
flexibility to make their internal URL structure different from the 
externally visible one.  I see no reason to foist this complexity off on 
the clients.

Regards,

---

From: Alessandro Vesely <[email-masked]>
Date: Tue, 21 Feb 2012 20:01:21 +0100

> only noticed one other contributor for whom anonymous access implies a

I have difficulties in understanding how this scaling issue can be
worded, and how authentication would solve it.  Please bear with my
ignorance of this problem.

I can understand that a service can be torn down by a huge number of
requests, which may or may not be caused by a DoS attack.  I also
understand publishing data --making it accessible to all-- versus
disclosing within a restricted community only.  In the latter case,
users with access privileges need to be trusted for not disclosing
that data in turn.

Data mining is more subtle.  There is some data that has to be
published while limiting the rate at which it is accessed.  Servers
that mash up that data, typically needed because of the difficulties
in locating the right WHOIS server, regularly hit rate limits.
Granting them unrestricted access would require to trust that they are
going to apply some rate limiting in turn, and at this point the whole
issue seems inextricably indirect to me.

At any rate, this issue stays the same whether one uses WEIRDS, EPP,
CRISP, or Authenticated WHOIS, and is not the most urgent issue.  The
question is if EPP (with suitable access limits, but certainly much
more relaxed than those it usually deploys) would be as good as WEIRDS
(with the same access limits) as a WHOIS replacement.

---

From: Steve Sheng <[email-masked]>
Date: Wed, 22 Feb 2012 11:47:51 -0800

Dear IETF colleagues,

  For your information, ICANN recently publishes a draft roadmap to
replace the WHOIS protocol. This roadmap is to implement Board resolution
that directs staff, in consultation with community, to develop a roadmap
for implementing SSAC's SAC 051 recommendations.

  The roadmap is available here:

http://www.icann.org/en/committees/security/sac051-draft-roadmap-18feb12-en
.pdf

  There is also a comment period, if you are interested to comment, please
write emails to [email-masked],

Regards,
Steve