Subject: [ire] DNRD CSV Draft

From: "Gould, James" <[email-masked]>
Date: Thu, 25 Oct 2012 20:43:21 +0000

All,

We have created a draft of the Domain Name Registration Data (DNRD) Comma-Separated Values (CSV) Objects Mapping that is attached for review and feedback.  We intend to post it to the IETF as an I-D once the submission page is available on November 5th.  This draft fully supports the Registry Data Escrow Specification (draft-arias-noguchi-registry-data-escrow-04).  It defines the CSV files and the order and format of the CSV fields for the data escrow of domain name, host, contact, registrar, and IDN language objects.  If there is interest in this model we can consider merging it with the Domain Name Registration Data (DNRD) Objects Mapping (draft-arias-noguchi-dnrd-objects-mapping-01).  The basis of using CSV files for DNRD objects includes:

  1.  CSV is a natural format for exporting and importing data from and to a database.  This could greatly simplify the generation of the data escrow files as well as the consumption of the files by an EBERO provider.
  2.  XML is a highly verbose format that will adversely affect the processing of large data sets .  With the draft, XML can be used for definition and CSV can be used for data, so the duplication of the descriptive information does not have to be used for every record.
     *   If you take the domain object (<rdeDomain:domain>) example from draft-arias-noguchi-dnrd-objects-mapping-01 and convert it to CSV files (domain, dnssec, and domainTransfer) you save around 75% uncompressed and 82% compressed using CSV.
     *   Extrapolating out the uncompressed size of a <rdeDomain:domain> records (1718 bytes versus 416 bytes uncompressed per record) for XML and CSV,  you get to 1.7 GB with XML and 443 MB for CSV with 1 million records and 170 GB with XML and 44.3 GB for CSV with 100 million records.
     *   The deposits are generated uncompressed, validated, compressed and transferred by registry, and uncompressed, validated and stored by the data escrow provider.  Both the size difference and the processing resources required for both the registry and the data escrow provider should be considered when comparing the two models.
     *   EBERO providers must transfer, uncompress, validate, and import the data into their database from the data escrow deposits, where the larger the files and the processing resources required, the longer it will take to recover the TLD.
     *   The full deposit is done weekly, so it is a weekly hit for all registries, where the larger the registry the bigger the hit.

Please review the attached draft and provide any feedback for consideration.

Thanks,

--

JG

James Gould
Principal Software Engineer
[email-masked]

703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

---

From: Gavin Brown <[email-masked]>
Date: Tue, 30 Oct 2012 10:45:41 +0000

On 25/10/2012 21:43, Gould, James wrote:
> We have created a draft of the Domain Name Registration Data (DNRD)
> Comma-Separated Values (CSV) Objects Mapping that is attached for review
> and feedback.  We intend to post it to the IETF as an I-D once the
> submission page is available on November 5th.  This draft fully supports
> the Registry Data Escrow Specification
> (draft-arias-noguchi-registry-data-escrow-04).  It defines the CSV files
> and the order and format of the CSV fields for the data escrow of domain
> name, host, contact, registrar, and IDN language objects.  If there is
> interest in this model we can consider merging it with the Domain Name
> Registration Data (DNRD) Objects Mapping
> (draft-arias-noguchi-dnrd-objects-mapping-01).

Using CVS instead of XML for DNRD makes an awful lot of sense to me.
This approach looks pretty good and I support (and try to help) any
effort to develop it further.

G.

-- 
Gavin Brown
Chief Technology Officer
CentralNic Ltd
Innovative, Reliable and Flexible Registry Services
for ccTLD, gTLD and private domain name registries
https://www.centralnic.com/

CentralNic Ltd is a company registered in England and Wales with company
number 4985780. Registered Offices: 35-39 Moorgate, London, EC2R 6AR.

---

From: "Gould, James" <[email-masked]>
Date: Tue, 30 Oct 2012 15:53:35 +0000

Gavin,

Thanks for your response and your support.  What has been created thus far
is just a straw man to determine if there is interest in the use of XML
for data escrow definition and CSV for data escrow data.  I'm looking for
feedback positive and negative from others on the list for this to
determine whether or not there is adequate interest in pursuing this
further.  Please weigh in on this either privately or on the list.

Thanks,

--
  
JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

---

From: Francisco Obispo <[email-masked]>
Date: Tue, 30 Oct 2012 10:37:18 -0700

I'm not totally against it,

However, I do think that the XML file add valuable metadata that can be used to recover a registry in a standardized way. It reduces the  guess-work and also reduces the amount of work on validation of the data, since it's much easier to validate an XML file than to examine all the data by hand to see if it conforms to a specific format.

I don't see the disk usage of the large XML files as a problem. Storage is cheap, people's time on the other hand is more valuable. The incremental cost of adding an object to the registry should account for the escrow space necessary to store it.

Francisco

> effort to develop it further.

Francisco Obispo 
Director of Applications and Services - ISC
email: [email-masked]
Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
PGP KeyID = B38DB1BE

---

From: Francisco Obispo <[email-masked]>
Date: Tue, 30 Oct 2012 10:46:11 -0700

On Oct 25, 2012, at 1:43 PM, "Gould, James" <[email-masked]> wrote:

> XML is a highly verbose format that will adversely affect the processing of large data sets .  With the draft, XML can be used for definition and CSV can be used for data, so the duplication of the descriptive information does not have to be used for every record.  
> 	• If you take the domain object (<rdeDomain:domain>) example from draft-arias-noguchi-dnrd-objects-mapping-01 and convert it to CSV files (domain, dnssec, and domainTransfer) you save around 75% uncompressed and 82% compressed using CSV.  

then you loose the ability to check/validate the data.

> 	• Extrapolating out the uncompressed size of a <rdeDomain:domain> records (1718 bytes versus 416 bytes uncompressed per record) for XML and CSV,  you get to 1.7 GB with XML and 443 MB for CSV with 1 million records and 170 GB with XML and 44.3 GB for CSV with 100 million records.
> 	• The deposits are generated uncompressed, validated, compressed and transferred by registry, and uncompressed, validated and stored by the data escrow provider.  Both the size difference and the processing resources required for both the registry and the data escrow provider should be considered when comparing the two models.  
> 	• EBERO providers must transfer, uncompress, validate, and import the data into their database from the data escrow deposits, where the larger the files and the processing resources required, the longer it will take to recover the TLD.  

This is not necessarily the only way to process it. You can technically decompress on the fly and process on a stream parser and generate events to write to a database, so there's no real need to do each step separately.

> 	• The full deposit is done weekly, so it is a weekly hit for all registries, where the larger the registry the bigger the hit.  

Yes, but for most business models, the bigger the registry the more resources they have to process larger files.

Francisco Obispo

---

From: David Conrad <[email-masked]>
Date: Tue, 30 Oct 2012 10:46:51 -0700

Hi,

I suspect the main advantage of XML over something like CSV is that when it come time to add additional fields to data objects, you run the risk of having to restructure all the data.

I agree with Francisco that a focus on data size is probably misplaced: disk space is cheap and I don't think there will be a need to parse the data that frequently. 

Regards,
-drc

On Oct 30, 2012, at 10:37 AM, Francisco Obispo <[email-masked]> wrote:

> _______________________________________________
> ire mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ire

---

From: "Gould, James" <[email-masked]>
Date: Tue, 30 Oct 2012 17:49:04 +0000

Francisco, 

There is no guess work involved in the CSV draft since the XML formally
defines the format of the CSV file and a validation program can be created
that validates the CSV data against the XML definition.  Although with XML
a standard XML parser could be used for validation, programatic validation
can be implemented without the data having to be in XML.  So the data
would not be examined "by hand".

There are two different advantages with the use CSV which includes it
being a better fit for database export and import than XML and the size of
the files (uncompressed and compressed).  Whatever solution is defined for
data escrow it should scale to medium to very large deposits, which is the
case for the use of CSV for data.  One of the goals should be to minimize
the time, cost, and complexity in producing and consuming the deposits,
where the use of CSV meets this goal better than XML.

--
  
JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 10/30/12 1:37 PM, "Francisco Obispo" <[email-masked]> wrote:

>However, I do think that the XML file add valuable metadata that can be
>used to recover a registry in a standardized way. It reduces the
>guess-work and also reduces the amount of work on validation of the data,
>since it's much easier to validate an XML file than to examine all the
>data by hand to see if it conforms to a specific format.
>
>I don't see the disk usage of the large XML files as a problem. Storage
>is cheap, people's time on the other hand is more valuable. The
>incremental cost of adding an object to the registry should account for
>the escrow space necessary to store it.
>
>Francisco
>
>

---

From: "Gould, James" <[email-masked]>
Date: Tue, 30 Oct 2012 17:52:44 +0000

Francisco,

My feedback is included below.

--
  
JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 10/30/12 1:46 PM, "Francisco Obispo" <[email-masked]> wrote:

>> XML is a highly verbose format that will adversely affect the
>>processing of large data sets .  With the draft, XML can be used for
>>definition and CSV can be used for data, so the duplication of the
>>descriptive information does not have to be used for every record.
>> 	€ If you take the domain object (<rdeDomain:domain>) example from
>>draft-arias-noguchi-dnrd-objects-mapping-01 and convert it to CSV files
>>(domain, dnssec, and domainTransfer) you save around 75% uncompressed
>>and 82% compressed using CSV.
>

JG - You don't lose the ability to check/validate the data.  You can't use
a standard XML parser, but a validation program can be created that
validates the CSV data against the XML definition.  That should not be
overly challenging.

>
>
>> 	€ Extrapolating out the uncompressed size of a <rdeDomain:domain>
>>records (1718 bytes versus 416 bytes uncompressed per record) for XML
>>and CSV,  you get to 1.7 GB with XML and 443 MB for CSV with 1 million
>>records and 170 GB with XML and 44.3 GB for CSV with 100 million records.
>> 	€ The deposits are generated uncompressed, validated, compressed and
>>transferred by registry, and uncompressed, validated and stored by the
>>data escrow provider.  Both the size difference and the processing
>>resources required for both the registry and the data escrow provider
>>should be considered when comparing the two models.
>> 	€ EBERO providers must transfer, uncompress, validate, and import the
>>data into their database from the data escrow deposits, where the larger
>>the files and the processing resources required, the longer it will take
>>to recover the TLD.
>
>This is not necessarily the only way to process it. You can technically
>decompress on the fly and process on a stream parser and generate events
>to write to a database, so there's no real need to do each step
>separately.
>
>> 	€ The full deposit is done weekly, so it is a weekly hit for all
>>registries, where the larger the registry the bigger the hit.
>
>Yes, but for most business models, the bigger the registry the more
>resources they have to process larger files.

JG - The approach used should easily scale to very large deposits for
creation and consumption independent of the available resources of the
various parties involved.

---

From: "Gould, James" <[email-masked]>
Date: Tue, 30 Oct 2012 18:00:58 +0000

David,

There is no difference between the use of XML or CSV for data when it
comes to changes.  The definition of each discrete deposit (XML) along
with the data (CSV) is provided together so if there is a new field it
would be included in a new combined deposit (definition and data).  If
there is a new field needed in XML a new deposit would include the new
definition (extension) along with the data associated with the extension.
I believe in both cases they are extensible.

The size is just one advantage of the use of CSV over XML, but why use
verbose XML for a large data deposit if you don't need to?  Correct me if
I'm wrong but I believe that the data escrow provider is supposed to
validate the deposits, meaning that the data would be decompressed (in
bulk or inline) and validated with both approaches regularly.  It also
makes sense for the party creating the deposit to validate it prior to
sending it.   

--
  
JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 10/30/12 1:46 PM, "David Conrad" <[email-masked]> wrote:

>Hi,
>
>I suspect the main advantage of XML over something like CSV is that when
>it come time to add additional fields to data objects, you run the risk
>of having to restructure all the data.
>
>I agree with Francisco that a focus on data size is probably misplaced:
>disk space is cheap and I don't think there will be a need to parse the
>data that frequently.
>
>Regards,
>-drc
>
>>guess-work and also reduces the amount of work on validation of the
>>data, since it's much easier to validate an XML file than to examine all
>>the data by hand to see if it conforms to a specific format.
>>

---

From: David Morris <[email-masked]>
Date: Tue, 30 Oct 2012 11:24:48 -0700 (PDT)

In addition to storage size, which is probably a minimal concern,
time on the wire for transfer and processing to create and parse the
information are concerns.

While conceptually, adding a field to XML seems simple, the processing
code changes are much more complex than those needed to process CSV
style data.

I would also observe, the TAB delimited is substantially better than
comma delimited as for most data sets, the TAB character can be excluded
from the set of valid data characters. Comma delimited generally requires
escaping rules.

---

From: "Gould, James" <[email-masked]>
Date: Tue, 30 Oct 2012 18:30:34 +0000

David,

The draft supports defining non-comma delimeters in the XML definition, so
it can be TAB delimited.  The default delimiter could be changed from a
comma.  

--
  
JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

---

From: John Boen <[email-masked]>
Date: Tue, 30 Oct 2012 11:31:30 -0700

One man's opinion...

I think the real consideration should be the amount of time it would take to get a registry up and running from an escrow file.  This is largely dependent upon how many skills are necessary to take a file of a particular specification, and import it into a registry system.  I think I could readily take a set of tabular data specified like this and do it without assistance.  I'm also sure I could unpack an XML document into various tabular files.

The work to transform the data into a registry schema shouldn't be that hard, but If I needed to perform an additional step of parsing the XML, it would take longer.  If we were performing wrok a multi-million domain environment, I think the XML document would be particularly unwieldy.

I consider the value by placing it into XML to be dubious; the work to transform it into XML (and ultimately back into row sets so it can be imported into a new registry environment) seems unnecessary.

If it were my decision alone, I would specify tabular data instead of XML.

John Boen
Database Engineer, Demand Media.

Please NOTE: This electronic message, including any attachments, may include privileged, confidential and/or inside information owned by Demand Media, Inc. Any distribution or use of this communication by anyone other than the intended recipient(s) is strictly prohibited and may be unlawful.  If you are not the intended recipient, please notify the sender by replying to this message and then delete it from your system. Thank you.

---

From: Francisco Obispo <[email-masked]>
Date: Tue, 30 Oct 2012 12:08:57 -0700

Hi John,

My response below:

> I think the real consideration should be the amount of time it would take to get a registry up and running from an escrow file.  This is largely dependent upon how many skills are necessary to take…

The CSV file. regardless of the field delimiter (tab, comma, ###, etc..) just represent "data", nothing else. To insert this data into a registry schema, you will need to validate it, regardless whether that's done by the XML parser itself, the program or the database.  Both approaches are "valid", the first one is just a matter of passing a XSD (XML Schema Definition - already provided by arias-draft) to the parser, the other one requires the programmer to go through every single column and checking for the proper type.

> The work to transform the data into a registry schema shouldn't be that hard, but If I needed to perform an additional step of parsing the XML, it would take longer. If we were performing wrok a mu…

It is not hard as long as the CSV is well formed and built according to the specs, if it's not, you'll end up with a bunch of garbage that needs to be fixed by hand (or by scripts most likely), so that it can later be inserted. 

The main problem is thinking that the XML will be parsed as a single data structure (like an config file for example). For large documents you will need a Stream parser (SAX, Expat, Apache Xerces, etc), that can handle large documents and separate them into parsing "events" which will ultimately become inserts into a database, etc.

> I consider the value by placing it into XML to be dubious; the work to transform it into XML (and ultimately back into row sets so it can be imported into a new registry environment) seems unnecess…

you don't need to transform the XML, the XML needs to be parsed and processed according to XML rules, worst case scenario, you can always have an XSLT to convert the XML to CSV if that's easier for you, however, the reverse it's usually more difficult (at least in generic terms).

> If it were my decision alone, I would specify tabular data instead of XML.

Perhaps we can all collaborate and write an Open Source suite to both generate and parse DNRD data? (in both CSV and XML)

Francisco Obispo

---

From: John Boen <[email-masked]>
Date: Tue, 30 Oct 2012 14:13:00 -0700

Thanks Francisco.

The steps to extract, transform to XML, and subsequently transform and import the data into a new system will necessarily be greater in complexity over just extracting data to a known tabular format (which could have data types, uniqueness, and cardinality specified as well) and importing.  I recognize a personal bias that comes from doing 20 years of database-specific work, but the logic is still sound.  It's more complicated to add XML into the mix - even if my personal bias is not considered.  I view the potential for additional transformation related bugs to be higher than the value added by placing it into XML.

That being said, we have a spec that people are working to, and different teams are at varying completeness along this path.  From a team perspective, I'd suggest we not change direction now.

// Perhaps we can all collaborate and write an Open Source suite to both generate and parse DNRD data? (in both CSV and XML)

I'll pass that idea around the team.

-----Original Message-----
From: Francisco Obispo [mailto:[email-masked]]
Sent: Tuesday, October 30, 2012 12:09 PM
To: John Boen
Cc: Gould, James; Gavin Brown; Thippeswamy, Chethan; [email-masked]
Subject: Re: [ire] DNRD CSV Draft

Hi John,

My response below:

Francisco Obispo
Director of Applications and Services - ISC
email: [email-masked]
Phone: +1 650 423 1374 || INOC-DBA *3557* NOC PGP KeyID = B38DB1BE

---

From: Gavin Brown <[email-masked]>
Date: Tue, 30 Oct 2012 21:18:40 +0000

> Perhaps we can all collaborate and write an Open Source suite to both generate and parse DNRD data? (in both CSV and XML)

+1 - we're doing this already (we have both import and export tools) and
would be very interested in contributing.

G.

---

From: Chris Wright <[email-masked]>
Date: Wed, 31 Oct 2012 10:21:48 +1100

Hey all, 

Lots to say on this topic, so apologies for the long email. Ill attempt to put in bullet point form to keep it short - please forgive me if they sound terse, just trying to be brief

 - We prefer CSV over XML, mainly because CSVs can be loaded into a database easier (external tables, SQL loader etc) - note we have applied to be an EBERO (so a consumer of the escrow data as well as a producer)
 - We are not convinced about the XML definition of fields, and other parts in the CSV proposal, however think CSV over XML is a better choice - more on that below
 - I think that realistically all we should care about is the format of the data that an escrow agent will provide to ICANN when asked - how the escrow provider gets the data from the registry is irrelevant - if I want to use database replication to the escrow provider and then provide him scripts to dump the data from my database to CSV then so be it, if I want to give full dumps all the time then I should be able to, if I want to send the data 24 times a day, then I should be able to, ICANN should only be mandating a maximum age for the data and the format that it is required to be given to ICANN in when they request it
 - this group should draft that format with input from registries and ICANN

- If this format must be used by registries to provide data to the escrow providers then:
     - We need to collect requirements form registries about the format, things such as:
           - it should be fast to generate
           - compressible so that there is not a lot of data to transfer on the wire
           - etc.
     - It needs to support all the different data models various registries have (so be extensible)

- Then we need to hear from ICANN about what their requirements are - which I believe are - be able to restore a registry onto another provider - but we need more detail than that, like, what happens to object pending transfer? What's the expectations around grace periods? Etc - I believe some of this is captured in the 'intention' of the current XML drafts produced, mainly, by ICANN - but I would prefer it to be more explicit. ICANN should develop these in conjunction with the EBERO providers who will be the ones actually tasked with reconstructing a registry from this data. ICANN probably needs to define some validation that the escrow provider needs to do on the data it receives each period (which at the moment I believe is daily).

As above, I think that really only ICANNs requirements need to matter, and that how the data gets to the Escrow provider, should be a matter between the provider and the registry, why does that need to be standardised at all, if I have an innovative way of doing it that saves cost, then why shouldn't I be allowed to do it, if it provides ICANN with all the data it needs. 

Thanks

Chris Wright
Chief Technology Officer
AusRegistry International Pty Ltd
Level 8, 10 Queens Road
Melbourne. Victoria, Australia. 3004
Phone: +61 3 9866 1990
Fax: +61 3 9866 1970
Mobile: +61 401 873 798
Email: [email-masked]
Web: www.ausregistry.com

The information contained in this communication is intended for the named recipients only. It is subject to copyright and may contain legally privileged and confidential information and if you are not an intended recipient you must not use, copy, distribute or take any action in reliance on it. If you have received this communication in error, please delete all copies from your system and notify us immediately.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Francisco Obispo
Sent: Wednesday, 31 October 2012 6:09 AM
To: John Boen
Cc: [email-masked]; Thippeswamy, Chethan
Subject: Re: [ire] DNRD CSV Draft

Hi John,

My response below:

Francisco Obispo

---

From: Francisco Obispo <[email-masked]>
Date: Tue, 30 Oct 2012 21:50:06 -0700

Hi Chris,

I'm  not sure if what you're stating here is exactly how ICANN is envisioning the system to work.

As far as I know, the Escrow provider is not going to process the data uploaded to it, they are just going to validate it's signature and store it until ICANN authorizes an EBERO or a new registry for a transition.. It is meant to be an escrow, not a registry.

Replica sets, backups, etc. is under the responsibility of the registry operator, not on the escrow provider, ICANN has to specify a format for the data, because I'm sure that if we were to disclose our registry schemas they differ significantly from each other.

The XML format is based on the fields specified on the EPP protocol specification, its objects and extensions, based on the Applicant's Guidebook requirements and even with an extension mechanism to map EPP extension data.

> - I think that realistically all we should care about is the format of the data that an escrow agent will provide to ICANN when asked - how the escrow provider gets the data from the registry is ir…

ICANN is mandating daily incremental deposits and full deposits on a weekly basis (sundays), refer to Applicant's Guidebook Specification 2, Part A, sections 1, 1.1 and 1.2

If they don't mandate how the data is to be formatted, everyone will end up representing it based on their understanding on what's required and thus, not useful in an emergency situation where you need to rapidly recover a registry.

> - this group should draft that format with input from registries and ICANN

The draft has been under discussion in the IETF, it has input from at least 2 registry operators, and it has been referenced from the applicants guidebook. I know everyone has been really busy with answering the technical questions, building the registries, etc. but going forward we should ask Francisco Arias (ICANN) about the best mailing list to engage on the discussions. 

The work that Francisco Arias has done on it as well as the other authors is remarkable.

I personally don't think that handling multiple gigabytes of data per day on a very large registry should be a problem. The draft also has an incremental dump which is meant to only provide the deltas between the last export and the current one, so the multiple gigabyte will only be a problem once a week, not necessarily on a daily basis.

Best regards,

Francisco Obispo

---

From: Chris Wright <[email-masked]>
Date: Wed, 31 Oct 2012 16:01:16 +1100

Thanks Francisco,

I think you have misunderstood the point. I am in no way saying that the escrow provider should be a registry.

Escrow is only required if the registry needs to be rebuilt somehow, thus ICANN only require the data in their specified format when they need it to rebuild the registry - so we have to ensure that the escrow provider can provide the data to ICANN in the format required if ever asked. I am not saying there shouldn’t be a format, but the format that is required is between the escrow provider and ICANN not anywhere else.

Why does that have to mean that I have to provide the data to my escrow provider in that format? Surely I can provide it anyway I please, as long as they have a way to provide it to ICANN in the format ICANN require, when/if ICANN ever ask? So as long as I have a process in place with my provider that can achieve that (which could be send the data to them in that format, send them a binary dump of my database and script that could be run to export it, or any other method) why does ICANN care? Surely that is a matter for me to resolve with my escrow provider in a way that works for me?

Also I know where the incremental 'requirements' come from, I am saying why should I go to the effort of doing incrementals if my registry only has 2000 domains in it? Why am I forced to do incrementals? I should just be allowed to do a full dump each day, how does that hurt anyone? In fact it makes things easier for whomever would have recover my registry later as they just load in the one dump and be done with it, they don’t have to try and write logic to interpret incrementals.

Thanks

c.

-----Original Message-----
From: Francisco Obispo [mailto:[email-masked]] 
Sent: Wednesday, 31 October 2012 3:50 PM
To: Chris Wright
Cc: John Boen; James Gould; [email-masked]; Thippeswamy, Chethan; Francisco Arias
Subject: Re: [ire] DNRD CSV Draft

Hi Chris,

On Oct 30, 2012, at 4:21 PM, Chris Wright <[email-masked]> wrote:

> - I think that realistically all we should care about is the format of 
> the data that an escrow agent will provide to ICANN when asked - how 
> the escrow provider gets the data from the registry is irrelevant - if 
> I want to use database replication to the escrow provider and then 
> provide him scripts to dump the data from my database to CSV then so 
> be it, if I want to give full dumps all the time then I should be able 
> to, if I want to send the data 24 times a day, then I should be able 
> to, ICANN should only be mandating a maximum age for the data and the 
> format that it is required to be given to ICANN in when they request 
> it

Best regards,

Francisco Obispo

---

From: Francisco Obispo <[email-masked]>
Date: Tue, 30 Oct 2012 22:11:38 -0700

That was exactly my feedback to ICANN ;-)

In fact, we decided that we were going to go for the full dumps every day, because the process of applying the deltas could really mess up things if not done right.

The feedback that I got was that as long as we provide something "better" than what is required, it should then be OK.

I see the point on the escrow provider, and I agree with you, it would be great if they offered that service, we would just set up a replica and let them do the dump on a daily or weekly basis.

But I have yet to find one that would do that.. (I'll talk to the one we chose to see if they are willing to do it)

Francisco Obispo

---

From: "Gould, James" <[email-masked]>
Date: Wed, 31 Oct 2012 18:01:12 +0000

Chris,

An additional step that the data escrow provider would need to perform per
deposit is to transform and validate the data in the standard format if
the deposit is provided in a non-standard format.  Having an agreement
with the data escrow provider to take data in one format to be transformed
to a standard format when needed can not be tested only once, but with
each deposit.  Having a standard format that supports validation by
multiple parties (registry, data escrow provider, EBERO provider), that
doesn't require transformation by the data escrow or EBERO provider, and
that scales makes sense to me.  Supporting more than just full deposits in
draft-arias-noguchi-registry-data-escrow is essential to scale to large
registries, but you are correct that for very small registries (e.g. 2000
domains) incremental deposits doesn't add much value.

What is your issue with providing XML definition of the CSV fields in the
draft?  The draft that we created is similar to the approach taken for
Fielded Text by formally defining the CSV fields.  In this case the
definition complies with draft-arias-noguchi-registry-data-escrow and
utilizes XSD elements to formally define the field format.  Do you have a
different approach that should be considered?  Do you have any specific
feedback to improve the draft?

Thanks,

--
  
JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 10/31/12 1:11 AM, "Francisco Obispo" <[email-masked]> wrote:

>In fact, we decided that we were going to go for the full dumps every
>day, because the process of applying the deltas could really mess up
>things if not done right.
>
>The feedback that I got was that as long as we provide something "better"
>than what is required, it should then be OK.
>
>I see the point on the escrow provider, and I agree with you, it would be
>great if they offered that service, we would just set up a replica and
>let them do the dump on a daily or weekly basis.
>
>But I have yet to find one that would do that.. (I'll talk to the one we
>chose to see if they are willing to do it)
>
>> Also I know where the incremental 'requirements' come from, I am saying
>>why should I go to the effort of doing incrementals if my registry only
>>has 2000 domains in it? Why am I forced to do incrementals? I should
>>just be allowed to do a full dump each day, how does that hurt anyone?
>>In fact it makes things easier for whomever would have recover my
>>registry later as they just load in the one dump and be done with it,
>>they don¹t have to try and write logic to interpret incrementals.
>> 
>> Thanks
>
>Francisco Obispo

---

From: Chris Wright <[email-masked]>
Date: Thu, 1 Nov 2012 13:17:21 +1100

On 1/11/12 5:01 AM, "Gould, James" <[email-masked]> wrote:

>that scales makes sense to me.

Understood, and if that’s the way Verisign wants to do it, then go for it,
you can deposit it in the format required to be provided to ICANN, no
issue what so ever, but why should everyone be forced to do that? It would
actually be really trivial for both me and my provider to do database
replication, and then they simply run a script to validate the data (that
I would supply them) - Im yet to actually see what the 'validation' they
are supposed to do actually is - if its just object counts - then I would
assert that dumping the data doesn't even really need to be done, but even
it it does, they run the dump scripts and get the data in the format
needed - not hard at all and completely automated.

I believe we should be defining the format of that the data is required to
be supplied by the Escrow provider to the EBERO, via ICANN and that’s it.
ICANN should define the maximum age of that data (which I believe is meant
to be no more than 24 hours). Then for pre-del testing ICANN can test this
by creating objects in the test registry, then 24 hours later asking for
the data from the Escrow provider and verifying it meets the format -
simple. How the Escrow provider gets this data to give to ICANN and
everything else is for the free market to solve - there is no justifiable
reason for ICANN to dictate any of it - all ICANN should concern
themselves with is the requirements outcomes.

>  Supporting more than just full deposits in

I do not agree that incremental deposites as defined is essential to large
registries. Depending on your technology doing full dumps may be quicker
and less code than incrementals. I may be doing DB replication which
doesn't involve 'deposites' at all - in fact DB replication is probably
easier for larger registries (if done right) - but realistically, this is
my point, each registry can solve this problem there own way - don't
dictate to me how to do it - just set the requirements. Which as I have
said above really are only 2:

 - the escrow provide must be able to supply to ICANN a complete copy of
all data in a registry (ICANN to define all data) to ICANN in a format to
be defined
 - this data should be no more than 24 hours old

How we achieve this is up to us.

>feedback to improve the draft?

Im wondering is a machine would ever read the field format? It seems to me
that a human (probably working for the EBERO) would be reading the field
formats when the trying to 'load' the data into the EBERO registry. I can
see anyone actually doing the work of writing code that parses the field
format from the XML and then tries to 'interpret' the CSV in such a way.

Surely there is a list of fixed fields which everyone must included and
these can be in a defined order, then we just need to deal with
extensions, which should probably be different files.

I see CSVs like this

Domains - the base details of all domains, that all registries have (roid,
domain name, expiry date, registrant contact ... Etc)
DomainNameServers - a 2 column CSV mapping domain ROIDs from the domains
file, to host roids in the host file (domain roid, host roid)
DomainContacts - a 3 column CSV mapping domain ROIDs, to contact ROIDs and
contact types eg. billing, tech, admin (domain roid, contact roid, contact
type)
DomainVariants - a 2 column CSV mapping domain ROIDs, to the provisioned
variants (roid, variant domain name)
Hosts - the base details of hosts (roid, host name ... Etc)
Contacts - the base details of contact (roid, loc street1, loc street2,
int street 1, int street 2 etc)
Registrar - the base details of registrars
DomainDNSSEC etc
Etc

Then for extensions each registry needs to define the format of the CSV in
documentation, I don't think it can be done pragmatically.

DomainExtension1 - a file that maps the domain roid to the extension
details (roid, extfield1, extfield2 etc...)

I have outstanding 'problems' which exist in all models:

For IDNs, I don’t want to have to escrow all blocked variants (I don't
generate them I use a unique key mapping form) - so how do we
'communicate' this to the EBERO, I think we need to deal with language
tables
DNSSEC - how do we transition this? (un-sign?)
And it goes on - topic of another email

Thanks for taking the time to read to the end.

Chris

---

From: Francisco Obispo <[email-masked]>
Date: Wed, 31 Oct 2012 20:39:04 -0700

Hi Chris,

Perhaps I'm reiterating what I said in my previous email, but I don't think that's how the escrow provider is supposed to work.

They are going to receive, verify and store an escrow dump from the registry. Verify means going through the items in Specification 2 of the Applicant's guidebook: http://archive.icann.org/en/topics/new-gtlds/rfp-clean-30may11-en.pdf

<specification2_extract>
Verification Procedure.

	• (1)  The signature file of each processed file is validated.

	• (2)  If processed files are pieces of a bigger file, the latter is put together.

	• (3)  Each file obtained in the previous step is then decrypted and uncompressed.

	• (4)  Each data file contained in the previous step is then validated against the format defined in [1].

	• (5)  If [1] includes a verification process, that will be applied at this step.

If any discrepancy is found in any of the steps, the Deposit will be considered incomplete.

[1] Domain Name Data Escrow Specification (work in progress), http://tools.ietf.org/html/draft-arias-noguchi-registry-data-escrow
[2] OpenPGP Message Format, http://www.rfc-editor.org/rfc/rfc4880.txt
[3] OpenPGP parameters, http://www.iana.org/assignments/pgp-parameters/pgp-parameters.xhtml

</specification2_extract>

The validation is an XML Schema Validation based on the IETF Draft "arias-noguchi". By passing XML Validation (which is a very simple check), the Escrow provider can be sure that the data not only is complete, but that the fields conform the data types specified by the spec.

Validating a dump wouldn't be more than just: 

$ xmllint --noout --sax --schema /path/to/arias-noguchi.xsd escrow_dump.xml

and if the file is compressed:

$ bzcat escrow_dump.xml.bz2 | xmllint --noout --sax --schema /path/to/arias-noguchi.xsd -

(that's an example using libxml2)

Having an CSV file would require the escrow provider write their own parser to validate comma separated values.. This is very dangerous, and time consuming, depending on how they do it, specially if they have to cope with custom formats from different registries.

If I was an escrow provider I wouldn't want to be liable in generating the dump file, because if something is missing, and the registry goes away, or a DELETE, TRUNCATE TABLE, DROP DATABASE, or anything like it happens and the replica throws the data away, there will be nowhere to fetch the data from (backups perhaps?)

On the variants issue, this would be part of the extension framework, and since this varies from registry to registry, it would be something that it would be have to be mapped and documented accordingly.

Best regards,

Francisco

Francisco Obispo

---

From: Chris Wright <[email-masked]>
Date: Thu, 1 Nov 2012 15:55:41 +1100

Thanks Francisco,

I 100% agree with what your saying, from the perspective of that is what
the agreement currently says.

However I am arguing that the requirements from that agreement are silly
and flawed, and that if we the technical community go back to ICANN and
tell them as such we can get that addressed.

It makes no sense to enforce a technical process like this in a legal
agreement, including down to the level of how to name files. All ICANN
needs to care about is how they get the data from the escrow provider.

The validation currently asked for by the legal agreement, which as you
rightly point our is trivial to do, is also almost pointless, schema
validation proves nothing except that I submitted schema valid XML, I
could fill schema valid XML with anything - it will not meet the
objectives of ensuring data is true and accurate based on what is actually
in my registry.

Im fine with the escrow providing having to write custom stuff to validate
data, they will need to to do 'real' validation, however again I wouldn't
go to that level from ICANN perspective, I would specify a set of things
the escrow provider needs to validate, and leave the 'how' to them.

If we are serious about validation we should validate things like,

Ensure every contact object referenced by a domain object is present
Ensure every host object reference by a domain object is present
Ensure that every sponsor referenced by domains, contacts and host is
present as a registrar object
Ensure every contact country code matches a country code on the ISO list

On and on...
Etc.. - we probably want to apply some checks to object properties as well
(e.g. Check email address are valid (syntactically))

Perhaps statistical analysis of object counts could be done, compared to
previous counts and anomalies investigated (if one day I have a million
domains, and then the next day I have 100,000 then the escrow provider
should look deeper into the data and have a discussion with the registry
to work out what went wrong, or if the massive drop is legit) - Escrow
providers should be more than just file repositories, otherwise they add
no value whatsoever, and ICANN might as well save us all some money and
just setup a central FTP server we all send the dump files to.

The validation we could(should?) do/expect goes on and on, only then will
we actually be more confident that we have better data - the thing is to
do these types of validation, you would need to reintegrate an incremental
file with a full file (as the incremental file may reference objects from
the full dump (or a previous incremental) so the escrow provider will need
to do more than schema validate a file anyway.

Obviously if the method I worked out between me and my Escrow provider was
database replication, they would need to ensure that if I replicated a
'drop database' statement to them, they would still have the data from the
previous replication, but again, that is a technical problem for the
escrow provider to work out with the registry, and I can think of many
technical ways to solve that problem.

The escrow provider will have a contractual requirement to provide data to
ICANN (in a format provided by ICANN - who will design the format in
conjunction with the technical community) that is no more than 24 hours
old, they will need to technically work out a way they can fulfil on that
obligation with the registry operator - that¹s all ICANN needs to day. The
escrow provider then needs a process where if they are not receiving the
data from the registry operator in a timely manner (thus not able to
fulfil on their obligation) or the registry operator is not doing
something they need to do, then they escalate this issue to ICANN who will
go the registry for contract breach - that¹s it. This situation exist if I
stop sending daily full or incremental files anyway.

Thanks

Chris

On 1/11/12 2:39 PM, "Francisco Obispo" <[email-masked]> wrote:

>Hi Chris,
>
>Perhaps I'm reiterating what I said in my previous email, but I don't
>think that's how the escrow provider is supposed to work.
>
>They are going to receive, verify and store an escrow dump from the
>registry. Verify means going through the items in Specification 2 of the
>Applicant's guidebook:
>http://archive.icann.org/en/topics/new-gtlds/rfp-clean-30may11-en.pdf
>
><specification2_extract>
>Verification Procedure.
>
>	€ (1)  The signature file of each processed file is validated.
>
>	€ (2)  If processed files are pieces of a bigger file, the latter is put
>together.
>
>	€ (3)  Each file obtained in the previous step is then decrypted and
>uncompressed.
>
>	€ (4)  Each data file contained in the previous step is then validated
>against the format defined in [1].
>
>	€ (5)  If [1] includes a verification process, that will be applied at
>this step.
>
>[1] Domain Name Data Escrow Specification (work in progress),
>http://tools.ietf.org/html/draft-arias-noguchi-registry-data-escrow
>[3] OpenPGP parameters,
>http://www.iana.org/assignments/pgp-parameters/pgp-parameters.xhtml
>
>The validation is an XML Schema Validation based on the IETF Draft
>"arias-noguchi". By passing XML Validation (which is a very simple
>check), the Escrow provider can be sure that the data not only is
>complete, but that the fields conform the data types specified by the
>spec.
>
>Having an CSV file would require the escrow provider write their own
>parser to validate comma separated values.. This is very dangerous, and
>time consuming, depending on how they do it, specially if they have to
>cope with custom formats from different registries.
>
>If I was an escrow provider I wouldn't want to be liable in generating
>the dump file, because if something is missing, and the registry goes
>away, or a DELETE, TRUNCATE TABLE, DROP DATABASE, or anything like it
>happens and the replica throws the data away, there will be nowhere to
>fetch the data from (backups perhaps?)
>
>On the variants issue, this would be part of the extension framework, and
>since this varies from registry to registry, it would be something that
>it would be have to be mapped and documented accordingly.
>
>Best regards,
>
>Francisco
>
>> Understood, and if that¹s the way Verisign wants to do it, then go for
>>it,