Subject: Re: [ire] DNRD CSV Draft

From: "Gould, James" <[email-masked]>
Date: Fri, 2 Nov 2012 09:59:06 +0000

Christopher,

Thank you for your detailed feedback.  I have the following clarification
comments and questions to your feedback below prefixed with "JG -".

--
  
JG

James Gould
Principal Software Engineer
[email-masked]
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 11/1/12 4:52 PM, "Christopher Browne" <[email-masked]> wrote:

>On Thu, Oct 25, 2012 at 4:43 PM, Gould, James <[email-masked]> wrote:
>> We have created a draft of the Domain Name Registration Data (DNRD)
>>Comma-Separated Values (CSV) Objects Mapping that is attached for review
>>and feedback.
>
>Preface: Sorry, my comments are pretty long.  And I reference
>footnotes considerably, as well as various later messages in the
>thread, to date.
>
>I'm finding myself considerably torn in all of this.
>
>On the one hand, I'm not a big fan of XML, am not particularly keen on
>the large size attendant to that.
>
>On the other hand, I see a few problems attendant with a CSV-based
>representation, which should be pointed out:
>
>- CSV is not a single, well-defined format (see: [1]), although there
>  is a relevant RFC [2].  We can say we'll be strict, but a hostile
>  reader might disagree.

JG - You are correct that CSV is not a terse standard like XML but it is a
defacto standard for database export and import.  I'm not sure if a
hostile reader would disagree that CSV is used for this purpose
extensively today. 

>
>- It implies a "flat" representation of information, and is not easily
>  amenable to extension.  Unlike with XML, there is not a built-in way
>  to introduce additional attributes and metadata to indicate such.

JG - Why is a flat representation not amenable to extension?  It is
correct that you need to use a relational model (one file per record type)
instead of an object oriented model (XML) to represent the data with CSV.
Since the source and the target databases are most likely relational it
makes sense to stick with a flat relational model.  Additional fields can
be easily added to the CSV field definition.  An EBERO provider would
understand specific standard fields and could support custom fields as
well.  

>
>That being said, there was an early IRE draft (which appears to have
>been lost?  [3] references unreferenceable document [4]), which I had
>some hand in, which took very much the approach that Chris Wright
>describes in [5], with a series of CSV files:

JG - I haven't seen draft that you're referring to, but I would be very
interested in reviewing it.

>
> - Domains
> - Contacts
> - Nameservers
> - Registrars
> - Nameserver IP addresses
> - Associations:
>   - Domain Statuses
>   - Contact Statuses
>   - Nameserver Statuses
>   - Domain/Nameserver Associations
>   - Domain/Contact Associations
> - Deletions
>   - Domains
>   - Contacts
>   - Nameservers
>
>An XML representation can readily directly express hierarchy; the
>above approach remaps that into series of "flattened structures."
>
>Onwards to comment on the Gould/Thippeswamy[6] document...  That
>document offers a structure largely analogous to the above, although
>it draws certain of the hierarchies into the objects.
>
>In particular, the 2010-01-25 proposal [4] separated object statuses
>into separate association files, whereas [6] has several multivalued
>attributes.
>
> - On Domains, to indicate
>   - Statuses (regular and RGP)
>   - Nameserver associations
>   - Contact associations
> - On hosts, to indicate
>   - IP addresses

JG - Yes, it was decided in draft-gould-thippeswamy-dnrd-csv-mapping to
support lists (not structures) in the CSV files.  If this is not desired
we can consider flattening it further to not support lists as well within
the CSV.  What are the thoughts to this?  I believe most of the feedback
below is related to supporting lists within the CSV files.  Let me know if
this is not the case.  The first draft is truly a straw man that can be
changed (even drastically) based on feedback from the list.

>
>There are arguments both pro- and con- either way:
>
> - Melding the associations into the object streams means there can
>   be no mismatches between those streams.
>
>   - In [6], you can be certain that the domain will have all of its
>     statuses, whereas
>
>   - in [4], the representation makes it possible to have entries in
>     the *Domain Status* stream that don't correspond to an existing
>     domain
>
> - On the other hand, [6] tortures the somewhat dodgy format
>   definition of CSV a bit harder, treating it, in effect as a
>   MultiValue Database[7]
>
> - Vis-a-vis contacts, the multivalue handling of [6] makes the
>   format less flexible to cope to cope with potential future need
>   for additional sorts of contacts, which would be more easily
>   handled if contact associations were handled as in [3], which is
>   trivial extensible as in the following "domain/contact
>   association" example:
>   | Domain   | Contact Handle | Contact Type |
>   |----------+----------------+--------------|
>   | abc.test | C-123          | Registrant   |
>   | abc.test | C-124          | Admin        |
>   | abc.test | C-125          | Billing      |
>   | abc.test | C-126          | Technical    |
>   | abc.test | C-127          | Anti-Abuse   |
>   | abc.test | C-128          | Extension #2 |
>
>   Introducing extra contact types is pretty trivial in this
>   [3]-based approach; in [6], it would require either another data
>   stream, or documenting an out-of-band way of indicating
>   additional columns for the Domains data stream.
>
>   The thing I want to point out here is /not/ that this implies
>   that [3] is superior to [6], but rather that representation
>   choices like this one will have considerable impact on how easy
>   or difficult later extension will be.  /That/ is a broadly
>   important issue.
>
>Something I find very interesting about [6] is in its use of XML to
>describe the data.  Francisco comments [8] on the validation process
>that he anticipates being used to validate escrow; it seems to me that
>the draft of [6] provides an interesting addition to this.  The XML
>schema that is described therein could be used, hopefully pretty
>directly, to do a well-formedness validation on the series of CSV
>files.
>
>For that to be possible implies that the "base" usage of [6] needs to
>be kept quite strict.  Adding additional fields would prevent using
>the XML schema directly, and perhaps it means that it would be
>necessary to declare that some set of functionality be declared a
>"core", and that extensions /must not/ alter anything in this core.  I
>wouldn't consider it a terrible thing to have a near-immutable "core,"
>but that certainly has implications for extensions, that they need to
>be separated.

JG - draft-gould-thippeswamy-dnrd-csv-mapping can support custom fields
via the <rdeCsv:fCustom> field with a required "name" attribute or a
custom set of fields can be defined with a custom XSD and referenced
within the XML field definition.  Is it not clear how
draft-gould-thippeswamy-dnrd-csv-mapping can be extended, since there is
not the intent to make it near-immutable for the deposits since we fully
understand that there different registries can and will contain custom
objects and custom attributes.

>
>For instance, if we took [6] directly as the "core", then adding an
>anti-abuse contact could not be done by adding it as an extra field,
>as that would violate the immutability of the "core."  Adding an
>additional data stream to associate in "Anti-Abuse" contacts would be
>perfectly straightforward, albeit with the challenge that /validating/
>this extension requires some new validation component downstream.
>(The approach of [3] is not inherently better in this regard - while
>makes it easier to mix the new contact type into the existing data
>streams, there is still a need to validate contact types.)

JG - Independent of the support for lists in CSV, a custom XSD can be
defined like an XSD with the namespace prefix csvCustom that defines an
Anti-Abuse contacts field as shown below.  You can also use the
<rdeCsv:fCustom> field with the required "name" attribute to define a
custom field that is of XSD token type.

<element name="fAntiAbuseContacts" type="rdeCsv:fContactsType"
substitutionGroup="rdeCsv:field">

This can be included in the <rdeCsv:fields> element and will have full XSD
format information included with it:

...
<rdeCsv:fields>
   <rdeCsv:fAdminContacts/>
   <rdeCsv:fTechContacts/>
   <rdeCsv:fBillingContacts/>
   <rdeCustom:fAntiAbuseContacts/>
   Š
</rdeCsv:fields>
Š

Does this address you extensibility concern?

>
>But none of this quite gets at an opinion as to whether we should go
>with the XML form (most recently laid out in version 3, [9]) or the
>Verisign proposal [6].
>
>As I already said, I'm torn.
>
>I am not thrilled at the verbosity of XML, for much the same reasons
>already laid out by others.
>
>And I have had decent success at drawing data into registries using
>"flat file-based" representations.  (And it's quite possible I have
>more experience at this than anyone else does; I have done this
>several times.)
>
>The flat files may be simplistic, but at the time that someone will
>most /need/ to use the escrow format, it will be because:
>
> - A registry has failed;
> - Clocks are ticking, the world is watching, as well as the
>   lawyers that released the data from escrow;
> - A new registry needs to be set up right away, if not sooner;
> - Time is of the essence, and anything outside of the "simplistic"
>   parts of the data is quite likely to need to go by the wayside;
> - If the failure was due to technical issues, it may be that it was
>   the fancy extras sitting in the extensions that induced failure,
>   and hence /losing/ that extended data is a prerequisite for
>   avoiding further failure.
>
>A dead simple "flat" representation of registry data will look mighty
>good at a moment like that.
>
>On the other hand, I worry that if we get anything wrong in
>representing data in (say) CSV, we may get stuck with terrible
>clumsiness in working around shortcomings.  If so, the workarounds
>might well make the "bloat" of XML look pretty good.
>
>It's certainly simpler to take a CSV file and toss it at a database
>and do some bulk processing.  The database guys (and I'm one of them)
>certainly like that.
>
>But it truly isn't vastly difficult to write a script to walk through
>an XML file, unravel, and re-ravel the flat file bits into a flat
>file.
>
>The most surprisingly bug-free code that I ever remember writing was a
>parser that used Expat to turn an XML bookmark stream into a Lisp
>list, followed by writing a bit of Lisp to grab elements I wanted to
>transform.  It was a pretty simple case; turning bookmarks from a
>(now-obsolete) browser's XML into DocBook/SGML.  I had the minor shock
>that the full-bore recursive bits of code ran, and gave me the result
>I was hoping for, on the very first run.  It *scares* me when code
>works the first time!
>
>I'm sure we have people that are *scared* that the XML form isn't
>shaped like the flat files that they're accustomed to, but there
>should be little problem in transforming the XML form of [9] into the
>set of flat files that won't scare them so much.  I wouldn't be at all
>surprised if this requires little more than a few XPath queries that
>are sufficiently conventional that they might get included as a
>"near-standard toolset."

JG - Why go through the transformation process at all?  The source and the
target databases are most likely relational, so why inject an object
representation in the middle for data escrow?  Just because you can
translate the XML to CSV doesn't mean that you should.

>
>So I am not terribly scared at using XML.
>
>What I think would be *terrible* would be if we wound up defining two
>standards for this that represent somewhat different sets of data in
>somewhat different ways, when what's wanted is one representation.  I
>hate the idea of including both [6] and [9] as a single
>"FrankenStandard."  That multiplies the complexities of validation and
>of trying to read and use the data.

JG - I agree and disagree on this one.  I see applicability of CSV for
medium to large registries and XML as an option for small registries.  I
would be interested to understand the interest in both models to help
drive whether one or both models are applicable.  The authors of
draft-arias-noguchi-dnrd-objects-mapping have expressed interest in
collaborating.     

>
>I suppose I'd be mollified if it could be proven that one form could
>always be automatically rewritten into the other, but I think I'd want
>that code to be publicly available before I'd be willing to believe in
>such an equivalence.
>
>I have thought a little bit about, but don't yet have terribly useful
>comment, regarding the full-versus-incremental issue.  Several others
>have commented that they haven't found it worthwhile to optimize
>things by doing partial/incremental dumps, and I can concur with that.
>
>Footnotes:
>
>[1]  [[http://en.wikipedia.org/wiki/Comma-separated_values][Wikipedia:
>Comma-separated values]]
>
>[2]  [[http://tools.ietf.org/html/rfc4180][RFC 4180 -  Common Format
>and MIME Type for Comma-Separated Values (CSV) Files]]
>
>[3] [[http://www.ietf.org/mail-archive/web/ire/current/msg00012.html][I-D
>Action:draft-arias-registry-data-escrow-00.txt]], indicating the
>release, 2010-01-25, of the following draft
>
>[4] 
>[[http://www.ietf.org/internet-drafts/draft-arias-registry-data-escrow-00.
>txt][draft-arias-registry-data-escrow-00.txt]],
>a draft document
>specifying the format and contents of Data Escrow deposits for Domain
>Registries, which used a CSV-based approach
>
>[5]  
>[[http://www.ietf.org/mail-archive/web/ire/current/msg00185.html][Chris
>Wright's message of 2012-10-31]]
>
>[6]  
>[[http://www.ietf.org/mail-archive/web/ire/current/msg00165.html][Email
>with draft-gould-thippeswamy-dnrd-csv-mapping-00]]
>
>[7]  [[http://en.wikipedia.org/wiki/MultiValue][MultiValue Database]]
>
>[8]  
>[[http://www.ietf.org/mail-archive/web/ire/current/msg00186.html][Francisc
>o
>Obispo comments on DNRD CSV Draft]]
>
>[9]  [[http://www.ietf.org/mail-archive/web/ire/current/msg00163.html][New
>version (3) of the spec and new draft on DNRD]]

---

From: Francisco Obispo <[email-masked]>
Date: Fri, 2 Nov 2012 11:25:14 -0700

Hi James, *,

The one thing I'm a bit worried about, is not only that we're putting all of this effort drafting a new document export format, but that the current spec (arias-noguchi-draft) is already part of the Applicant's guidebook, and since we can't just modify the guidebook at this point, because there were questions regarding Data Escrow in the new gTLD program that clearly asked the registry operators how they would comply with such requirements, what are we expecting to happen?

If the task is to pursue a more 'optimal' format to suit mid to large registries better, we will have to be aware that there is an ICANN component which will need to be addressed. ICANN is not bound to what IETF says with regards to an escrow dump format. Changing the spec will really impact a whole bunch of applications who already have an answer on this section and for which we cannot change at this point.

We might need to support arias-noguchi-draft until a new format is adopted by ICANN as the official format, however, this changes the dynamics of a fundamental section of the commitments made by the registry backend operator.

Are we looking at this as a post-delegation, post 1st round of new-gtld work? or are we looking to somehow get ICANN to approve a change to the guidebook?

Best regards

Francisco Obispo 
Director of Applications and Services - ISC
email: [email-masked]
Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
PGP KeyID = B38DB1BE

---

From: "Gould, James" <[email-masked]>
Date: Fri, 2 Nov 2012 19:33:12 +0000

Francisco,

Everything being proposed fully meets the Application Guidebook. Both
drafts (draft-arias-noguchi-dnrd-objects-mapping and
draft-gould-thippeswamy-dnrd-csv-mapping) fully comply with
draft-arias-noguchi-registry-data-escrow.  I'm not requesting a change to
the Application Guidebook rather I'm requesting to apply the IETF
community process to define the appropriate protocol for the dnrd objects.
 Hopefully you are not stating that any draft produced by ICANN that is
based on draft-arias-noguchi-registry-data-escrow is an implied extension
of the Application Guidebook and an extension of the commitments made by
the registry backend operators.  You previously stated to me that if there
was interest on the IRE list for the CSV model it would be worth
consideration.  Since interest has been shown I would like us to take the
next step of merging the two drafts or create a new draft that meets the
interest of the community.  I encourage further discussion on this list on
the drafts created in the hope that we can achieve the goals of meeting
the needs of small and large registries.

Thanks,  

--
  
JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 11/2/12 2:25 PM, "Francisco Obispo" <[email-masked]> wrote:

>Hi James, *,
>
>The one thing I'm a bit worried about, is not only that we're putting all
>of this effort drafting a new document export format, but that the
>current spec (arias-noguchi-draft) is already part of the Applicant's
>guidebook, and since we can't just modify the guidebook at this point,
>because there were questions regarding Data Escrow in the new gTLD
>program that clearly asked the registry operators how they would comply
>with such requirements, what are we expecting to happen?
>
>If the task is to pursue a more 'optimal' format to suit mid to large
>registries better, we will have to be aware that there is an ICANN
>component which will need to be addressed. ICANN is not bound to what
>IETF says with regards to an escrow dump format. Changing the spec will
>really impact a whole bunch of applications who already have an answer on
>this section and for which we cannot change at this point.
>
>We might need to support arias-noguchi-draft until a new format is
>adopted by ICANN as the official format, however, this changes the
>dynamics of a fundamental section of the commitments made by the registry
>backend operator.
>
>Are we looking at this as a post-delegation, post 1st round of new-gtld
>work? or are we looking to somehow get ICANN to approve a change to the
>guidebook?
>
>Best regards
>
>
>Francisco Obispo

---

From: Christopher Browne <[email-masked]>
Date: Thu, 1 Nov 2012 16:52:39 -0400

On Thu, Oct 25, 2012 at 4:43 PM, Gould, James <[email-masked]> wrote:
> We have created a draft of the Domain Name Registration Data (DNRD) Comma-Separated Values (CSV) Objects Mapping that is attached for review and feedback.

 - Domains
 - Contacts
 - Nameservers
 - Registrars

 - A registry has failed;

Footnotes:

[4] [[http://www.ietf.org/internet-drafts/draft-arias-registry-data-escrow-00.txt][draft-arias-registry-data-escrow-00.txt]],
a draft document
specifying the format and contents of Data Escrow deposits for Domain
Registries, which used a CSV-based approach

[5]  [[http://www.ietf.org/mail-archive/web/ire/current/msg00185.html][Chris
Wright's message of 2012-10-31]]

[6]  [[http://www.ietf.org/mail-archive/web/ire/current/msg00165.html][Email
with draft-gould-thippeswamy-dnrd-csv-mapping-00]]

[8]  [[http://www.ietf.org/mail-archive/web/ire/current/msg00186.html][Francisco
Obispo comments on DNRD CSV Draft]]

---

From: Francisco Obispo <[email-masked]>
Date: Fri, 2 Nov 2012 15:21:03 -0700

Hi James,

On Nov 2, 2012, at 12:33 PM, "Gould, James" <[email-masked]> wrote:

> the registry backend operators.

Well, the draft is mentioned in the guidebook as the guideline for the registry escrow.

The guidebook in Specification 2, section 3.1 states:

<guidebook>

Deposit’s Format. Registry objects, such as domains, contacts, name servers, registrars, etc. will be compiled into a file constructed as described in draft-arias-noguchi-registry-data-escrow, see [1]. The aforementioned document describes some elements as optional; Registry Operator will include those elements in the Deposits if they are available. Registry Operator will use the draft version available at the time of signing the Agreement, if not already an RFC. Once the specification is published as an RFC, Registry Operator will implement that specification, no later than 180 days after. UTF-8 character encoding will be used.

</guidebook>

So if we are now to propose a new format, technically we would either need to:

1) Include it as part of the draft in the new version; or

2) Use an ICANN process to adopt the new format as an additional method.

I personally think that having two formats for the dump generates additional burden to both EBEROs and Escrow providers, but that's my personal opinion. 

>  You previously stated to me that if there

Yes, we would have to be careful to make sure that we don't loose functionality by having CSV dumps, but on the other hand, as long as we can keep using the XML file, I'm ok with having an alternative for those who don't want it.

I don't want to force people to use what I think its best, I prefer to have alternatives.

Francisco Obispo

---

From: "Gould, James" <[email-masked]>
Date: Fri, 2 Nov 2012 22:54:08 +0000

Francisco,

My feedback is below.

--
  
JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 11/2/12 6:21 PM, "Francisco Obispo" <[email-masked]> wrote:

>Deposit¹s Format. Registry objects, such as domains, contacts, name
>servers, registrars, etc. will be compiled into a file constructed as
>described in draft-arias-noguchi-registry-data-escrow, see [1]. The
>aforementioned document describes some elements as optional; Registry
>Operator will include those elements in the Deposits if they are
>available. Registry Operator will use the draft version available at the
>time of signing the Agreement, if not already an RFC. Once the
>specification is published as an RFC, Registry Operator will implement
>that specification, no later than 180 days after. UTF-8 character
>encoding will be used.
>
></guidebook>

draft-arias-noguchi-registry-data-escrow is mentioned in the guidebook,
but draft-arias-noguchi-dnrd-objects-mapping is not.  It does refer to the
the various objects that need to be escrowed.  Both drafts
draft-arias-noguchi-dnrd-objects-mapping and
draft-gould-thippeswamy-dnrd-csv-mapping fully support the objects listed
and follow draft-arias-noguchi-dnrd-objects-mapping that is referenced.

>2) Use an ICANN process to adopt the new format as an additional method.

There is not a new format in my interpretation of the guidebook since
draft-arias-noguchi-registry-data-escrow is being followed.

>
>I personally think that having two formats for the dump generates
>additional burden to both EBEROs and Escrow providers, but that's my
>personal opinion. 

I agree with you.  It would be interesting to ask those that need to
implement the data escrow (registries and EBERO providers) which approach
they prefer.  I have posted previously on this list that XML is not
scalable for large registries.  I believe that CSV will be much easier for
the creation of the data escrow and the consumption of the data escrow in
a more compressed format.  We created the CSV model as a straw man to
address the concern that I've expressed on this list and to see if other
implementors of the data escrow have interest in it as well.

>Yes, we would have to be careful to make sure that we don't loose
>functionality by having CSV dumps, but on the other hand, as long as we
>can keep using the XML file, I'm ok with having an alternative for those
>who don't want it.
>

I would leverage the IETF community process to let these drafts get to a
natural conclusion.  Ideally this could have been an IETF working group.
This list is the best sounding board to work things out.  I have received
two thumbs up publicly and one privately for the use of CSV.  I would like
to hear from others that feel for or against the use of CSV.  I believe
there is already adequate interest, but I would like to know how strong
the interest is to help drive the next steps.

---

From: Chris Wright <[email-masked]>
Date: Mon, 5 Nov 2012 12:13:54 +1100

I don’t think we have to be too concerned about the guidebook referencing the draft (the main problem is the contract references the draft).

However, if we can show all the registries, and the EBEROs (when they are appointed) consenting to an approach I am confident ICANN will see reason and update the approach in the contract. I don’t think this should stop us working on this.

Thanks

c.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Gould, James
Sent: Saturday, 3 November 2012 9:54 AM
To: Francisco Obispo
Cc: Christopher Browne; Thippeswamy, Chethan; [email-masked]
Subject: Re: [ire] DNRD CSV Draft

Francisco,

My feedback is below.

--
  
JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 11/2/12 6:21 PM, "Francisco Obispo" <[email-masked]> wrote:

>> Hopefully you are not stating that any draft produced by ICANN that 
>>is  based on draft-arias-noguchi-registry-data-escrow is an implied 
>>extension  of the Application Guidebook and an extension of the 
>>commitments made by  the registry backend operators.
>
>available. Registry Operator will use the draft version available at 
>the time of signing the Agreement, if not already an RFC. Once the 

draft-arias-noguchi-registry-data-escrow is mentioned in the guidebook, but draft-arias-noguchi-dnrd-objects-mapping is not.  It does refer to the the various objects that need to be escrowed.  Both drafts draft-arias-noguchi-dnrd-objects-mapping and draft-gould-thippeswamy-dnrd-csv-mapping fully support the objects listed and follow draft-arias-noguchi-dnrd-objects-mapping that is referenced.

>2) Use an ICANN process to adopt the new format as an additional method.

There is not a new format in my interpretation of the guidebook since draft-arias-noguchi-registry-data-escrow is being followed.

>additional burden to both EBEROs and Escrow providers, but that's my 

I agree with you.  It would be interesting to ask those that need to implement the data escrow (registries and EBERO providers) which approach they prefer.  I have posted previously on this list that XML is not scalable for large registries.  I believe that CSV will be much easier for the creation of the data escrow and the consumption of the data escrow in a more compressed format.  We created the CSV model as a straw man to address the concern that I've expressed on this list and to see if other implementors of the data escrow have interest in it as well.

>
>
>>  You previously stated to me that if there  was interest on the IRE 
>>list for the CSV model it would be worth  consideration.  Since 
>>interest has been shown I would like us to take the  next step of 
>>merging the two drafts or create a new draft that meets the  interest 
>>of the community.  I encourage further discussion on this list on  the 
>>drafts created in the hope that we can achieve the goals of meeting  
>can keep using the XML file, I'm ok with having an alternative for 
>those who don't want it.
>

I would leverage the IETF community process to let these drafts get to a natural conclusion.  Ideally this could have been an IETF working group.
This list is the best sounding board to work things out.  I have received two thumbs up publicly and one privately for the use of CSV.  I would like to hear from others that feel for or against the use of CSV.  I believe there is already adequate interest, but I would like to know how strong the interest is to help drive the next steps.

>Phone: +1 650 423 1374 || INOC-DBA *3557* NOC PGP KeyID = B38DB1BE
>

_______________________________________________
ire mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ire

---

From: Chris Wright <[email-masked]>
Date: Mon, 5 Nov 2012 12:16:47 +1100

As a company that applies to be an EBERO (hopefully announcements soon as to which companies have been selected) I am a BIG fan of the CSVs, and I agree with Christopher in that I don’t like the 'multi value' CSVs presented in the versign draft as they are difficult to use as external tables in databases.

Doing it broken up as I described and Christopher elaborated on below, is the simplest approach, especially when having to load data into registries straight away.

We would simply reference the CSVs as external tables (all good RDBMS systems have this ability) and then use SQL statements to import the data - any transformations required can be done using stored procedures (PLSQL in our case within Oracle). This will be the fastest way to get data into a registry in a short time.

Thanks

c.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Christopher Browne
Sent: Friday, 2 November 2012 7:53 AM
To: Gould, James
Cc: Thippeswamy, Chethan; [email-masked]
Subject: Re: [ire] DNRD CSV Draft

> We have created a draft of the Domain Name Registration Data (DNRD) Comma-Separated Values (CSV) Objects Mapping that is attached for review and feedback.

Preface: Sorry, my comments are pretty long.  And I reference footnotes considerably, as well as various later messages in the thread, to date.

On the one hand, I'm not a big fan of XML, am not particularly keen on the large size attendant to that.

On the other hand, I see a few problems attendant with a CSV-based representation, which should be pointed out:

That being said, there was an early IRE draft (which appears to have been lost?  [3] references unreferenceable document [4]), which I had some hand in, which took very much the approach that Chris Wright describes in [5], with a series of CSV files:

 - Domains
 - Contacts
 - Nameservers
 - Registrars

An XML representation can readily directly express hierarchy; the above approach remaps that into series of "flattened structures."

Onwards to comment on the Gould/Thippeswamy[6] document...  That document offers a structure largely analogous to the above, although it draws certain of the hierarchies into the objects.

In particular, the 2010-01-25 proposal [4] separated object statuses into separate association files, whereas [6] has several multivalued attributes.

Something I find very interesting about [6] is in its use of XML to describe the data.  Francisco comments [8] on the validation process that he anticipates being used to validate escrow; it seems to me that the draft of [6] provides an interesting addition to this.  The XML schema that is described therein could be used, hopefully pretty directly, to do a well-formedness validation on the series of CSV files.

For that to be possible implies that the "base" usage of [6] needs to be kept quite strict.  Adding additional fields would prevent using the XML schema directly, and perhaps it means that it would be necessary to declare that some set of functionality be declared a "core", and that extensions /must not/ alter anything in this core.  I wouldn't consider it a terrible thing to have a near-immutable "core,"
but that certainly has implications for extensions, that they need to be separated.

For instance, if we took [6] directly as the "core", then adding an anti-abuse contact could not be done by adding it as an extra field, as that would violate the immutability of the "core."  Adding an additional data stream to associate in "Anti-Abuse" contacts would be perfectly straightforward, albeit with the challenge that /validating/ this extension requires some new validation component downstream.
(The approach of [3] is not inherently better in this regard - while makes it easier to mix the new contact type into the existing data streams, there is still a need to validate contact types.)

But none of this quite gets at an opinion as to whether we should go with the XML form (most recently laid out in version 3, [9]) or the Verisign proposal [6].

I am not thrilled at the verbosity of XML, for much the same reasons already laid out by others.

And I have had decent success at drawing data into registries using "flat file-based" representations.  (And it's quite possible I have more experience at this than anyone else does; I have done this several times.)

The flat files may be simplistic, but at the time that someone will most /need/ to use the escrow format, it will be because:

 - A registry has failed;

A dead simple "flat" representation of registry data will look mighty good at a moment like that.

On the other hand, I worry that if we get anything wrong in representing data in (say) CSV, we may get stuck with terrible clumsiness in working around shortcomings.  If so, the workarounds might well make the "bloat" of XML look pretty good.

It's certainly simpler to take a CSV file and toss it at a database and do some bulk processing.  The database guys (and I'm one of them) certainly like that.

But it truly isn't vastly difficult to write a script to walk through an XML file, unravel, and re-ravel the flat file bits into a flat file.

The most surprisingly bug-free code that I ever remember writing was a parser that used Expat to turn an XML bookmark stream into a Lisp list, followed by writing a bit of Lisp to grab elements I wanted to transform.  It was a pretty simple case; turning bookmarks from a
(now-obsolete) browser's XML into DocBook/SGML.  I had the minor shock that the full-bore recursive bits of code ran, and gave me the result I was hoping for, on the very first run.  It *scares* me when code works the first time!

I'm sure we have people that are *scared* that the XML form isn't shaped like the flat files that they're accustomed to, but there should be little problem in transforming the XML form of [9] into the set of flat files that won't scare them so much.  I wouldn't be at all surprised if this requires little more than a few XPath queries that are sufficiently conventional that they might get included as a "near-standard toolset."

What I think would be *terrible* would be if we wound up defining two standards for this that represent somewhat different sets of data in somewhat different ways, when what's wanted is one representation.  I hate the idea of including both [6] and [9] as a single "FrankenStandard."  That multiplies the complexities of validation and of trying to read and use the data.

I suppose I'd be mollified if it could be proven that one form could always be automatically rewritten into the other, but I think I'd want that code to be publicly available before I'd be willing to believe in such an equivalence.

I have thought a little bit about, but don't yet have terribly useful comment, regarding the full-versus-incremental issue.  Several others have commented that they haven't found it worthwhile to optimize things by doing partial/incremental dumps, and I can concur with that.

Footnotes:

[2]  [[http://tools.ietf.org/html/rfc4180][RFC 4180 -  Common Format and MIME Type for Comma-Separated Values (CSV) Files]]

[3] [[http://www.ietf.org/mail-archive/web/ire/current/msg00012.html][I-D
Action:draft-arias-registry-data-escrow-00.txt]], indicating the release, 2010-01-25, of the following draft

[4] [[http://www.ietf.org/internet-drafts/draft-arias-registry-data-escrow-00.txt][draft-arias-registry-data-escrow-00.txt]],
a draft document
specifying the format and contents of Data Escrow deposits for Domain Registries, which used a CSV-based approach

[9]  [[http://www.ietf.org/mail-archive/web/ire/current/msg00163.html][New
version (3) of the spec and new draft on DNRD]] _______________________________________________
ire mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ire

---

From: Francisco Obispo <[email-masked]>
Date: Sun, 4 Nov 2012 17:29:52 -0800

> We would simply reference the CSVs as external tables (all good RDBMS systems have this ability) and then use SQL statements to import the data - any transformations required can be done using stor…

Well, I would argue that, you can also write an extension to PgSQL to consume the XML directly, using Foreign Data Wrappers (FDW), here's an example on how to do it with a twitter feed: https://github.com/umitanuki/twitter_fdw, in which case it can decompress, parse, validate and then just leave a plan view to be queried from where you can do a copy to another table ;-)

But I see your point, this is all agnostic to the gTLD program and if we are going to have multiple formats, lets do it, there is no one size fits all.

(we also applied to be an EBERO)

Francisco

Francisco Obispo

---

From: "Gould, James" <[email-masked]>
Date: Mon, 5 Nov 2012 14:28:57 +0000

As stated in an earlier posting, the draft has been posted as an I-D at
the URL below:

http://tools.ietf.org/html/draft-gould-thippeswamy-dnrd-csv-mapping

We intend to create an 01 draft that addresses Christopher's and Chris's
concern related to the support for list CSV fields.

--
  
JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 11/4/12 8:16 PM, "Chris Wright" <[email-masked]> wrote:

>As a company that applies to be an EBERO (hopefully announcements soon as
>to which companies have been selected) I am a BIG fan of the CSVs, and I
>agree with Christopher in that I don¹t like the 'multi value' CSVs
>presented in the versign draft as they are difficult to use as external
>tables in databases.
>
>Doing it broken up as I described and Christopher elaborated on below, is
>the simplest approach, especially when having to load data into
>registries straight away.
>
>We would simply reference the CSVs as external tables (all good RDBMS
>systems have this ability) and then use SQL statements to import the data
>- any transformations required can be done using stored procedures (PLSQL
>in our case within Oracle). This will be the fastest way to get data into
>a registry in a short time.
>
>Thanks
>
>c.
>
>
>Preface: Sorry, my comments are pretty long.  And I reference footnotes
>considerably, as well as various later messages in the thread, to date.
>
>That being said, there was an early IRE draft (which appears to have been
>lost?  [3] references unreferenceable document [4]), which I had some
>hand in, which took very much the approach that Chris Wright describes in
>[5], with a series of CSV files:
>
> - Domains
> - Contacts
> - Nameservers
> - Registrars
>An XML representation can readily directly express hierarchy; the above
>approach remaps that into series of "flattened structures."
>
>Onwards to comment on the Gould/Thippeswamy[6] document...  That document
>offers a structure largely analogous to the above, although it draws
>certain of the hierarchies into the objects.
>
>In particular, the 2010-01-25 proposal [4] separated object statuses into
>separate association files, whereas [6] has several multivalued
>attributes.
>
>describe the data.  Francisco comments [8] on the validation process that
>he anticipates being used to validate escrow; it seems to me that the
>draft of [6] provides an interesting addition to this.  The XML schema
>that is described therein could be used, hopefully pretty directly, to do
>a well-formedness validation on the series of CSV files.
>
>For that to be possible implies that the "base" usage of [6] needs to be
>kept quite strict.  Adding additional fields would prevent using the XML
>schema directly, and perhaps it means that it would be necessary to
>declare that some set of functionality be declared a "core", and that
>extensions /must not/ alter anything in this core.  I wouldn't consider
>it a terrible thing to have a near-immutable "core,"
>anti-abuse contact could not be done by adding it as an extra field, as
>that would violate the immutability of the "core."  Adding an additional
>data stream to associate in "Anti-Abuse" contacts would be perfectly
>straightforward, albeit with the challenge that /validating/ this
>extension requires some new validation component downstream.
>But none of this quite gets at an opinion as to whether we should go with
>the XML form (most recently laid out in version 3, [9]) or the Verisign
>proposal [6].
>
>And I have had decent success at drawing data into registries using "flat
>file-based" representations.  (And it's quite possible I have more
>experience at this than anyone else does; I have done this several times.)
>
>The flat files may be simplistic, but at the time that someone will most
>/need/ to use the escrow format, it will be because:
>
> - A registry has failed;
>On the other hand, I worry that if we get anything wrong in representing
>data in (say) CSV, we may get stuck with terrible clumsiness in working
>around shortcomings.  If so, the workarounds might well make the "bloat"
>of XML look pretty good.
>
>It's certainly simpler to take a CSV file and toss it at a database and
>do some bulk processing.  The database guys (and I'm one of them)
>certainly like that.
>
>But it truly isn't vastly difficult to write a script to walk through an
>XML file, unravel, and re-ravel the flat file bits into a flat file.
>
>parser that used Expat to turn an XML bookmark stream into a Lisp list,
>followed by writing a bit of Lisp to grab elements I wanted to transform.
> It was a pretty simple case; turning bookmarks from a
>that the full-bore recursive bits of code ran, and gave me the result I
>was hoping for, on the very first run.  It *scares* me when code works
>the first time!
>
>I'm sure we have people that are *scared* that the XML form isn't shaped
>like the flat files that they're accustomed to, but there should be
>little problem in transforming the XML form of [9] into the set of flat
>files that won't scare them so much.  I wouldn't be at all surprised if
>this requires little more than a few XPath queries that are sufficiently
>conventional that they might get included as a "near-standard toolset."
>
>"FrankenStandard."  That multiplies the complexities of validation and of
>trying to read and use the data.
>
>have commented that they haven't found it worthwhile to optimize things
>by doing partial/incremental dumps, and I can concur with that.
>
>Footnotes:
>
>[2]  [[http://tools.ietf.org/html/rfc4180][RFC 4180 -  Common Format and
>MIME Type for Comma-Separated Values (CSV) Files]]
>

---

From: "Gould, James" <[email-masked]>
Date: Mon, 5 Nov 2012 17:09:41 +0000

Christopher,

My feedback is below.  Thanks for your thoughtful feedback.

--
  
JG

James Gould
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 11/5/12 11:37 AM, "Christopher Browne" <[email-masked]> wrote:

>On Fri, Nov 2, 2012 at 5:59 AM, Gould, James <[email-masked]> wrote:
>> Christopher,
>I add an extra field to my escrow, and you add a different extra field
>to your escrow, there's a conflict there that needs to get resolved,
>somehow.

JG - What conflict are you referring to?  Assume that you create a new
field called "foo" and I create a new field call "bar" that is to be
included in the "domain" CSV deposit, then each of us will create our own
XML schema definition to formally define the fields with a unique XML
namespace and referenced prefix like prefix "you" for your custom fields
and "my" for my custom fields, and with the field names <you:fFoo> and
<my:fBar> respectively.  The custom field would be included in the
<rde:csv:fields> list for the "domain" <rdeCsv:csv> element.  So, your
definition could look something like:

<rdeCsv:csv name="domain">
...
   <rdeCsv:fields>
   ...
      <you:fFoo>
   ...
   </rdeCsv:fields>
...
</rdeCsv:csv>

My deposit would look lie

<rdeCsv:csv name="domain">
...
   <rdeCsv:fields>
   ...
      <my:fBar>
   ...
   </rdeCsv:fields>
...
</rdeCsv:csv>

The EBERO would most likely support a defined set of fields to import like
<rdeCsv:fName/><rdeCsv:fRoid/> but might decide to handle custom field
references differently.  XML doesn't magically handle the processing of
unique extensions.  In both cases the fields and extensions are defined
via XSD and need to be handled is a custom way by the EBERO provider.

>
>If we're using XML, entities and attributes are named, which provides
>conflict resolution, and that's inherent to using XML for this.

JG - The fields are equally named as shown about in the CSV draft.

>
>In a CSV-based model, there needs to be some convention declared for
>how to recognize these attribute/field additions.  That can be done,
>to be sure; it's an extra body of material to add to the standard.

JG - I don't see the definition of the custom fields in the CSV draft to
be that much different from the extensibility defined in the XML draft.

>
>Here's what I wrote, way back.
>------- snip here -------
>Extensions
>If a particular registry operator's contract requires submission of
>additional data, not included above, additional "extension" files may
>be defined to represent that data which may use Domain, Contact,
>Nameserver, and Registrar handles in order to associate that data with
>these objects, and which may introduce new objects, with their own
>handles that may, in turn, be used to allow extension files to
>indicate references to these new objects.
>If there turn out to be extensions that are to be imposed against all
>registry operators, then it might be reasonable to consider revising
>this escrow format specification to add additional fields to the
>existing files.
>
>If, however, an extension is not universally imposed, however, then it
>should be applied by adding additional "extension files." That way,
>existing escrow validation systems will continue to be applicable. The
>extension would only require adding additional validation "extensions"
>to support the additional data.
>------- snip here -------
>
>The excerpted material assumes that new attributes would be added by
>adding additional file streams, and the last paragraph explains why
>that would be desirable.
>
>I don't seem to have a copy of what was ultimately published, only
>earlier drafts that were pretty significantly changed before the
>published draft.  Perhaps Francisco can provide a copy?
>
>>> - Domains
>>> - Contacts
>>> - Nameservers
>>> - Registrars
>Yes, I think you're reading things right.
>
>I prefer the "flattened" forms, from an extensibility perspective,
>particularly vis-a-vis contact associations, as the [ Domain
>reference, Contact reference, Contact type ] representation makes it
>simpler to extend things to support additional contact types. The use
>of new sorts of contacts appears to be getting popular as a way to
>extend registry functionality.
>
>The one place where it seems to me that functionality was missing in
>your representation was vis-a-vis statuses.  The EPP standard allows
>accompanying statuses with a string indicating rationale for the
>status; if a registry captures the accompanying data, then it would be
>reasonable for escrow to also capture that.  (It's also an interesting
>idea to propose that the "human readable rationale" should *NOT* be
>part of escrow; that would be interesting to debate...)

JG - Good point, this can be addressed.

>
>I feel less strongly about host and IP associations.
>
>There is a good argument to be made in favour of the list-based
>approach, namely that this eliminates several data streams, and
>eliminates the possibility of mismatches.  (e.g. - the Domain/Contact
>stream has entries for contacts or domains that do not exist, or
>misses entries.  One could induce this to happen, in an SQL context,
>if the respective streams were constructed in separate SQL
>transactions.)  Of course, "good argument" is not the same as
>"forcible argument."

JG - The advantage of supporting list fields directly in the CSV is that
you require less CSV files, but I believe the advantage of truly modeling
CSV with the link tables held in separate CSV will be easier to produce
and consume.  We will create an 01 draft that addresses this.

>Yes; it introduces some extra details, though.
>
>If it is intended that we extend via custom XSDs, then there needs to
>be a way to communicate the custom XSDs to EBERO providers.  In
>effect, this means that the XSD being used needs to be a part of the
>escrow.

JG - Yes that is correct, the custom XSD files should be included in the
deposit.  This is the same case for the XML approach.

>Well, this diverges, for the moment, from discussing the merits of CSV.
>
>If I were given a dump in the *other* proposed format, which is, in
>effect, an XML "tree", my inclination would be to write some
>transformations to unravel it into a bunch of flat streams to load
>into a database, and I expect that to work perfectly well.  Of course,
>if I start with CSV, there's no unravelling required.
>
>Some of my colleagues might rather collect data for a domain object,
>and do an "object-oriented" loading of that data.  And there's nothing
>deeply wrong about doing that instead.

JG - You are correct that the data could be handled via "object-oriented"
loading, but I believe this is more complex then using the CSV files.  It
would be great to hear from those that prefer to do the OO mapping either
for the production of the data escrow or the consumption of the data
escrow.   

>This is a small enough community that it seems to me that if two
>standards are created, then it is unlikely that the effort would go in
>to make them really consistent with one another, and unhappiness will
>come from inconsistency.  I'd like to be wrong about that.

JG - My hope is that we can have the discussion of the merits of both
approaches and agree upon the right solution going forward (XML, CSV, or
both) based on the interests of the producers (registries) and the
consumers (EBERO providers).  I might be an optimist.

---

From: Christopher Browne <[email-masked]>
Date: Mon, 5 Nov 2012 11:37:49 -0500

...

------- snip here -------
Extensions

---

From: Gustavo Lozano <[email-masked]>
Date: Mon, 5 Nov 2012 12:47:31 -0800

Christopher,

>published draft.  Perhaps Francisco can provide a copy?

On the following link you can find it:

http://tools.ietf.org/html/draft-arias-registry-data-escrow-00

Regards,
Gustavo