Subject: Re: [ire] DNRD CSV Draft

From: "Gould, James" <[email-masked]>
Date: Fri, 2 Nov 2012 09:03:52 +0000

Chris,

My feedback is below.

--
  
JG

James Gould
Principal Software Engineer
[email-masked]
 
703-948-3271 (Office)
12061 Bluemont Way
Reston, VA 20190
VerisignInc.com

On 10/31/12 10:17 PM, "Chris Wright" <[email-masked]> wrote:

>
>
>On 1/11/12 5:01 AM, "Gould, James" <[email-masked]> wrote:
>
>>Chris,
>>
>>An additional step that the data escrow provider would need to perform
>>per
>>deposit is to transform and validate the data in the standard format if
>>the deposit is provided in a non-standard format.  Having an agreement
>>with the data escrow provider to take data in one format to be
>>transformed
>>to a standard format when needed can not be tested only once, but with
>>each deposit.  Having a standard format that supports validation by
>>multiple parties (registry, data escrow provider, EBERO provider), that
>>doesn't require transformation by the data escrow or EBERO provider, and
>>that scales makes sense to me.
>
>Understood, and if that’s the way Verisign wants to do it, then go for it,
>you can deposit it in the format required to be provided to ICANN, no
>issue what so ever, but why should everyone be forced to do that? It would
>actually be really trivial for both me and my provider to do database
>replication, and then they simply run a script to validate the data (that
>I would supply them) - Im yet to actually see what the 'validation' they
>are supposed to do actually is - if its just object counts - then I would
>assert that dumping the data doesn't even really need to be done, but even
>it it does, they run the dump scripts and get the data in the format
>needed - not hard at all and completely automated.
>
>I believe we should be defining the format of that the data is required to
>be supplied by the Escrow provider to the EBERO, via ICANN and that’s it.
>ICANN should define the maximum age of that data (which I believe is meant
>to be no more than 24 hours). Then for pre-del testing ICANN can test this
>by creating objects in the test registry, then 24 hours later asking for
>the data from the Escrow provider and verifying it meets the format -
>simple. How the Escrow provider gets this data to give to ICANN and
>everything else is for the free market to solve - there is no justifiable
>reason for ICANN to dictate any of it - all ICANN should concern
>themselves with is the requirements outcomes.

JG - You can workout the content and the understanding of the application
guidebook with ICANN, since I don't believe this is the list to address
it.  No matter if you provide a custom format or a standard format to the
data escrow provider, that deposit should be validated against the
standard format expected by the EBERO provider to ensure that the EBERO
provider doesn't get garbage when it recovers the data.  Since there are
two separate parties (registry and data escrow provider) it makes sense to
have the data escrow provider ensure the integrity of the data and ensure
that it meets the target format for recovery.

>
>
>>  Supporting more than just full deposits in
>>draft-arias-noguchi-registry-data-escrow is essential to scale to large
>>registries, but you are correct that for very small registries (e.g. 2000
>>domains) incremental deposits doesn't add much value.
>
>I do not agree that incremental deposites as defined is essential to large
>registries. Depending on your technology doing full dumps may be quicker
>and less code than incrementals. I may be doing DB replication which
>doesn't involve 'deposites' at all - in fact DB replication is probably
>easier for larger registries (if done right) - but realistically, this is
>my point, each registry can solve this problem there own way - don't
>dictate to me how to do it - just set the requirements. Which as I have
>said above really are only 2:
>
> - the escrow provide must be able to supply to ICANN a complete copy of
>all data in a registry (ICANN to define all data) to ICANN in a format to
>be defined
> - this data should be no more than 24 hours old
>
>How we achieve this is up to us.

JG - Assuming that you do agree that the format provided to the data
escrow must follow the standard format, then would you not agree that
incremental deposits are essential to large registries?  Doing a daily
full database dump would be simple; although you're back to the same
discussion of whether the target format must be validated by the data
escrow provider which will simply move the deposit format requirement and
task from the registry to the data escrow provider.  The problem still
exists if the data must be validated, so incremental updates would reduce
the cost and effort in the generation and validation of a deposit in the
standard format.   

>
>>
>>What is your issue with providing XML definition of the CSV fields in the
>>draft?  The draft that we created is similar to the approach taken for
>>Fielded Text by formally defining the CSV fields.  In this case the
>>definition complies with draft-arias-noguchi-registry-data-escrow and
>>utilizes XSD elements to formally define the field format.  Do you have a
>>different approach that should be considered?  Do you have any specific
>>feedback to improve the draft?
>
>Im wondering is a machine would ever read the field format? It seems to me
>that a human (probably working for the EBERO) would be reading the field
>formats when the trying to 'load' the data into the EBERO registry. I can
>see anyone actually doing the work of writing code that parses the field
>format from the XML and then tries to 'interpret' the CSV in such a way.
>
>Surely there is a list of fixed fields which everyone must included and
>these can be in a defined order, then we just need to deal with
>extensions, which should probably be different files.

JG - A machine certainty can read the field format to interpret the CSV
files.  This is not much different from the software that would validate
the CSV data against the definition.  There would be a predefined set of
standard fields that the software can key off of for importing the data
contained the the CSV into the database.  In using the example
<rdeCsv:csv> element for domain name records in the draft, software can
identify the order and the fields that it supports (standard versus custom
extensions) for importing. Each of these fields have a predefined meaning
and format that should be simple for software to process.

<rdeCsv:fields>
   <rdeCsv:fName/>
   <rdeCsv:fRoid/>
   <csvDomain:fIdn/>
   <rdeCsv:fLanguage/>
   <csvDomain:fVariant/>
   <csvDomain:fOriginalName/>
   <csvDomain:fReserved/>
   <rdeCsv:fStatuses/>
   <csvDomain:fRgpStatuses/>
   <rdeCsv:fRegistrant/>
   <rdeCsv:fAdminContacts/>
   <rdeCsv:fTechContacts/>
   <rdeCsv:fBillingContacts/>
   <csvDomain:fNsObjs/>
   <rdeCsv:fClID/>
   <rdeCsv:fCrID/>
   <rdeCsv:fCrDate/>
   <rdeCsv:fUpID/>
   <rdeCsv:fUpDate/>
   <rdeCsv:fExDate/>
   <rdeCsv:fPwAuthInfo/>
  </rdeCsv:fields>

>
>I see CSVs like this
>
>Domains - the base details of all domains, that all registries have (roid,
>domain name, expiry date, registrant contact ... Etc)
>DomainNameServers - a 2 column CSV mapping domain ROIDs from the domains
>file, to host roids in the host file (domain roid, host roid)
>DomainContacts - a 3 column CSV mapping domain ROIDs, to contact ROIDs and
>contact types eg. billing, tech, admin (domain roid, contact roid, contact
>type)
>DomainVariants - a 2 column CSV mapping domain ROIDs, to the provisioned
>variants (roid, variant domain name)
>Hosts - the base details of hosts (roid, host name ... Etc)
>Contacts - the base details of contact (roid, loc street1, loc street2,
>int street 1, int street 2 etc)
>Registrar - the base details of registrars
>DomainDNSSEC etc
>Etc
>
>Then for extensions each registry needs to define the format of the CSV in
>documentation, I don't think it can be done pragmatically.
>
>DomainExtension1 - a file that maps the domain roid to the extension
>details (roid, extfield1, extfield2 etc...)

JG - So how is this so much different from what is in the
draft-gould-thippeswamy-dnrd-csv-mapping draft?  We defined a standard set
of global fields that apply across many objects, a standard set of fields
that apply per object for definition, and a standard set of CSV files with
references to the fields that SHOULD be used.  Extensions are supported by
either utilizing a <rdeCsv:fCustom> fields definition with a required
"name" attribute or by defining your own custom field types using your own
custom XSD.  The EBERO software would know to expect the standard set of
fields and can then use the definition to process the CSV files.  It
doesn't sound like you're in agreement to define the format of the CSV
files via XML and supporting draft-arias-noguchi-registry-data-escrow.  Do
you have proposed changes to draft-gould-thippeswamy-dnrd-csv-mapping to
meet what you're looking for or do you propose a completely different
approach and if so do you intend to create a draft for it?

>
>
>I have outstanding 'problems' which exist in all models:
>
>For IDNs, I don’t want to have to escrow all blocked variants (I don't
>generate them I use a unique key mapping form) - so how do we
>'communicate' this to the EBERO, I think we need to deal with language
>tables
>DNSSEC - how do we transition this? (un-sign?)
>And it goes on - topic of another email

JG - Yes, I agree that the escrow should not contain the blocked variants.
 The canonical variant in the data escrow deposit is of little use to the
EBERO provider.  I believe the best that can be done for IDN domains is to
reference the language tag and then for each language tag reference the
language table URI for the code points.  In
draft-gould-thippeswamy-dnrd-csv-mapping, the <rdeCsv:fLanguage> field is
included in the "domain" CSV that references the language for the IDN, and
the IDN Language Object includes the same <rdeCsv:fLanguage> field as the
key and a <rdeCsv:fUrl> field that includes the URL of the language table.
 There is no canonical variant and no inclusion of blocked variants, since
it would be up to the EBERO provider to produce the appropriate canonical
variants from the data escrow deposit.

>
>Thanks for taking the time to read to the end.
>
>Chris
>
>>
>>Thanks,
>>
>>--
>>  
>>JG
>> 
>>
>> 
>>James Gould
>>Principal Software Engineer
>>[email-masked]
>> 
>>703-948-3271 (Office)
>>12061 Bluemont Way
>>Reston, VA 20190
>>VerisignInc.com
>>
>>
>>
>>
>>
>>
>>
>>On 10/31/12 1:11 AM, "Francisco Obispo" <[email-masked]> wrote:
>>
>>>That was exactly my feedback to ICANN ;-)
>>>
>>>In fact, we decided that we were going to go for the full dumps every
>>>day, because the process of applying the deltas could really mess up
>>>things if not done right.
>>>
>>>The feedback that I got was that as long as we provide something
>>>"better"
>>>than what is required, it should then be OK.
>>>
>>>I see the point on the escrow provider, and I agree with you, it would
>>>be
>>>great if they offered that service, we would just set up a replica and
>>>let them do the dump on a daily or weekly basis.
>>>
>>>But I have yet to find one that would do that.. (I'll talk to the one we
>>>chose to see if they are willing to do it)
>>>
>>>On Oct 30, 2012, at 10:01 PM, Chris Wright <[email-masked]>
>>>wrote:
>>>
>>>> Also I know where the incremental 'requirements' come from, I am
>>>>saying
>>>>why should I go to the effort of doing incrementals if my registry only
>>>>has 2000 domains in it? Why am I forced to do incrementals? I should
>>>>just be allowed to do a full dump each day, how does that hurt anyone?
>>>>In fact it makes things easier for whomever would have recover my
>>>>registry later as they just load in the one dump and be done with it,
>>>>they don¹t have to try and write logic to interpret incrementals.
>>>> 
>>>> Thanks
>>>
>>>Francisco Obispo
>>>Director of Applications and Services - ISC
>>>email: [email-masked]
>>>Phone: +1 650 423 1374 || INOC-DBA *3557* NOC
>>>PGP KeyID = B38DB1BE
>>>
>>
>