Subject: [APPS-REVIEW] Metalink XML Download Description Format (draft-bryan-metalink-01)

From: "Anthony Bryan" <[email-masked]>
Date: Mon, 1 Sep 2008 01:44:23 -0400

Greetings,

Here is the Internet Draft for Metalink, available at
http://tools.ietf.org/html/draft-bryan-metalink-01
with interim revisions at
http://metalinks.svn.sourceforge.net/viewvc/metalinks/internetdraft/ .
We're looking for review and public comments.

Metalink is currently supported by some 35 applications and used by
projects such as OpenOffice.org, openSUSE, Ubuntu, cURL, and others.

  Metalink is an XML-based document format that describes a file or
  lists of files to be added to a download queue.  Lists are composed
  of a number of files, each with an extensible set of attached
  metadata.  For example, each file can have a description, checksum,
  and list of URIs that it is available from.

  The primary use case that Metalink addresses is the description of
  downloadable content in a format so download agents can act
  intelligently and recover from common errors with little or no user
  interaction necessary.  These errors can include multiple servers
  going down and data corrupted in transmission.

Thank you,
-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
 )) Easier, More Reliable, Self Healing Downloads
_______________________________________________
APPS-REVIEW mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/apps-review

---

From: Julian Reschke <[email-masked]>
Date: Tue, 02 Sep 2008 11:35:17 +0200

Dave Cridland wrote:
> ...
> 11) Section 4.2.17: Does that "type" attribute totally suck? Of course, 
> you have to have it because every man, his dog, and his pet hampster has 
> decided that only HTTP is allowed, these days, for absolutely 
> everything, leading to a totally useless URI scheme which essentially 
> fails to describe the actual resource it's supposedly a locator for. 
> Okay, rant over, back to the review.
> ...

"4.2.17. The "metalink:url" Element

    The "metalink:url" element contains the IRI of a file.  All IRIs
    should lead to identical files, except in the case of type
    "bittorrent" where the IRI should lead to a .torrent file."

Actually, don't do that. Don't make the meaning of an element totally 
dependent on an attribute on it. If torrent files are a special case 
(can there be more...?), then define a separate element.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 02 Sep 2008 23:13:26 +0200

Brian Smith wrote:
> Dave Cridland wrote:
>> 1) I'm not mad keen on namespaces pointing to privately owned 
>> domain names. You have a different one in the examples as is 
>> specified in section 1.2, by the way.
> 
> What namespace URI namespace should IETF specifications use?
> urn:ietf:params:xml:ns:<name>?
> 
> http://tools.ietf.org/html/rfc3553
> http://tools.ietf.org/html/rfc3688

That's one possible choice.

Another one is to keep using HTTP URIs, but to move them to a more 
neutral location (purl.org has been used in the past, so has www.w3.org).

BR, Julian

---

From: "Anthony Bryan" <[email-masked]>
Date: Tue, 2 Sep 2008 19:36:29 -0400

Dave, thank you for these comments! You caught some issues that needed
to be addressed & that has improved this draft IMO. Changes are in
SVN.

On Tue, Sep 2, 2008 at 4:46 AM, Dave Cridland <[email-masked]> wrote:
> On Mon Sep  1 06:44:23 2008, Anthony Bryan wrote:
>>
>> http://metalinks.svn.sourceforge.net/viewvc/metalinks/internetdraft/.
> 1) I'm not mad keen on namespaces pointing to privately owned domain names.
> You have a different one in the examples as is specified in section 1.2, by
> the way.

I understand & feel the same thing. The draft didn't seem like the
right place to put this, but I want the IETF to change the namespace
to whatever is appropriate. I don't know if this is something that
should be changed late or early in the process, that I change, or the
RFC editor changes?

The Atom namespace is "http://www.w3.org/2005/Atom". I don't know the
history behind it, but it doesn't seem my place to suggest something
like "http://www.w3.org/2008/Metalink" for Metalink. If that or
something similar is correct, perfect!

I do not want the namespace pointing to a domain name I own. More
responsibility than I need. :)

That was an error in the example, thanks for catching that!

> 2) The type attribute of the hash element ought to have values selected from
> the IANA registry of textual hash names;
> http://www.iana.org/assignments/hash-function-text-names/ - section 4.1.6.1
> & 4.2.4.1.

Wow, thank you so much! I didn't know about that. Should I list the
same hash types, or just have this:

   The IANA registry named "Hash Function Textual Names" defines values
   for hash types.

> 3) Awesome:
>
> """
>  For convenience, this data format may be referred to as "Metalink
>  3.0".  This specification uses "Metalink" internally.
> """
>
> Yeah, so you use a longer, more formal name for *convenience*? And then in
> the specification itself, use a short-hand for, what, inconvenience? :-)

HAH! Glad you enjoyed that, an inside semi-joke :) Pretty ridiculous.

I've changed it to:
   For convenience, this data format may be referred to as "Metalink",
   which this specification uses internally.

> 4) Notational convention defines a different convention than is used in the
> examples.

The RELAX NG Compact schema doesn't match the rest of the spec? Or
something else? I'm still working on that, & if anyone wants to help
out please do because I don't have much experience.

> 5) Section 2, "Metalink Documents MUST be well-formed
>  XML." - and presumably "namespace well-formed", as specified in
> [XML-NAMES]? Or not?

Yes.
Is this something I need to add to the draft, or are you mentioning it
because of internal inconsistencies in the examples?

> 6) Section 2: I'd personally kick out xml:base - I don't see it solving
> anything here.

To be honest, most of the XML boilerplate, "Securing Metalink
Documents", and "Extending Metalink" sections are stolen directly from
the Atom RFC.

xml:base hasn't been used in Metalink. Could it be useful? I found
this blog about it
http://feedblog.org/2005/12/05/xmlbase-was-a-bad-addition-to-atom/

> 7) Section 4.1.3.1: I'm deeply unconvinced about this "name" attribute
> actually containing a path - I don't see this being needed at all. A
> filename, sure. But otherwise, someone somewhere will inadvertantly allow
> /etc/passwd to be overwritten, or something equally hideous. Just ban '/'.

I think it's potentially useful to be able to create a sub-directory
structure. For instance, this is done with jigdo or torrents.
Clients have security holes, I agree, but they should be warned about
the possibilities so they can be fixed.

> 8) Section 4.1.4: "MUST contain one [good] and SHOULD contain more than one"
> - you mean if my download is only available from one location then this may
> provide a cause of poor interoperability? I don't think this is what you
> mean. I think you mean: "MUST contain at least one".

Exactly, that was an error.

      metalink:resources element MUST contain at least one metalink:url
      element and SHOULD contain more than one metalink:url elements.

> 9) Section 4.2.3: I'm personally wary of doing this on two grounds. Firstly,
> people invariably use them for advertising, rather than debugging. Whilst
> not all marketing is, in fact, evil, I'm never truly keen on adding a bunch
> of octets for no good reason. Secondly, I'd live in perpetual fear that
> processors will change behaviour dependent on the agent's foibles.

This is the same as the Atom RFC. Has this been a problem in the real
world? (I don't know, just asking).

> 10) Section 4.2.13: I had to read this twice to figure out it referred to a
> digital signature of the file to be downloaded. It might be as well to
> define a term for that in section 2, and use it throughout. I suspect the
> nice security people might have something to say about specifying PGP-only
> detatched signatures.

Can you suggest wording that is more clear? I've changed it to:

   The "metalink:signature" element is a Text construct that conveys a
   digital signature for a file described in a Metalink Document.

I hope some security people will have criticism (& additions) for this
section. It describes how Metalink functions now, but we don't want to
be PGP specific. I am not familiar enough with other options to write
about it :)

> 11) Section 4.2.17: Does that "type" attribute totally suck? Of course, you
> have to have it because every man, his dog, and his pet hampster has decided
> that only HTTP is allowed, these days, for absolutely everything, leading to
> a totally useless URI scheme which essentially fails to describe the actual
> resource it's supposedly a locator for. Okay, rant over, back to the review.

Do you suggest removing the "type" attribute & having a new separate
element as Julian suggested?

> 12) Section 4.2.17.1: Perhaps preference and weight? See SRV.

I'm not against that if others think that it is necessary/useful, but
I haven't added it.

> 13) Section 4.2.17.2: Hang on, maybe this is a bit too far - I can see the
> need for type, sure, but I don't see the need to be able to specify "Use FTP
> to retrieve this HTTP URI", because that's just introducing a silly state.
>
> 14) Section 5 - I'd personally just ditch the entire thing, you effectively
> say all that's needed in Section 8.4.

This is taken from the Atom RFC. If it needed to be there, does it
need to be in this draft as well?

> 15) I'm sure a lot of Section 6.1~6.3 could be summed up as "Ignore any
> markup you don't understand, whether from the metalink namespace or not." Of
> course, I'm not really an XML expert, so if this language is really needed,
> that's all fine and good.

Same as above. I'm not an XML expert either, so I'll defer to others. :)

> Hope this helps, somewhat. Feel free to ignore the bits you disagree with.

It helps a lot! Very nice of you to take the time to offer a helpful critique!

---

From: "Anthony Bryan" <[email-masked]>
Date: Tue, 2 Sep 2008 19:41:25 -0400

On Tue, Sep 2, 2008 at 5:35 AM, Julian Reschke <[email-masked]> wrote:
>> decided that only HTTP is allowed, these days, for absolutely everything,
>> leading to a totally useless URI scheme which essentially fails to describe
>> the actual resource it's supposedly a locator for. Okay, rant over, back to
>> the review.
>> ...
>
> dependent on an attribute on it. If torrent files are a special case (can
> there be more...?), then define a separate element.

I'm sure there COULD be more, but I don't know of any other right now.

Do you suggest a torrent specific element, or something generic?

Thanks Julian.

---

From: "Anthony Bryan" <[email-masked]>
Date: Tue, 2 Sep 2008 22:25:45 -0400

On Tue, Sep 2, 2008 at 8:05 PM, Dave Cridland <[email-masked]> wrote:
> On Wed Sep  3 00:36:29 2008, Anthony Bryan wrote:
>>
> The latter - just point to the registry.

Done. Thanks again, that seems to mesh well.

I've changed the schema to just metalinkTextConstruct instead of the
individual hash types too.

> No, your conventions state you'll use a prefix of "metalink", but your
> examples don't. There might have been something else, I forget now.

I see, the schema at the end. I've fixed it.

> No inconsistency, I simply thought that might be what you meant.

I'm just asking if this is something that needs to be added, since it
copies the Atom RFC text. Isn't it at least somewhat implied? I don't
know enough here, so your (or others) input is appreciated.

> No, not what I meant. That "SHOULD" is wrong - it can be a "MAY", but
> "SHOULD" has a particular meaning - it means that if you only list one URL
> here, you risk an interop failure.
>
> RFC 2119 is an addictive drug, but like most strong medicine, best used
> sparingly.
>
> I suspect what you want to write is:
>
> The metalink:resources element MUST contain at least one metalink:url
> element. Typically, such elements contain more than one metalink:url element
> to provide multiple download sources.

Yes, changed.

> Much more obvious.

Great.

> I'm sure the security people will chip in.

Looking forward to it.

> Julian knows a lot more about XML than I do, I'd go along with what he says.

Will do.

> Might be worth discussing, at least - I've actually no opinion on the
> subject, I just note that SRV, for one, opted for both, presumably based on
> experience with MX.

Good to have the input.

> I'm not convinced it needs to be there either.

I'll be conservative and leave it for now, depending on comments from others.

Updated version at
http://metalinks.svn.sourceforge.net/viewvc/metalinks/internetdraft/

---

From: Julian Reschke <[email-masked]>
Date: Wed, 03 Sep 2008 11:17:57 +0200

> Do you suggest a torrent specific element, or something generic?

If it could be phrased as something more generic, and then state the 
current use case for torrent files, that probably would be best.

So, in general, this would be for IRIs that do not identify a resource 
to download, but metadata about a resource to download? Strictly 
speaking, isn't metalink not yet another format for that?

BR, Julian

---

From: "Anthony Bryan" <[email-masked]>
Date: Wed, 3 Sep 2008 09:12:32 -0400

On Wed, Sep 3, 2008 at 5:17 AM, Julian Reschke <[email-masked]> wrote:
>>>> everything,
> If it could be phrased as something more generic, and then state the current
> use case for torrent files, that probably would be best.
>
> So, in general, this would be for IRIs that do not identify a resource to
> download, but metadata about a resource to download? Strictly speaking,
> isn't metalink not yet another format for that?

Yes, it is.

Do you think about a "metadata" element (a sub-element of
"resources")with a required "type" attribute of MIME type is
appropriately generic?
This could then be used to describe Metalinks if needed, and other
types that may come later.

I don't think BitTorrent's MIME type is in the is listed in the IANA
MIME Media Types. Would this be a problem?

   <?xml version="1.0" encoding="UTF-8"?>
   <metalink xmlns="http://www.metalinker.org">
     <files>
       <file name="example.ext">
         <resources>
           <url>ftp://ftp.example.com/example.ext</url>
           <url>http://example.com/example.ext</url>
           <metadata
type="application/x-bittorrent">http://example.com/example.ext.torrent
           </metadata>
         </resources>
       </file>
     </files>
   </metalink>

---

From: Julian Reschke <[email-masked]>
Date: Wed, 03 Sep 2008 15:22:30 +0200

> "resources")with a required "type" attribute of MIME type is

Sounds good.

> types that may come later.

Right.

> type="application/x-bittorrent">http://example.com/example.ext.torrent

One of these issues that regularly come up :-)

One way out of that would by to "grab" special type names like 
"torrent", and hardwire them. That should be ok as long as they can't 
collide with registered names (thus no "/" allowed).

BR, Julian

---

From: "Anthony Bryan" <[email-masked]>
Date: Wed, 3 Sep 2008 12:01:31 -0400

On Wed, Sep 3, 2008 at 9:22 AM, Julian Reschke <[email-masked]> wrote:
> One way out of that would by to "grab" special type names like "torrent",
> and hardwire them. That should be ok as long as they can't collide with
> registered names (thus no "/" allowed).

Great, thanks Julian! Thanks to the other people that mentioned this
as well, it seems much cleaner & tightened up now. :)

Do I need to explicitly state no "/" allowed, or just know any that
ones I'm defining can't be "/" ?
And would it be better to just allow the unofficial MIME type, instead
of essentially creating an unofficial registry with one or a handful
of entries? Or is MIME types plus hardwired entries better?

Here is the changed text:

4.2.9.  The "metalink:metadata" Element

   The "metalink:metadata" element contains the IRI of metadata about a
   resource to download.  For example, this could be the IRI of a
   BitTorrent .torrent file or a Metalink Document.

   metalinkMetadata =
      element metalink:metadata {
         metalinkCommonAttributes,
         attribute preference { xsd:integer }?,
         attribute type { metalinkTextConstruct },
         metalinkUri
      }+

4.2.9.1.  The "preference" Attribute

   metalink:metadata elements MAY have a preference attribute, whose
   value MUST be a number from 1 to 100 for priority, with 100 used
   first and 1 used last.  See the "preference" attribute of the
   metalink:url element for more information.

4.2.9.2.  The "type" Attribute

   metalink:metadata elements MUST have a "type" attribute that
   indicates the MIME type of the metadata available at the IRI.  In the
   case of BitTorrent as specified in [BITTORRENT], the value "torrent"
   is required.

   Metalink Processors that do not support a specified type of metadata
   about a resource to download MUST ignore that metadata

4.2.18.  The "metalink:url" Element

   The "metalink:url" element contains the IRI of a file.  All IRIs MUST
   lead to identical files.

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 2 Sep 2008 16:03:49 -0500

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 03 Sep 2008 20:54:29 +0200

> ones I'm defining can't be "/" ?

I'd say that types without "/" are reserved, and the only reserved value 
for now is "torrent".

> of entries? Or is MIME types plus hardwired entries better?

I think you'll see pushback once you start to make normative 
requirements related to unregistered mime types. So, you can also try to 
get a type registered, which of course would be the cleaner solution...

BR, Julian

---

From: Dave Cridland <[email-masked]>
Date: Tue, 02 Sep 2008 09:46:25 +0100

On Mon Sep  1 06:44:23 2008, Anthony Bryan wrote:
> .

1) I'm not mad keen on namespaces pointing to privately owned domain  
names. You have a different one in the examples as is specified in  
section 1.2, by the way.

2) The type attribute of the hash element ought to have values  
selected from the IANA registry of textual hash names;  
http://www.iana.org/assignments/hash-function-text-names/ - section  
4.1.6.1 & 4.2.4.1.

3) Awesome:

"""

Yeah, so you use a longer, more formal name for *convenience*? And  
then in the specification itself, use a short-hand for, what,  
inconvenience? :-)

4) Notational convention defines a different convention than is used  
in the examples.

6) Section 2: I'd personally kick out xml:base - I don't see it  
solving anything here.

7) Section 4.1.3.1: I'm deeply unconvinced about this "name"  
attribute actually containing a path - I don't see this being needed  
at all. A filename, sure. But otherwise, someone somewhere will  
inadvertantly allow /etc/passwd to be overwritten, or something  
equally hideous. Just ban '/'.

8) Section 4.1.4: "MUST contain one [good] and SHOULD contain more  
than one" - you mean if my download is only available from one  
location then this may provide a cause of poor interoperability? I  
don't think this is what you mean. I think you mean: "MUST contain at  
least one".

9) Section 4.2.3: I'm personally wary of doing this on two grounds.  
Firstly, people invariably use them for advertising, rather than  
debugging. Whilst not all marketing is, in fact, evil, I'm never  
truly keen on adding a bunch of octets for no good reason. Secondly,  
I'd live in perpetual fear that processors will change behaviour  
dependent on the agent's foibles.

10) Section 4.2.13: I had to read this twice to figure out it  
referred to a digital signature of the file to be downloaded. It  
might be as well to define a term for that in section 2, and use it  
throughout. I suspect the nice security people might have something  
to say about specifying PGP-only detatched signatures.

11) Section 4.2.17: Does that "type" attribute totally suck? Of  
course, you have to have it because every man, his dog, and his pet  
hampster has decided that only HTTP is allowed, these days, for  
absolutely everything, leading to a totally useless URI scheme which  
essentially fails to describe the actual resource it's supposedly a  
locator for. Okay, rant over, back to the review.

13) Section 4.2.17.2: Hang on, maybe this is a bit too far - I can  
see the need for type, sure, but I don't see the need to be able to  
specify "Use FTP to retrieve this HTTP URI", because that's just  
introducing a silly state.

14) Section 5 - I'd personally just ditch the entire thing, you  
effectively say all that's needed in Section 8.4.

15) I'm sure a lot of Section 6.1~6.3 could be summed up as "Ignore  
any markup you don't understand, whether from the metalink namespace  
or not." Of course, I'm not really an XML expert, so if this language  
is really needed, that's all fine and good.

Hope this helps, somewhat. Feel free to ignore the bits you disagree  
with.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
_______________________________________________
APPS-REVIEW mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/apps-review

---

From: Dave Cridland <[email-masked]>
Date: Wed, 03 Sep 2008 01:05:02 +0100

> out please do because I don't have much experience.
No, your conventions state you'll use a prefix of "metalink", but  
your examples don't. There might have been something else, I forget  
now.

No, not what I meant. That "SHOULD" is wrong - it can be a "MAY", but  
"SHOULD" has a particular meaning - it means that if you only list  
one URL here, you risk an interop failure.

RFC 2119 is an addictive drug, but like most strong medicine, best  
used sparingly.

The metalink:resources element MUST contain at least one metalink:url  
element. Typically, such elements contain more than one metalink:url  
element to provide multiple download sources.

> referred to a
Much more obvious.

> element as Julian suggested?
Julian knows a lot more about XML than I do, I'd go along with what  
he says.

Might be worth discussing, at least - I've actually no opinion on the  
subject, I just note that SRV, for one, opted for both, presumably  
based on experience with MX.

Dave.

---

From: "Anthony Bryan" <[email-masked]>
Date: Sat, 20 Sep 2008 19:26:22 -0400

Greetings,

An updated Internet Draft for Metalink is available at
http://tools.ietf.org/html/draft-bryan-metalink-03
with interim revisions at
http://metalinks.svn.sourceforge.net/viewvc/metalinks/internetdraft/ .

This updated version incorporates comments:

- urn:ietf:params:xml:ns:metalink namespace & registration.
- IANA "Operating System Names" registry for listing the OS that a
file listed in a metalink is for. This registry would need to be
updated to be useful.
- new "Client Implementation Considerations" . Currently, this section
mentions transparent content negotiation.

Issues:
- Inclusion of digital signatures other than PGP of files listed in
the Metalink. (Only PGP is currently supported). What do we need to
add and how might we need to restructure this for other digital
signatures to be included/used?
- Proxies. If a file listed in a metalink is already in a cache, but a
different URI is used then I assume it won't be recognized and will be
requested again since it seems to be an uncached file. I'm wondering
if the checksums in a metalink or something else could be used to see
if the file is the same or not.

Thank you,

---

From: "Anthony Bryan" <[email-masked]>
Date: Sat, 20 Sep 2008 19:26:22 -0400

Greetings,

Issues:

Thank you,

_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: [email-masked]
subscribe: [email-masked]
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php