Subject: Re: [apps-discuss] Fwd: Last Call: draft-hammer-hostmeta (Web Host Metadata) to Proposed Standard

From: Eran Hammer-Lahav <[email-masked]>
Date: Sat, 24 Jul 2010 00:18:54 -0700

> -----Original Message-----
> From: John Panzer [mailto:[email-masked]]
> Sent: Friday, July 23, 2010 6:37 PM
> To: Blaine Cook
> Cc: Eran Hammer-Lahav; [email-masked]
> Subject: Re: [apps-discuss] Fwd: Last Call: draft-hammer-hostmeta (Web
> Host Metadata) to Proposed Standard
> 
> That's also why I don't see "stop when you see the rel you're looking for" as a
> big help - you actually are looking for a collection where rel=x in the general
> case (think rel=alternate for a common example from other formats).  So you
> have to get the set of links in some context, which could be the host-meta
> document or the target of the lrdd template.

The only way to avoid this is to forbid using link templates for anything other than a single 'lrdd' link template, or even remove templates completely and using a single link (non template) with relation 'lrdd' and a pre-defined way to include the context URI as a parameter. Otherwise, link templates at the host-meta level are semantically equivalent to those in a LRDD document, not matter how much you try to pretend they are to be ignored.

Can you offer some useful and predictable guidelines when a client should use link templates in host-meta and when it should not? When it should only look for a single 'lrdd' document for all its metadata needs? How should an application developer decide where to put resource-specific links, if we follow your approach and treat the two sets (link templates in host-meta, and links in a LRDD document) differently?

For your approach to work, we need to be able to solve these questions because semantically, all these links share the same context URI. We created a mechanism using link templates and defined it well. The processing rules in 4.2 are just the logical implementation of this framework.
 
> But in general I agree with Blaine's primary objection; this adds an include
> model that has not been proven and which forces clients to provision for a
> nearly arbitrary number of worst-case http fetches.

Servers can always do stupid things, like use too many redirections which requires the same kind of handling. From a code perspective, processing links from both host-meta and LRDD isn't that much more work - you already have to process both. The ability to link to more than one LRDD document is something that keeps coming up again and again when people want to reuse a descriptor document available elsewhere.

'lrdd' links have a very limited include semantic by limiting it to one level.

> Query:  Is it at least possible for a server to automatically merge multiple
> rel=lrdd targets and serve up a single composite rel=lrdd target resource to
> clients without impacting semantics?  It's not clear to me from the spec.

No idea what you mean.

> On Friday, July 23, 2010, Blaine Cook <[email-masked]> wrote:
> > On 23 July 2010 21:12, Eran Hammer-Lahav <[email-masked]>
> wrote:
> >> ** If this is not clear, I am happy to make it more explicit in section 4.2,
> including adding a full procedure for clients looking for a specific link.
> >
> > +1 - In my reading of 4.2, I had overlooked the sentence in 4 that
> > describes stopping whenever a suitable link relation is found. With
> > that processing rule in place, the semantic is clear and simple.
> >
> >> However, if people would like to simplify this a bit, I am open to (but not
> excited about) making a small change in which clients will first look for (any)
> 'lrdd' link templates, process them, and only then look for other link-
> templates in host-meta. This exception will deviate from the normal XRD
> processing rules of document order, but will move from a potential complex
> mix from link sources to a two stage procedure (LRDD links, then host-meta
> link templates). Note that this does not solve the basic requirement to
> construct the full resource descriptor from multiple location, only the order in
> which that is done.
> >
> > I'm not sure what others think about this, but I personally prefer
> > this approach (LRDD-first, followed by host-level templates). The
> > reasoning is that this gives LRDD profiles the ability to over-ride
> > host-level decisions, whereas strictly following document ordering
> > means that hosts could potentially force all users to use a single URL
> > for a given rel value.
> >
> > The challenge is not so much that the hosts could force this (because
> > a "malicious" host could over-write the rel value in ever user's LRDD
> > document), but that interop becomes more challenging since any parser
> > that does LRDD-first *might* disagree with a strictly ordered parser
> > (an example where this is a real possibility: three parsers, one
> > SAX-based, one XPath based, and one DOM-based, may come up with very
> > different results if we rely on document ordering alone).
> >
> > b.
> > _______________________________________________
> > apps-discuss mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/apps-discuss
> >
> 
> --
> --
> John Panzer / Google
> [email-masked] / abstractioneer.org / @jpanzer

---

From: "Manger, James H" <[email-masked]>
Date: Sun, 25 Jul 2010 23:51:56 +1000

I don't think "lrdd-first followed by host-meta templates" is a better approach than
the current host-meta rules (host-meta templates and 'lrdd' includes in document order).

The "lrdd-first" proposal means a client always has to make the extra request to get an lrdd (when present),
even when the required link is already in the host-meta that the client already has.

host-meta is a mechanism where the host is in charge (ahead of resources).
"lrdd-first" would confuse that.
A host would be in charge while it doesn't put a 'lrdd' link in host-meta, but as soon as it includes
such a link all the host's other link templates can be overridden.
That is a bad choice to force a host to make.

I don't think SAX-based, XPath-based, or DOM-based parsers make that much difference here.
The current host-meta requires looping through link templates in document order.
All parser varieties can do that. It isn't much (if any) harder than separately finding
'lrdd' templates and templates for a specific rel value.

Adding a rule that "host-meta can have, at most, a single 'lrdd' link" would
reduce the risk that lots of hosts include lots of 'lrdd's without considering
the performance impact on clients.
I doubt this is that helpful, however. A host can be slow (bandwidth, redirects, no caching...)
even with 0 or 1 'lrdd' links. Other hosts can be tuned for performance.

I recommend keeping the current host-meta rules.

P.S. One minor typo: section 4.2, item 3.2, change
 "The client adds any link found"
to
 "The client adds any links found"

--
James Manger

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Eran Hammer-Lahav [[email-masked]]
Sent: 24 July 2010 17:18
To: John Panzer; Blaine Cook
Cc: [email-masked]
Subject: Re: [apps-discuss] Fwd: Last Call: draft-hammer-hostmeta (Web Host Metadata) to Proposed Standard

No idea what you mean.

---

From: John Panzer <[email-masked]>
Date: Mon, 26 Jul 2010 13:22:33 -0700

On Sat, Jul 24, 2010 at 12:18 AM, Eran Hammer-Lahav <[email-masked]>wrote:

> The only way to avoid this is to forbid using link templates for anything
> other than a single 'lrdd' link template, or even remove templates
> completely and using a single link (non template) with relation 'lrdd' and a
> pre-defined way to include the context URI as a parameter. Otherwise, link
> templates at the host-meta level are semantically equivalent to those in a
> LRDD document, not matter how much you try to pretend they are to be
> ignored.
>
> Can you offer some useful and predictable guidelines when a client should
> use link templates in host-meta and when it should not? When it should only
> look for a single 'lrdd' document for all its metadata needs? How should an
> application developer decide where to put resource-specific links, if we
> follow your approach and treat the two sets (link templates in host-meta,
> and links in a LRDD document) differently?
>
> For your approach to work, we need to be able to solve these questions
> because semantically, all these links share the same context URI. We created
> a mechanism using link templates and defined it well. The processing rules
> in 4.2 are just the logical implementation of this framework.
>
> Servers can always do stupid things, like use too many redirections which
> requires the same kind of handling. From a code perspective, processing
> links from both host-meta and LRDD isn't that much more work - you already
> have to process both. The ability to link to more than one LRDD document is
> something that keeps coming up again and again when people want to reuse a
> descriptor document available elsewhere.
>

If I have a host-meta file to which a user has added, say, 5 rel='lrdd'
templates, is it possible for me to replace all 5 with a single composite
(generated) document and a single (generated) rel='lrdd' link to same,
without breaking semantics?

Why one might want to do this:  Performance, for the same reason that web
servers do things like dynamic spriting of images by combining multiple
images into one larger image for delivery over the wire.  And XRDs are
things that you typically won't be loading lazily as you would images --
things are going to block on pulling this information down.

I don't think this is possible in the presence of XRD signatures.  Those
appear to have been shelved though.

I am also not really a fan of "stop when you see what you want" because I
think it will make things harder to code and will lead to user surprises.
 For example, if a user _removes_ a single link, they may suddenly see
dozens of links _added_ that were hidden or "shadowed" before.  In the
meantime, the code itself already has to handle multiple links to work
correctly (think rel="alternate").  If it can just build up one dictionary
(multi-valued) per subject this isn't too bad, but if you also add in
multiple "domains" with priority ordering between them this makes the code
much more complicated and bug-prone.  The slight gain in performance isn't
really worth it compared to just advising people not to do that.  I wish we
could prevent them from doing it, is all.

I'm also not a big fan of relying on document ordering, as it's inherently
fragile.  The Atom working group rejected using it as a primary ordering
criteria (in favor of atom:updated by default).  It will also cause problems
for the mooted XRD JSON serialization.  The most natural JSON serialization
is a map from relvalue strings to link record lists (see discussions on
activity streams mailing list for details on why this is so).  While JSON
clearly does have a textual layout order for these maps, that information is
typically totally lost by most (all?) current JSON parsers.  So you're faced
with a Hobson's choice of using an unnatural JSOn serialization (a list)
which is annoying and difficult for JSON users, or adding additional
annoying and difficult priority fields.

Since at the end of the day you're interested in getting a set of links, and
the relative ordering of the links within a relvalue is almost _never_
relevant, I think this spec should follow the lead of the Atom specification
and ignore the document ordering.

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Mon, 26 Jul 2010 14:03:27 -0700

On 7/26/10 1:22 PM, "John Panzer" <[email-masked]> wrote:

>>> From: John Panzer
>> other than a single 'lrdd' link template, or even remove templates completely
>> and using a single link (non template) with relation 'lrdd' and a pre-defined
>> way to include the context URI as a parameter. Otherwise, link templates at
>> the host-meta level are semantically equivalent to those in a LRDD document,
>> not matter how much you try to pretend they are to be ignored.
>>
>> Can you offer some useful and predictable guidelines when a client should use
>> link templates in host-meta and when it should not? When it should only look
>> for a single 'lrdd' document for all its metadata needs? How should an
>> follow your approach and treat the two sets (link templates in host-meta, and
>> links in a LRDD document) differently?

What should I conclude from your silence on this?

>> a mechanism using link templates and defined it well. The processing rules in
>> 4.2 are just the logical implementation of this framework.
>>
>> requires the same kind of handling. From a code perspective, processing links
>> from both host-meta and LRDD isn't that much more work - you already have to
>> process both. The ability to link to more than one LRDD document is something
>> that keeps coming up again and again when people want to reuse a descriptor
>> document available elsewhere.
>>
> (generated) document and a single (generated) rel='lrdd' link to same, without
> breaking semantics?

I don't understand how the "user" can "add" link templates to your host-meta
document. Who is this user and what is her relation to the host?

Either way, multiple XRD documents can be easily combined into a single
document (assuming you resolve the relationships between conflicting
<Subject>s, demoting some to <Alias>es is there are conflicts).

XRD documents resolve into a set of links, where the XRD subject is the
context URI of the link. Therefore, semantically, it is trivial to take
links from multiple XRD with the same subject and combine them (ideally,
maintaining the order of links across documents).

> servers do things like dynamic spriting of images by combining multiple images
> into one larger image for delivery over the wire.  And XRDs are things that
> you typically won't be loading lazily as you would images -- things are going
> to block on pulling this information down.
>

Well, it depends who published the documents and who is the authority for
signing them.

> I am also not really a fan of "stop when you see what you want"

That's reality. In practice, clients either look for the first matching link
and ignore everything else, or they collect all matching links and use them
in some UI where a human selects a link. Do you have actual examples of
cases where clients seek multiple links of the same type and choose one
based on some external criteria (i.e. not document order or other link
attributes that would narrow the selection back to one)?

> because I
> think it will make things harder to code and will lead to user surprises.  For
> example, if a user _removes_ a single link, they may suddenly see dozens of
> links _added_ that were hidden or "shadowed" before.

That's always the came when you pick one link out of many, and that link
goes away. The only way to avoid it is to only allow one and break on any
errors.

> In the meantime, the
> code itself already has to handle multiple links to work correctly (think
> rel="alternate").

Can you give real world examples where 'alternate' or other relation types
are used to provide multiple links with the same relation type for a single
resource? How does the client decide which link to use?

> If it can just build up one dictionary (multi-valued) per
> subject this isn't too bad, but if you also add in multiple "domains" with
> priority ordering between them this makes the code much more complicated and
> bug-prone.

You end up with one collection of links which you can implement as a
dictionary of ordered lists for each relation type (assuming you only care
about the relation type as key). This is true no matter how many link
templates are in host-meta, or how many LRDD documents are linked to.

I don't know what you mean by 'multiple "domains"'

> The slight gain in performance isn't really worth it compared to
> just advising people not to do that.  I wish we could prevent them from doing
> it, is all.

What gain in performance? I'm happy to include a note about servers SHOULD
consolidate links as much as possible to make client processing faster.

> for the mooted XRD JSON serialization.  The most natural JSON serialization is
> a map from relvalue strings to link record lists (see discussions on activity
> streams mailing list for details on why this is so).  While JSON clearly does
> have a textual layout order for these maps, that information is typically
> totally lost by most (all?) current JSON parsers.  So you're faced with a
> Hobson's choice of using an unnatural JSOn serialization (a list) which is
> annoying and difficult for JSON users, or adding additional annoying and
> difficult priority fields.

We used to have a 'priority' attribute in XRD but the feedback was to drop
it in favor of document ordering. It is very useful to be able to express
precedence when dealing with an otherwise flat list of links. At the end of
the day, if the client ends up with multiple qualifying links, it needs to
know how to pick one. If you need multiple clients to pick the same one for
consistency, you need to come up with rules.

> and ignore the document ordering.

Applications can do that. But as a general rule, I think document order
expresses some intend and can help clients make a better, more relevant
decision. For example, get the first link and try it - if it doesn't work
move on to the next, is a very common strategy.

I am still not sure what actual changes you are proposing for the draft.

EHL

---

From: "Manger, James H" <[email-masked]>
Date: Tue, 27 Jul 2010 11:25:34 +1000

John,

> If I have a host-meta file to which a user has added, say,

> 5 rel='lrdd' templates, is it possible for me to replace all 5 with

> a single composite (generated) document and a single (generated) rel='lrdd' link

> to same, without breaking semantics?

Yes.

You could make host-meta consist solely of a single rel='lrdd' template (and indicate that host-meta can be cached for a year) and serve one XRD for each resource combining links from all 5.

> ...the relative ordering of the links within a relvalue is almost _never_ relevant,

Except when you are only expecting (or interested in) 1 link. The link (within a relvalue) that comes first does matter.

Controlling which link comes first and ignoring the ordering of the rest is no easier to implement than controlling the order of all links.

--

James Manger

---

From: John Panzer <[email-masked]>
Date: Mon, 26 Jul 2010 20:53:09 -0700

On Mon, Jul 26, 2010 at 2:03 PM, Eran Hammer-Lahav <[email-masked]>wrote:

> > On Sat, Jul 24, 2010 at 12:18 AM, Eran Hammer-Lahav <[email-masked]
> >
> > wrote:
> >>
> >>
> >>> That's also why I don't see "stop when you see the rel you're looking
> for"
> >>> as a
> completely

Mostly, that I have a day job :).  I'm not sure what the rationale behind
link templates in host-meta are exactly.  If I knew that I might be able to
answer better.

> links

The user would control the host-meta file for the host, which might be a
third level hostname (x.appspot.com for example).

> maintaining the order of links across documents).

Great!

> images

Since this isn't deployed, it's hard to come up with actual examples in the
wild.  Do you mean in analogous systems?  Sure, link rel="alternate"
type="appication/atom+xml" title=$TITLE.  Decent feed readers show the list
of $TITLEs to a user attempting to subscribe to a 'web page' and ask them
which variant they want.

>  For

If the cardinality of the link is always 1, I'd suggest it would be a good
idea to tell the user they've got something misconfigured if they supply
more than 1, rather than guessing what they meant.  Or at least, leave the
decision up to higher level pieces of code that have an opportunity to talk
to the user if appropriate/necessary.

If the cardinality is N, I suggest that it is a very bad idea to pick the
set you're going to pay attention to at random.

> resource? How does the client decide which link to use?

See above.  Also, consider rel="me" today with the Social Graph API.

> templates are in host-meta, or how many LRDD documents are linked to.
This is not true.  If I am building up a global view of _all_ links for
$SUBJECT, then I need to keep track of where the links came from & keep them
properly ordered under at least some of these proposals.  E.g., if link A
came from host-meta and preceded link B which came indirectly from link
rel="lrdd", then I need to correctly separate A and B when queried later on.

> it in favor of document ordering.

Note that this feedback was considering only the XML format.

> It is very useful to be able to express

I'd like at least a note about minimizing and consolidating LRDD external
references wherever possible.

> >>>>> ** If this is not clear, I am happy to make it more explicit in
> section
> >>>>> 4.2,
> >>>>> However, if people would like to simplify this a bit, I am open to
> (but
> >>>>> not
> >>> excited about) making a small change in which clients will first look
> for
> >>> (any)