Subject: Re: Exploring a new WG around Atom

From: Mark Nottingham <[email-masked]>
Date: Mon, 13 Jul 2009 14:56:27 +1000

WRT queries, just in case you weren't aware:
   http://tools.ietf.org/html/draft-nottingham-atompub-fiql-00

Cheers,

On 13/06/2009, at 12:12 AM, Colm Divilly wrote:

>
> I'm interested, specific areas of interest would be:
>
> - meta-publishing
> - query-feed discovery/navigation, i.e. discovering what queries can  
> be applied to a collection, or even a mechanism for end-users to  
> publish their own custom queries searching over AtomPub  
> collection(s), and advertise those queries to other users.
>
> Regards,
> Colm Divilly
>
> Nikunj R. Mehta wrote:
>>
>> Recent discussions on the atom-syntax mailing list has revealed  
>> interest in creating a new WG to look at several extensions of  
>> Atom. Here are several extensions that have been discussed:
>>
>> 1. Hierarchy operations including meta-publishing
>> 2. In-line representation
>> 3. Bi-directional text
>>
>> Please respond to this email if you are interested in helping to  
>> explore a new WG further. Please also identify if you have a  
>> specific interest area to consider for a potential new Atom WG.
>>
>> Nikunj
>> http://o-micron.blogspot.com
>>
>

--
Mark Nottingham     http://www.mnot.net/

---

From: Colm Divilly <[email-masked]>
Date: Tue, 14 Jul 2009 12:57:46 +0100

Hi Mark,
 yes I'd seen this before, but neglected to give you feedback on it, I 
basically concur with feedback given by Brian Smith [1], James Snell [2] 
previously.

 * I'm not convinced of the utility of being just able to filter within 
a single feed. I think a query language (e.g XQuery) that permitted 
defining a feed composed of entries from many disparate Collections of 
entries might be more useful.
 * Don't like the prefix based selector mechanism at all, as Brian said 
it goes contrary to the normal practices for dealing with QNames. I also 
would prefer an XPath subset syntax.
 * The fq:interface idea has some merit, although I think James' 
link-template mechanism [2] is more in line with what I had been 
thinking about. Both also seem similar to Open Search [3]
  * I'd like to be able to go further, in a similar fashion to how 
categories can be defined in-line and out-of-line in an app:collection 
element, I would like to be able to link to have a 'rel="queries"' link 
that links to a document that enumerates all the queries that could be 
applied. This is would help reduce duplication (where two or more feeds 
wish to reference the same set of queries),  and also avoid cluttering 
the feed when the number of queries starts to grow large.
 * I also feed that any query mechanism should be able to select on 
values of HTTP headers supplied with the request in addition to any uri 
parameters.

A minor nit: Section 4 has the following example:
 http://example.com/feed.atom?query=title==*new*,author==bob*
  Shouldn't that be name==bob* ? (/entry/author/name...)

In summary I think we need a few specifications:

 * The first defines a means of discovering what queries are possible, 
ala link-templates or OpenSearch.
 * The second defines a means of defining queries, It should be possible 
to do this in such a way that the specification supports multiple query 
lanaguages. I think of this as 'Published Queries' (JCR has a similar 
concept: persistent queries [4]), a user submits a query specification 
which enumerates all the 'link-template' meta-data plus an actual query 
that can be processed by the server to generate the required results, 
plus meta-data to bind the parameters specified in the link-template to 
parameters in the actual query.
 *  A third specification might define a lingua-franca query language 
that could be used instead of a server dependent query language, to be 
used when publishing queries.
(Ordered from most likely to see widespread adoption to least likely)

Have you had any implementation experience with FIQL since you wrote the 
draft? any insights to share with us?

Regards,
Colm

[1] http://www.imc.org/atom-syntax/mail-archive/msg20131.html
[2] http://www.imc.org/atom-syntax/mail-archive/msg20146.html
[3] http://www.opensearch.org/Specifications/OpenSearch/1.1
[4] 
http://jcp.org/aboutJava/communityprocess/maintenance/jsr170/index.html 
(jsr-170-1.0.1.pdf, section 6.6.11 p128)

---

From: Erik Wilde <[email-masked]>
Date: Tue, 14 Jul 2009 10:40:30 -0700

hello.

Colm Divilly wrote:
> * I'm not convinced of the utility of being just able to filter within a 
> single feed. I think a query language (e.g XQuery) that permitted 

i do agree that a feed query language should be able to do that, but not 
in the sense that it actually takes multiple feeds as input. you can 
always aggregate feeds into one feed, retain their source information, 
and then query that aggregate feed. there are two main advantages to 
this approach:

- aggregation and querying are independent, and you can chain and 
combine both as you like. since feeds have the built0in ability to be 
aggregated, there is no need to replicate that in the query language.

- by using only one feed as input it is easily possible to use a query 
in the feed's URI. you loose that ability if you can have multiple 
feeds, and i think this is a feature that is very powerful.

> would prefer an XPath subset syntax.

same here. more importantly, FIQL is operating very much on the feed 
syntax level. i think we need something like a "feed infoset", i.e. an 
abstract data model of a feed that does ignore certain syntax issues. 
also, in the end i'd like to see a language that essentially would be a 
"collection query language", because in the end, you don't really query 
the feed, but the underlying collection, and you get the query results 
as a feed.

wrt prefixes and namespaces: XML namespaces always are hard to deal 
with. my approach is to assume that the prefixes used in the feed can be 
used as defined by the feed, but that queries can also define their own 
prefixes and use those. this is a bit brittle if feeds change their 
namespace declarations, but that will not happen too often.

> thinking about. Both also seem similar to Open Search [3]

some sort of "service description" of a feed definitely is very useful, 
so that services can publish what sort of query they support. it may be 
something as simple as a simple keyword search (a la opensearch), or it 
may be something as complex as a join spanning multiple entries.

> the feed when the number of queries starts to grow large.

nice idea, "feed query interfaces" thus would become standalone documents.

> values of HTTP headers supplied with the request in addition to any uri 

hm. i am not sure about this one, even though i see that it might be 
convenient. one of the things i think should happen for a feed query 
language would be to rather see it as a collection query language and 
make it a little less dependent on the exact feed syntax. allowing HTTP 
into the language would couple it more tightly with its implementation 
details. we might consider a magic prefix, though, that (unless it is 
bound to something else) could be used to select HTTP parameters in the 
query.

> ala link-templates or OpenSearch.

part of that would be to first define the data model we are querying on. 
in essence, it would be the infoset or XDM for feeds, and i think in the 
long run, we really need something like that. tools like the universal 
feed parser give you some similar to such a model, but these models are 
tool-specific, and sometimes not very convenient (the universal feed 
parser, for example, has no standard way how to expose extensions). i 
think we should have such a "feed infoset" first, but my guess is that 
some might see that as too much of a top-down approach.

> parameters in the actual query.

that would be the query language operating on the feed infoset, right? 
if we think in XML terms, this would be XPath. the input of such a 
language should be a feed infoset, the output probably one as well.

> (Ordered from most likely to see widespread adoption to least likely)

i would see that the third thing might be the query interface that would 
allow providers to publish query templates, if they wish to do so. this 
might be particularly important for all the query parameters that 
providers might support that are not represented in the feed itself, but 
should be queryable anyway.

cheers,

erik wilde   tel:+1-510-6432253 - fax:+1-510-6425814
        [email-masked]  -  http://dret.net/netdret
        UC Berkeley - School of Information (ISchool)

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 15 Jul 2009 15:09:15 +1000

It has been implemented, but not publicly.

 From what I've seen, people tend to either think it hits the 80/20  
point right on, or they want a *lot* more (e.g., full XQuery, YQL,  
etc.).

Specific responses below.

On 14/07/2009, at 9:57 PM, Colm Divilly wrote:

> Hi Mark,
> yes I'd seen this before, but neglected to give you feedback on it,  
> I basically concur with feedback given by Brian Smith [1], James  
> Snell [2] previously.
>
> * I'm not convinced of the utility of being just able to filter  
> within a single feed. I think a query language (e.g XQuery) that  
> permitted defining a feed composed of entries from many disparate  
> Collections of entries might be more useful.

FIQL presumes that the definition of what the target feed up is up to  
its publisher, so that a collection of feeds can become a single feed  
if desired.

> * Don't like the prefix based selector mechanism at all, as Brian  
> said it goes contrary to the normal practices for dealing with  
> QNames. I also would prefer an XPath subset syntax.

I have little regard for QNames in general, so I don't mind abusing  
them.

> * The fq:interface idea has some merit, although I think James' link- 
> template mechanism [2] is more in line with what I had been thinking  
> about. Both also seem similar to Open Search [3]
> categories can be defined in-line and out-of-line in an  
> app:collection element, I would like to be able to link to have a  
> 'rel="queries"' link that links to a document that enumerates all  
> the queries that could be applied. This is would help reduce  
> duplication (where two or more feeds wish to reference the same set  
> of queries),  and also avoid cluttering the feed when the number of  
> queries starts to grow large.
> values of HTTP headers supplied with the request in addition to any  
> uri parameters.

What's your use case?

> * The first defines a means of discovering what queries are  
> possible, ala link-templates or OpenSearch.

Agreed.

> * The second defines a means of defining queries, It should be  
> possible to do this in such a way that the specification supports  
> multiple query lanaguages. I think of this as 'Published  
> Queries' (JCR has a similar concept: persistent queries [4]), a user  
> submits a query specification which enumerates all the 'link- 
> template' meta-data plus an actual query that can be processed by  
> the server to generate the required results, plus meta-data to bind  
> the parameters specified in the link-template to parameters in the  
> actual query.

I think that sounds interesting.

> that could be used instead of a server dependent query language, to  
> be used when publishing queries.

Possibly.

> Have you had any implementation experience with FIQL since you wrote  
> the draft? any insights to share with us?
>
> Regards,
> Colm
>
> [4] http://jcp.org/aboutJava/communityprocess/maintenance/jsr170/index.html

---

From: Colm Divilly <[email-masked]>
Date: Mon, 20 Jul 2009 15:14:32 +0100

Mark Nottingham wrote:
> point right on, or they want a *lot* more (e.g., full XQuery, YQL, etc.).
:) Feels like I'm falling in the latter group, but I think it wouldn't 
need much changing to make me happy:

To recap, FIQL does the following:
 
 - enumerates the available queries
 - for each query enumerate the parameters for the query
 - for each parameter enumerate its type
 - for each parameter define how it 'selects' from the feed.

It is this last bullet that I am concerned with, I'm not convinced that 
a query definition needs to tell clients how the query selects from the 
feed. Were there particular use cases that would be inhibited by the 
client not knowing how selection happens? Any reasons the following 
would not suffice ?

     <fq:interface
      xmlns:fiql="http://purl.org/syndication/query"
      template="/feed-search?{-join|&|title,foo,bar}">
       <fq:param name="title"/>
       <fq:param name="foo"
        type="http://purl.org/syndication/query/date"/>
       <fq:param name="bar"
        type="http://purl.org/syndication/query/numeric"/>
     </fq:interface>

In the above example the query interface just describes the inputs to 
the query but says nothing about how those inputs are applied to 
generate a result, the name attribute is no longer overloaded with a 
'selector' semantic.

If a feed provider wishes to advertise how query parameters are bound to 
entry elements/attributes they could include a fq:param/@path attribute 
in order to communicate that information.

Regards,
Colm

>> values of HTTP headers supplied with the request in addition to any 
Just a couple of messy things:

 - Query needs to re-write URIs and/or xml:base decls in the response, 
depending on the value of a custom HTTP header (in order to re-write 
URIs to their publicly accesible endpoints)
 - Query needs to filter results based on user identity, which is 
provided by means of yet another custom HTTP header (provided by a web 
SSO mechanism).

Suggestions for better ways to handle these very welcome...

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 23 Jul 2009 12:14:21 +1000

On 21/07/2009, at 12:14 AM, Colm Divilly wrote:

> :) Feels like I'm falling in the latter group, but I think it  
> wouldn't need much changing to make me happy:
>
> It is this last bullet that I am concerned with, I'm not convinced  
> that a query definition needs to tell clients how the query selects  
> from the feed. Were there particular use cases that would be  
> inhibited by the client not knowing how selection happens? Any  
> reasons the following would not suffice ?
>
>    <fq:interface
> In the above example the query interface just describes the inputs  
> to the query but says nothing about how those inputs are applied to  
> If a feed provider wishes to advertise how query parameters are  
> bound to entry elements/attributes they could include a fq:param/ 
> @path attribute in order to communicate that information.
>
> Regards,
> Colm
>
>>>
>>> values of HTTP headers supplied with the request in addition to  
>>> any uri parameters.
>>
>> What's your use case?
>>
>>
> - Query needs to re-write URIs and/or xml:base decls in the  
> response, depending on the value of a custom HTTP header (in order  
> to re-write URIs to their publicly accesible endpoints)

Hmm. Probably best to do it on the client side, or with a URI query  
parameter...

> provided by means of yet another custom HTTP header (provided by a  
> web SSO mechanism).

I think this is more about feed selection (usually by the URI) than  
actual query processing...

Cheers,