Subject: Re: [ESDS] IF-MAP Intro (one more time)

From: Mark Harrison <[email-masked]>
Date: Fri, 30 May 2008 13:52:56 +0100

Dear Stuart,

Many thanks for your further explanation about IF-MAP - and sorry for  
the delayed response.  I'll try to respond to some of these now - then  
continue with an attempt to consider the underlying ESDS / Discovery  
Services data model as a map, so you can assess how well it might fit  
to the IF-MAP protocol.  I have also tried to highlight some of the  
specific needs that a DS would require, which might not be such strong  
needs for the use of IF-MAP within a single organization.

Search vs Query:

You explained that query is more suited to structured databases and  
directories, whereas search is better suited for maps with more semi- 
structured data.  The underlying data model for ESDS can probably be  
expressed as a fairly well-defined structure/schema, even though some  
meta-data fields might be optional or allow for null values.

Directories vs Maps:

You said that:
Directories are distributed, hierarchical, read-mostly (fairly  
static), centrally updated and queried
MAPS are distributed, graphical/heterarchical, read/write (dynamic),  
updateable anywhere, searched.
I think the technical properties that apply to Discovery Services are:
DS are distributed, peer-to-peer rather than hierarchical,  
structurally fairly static, but with dynamic read/write of content,  
updateable by multiple entities (with each entity maintaining control  
over who can read/write the links and metadata it publishes), able to  
be queried primarily on the ID of the individual physical object (and  
optionally some additional metadata constraints, but not primarily on  
the ID of the resource, nor starting from arbitrary pieces of metadata  
information) - and with the query results being filtered to comply  
with the access control policies set by each resource owner, before  
being returned to the client.

Real-time vs Historical data.

A DS should also support subscription-based queries (standing queries)  
by clients, so that they can receive updates about future events/ 
records/links related to the objects they previously handled.  In this  
sense, there is an analogy with the Security Event Managers (SEM)  
subscribing to the map.

However, the problem is that a company receiving an individual object  
for the first time may only become aware of its object ID when it  
physically arrives or only shortly before then (e.g. via an Advanced  
Shipping Notice or Despatch Advice).  Until the company becomes aware  
of the object's ID, it is very difficult for it to subscribe to a DS  
to collect event information about that object - other than  
subscribing to all event information (and at the time that the event  
information is generated, it may not yet have been granted access  
permission to even receive those events, since it might not yet have a  
relationship with the object's originator/manufacturer or with the  
company that shipped the object to them).

So, the problem for ESDS is that when an individual object arrives,  
the company that receives it might need to check where it has come  
from (i.e. to do a historical 'trace') - and may have only just  
recently been granted access rights to make such a trace query or  
receive the results of such a trace query.

Not until no further companies are likely to handle the object can we  
realistically purge and archive the historical link information from a  
DS.

Regarding your comment about the consolidated information broker  
(CIB), this probably will consist of a number of modules. One module  
may be an 'event gathering layer' that primarily gathers all link  
data, then follow each link to obtain finer grained location data from  
each resource.  It may also need to make iterative queries to follow  
changes of the ID of the object to track (e.g. upon changes of  
containment/ aggregation and re-labeling of objects).  Another module  
may perform the analysis of the data, identifying correlations and  
deviations and making predictions about future locations or routing  
options.  We are doing work in the EU BRIDGE project about a framework  
for enhanced track & trace and supply chain control at the granularity  
of individual objects - so this is somewhat similar to the idea of the  
CIB that David Potter introduced.  There could well be a use for IF- 
MAP in that area.

One-to-many links:

Thanks for the explanation.  As you can see from the diagram below, we  
will likely have 1-many links from objects to resources and also 1- 
many links from resource to objects - i.e. many-many links.  If as you  
say, the meta-data tags on links are published at runtime, then there  
are ESDS metadata fields such as eventTime, businessStep and  
recordTime which would fit nicely as run-time published meta-data.   
Would I be right in thinking that IF-MAP also supports run-time  
creation of links, as well as run-time attaching of meta-data to links?

Scale:

Discovery Services  will need to be scalable to trillions of new  
objects per year and perhaps millions or even billions of resources,  
with many-many links between them, probably somewhere between 10-100  
links per individual object, and in many cases, links remaining live  
for between 1-50 years, depending on the longevity of the object.  I  
have some concerns about whether search might be significantly more  
time-consuming than query, given that our schema/structure is probably  
fairly static, non-emergent - just extremely voluminous.

I have tried to represent the underlying data model for ESDS /  
Discovery Services as a map.

The main identifiers would be:

1) identifiers of individual physical objects (shown as the yellow  
ovals).
	These might be identified by strings or by URNs and may be globally  
or only locally unique IDs.

2) identifiers of resources
	These identifiers should probably not be based on the URL of the  
resource (since that might need to be updated over time)
	- but preferably an ID that is locally unique for all time
	To each resource, standard metadata fields would include 'service  
Type' - indicating the kind of service available at each resource.
	serviceType would usually be a string from an enumerated list  
representing one choice from among
	(web pages, XML data, web services, EPC Information Services,  
Discovery Services [ESDS], ...)
	Each resource would also have a metadata field for its current URL.

	The serviceType of a resource should never need to change.
	However, the URL may need to be updateable *only* by the owner of the  
resource.
	For audit reasons, we may need to be able to maintain a log of the  
previous URLs of resources and the time ranges over which they were in  
effect.  However, that might not need to be stored in the live map.

Regarding publishing, it would usually be the owner of a resource who  
creates a link from a resource to one or more objects that they have  
handled.  The publisher would normally also be able to add metadata  
fields to the link to indicate:

	eventTime - a timestamp corresponding to when they actually handled  
the object
	business Step - a string or URN indicating the business step (e.g.  
shipping or receiving) - a.k.a. lifecycleStepID in ESDS drafts

At the same time that the resource owner creates the link from the  
resource to an object and adds the metadata  to the link, the DS  
itself should be triggered to add an additional piece of metadata to  
the newly created link to record the recordTime - a timestamp  
corresponding to when the link was created within the DS.

In general, the number of identifiers of individual objects will be  
several orders of magnitude greater than the number of resources.
However, there will be many-many mappings between objects and resources.

For searching (or querying a DS), it will be usual to begin by  
specifying the ID of an individual physical object and then to  
retrieve links to the various information resources that are linked to  
it, together with the metadata associated with the link and with each  
information resource.  Like in the IF-MAP protocol, this information  
will be returned to the query client in XML format.

I might be wrong but I think there are probably few cases where a DS  
might want to allow a search to begin from another arbitrary starting  
point on the map.  For example, if it were allowed to start a search  
from the ID or URL of a particular resource, it might be possible to  
abuse the DS, to mine it to extract information about volumes and  
flows of goods.

Likewise, starting a search based only on an event Time would probably  
produce many spurious results - although it would be perfectly  
reasonable to begin a search with the ID of an individual physical  
object and to bound this with additional constraints, such as a time  
range for the eventTime.

However, Discovery Services do have some rather specific access  
control features, which might not be such strong requirements for  
general IF-MAP use within an organization.  These are:

1) Only the owner of a resource is allowed to create a link from the  
resource to an object.
2) The owner of a resource must have complete control over who has  
visibility of the links from its resources.
	This means that the results of a search of the map may need to  
undergo some post-search filtering to suppress links and metadata that  
the resource owner does not wish to reveal to a particular client.
This also means that in most cases, each different client may receive  
a different result-set, even for the same query, depending on the  
access privileges that have been granted to them, as defined in a  
policy specified by the resource owner, based on the trust  
relationship between the resource owner and the client, together with  
the ID ranges/patterns of the objects that are linked and other  
metadata parameters.
3) As an individual physical object passes from one resource owner to  
another, the links to previous resources visited must be maintained in  
the live map until the physical object has completed its journey  
through the supply chain or lifecycle.  In some industry sectors,  
there may even be a requirement for these links to be readily  
searchable for 10-20 years after the object has reached the end of its  
life.

Unlike the example on slide 16 of the IF-MAP tutorial, the map for the  
underlying ESDS data set might not increase in complexity over time  
(in the sense of different kinds of links being established or changes  
in the schema) - but it will certainly grow in volume over time,  
although most of those links will be between physical objects and  
information resources and most of those links will have the same set  
of metadata attribute keys attached to each link, albeit with  
different values and sometimes null values.

So far, I've considered how the underlying data model for DS might be  
expressed as a map.  However, the major complexity in Discovery  
Services is not (only) in supporting massive scalability of the  
underlying records - but in the expression, interpretation and  
enforcement of the access control policies (specified by each resource  
owner), which should limit visibility.  Because these access control  
policies may be quite expressive and involve a number of different  
parameters such as ID of resource owner, ID of client, time range,  
pattern or range of IDs of objects the client is allowed to see,  
business steps the client is allowed to see (e.g. only shipping but  
not receiving or vice versa), IF-MAP could be helpful for handling  
these policies if a search paradigm or RDF-like reasoning is needed to  
interpret them.

I hope this attempt to consider Discovery Services within the IF-MAP  
framework helps us to continue the conversation about the  
applicability of IF-MAP to support ESDS.

Best regards,

- Mark

_______________________________________________
ESDS mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/esds