Subject: [ESDS] IF-MAP Intro (one more time)

From: "Stuart Bailey" <[email-masked]>
Date: Thu, 8 May 2008 16:52:33 -0700

I apologize for all the resends.  I believe I was having issues with
the list-serv accepting my mail formatting.  I hope this works. :-)
==============
Hello,

I am a co-editor for a new open specification from Trusted Computing
Group (TCG) called IF-MAP.  It would appear to me that the CIB may be
functionally equivalent to what TCG has defined as a Metadata Access
Point (MAP).

The protocol, IF-MAP, defines a simple publish/search/subscribe
interface to the MAP.  The MAP is a common metadata database.  Within
the MAP, metadata automatically aggregates with respect to common
unique keys (i.e. identifiers).

While IF-MAP's initial use-cases are in coordinated security, it was
designed to meet other coordinated computing use-cases and may be
appropriate for ESDS.  The ESDS use-cases may be met simply by
defining metadata labels appropriate for the use-case using XML
Schema.  IF-MAP was designed to be modular in this way.

The binding to SOAP/HTTPS defined in IF-MAP 1.0 also makes adoption simple.

The spec can be downloaded here:
https://www.trustedcomputinggroup.org/specs/TNC/TNC_IFMAP_v1_0_r25.pdf

A white paper from InteropLabs can be downloaded here:
http://www.opus1.com/nac/teamwhitepapers/2008-06MAP.pdf

Thoughts?

Stu
_______________________________________________
ESDS mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/esds

---

From: Mark Harrison <[email-masked]>
Date: Tue, 13 May 2008 17:28:45 +0100

Hello Stuart, ESDS folks,

Many thanks for posting the info about the IF-MAP protocol.  Very  
helpful.
I've read through the white paper and through the protocol and there  
seem to be some similarities and some differences, compared with what  
ESDS is trying to do.

Having said that, I am sure that it will be worthwhile studying the  
security threats and recommendations for IF-MAP in more detail, since  
many of these will be relevant for ESDS too.

Below are a few notes I made about similarities and differences while  
reading through the IF-MAP white paper and protocol.

Similarities:
=============

Interface supports the methods:
	publish
	search (synchronous query)
	subscribe (establish subscription query)
	poll

Both are concerned with links between identifiers and metadata.

Use of a constraint-based query language keyed on identifiers.
EPCglobal's EPCIS 1.0 standard also used a constraint-based query  
language, although somewhat less flexible than what IF-MAP supports.   
EPCIS allows filtering based on multiple constraints with a logical  
AND among them - but where each individual constraint may allow  
multiple values (i.e. logical OR among values within a constraint).

Similar requirements
(support of both synchronous response as well as asynchronous  
notification to queries)
(strong support for distributed updates as well as distributed queries)

High numbers of updates per second expected

ESDS should consider IF-MAP security requirements and threat analysis,  
since it is a shared-facility referral / cross-reference service that  
may also be vulnerable to similar types of attack.

Differences:
============

IF-MAP does not provide historical information (according to section  
2.2 of the IF-MAP protocol).  A major use for ESDS is to maintain  
historical link information between the ID of a physical object (or a  
discussion topic or keyword) and multiple entities who have collected/ 
generated some information about that individual object (or discussion  
topic or keyword) at previous times.
IF-MAP does allow for linking to events, whereas ESDS primarily  
handles events (where events are tuples that contain identifiers and  
URLs of resources)

IF-MAP is concerned with building a graph to build associations  
between different identifiers and pieces of meta-data.
With ESDS, the internal data model is concerned primarily with  
timestamped events containing associations between:

	ID of object (or keyword/topic)
	URL of resource that holds more detailed information
	[ timestamps to help with sorting into chronological order and  
completeness of resultsets ]
	[ serviceType indicating what kind of resource to expect at the URL ]
	[ other metadata that provides additional context or is helpful for  
defining access control policies ]
	
Furthermore, ESDS may also need to handle an additional data model of  
the access control policies it is expected to enforce by the  
publishers of information.

Authorization model is outside the scope of IF-MAP (according to the  
footnote 3 on p49 of the IF-MAP protocol), "3 An IF-MAP server  
implementation may provide an authorization model which protects data
published by one IF-MAP client from being visible to another IF-MAP  
client. The specifics of such
an authorization model are outside the scope of this specification."

The ability to enforce an authorization model based on policies  
defined by individual publishers is a key functional requirement for  
ESDS.  Without this, many end-users will not publish data to it if it  
cannot be protected (hidden) from other end-users with whom they do  
not wish to share this information.

In IF-MAP, the link is an un-named bi-directional binding relationship  
between two identifiers. (§2.6.2 of IF-MAP protocol).
In ESDS, there will be 1-many links:

One entity (or information provider) may hold information about many  
different individual physical objects (or many different topics /  
keywords).

One individual physical object (or topic/keyword) may be linked [via  
ESDS] to multiple entities that hold more detailed information  
fragments about it.

I'm not sure whether we can align the two by using IF-MAP's multi- 
valued metadata types.  The problem might be in the granularity of  
access control, since each publisher will want to be able to specify  
which clients have visibility of the link information they asserted -  
even when the IDs of those clients are not known in advance.

Scaling requirements - ESDS may need to handle trillions of  
identifiers for physical objects (or keywords, discussion topics) and  
potentially billions of addresses of websites and corporate  
information services that hold the detailed information.
IF-MAP is currently talking about scalability to thousands or millions  
of devices and users.

Stuart - thanks again for contributing this.  I am sure that there are  
relevant ideas we can learn from the IF-MAP work.  If you feel I have  
misunderstood something in the summary above, please correct me.   
Also, if you can see a way to develop ESDS as an extension of IF-MAP,  
we'd certainly be interested to hear your views on this.

Best regards,

- Mark Harrison

On 9 May 2008, at 00:52, Stuart Bailey wrote:

The binding to SOAP/HTTPS defined in IF-MAP 1.0 also makes adoption  
simple.

Thoughts?

Stu

---

From: "Stuart Bailey" <[email-masked]>
Date: Fri, 16 May 2008 15:02:26 -0700

Mark,

Thank you for the detailed response and thoughts regarding IF-MAP's
potential for ESDS.  In response, I have  some thoughts for you and
the ESDS group to consider.  My intention is to frame out some ideas
to aid us in guiding the discussion.

Search vs. Query

While this may seem like a nit, we found in both standards discussions
and in technical discussions making a distinction between search and
query was helpful and allowed the use-cases to be met while keeping
IF-MAP simple and potentially applicable to other use-cases.

Specifically, we considered that a query requires a priori knowledge
of the schema (including both labels and structure).   Because of the
emergent structures involved in coordinated computing use-cases like
security and potentially ESDS, it may be helpful to define and use the
search metaphor as acting on semistructured data and emergent data
structures.  We felt that query is (at the high level) applicable to
structured databases and directories and search was more appropriate
for MAPs.

Directories vs. MAPs

Since it has been typical for real-time network (machine-to-machine)
database requirements to be met with directories, we also spent a lot
of time discussing why DNS, LDAP or even SNMP would not fit the
use-cases.  Here are the differences in technical properties:
directories are distributed, hierarchical, read-mostly (fairly
static), centrally updated, and queried (e.g. classic DNS and LDAP).
While MAPs are distributed, graphical (not in the visual
sense)/heterarchical, read/write (dynamic), updatable anywhere,
searched.  MAPs are similar (but not the same as) multi-master DNS
(which is only available in a proprietary implementation from MSFT)
combined with pub/sub systems.

Real-time state vs. Historical data

We felt it was prudent to follow the general database community in
separating the real-time state data from historical data.  Therefore
you are correct that a MAP does not support historical data,
analytics, or any of the properties generally associated with logging
systems or data warehousing.  In our security use-cases, those systems
like Security Event Managers (SEM) subsribe to the MAP and are able to
get more information and publish actionable "knowledge" back to the
MAP as a result of analytics.    Generally this separation is due to
implementation realities including the type of persistence,
query/search patterns, and computational demands.  Historical stores
tend to have monotonic non-transactional or "less" transactional
persistence and while the real-time state data (like MAP) tends to
live in read/write transactional stores.

I would suggest considering a similar separation for the ESDS
use-cases.  This may mean that the CIB has one function (like a MAP)
and there is another piece of the architecture for correlation and
analytics of historical information which interfaces with the CIB
using something like IF-MAP.

One-to-many Links

Maybe it is not clear in the spec or I was unclear about your comment;
so I would like to spend time clarifying this topic collaboratively.
IF-MAP does support 1-many links.  This may be unclear because the
relationships between identifiers are published (at run-time) as
metadata tags on links.   This subtle feature of IF-MAP is what allows
the schema to be run-time dynamic and emergent and leads us to a
search metaphor rather than query as stated above.    In my opinion,
this is a key differentiator of IF-MAP.  To my knowledge (and many of
the folks who have studied IF-MAP thus far) there are no other systems
that work this way though it is somewhat akin to RDF.

Scale

I suspect that the next version of IF-MAP spec will also say
"trillions and beyond".  That prose was written in 2006 and even the
security use-cases are facing massive scale requirements.  That being
said, we attempted not limit scale in the initial specification, and I
would be grateful if you could point out areas of the spec which you
were concerned about scale as a practical matter.

I hope this clarifies some of the points you mentioned.   Below is a
link to an IF-MAP tutorial (Draft) which highlights some of these
points and introduces some technical aspects of IF-MAP.

http://www.infoblox.com/solutions/pdf/IFMAP-tutorial-ppt.pdf

We have hoped, in the design of IF-MAP, we have the start of a
protocol which could be to "coordinated computing" what TCP/IP is to
connectivity: simple and meets more than 80% of the requirements.

Thoughts?

Thanks much,
Stu