Subject: [Ecrit] LoST Sync Update

From: Hannes Tschofenig <[email-masked]>
Date: Sat, 06 Mar 2010 22:44:51 +0200

I have re-submitted a version of the LoST sync document.
The update reflects the comments from Karl Heinz and Alex.

As you have seen, I have solicited feedback from the XML / Relax NG 
experts regarding the schema. I hope to get a response soon.

I would also like to get some feedback from comments I filed against the 
NENA i3 stage 3 specification. The NENA spec hints to a different 
synchronization mechanisms (as Brian mentioned already many times) but I 
failed to understand how it works. I would like to see what the 
similarity and the differences are. Since the i3 spec is supposed to be 
completed very soon I assume clarifications regarding the 
synchronization aspects will have to get discussed as well.

Ciao
Hannes

---

From: Brian Rosen <[email-masked]>
Date: Sat, 06 Mar 2010 16:33:50 -0500

In the NENA i3 scheme, the interface to the LoST server is a set of GIS
layers, and specifically, there are sets of polygons that define service
boundaries.  Each of these boundaries has attributes for the service URN
URI, etc that are returned in the LosT response when the query location is
found within the boundary.

For a civic address on the query, in theory, the LoST server does a geocode
and then point-in-polygon, but an implementation can compute a table to do
lookups for civics.

The provisioning interface is a layer replication mechanism that uses two
parts:
1. A Web Feature Service (an OGC standard that does insert/delete/modify
operations)
2. An RSS feed of WFS commands

We have a "master GIS" that contains the layers that the LoST server needs.
These include polygons for the political boundaries, plus either (or both)
centerlines or road geometry, address points or parcel/subparcel polygons,
etc.  There is a feature layer or attribute on a feature for each of the
CAtypes.

As normal GIS maintenance is performed on the master (adding streets,
changing boundaries, etc.), the WFS commands that replicate that change are
sent as an RSS feed to the LoST server, which replicates the edits on its
database, maintaining synchrony with the master.

We allow the LoST server to have an OUTPUT layer replication service, which
is used on an aggregating LoST server.  For example, each county in a state
could have a LoST server, and the state could have a LoST server which is
the aggregate of the individual counties.  There is a gap/overlap reporting
mechanism for this.

One of the primary goals for this mechanism is to enable the boundaries to
be changed on the GIS system by a normal edit function, and post the cache
TTL, routes changes happen immediately.  We use this in disaster scenarios
when our other mechanisms to reroute calls don't work for the actual
situation on the ground.

Brian

On 3/6/10 3:44 PM, "Hannes Tschofenig" <[email-masked]> wrote:

> _______________________________________________
> Ecrit mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ecrit

---

From: Henning Schulzrinne <[email-masked]>
Date: Sat, 6 Mar 2010 17:13:16 -0500

An RSS (I hope you're at least following IETF standards and using ATOM...) feed is, as far as I understand, one long set of XML records. This isn't a problem for a newsfeed on Slashdot, since the system presumably exports only a day's worth of stories, but it's unclear to me how this would work for updates that could span years. Would each server get the full list of operations going back to the dawn of time and then has to sort out which ones it has already performed? If not, what happens when a new server joins? (RSS is also a "pull" format, i.e., involves polling, which isn't ideal, but probably won't matter much for the small number of RSS clients hanging of a GIS database.)

Henning

---

From: Brian Rosen <[email-masked]>
Date: Sat, 06 Mar 2010 19:46:56 -0500

Yes, it's ATOM.  I'm not an ATOM expert, but there is some kind of record ID
and some way to filter updates.

We will "checkpoint" from time to time: get a fresh copy of the entire
layer, and start implementing updates from that point.

Brian

On 3/6/10 5:13 PM, "Henning Schulzrinne" <[email-masked]> wrote:

> An RSS (I hope you're at least following IETF standards and using ATOM...)
> feed is, as far as I understand, one long set of XML records. This isn't a
> problem for a newsfeed on Slashdot, since the system presumably exports only a
> day's worth of stories, but it's unclear to me how this would work for updates
> that could span years. Would each server get the full list of operations going
> back to the dawn of time and then has to sort out which ones it has already
> performed? If not, what happens when a new server joins? (RSS is also a "pull"
> format, i.e., involves polling, which isn't ideal, but probably won't matter
> much for the small number of RSS clients hanging of a GIS database.)
> 
> Henning
>

---

From: Hannes Tschofenig <[email-masked]>
Date: Sun, 07 Mar 2010 09:44:26 +0200

Hi Brian,

> found within the boundary.

The sentence "the interface to the LoST server is a set of GIS layers" 
sounds nice but does not make a lot of sense to me.
An interface is a protocol or an API. A GIS layer, as you describe it in 
the following sentences, is essentially mapping data.

   Mapping Data:
      Mapping data refers to a data structure that is defined in
      Section 5 <http://tools.ietf.org/html/draft-tschofenig-ecrit-rfc5222bis-00#section-5> of RFC 5222.

> and then point-in-polygon, but an implementation can compute a table to do
The question here should be how do you describe civic boundaries. If you 
have no story for it then it is a bit mood to return civic boundaries in 
a LoST response to the LoST client as well.

> The provisioning interface is a layer replication mechanism that uses two

I hate the term "layer replication" mechanism because it is so 
meaningless in our concepts. We replicate mapping data because that is 
what LoST servers at the end need. If you replicate something else then 
there is the question of how this maps to the fields required for the 
LoST mapping data.

> 2. An RSS feed of WFS commands
Nice. Solutions by referencing other standards that nobody understands.

While I am confident that the OGC knows what they want their concept to 
use for I am not sure how well the story is complete for dealing with 
LoST servers. To me this concept sounds like being at the stage of early 
design work.

I want to have an example of how this works. Take a look at the example 
we have in the LoST sync document and try to create something with these 
two technologies.

There are two implementations of LoST Sync; the code is available.
Are there implementions of what you describe for usage with LoST?

> We have a "master GIS" that contains the layers that the LoST server needs.
How well does this concept map to the term defined in the LoST 
architecture document called "authoritative mapping server" (see Section 
3 of RFC 5582)?

> etc.  There is a feature layer or attribute on a feature for each of the

Why do these things matter for LoST operation? What matters in LoST is 
the mapping data in a LoST server. Whether these boundaries relate to 
political boundaries does not matter.

> the aggregate of the individual counties.  There is a gap/overlap reporting
Aggregating the mapping data in LoST servers all the way up to FGs is 
OK. That's what we also describe in the LoST Sync document.

> when our other mechanisms to reroute calls don't work for the actual
Here is another disconnect, namely the believe that changing to mapping 
data at a LoST server (even when replicated immediately to other LoST 
servers) leads to immediate route changes. No, it doesn't since we have 
built into LoST the caching concept to improve reliability, robustness 
and scalability. The downside of it is that, when used of course, 
changes to the mapping data are in effect only after a certain amount of 
time.

Unless you make some further assumptions, which you did not state in 
your paragraph above.

Ciao
Hannes

---

From: Brian Rosen <[email-masked]>
Date: Sun, 07 Mar 2010 10:52:49 -0500

Hannes

I am not sure of the point of this thread.  No one has suggested that the
NENA LoST provisioning idea be somehow adopted by the IETF (although I am
sure we wouldn't object if it did).  No one has suggested that there is
anything "wrong" with lost-sync.  This is a thread on the ecrit list, not
the NENA i3 list.

See replies in-line

> the following sentences, is essentially mapping data.
Yes, it is "mapping data".  It is the data that drive LoST query resolution.
It is represented as a set of GIS layers that are maintained on the LoST
server in sync with the same layers in the master GIS system which is the
authoritative source for the data.  The data is the layers, the transport
protocol is WFS with an ATOM feed
>       Section 5 
> <http://tools.ietf.org/html/draft-tschofenig-ecrit-rfc5222bis-00#section-5> of
> RFC 5222.
I am not even going to go look to see if the definition in there is a
complete overlap in definitions.
> a LoST response to the LoST client as well.
Some GIS basics, and please remember that I am not a GIS expert.

A GIS system is a database.

The database has records.  The geodetic records are called "features".  A
feature is fundamentally what we often call a 'geoshape' = point, line,
circle, polygon,...  A feature can have attributes (=values associated with
the feature, for example, a name, a record-id, or some other conventional
data item)

Conventionally, features are grouped into "layers" which is a set of
features that have some relationship.  Layers can have attributes, although
we often describe a set of attributes that each feature in the layer must
have.  For example, consider a 'street centerline' layer.  The features in
this layer are lines that represent a segment of a street.  Each of the
lines in the layer would have attributes for (among other things)
 * The name of the street
 * The municipality it lies in
 * The first and last address number for each side of the street

A simpler example would be a county layer, which has polygons that represent
the boundary of the county, with attributes for the name of the county, a
code, and perhaps the state it belongs to.

Another interesting layer would be an address point layer. The features are
points.  The attributes are "house number", street name, etc.

So, we have a layer for each of the CAtypes
  * A State layer with polygons for a state
  * A county layer with polygons for counties
  * A municipality layer with polygons for cites, towns
  * A district layer
  * A neighborhood layer
  * A street layer, which could be a street centerline layer as described
     above, or a set of polygons that describe the actual street geometry
  * An address layer, or parcel and subparcel layers that represent
     addressable parts of a road
  ...

This is a map: it lets us locate a position either by a geo or a civic.  To
find the location of a civic, you do a common GIS operation which does
successive searches to find the address point or polygon on the road in the
neighborhood (if specified) in the district (if specified) in the
municipality, in the county (if specified) in the state, using the geometry
and/or the attributes.  In most GIS systems, this would be a conventional
multi-table JOIN, because there are attributes in the features that relate
them to the next highest level.

If the result of the search was exactly one feature, then we have a unique
location for the given civic address, and we take the point (if it is an
address point) or a polygon (if it is a parcel/subparcel/...) that
represents it (that is, we "geocode" to go from a civic to a geo).  We then
do PinP on the PSAP boundary layer to determine the LoST mapping.  In a bit
more detail, we do PinP with the sets of polygons that have a service URN
attribute matching the service urn in the LoST request, and return the URI
and other information in the corresponding attributes on the selected
service boundary polygon set.

This makes LosT mapping either a straight PinP for a geo, or geocode
followed by PinP for a civic.

One important reason we do it this way is that the GIS system with all these
layers already exist in most PSAPs in North America.  The database was
created to handle wireless calls, and is used to first display where a
mobile caller is calling from, and then to determine the civic address to
dispatch responders to (reverse geocode).  Following the latest NENA
guidance, even the current table driven routing database (the "MSAG") that
the 9-1-1 system now uses is DERIVED from the GIS system.  When there are
changes in addressing: perhaps a new street is created, or a
"un-incorporated" portion of a county boundary is annexed to a city, the
change is made on the master GIS system, and a new MSAG is generated from
that.  Going forward, essentially ALL of the location data that drives 9-1-1
is derived directly from the GIS.  For i3 that means the LoST mapping data
is derived from the master GIS system.  Using layer replication makes this
simple, and we use the same mechanism for other services in i3.

> there is the question of how this maps to the fields required for the
Too bad.  It describes exactly what we do.  We have a set of layers in the
master GIS system that represent the same layers in the LoST server used to
do the LoST mapping.  We replicate those layers from the master GIS to the
slave LoST server, so that any edits to the master GIS quickly replicate on
the LoST server.  The LoST server IS a GIS system in this view.
> Nice. Solutions by referencing other standards that nobody understands.
Hello?  The GIS folks know what a WFS is.  NENA is not restricted to IETF
standards.  The GIS guys don't know what an ATOM feed is, but it's a simple
enough (IETF!) standard to explain to them.  WFS isn't that simple, but for
your purposes, it's an add/delete/edit mechanism for a GIS system that
operates on features.  It uses web services for transport.

> LoST servers. To me this concept sounds like being at the stage of early
As is lost-sync, the difference being that the mechanism for layer sync (WFS
+ ATOM) has been implemented, trialed, and found to work in Canada.

> we have in the LoST sync document and try to create something with these
I explained this above.  If you are still confused, ask any GIS expert how
geocoding works.  After that, it works the same way a geo query to LoST
would work.
> Are there implementions of what you describe for usage with LoST?
Yes, although not applied to a LoST server.  However, the idea of geocode
then PinP is not exactly stretching anything very far.

> architecture document called "authoritative mapping server" (see Section
The master GIS system is not a LoST server.  It is the source of data the
authoritative mapping server derives mappings from.  It is how we provision
the authoritative mapping server.  We use the same protocol to provision any
LoST server.
> political boundaries does not matter.
See above.  We use the layers to geocode civic to geo.  Then we use the PSAP
service boundary polygons to determine mapping in geo-space.

> changes to the mapping data are in effect only after a certain amount of
Did you see the "post the cache TTL" phrase in what I wrote?

We keep the TTLs short, for this reason.  We don't think there is any
problem maintaining the servers reliable or scalable with short TTLs.

---

From: "Tschofenig, Hannes (NSN - FI/Espoo)" <[email-masked]>
Date: Wed, 10 Mar 2010 11:06:25 +0200

Hi Brian, 

Thanks for your detailed description. Just with the current text in NENA
i3 stage 3 I was not able to capture the full picture. Your additional
description at the EENA NG112 TC conference call helped me as well. 

>From an LoST Sync document point of few I believe we need to cover both
civic as well as geo boundaries. If I would like to re-use the GeoRSS
mechanism my understanding is that GeoRSS only supports the encoding of
GML and would not allow me to carry any civic information in there. 

The other question that I have is whether the push mechanism is
necessary with LoST Sync given that you are using GeoRSS, which utilizes
a poll approach. 

Ciao
Hannes 

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] 
>On Behalf Of ext Brian Rosen
>Sent: 07 March, 2010 17:53
>To: Hannes Tschofenig
>Cc: 'ECRIT'
>Subject: Re: [Ecrit] LoST Sync Update
>
>Hannes
>
>I am not sure of the point of this thread.  No one has 
>suggested that the NENA LoST provisioning idea be somehow 
>adopted by the IETF (although I am sure we wouldn't object if 
>it did).  No one has suggested that there is anything "wrong" 
>with lost-sync.  This is a thread on the ecrit list, not the 
>NENA i3 list.
>
>See replies in-line
>
>
>>> In the NENA i3 scheme, the interface to the LoST server is a set of 
>>> GIS layers, and specifically, there are sets of polygons 
>that define 
>>> service boundaries.  Each of these boundaries has 
>attributes for the 
>>> service URN URI, etc that are returned in the LosT response 
>when the 
>>> query location is found within the boundary.
>>>   
>> 
>> An interface is a protocol or an API. A GIS layer, as you 
>describe it 
>> in the following sentences, is essentially mapping data.
>It is represented as a set of GIS layers that are maintained 
>on the LoST server in sync with the same layers in the master 
>GIS system which is the authoritative source for the data.  
>The data is the layers, the transport protocol is WFS with an ATOM feed
>> 
>>    Mapping Data:
><http://tools.ietf.org/html/draft-tschofenig-ecrit-rfc5222bis-00#secti
>> on-5> of RFC 5222.
>I am not even going to go look to see if the definition in 
>there is a complete overlap in definitions.
>> 
>> 
>> 
>> 
>>> For a civic address on the query, in theory, the LoST server does a 
>>> geocode and then point-in-polygon, but an implementation 
>can compute 
>>> a table to do lookups for civics.
>>> 
>>>   
>> The question here should be how do you describe civic boundaries. If 
>> you have no story for it then it is a bit mood to return civic 
>> boundaries in a LoST response to the LoST client as well.
>The database has records.  The geodetic records are called 
>"features".  A feature is fundamentally what we often call a 
>'geoshape' = point, line, circle, polygon,...  A feature can 
>have attributes (=values associated with the feature, for 
>example, a name, a record-id, or some other conventional data item)
>
>Conventionally, features are grouped into "layers" which is a 
>set of features that have some relationship.  Layers can have 
>attributes, although we often describe a set of attributes 
>that each feature in the layer must have.  For example, 
>consider a 'street centerline' layer.  The features in this 
>layer are lines that represent a segment of a street.  Each of 
>the lines in the layer would have attributes for (among other things)
> * The name of the street
>A simpler example would be a county layer, which has polygons 
>that represent the boundary of the county, with attributes for 
>the name of the county, a code, and perhaps the state it belongs to.
>
>Another interesting layer would be an address point layer. The 
>features are points.  The attributes are "house number", 
>street name, etc.
>
>This is a map: it lets us locate a position either by a geo or 
>a civic.  To find the location of a civic, you do a common GIS 
>operation which does successive searches to find the address 
>point or polygon on the road in the neighborhood (if 
>specified) in the district (if specified) in the municipality, 
>in the county (if specified) in the state, using the geometry 
>and/or the attributes.  In most GIS systems, this would be a 
>conventional multi-table JOIN, because there are attributes in 
>the features that relate them to the next highest level.
>
>If the result of the search was exactly one feature, then we 
>have a unique location for the given civic address, and we 
>take the point (if it is an address point) or a polygon (if it 
>is a parcel/subparcel/...) that represents it (that is, we 
>"geocode" to go from a civic to a geo).  We then do PinP on 
>the PSAP boundary layer to determine the LoST mapping.  In a 
>bit more detail, we do PinP with the sets of polygons that 
>have a service URN attribute matching the service urn in the 
>LoST request, and return the URI and other information in the 
>corresponding attributes on the selected service boundary polygon set.
>
>This makes LosT mapping either a straight PinP for a geo, or 
>geocode followed by PinP for a civic.
>
>One important reason we do it this way is that the GIS system 
>with all these layers already exist in most PSAPs in North 
>America.  The database was created to handle wireless calls, 
>and is used to first display where a mobile caller is calling 
>from, and then to determine the civic address to dispatch 
>responders to (reverse geocode).  Following the latest NENA 
>guidance, even the current table driven routing database (the 
>"MSAG") that the 9-1-1 system now uses is DERIVED from the GIS 
>system.  When there are changes in addressing: perhaps a new 
>street is created, or a "un-incorporated" portion of a county 
>boundary is annexed to a city, the change is made on the 
>master GIS system, and a new MSAG is generated from that.  
>Going forward, essentially ALL of the location data that 
>drives 9-1-1 is derived directly from the GIS.  For i3 that 
>means the LoST mapping data is derived from the master GIS 
>system.  Using layer replication makes this simple, and we use 
>the same mechanism for other services in i3.
>
>> 
>> 
>>> The provisioning interface is a layer replication mechanism 
>that uses 
>>> two
>>> parts:
>>>   
>> 
>> what LoST servers at the end need. If you replicate something else 
>> then there is the question of how this maps to the fields 
>required for 
>> the LoST mapping data.
>Too bad.  It describes exactly what we do.  We have a set of 
>layers in the master GIS system that represent the same layers 
>in the LoST server used to do the LoST mapping.  We replicate 
>those layers from the master GIS to the slave LoST server, so 
>that any edits to the master GIS quickly replicate on the LoST 
>server.  The LoST server IS a GIS system in this view.
>> 
>Hello?  The GIS folks know what a WFS is.  NENA is not 
>restricted to IETF standards.  The GIS guys don't know what an 
>ATOM feed is, but it's a simple enough (IETF!) standard to 
>explain to them.  WFS isn't that simple, but for your 
>purposes, it's an add/delete/edit mechanism for a GIS system 
>that operates on features.  It uses web services for transport.
>
>> 
>> While I am confident that the OGC knows what they want their concept 
>> to use for I am not sure how well the story is complete for dealing 
>> with LoST servers. To me this concept sounds like being at the stage 
>> of early design work.
>> I want to have an example of how this works. Take a look at the 
>> example we have in the LoST sync document and try to create 
>something 
>> with these two technologies.
>I explained this above.  If you are still confused, ask any 
>GIS expert how geocoding works.  After that, it works the same 
>way a geo query to LoST would work.
>> 
>Yes, although not applied to a LoST server.  However, the idea 
>of geocode then PinP is not exactly stretching anything very far.
>
>> 
>The master GIS system is not a LoST server.  It is the source 
>of data the authoritative mapping server derives mappings 
>from.  It is how we provision the authoritative mapping 
>server.  We use the same protocol to provision any LoST server.
>> 
>>> These include polygons for the political boundaries, plus 
>either (or 
>>> both) centerlines or road geometry, address points or 
>>> parcel/subparcel polygons, etc.  There is a feature layer or 
>>> attribute on a feature for each of the CAtypes.
>>>   
>> 
>See above.  We use the layers to geocode civic to geo.  Then 
>we use the PSAP service boundary polygons to determine mapping 
>in geo-space.
>
>> 
>> 
>>> changing boundaries, etc.), the WFS commands that replicate that 
>>> change are sent as an RSS feed to the LoST server, which replicates 
>>> the edits on its database, maintaining synchrony with the master.
>>> 
>>> We allow the LoST server to have an OUTPUT layer 
>replication service, 
>>> which is used on an aggregating LoST server.  For example, each 
>>> county in a state could have a LoST server, and the state 
>could have 
>>> a LoST server which is the aggregate of the individual counties.  
>>> There is a gap/overlap reporting mechanism for this.
>>> 
>>>   
>>> One of the primary goals for this mechanism is to enable the 
>>> boundaries to be changed on the GIS system by a normal edit 
>function, 
>>> and post the cache TTL, routes changes happen immediately.  We use 
>>> this in disaster scenarios when our other mechanisms to 
>reroute calls 
>>> don't work for the actual situation on the ground.
>>>   
>> Here is another disconnect, namely the believe that changing to 
>> mapping data at a LoST server (even when replicated immediately to 
>> other LoST
>> servers) leads to immediate route changes. No, it doesn't since we 
>> have built into LoST the caching concept to improve reliability, 
>> robustness and scalability. The downside of it is that, when used of 
>> course, changes to the mapping data are in effect only after 
>a certain 
>> amount of time.
>We keep the TTLs short, for this reason.  We don't think there 
>is any problem maintaining the servers reliable or scalable 
>with short TTLs.
>
>> 
>>>> I would also like to get some feedback from comments I 
>filed against 
>>>> the NENA i3 stage 3 specification. The NENA spec hints to a 
>>>> different synchronization mechanisms (as Brian mentioned already 
>>>> many times) but I failed to understand how it works. I 
>would like to 
>>>> see what the similarity and the differences are. Since the i3 spec 
>>>> is supposed to be completed very soon I assume clarifications 
>>>> regarding the synchronization aspects will have to get 
>discussed as well.
>>>> 
>>>> Ciao
>>>> Hannes
>>>>