Subject: [netlmm] #55: Define desired scalability more precisely

From: netlmm-tracker <[email-masked]>
Date: Tue, 23 May 2006 10:57:10 -0000

#55: Define desired scalability more precisely
-----------------------------------------+----------------------------------
 Reporter:  [email-masked]     |       Owner:  [email-masked]
     Type:  defect                       |      Status:  new                     
 Priority:  minor                        |   Milestone:  milestone2              
Component:  draft-ietf-netlmm-nohost-ps  |     Version:  1.0                     
 Keywords:  PS draft                     |  
-----------------------------------------+----------------------------------
 Pekka Savola

 1) scaling requirements/problem space is not described

 you do not define anywhere (here or in the requirements) what is the
 scaling factor or the scenarios you're looking at.  Both documents
 mention the WLAN switch markets, but that makes me think of O(1000) to
 up to O(50,000) nodes.  However, now I have heard that some folks are
 fantasizing about O(10,000,000) nodes per subnet as well.  These
 assumptions are very important and need to be made in the problem
 statement.

 My own belief is that we should not stretch a single IP subnet that
 far.  If you go down that route, even O(10M) isn't going to be enough:
 a service provider operating globally will want to provide seamless
 mobility even when you move between cities (for example) -- because it
 if wouldn't need to do that, it wouldn't need this kind of scaling as
 it could use some other mobility techniques to ensure smooth
 transitions between IP subnets.  Hence, the whole service provider
 would need to be in the same IP subnet.  And the next logical step
 would obviously be extending this to inter-domain, like LxVPNs have
 been extended to inter-domain.

 This has to stop somewhere.  There's no escaping the fact that if you
 want seamless mobility beyond a somehow defined topological area, some
 other mechanisms than extending the IP subnet will need to be used,
 and that likely requires host modifications.

-- 
Ticket URL: <http://www1.tools.ietf.org/wg/netlmm/trac/ticket/55>
netlmm <http://tools.ietf.org/wg/netlmm/>

_______________________________________________
netlmm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/netlmm

---

From: netlmm-tracker <[email-masked]>
Date: Tue, 23 May 2006 11:13:59 -0000

#55: Define desired scalability more precisely
------------------------------------------+---------------------------------
  Reporter:  [email-masked]     |       Owner:  [email-masked]
      Type:  defect                       |      Status:  assigned                
  Priority:  minor                        |   Milestone:  milestone2              
 Component:  draft-ietf-netlmm-nohost-ps  |     Version:  1.0                     
Resolution:                               |    Keywords:  PS draft                
------------------------------------------+---------------------------------
Changes (by [email-masked]):

  * status:  new => assigned

> and that likely requires host modifications.

New description:

 Pekka Savola writes:

Comment:

 Would it be wise to put actual numbers into the text? Is the actual upper
 limit in scalability a protocol issue, or is an implementation issue,
 within the rather loose bounds currently specified? If the protocol is
 designed for upward scalability, then it is quite possilbe for
 implementors to work out technology and products that scale. Competition
 in this space could result in rapid improvement in the scalability of
 products. Obviously, the protocol won't scale to the entire Internet, and
 it isn't intended to.

 As to whether technology exists so that the NETLMM architecture specified
 in the charter could handle very large numbers of mobile nodes, one needs
 to look outside of IETF conventional wisdom for evidence. Many 3GPP
 operators deploy a single GGSN for an entire country, with on the order of
 10's of millions of terminals. Admittedly, these networks today handle
 primarily standard bursty data traffic and not constant bit rate traffic,
 but it is not clear that, as mentioned above, technology improvements
 won't make that possible. In addition, some Asian wireless operators do
 realtime examination of application layer information in packets for
 charging purposes. The technology for this kind of realtime packet
 examination is quite advanced, though of course this kind of treatment of
 traffic is quite anathama to the IETF conventional wisdom.

 The point is not that these kinds of practices are good, but rather that
 the technology for rapid, scalable examination of packets in the market
 today is far beyond what most IETF particpants today are willing to
 acknowledge, and that it is quite possible that vendors will bring this
 technology to NETLMM once a standard is in place, and that competition
 will cause rapid upwardly scalable improvements.

 The recommendation is therefore to not include any specific numbers on
 scalability. Adding some wording indicating more clearly that scalability
 to the entire Internet is not intended, beyond the current definition of
 localized mobility which is intended to bound scalability at that level,
 may be desirable.

---

From: netlmm-tracker <[email-masked]>
Date: Tue, 23 May 2006 11:15:16 -0000

---

From: Pekka Savola <[email-masked]>
Date: Thu, 25 May 2006 15:18:04 +0300 (EEST)

On Tue, 23 May 2006, netlmm-tracker wrote:
>> ...

Obviously, scalability is both a protocol and an implementation issue. 
We should be concerned about the protocol.  We shouldn't be 
over-engineering an ultra-scalable solution if one isn't necessary for 
a sane NETLMM architecture.

> localized mobility which is intended to bound scalability at that level,

I'm not sure if that's good enough.

As I see NETLMM architecture, it makes sense ONLY in *localized* 
mobility management.  We need to realize what "localized" 
means here.

Let me put in a few high level requirements from the user's 
perspective:

Without turning the terminal on/off and without disturbing existing 
connectivity, the user must be able to move:

a) from one city block to another
b) from one part of a city to another part of the city
c) from one city to another
d) from one country to another
e) from one continent to another
f) from one service provider's network to another (in "roaming" case)

There's no escaping the fact that for some of a)-e) (i.e., inside the 
service provider's network), the user's terminal needs to have a 
non-LMM mobility management solution.  Personally, I'd put the border 
(depending on deployment) somewhere between b) and c).

If the terminal needs to support non-LMM mobility management in any 
case, why are we putting cycles to designing a solution that should 
scale up to O(10M) users?

I'd rather state explicitly that the solution must be able to scale 
to, say, O(10K) or O(50K) simultaneous users.  If the implementations 
can work on the scalability to make that much higher -- that's great, 
but not something we in the IETF should be concerned about, because 
networks can just deploy more boxes without significant disadvantage 
to the user if the limit is reached.

It seems to me that as soon as we discard Mobile IP as a basis for 
localized mobility management, there seems to be push to make the LMM 
domain bigger and bigger so that you could make do without any non-LMM 
mobility management solution.  As shown in the list a)-f), building 
one LMA/MAP big enough to cover the whole world (or break users' 
session if (s)he moves e.g. from one city to another) just doesn't 
seem feasible, so I can't figure out why we're trying.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: "James Kempf" <[email-masked]>
Date: Mon, 29 May 2006 05:05:23 -0700

Pekka,

I'm sorry I took so long to respond to this. I was in transit and I wanted 
to think a bit more about what to say. Vidya made some of the points that I 
wanted to make in her note.

FWIW, I find the boundary to be firmly at d) , depending on the size of the 
country, with c) being the most likely deployment scenerio.

AFICT, the only argument you're making against designing the protocol for 
good upward scalability is that it would take too long to do the design. 
Since most of the operators who are involved in the WG are interested in 
large scale upward scalability, I think they are willing to do the work to 
see it through. I have not heard from any vendors that such an 
implementation is technically infeasible, so I think they are willing to do 
it too. In addition, I disagree about this point, since I believe we can 
converge on a nicely scalable design quite quickly. I think I've made clear 
in earlier postings and in my recommendation for issue resolution that the 
protocol is not intended to scale up to the entire Internet or even large 
part of it. Thus, it would not run between two ASes, for example.

With regard to your comment:

>It all boils down to the requirements for the protocol.  As I said, I think 
>protocol should be able to easily scale to, say, O(10K) simultaneous users 
>in a single subnet.  Is there even experience of anything even close to 
>that range?

In fact, there is. DoCoMo runs a single GGSN for all of our approx. 20 
million GPRS customers. We have one GGSN to serve all of Japan. Now, I am 
not sure what the exact deployment looks like (I'm not in operations), and 
certainly there are not 20 million GPRS data connections up at one time, but 
this kind of deployment is quite common for GSM operators. The NETLMM 
architecture is very similar to GPRS, but the protocol is intended to be 
considerably more lightweight.

You also mention Mobile IP scalability. We did an implementation of HMIP, 
which uses a Mobile IP-like model for LMM, and an implementation of our Edge 
Mobility Protocol (EMP), which was intended to be a prototype of NETLMM. The 
implementations were done on Linux. We wanted to compare scalability and 
performance. The HMIP implementation required approx. 1.6K of memory on the 
MAP per mobile node, while the EMP implemenation required approx. 600 bytes. 
This indicates that memory usage scalability for NETLMM is likely to be 
good. The reason for the difference is that HMIP requires a tunnel interface 
per mobile node while NETLMM requires a tunnel interface per AR. The HMIP 
implementation did not include IPsec, if that is included, it will be more. 
Of course, your milage may vary and on another operating system or with 
clever implementation tricks, it might be possible to get the HMIP memory 
usage down some.

Anybody else have an opinion on this topic?

            jak

----- Original Message ----- 
From: "Pekka Savola" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, May 25, 2006 5:18 AM
Subject: Re: [netlmm] #55: Define desired scalability more precisely

> Obviously, scalability is both a protocol and an implementation issue. We 
> should be concerned about the protocol.  We shouldn't be over-engineering 
> an ultra-scalable solution if one isn't necessary for a sane NETLMM 
> architecture.
>
> As I see NETLMM architecture, it makes sense ONLY in *localized* mobility 
> management.  We need to realize what "localized" means here.
>
> Let me put in a few high level requirements from the user's perspective:
>
> service provider's network), the user's terminal needs to have a non-LMM 
> mobility management solution.  Personally, I'd put the border (depending 
> on deployment) somewhere between b) and c).
>
> If the terminal needs to support non-LMM mobility management in any case, 
> why are we putting cycles to designing a solution that should scale up to 
> O(10M) users?
>
> I'd rather state explicitly that the solution must be able to scale to, 
> say, O(10K) or O(50K) simultaneous users.  If the implementations can work 
> on the scalability to make that much higher -- that's great, but not 
> something we in the IETF should be concerned about, because networks can 
> just deploy more boxes without significant disadvantage to the user if the 
> limit is reached.
>
> mobility management solution.  As shown in the list a)-f), building one 
> LMA/MAP big enough to cover the whole world (or break users' session if 
> (s)he moves e.g. from one city to another) just doesn't seem feasible, so 
> I can't figure out why we're trying.
>
> --

---

From: Pekka Savola <[email-masked]>
Date: Mon, 29 May 2006 15:26:55 +0300 (EEST)

> country, with c) being the most likely deployment scenerio.

How do you expect how mobility is handled between countries or cities, 
but within the same ISP?

If I were convinced that the terminals and networks actually would 
implement something for this, I'd be a lot less concerned about this.

> good upward scalability is that it would take too long to do the design.

No.  I'm saying that it could exclude designs or design approaches 
which might even already exist (remember the reuse argument), and that 
IP subnet or link emulation after certain point is doubtful to work 
well anymore.

> million GPRS customers. We have one GGSN to serve all of Japan. Now, I am not 
> sure what the exact deployment looks like (I'm not in operations), and 

Note thatI said "in a single subnet".  I'm not familiar with typical 
GGSN and GPRS designs, but do those indeed use a subnet?

I see a lot of issues with trying to perform IP subnet (or even link) 
emulatation when the subnet size is scaled up dramatically.  If the 
solution is different (e.g., each host gets basically just addresses 
for itself and possibly its connected devices, like prefix delegation) 
so that there is no IP subnet emulation (or it can be decided upon 
deployment), my concerns from the Internet Protocol scaling point of 
view are significantly reduced.

There are benefits of using IP subnet emulation (e.g., possibly more 
optimal multicast distribution when a link layer is shared), but if 
the design needs to be scaled up a lot, I'm not confident it's going 
to work very well anymore.

---

From: "James Kempf" <[email-masked]>
Date: Mon, 29 May 2006 06:50:10 -0700

> How do you expect how mobility is handled between countries or cities, but 
> within the same ISP?
>

Using a global mobility protocol. For IETF, that would be either Mobile IP, 
HIP, or Mobike. I assume for most commercial, public access deployments it 
would be Mobile IP, but it really depends on what the other SDOs that adopt 
this (if any do) decide on.

> implement something for this, I'd be a lot less concerned about this.

OK.

> No.  I'm saying that it could exclude designs or design approaches which 
> might even already exist (remember the reuse argument), and that IP subnet 
> or link emulation after certain point is doubtful to work well anymore.
>

Reuse isn't the primary goal, it is only one of many, and it is primarily 
indented to be practical, like not inventing a new transport protocol when 
an existing one will do. The gap analysis has convinced most of us who have 
been involved in the BOF process (almost 9 months), that we do need a new 
protocol, but we want to make it as simple and uncomplicated as possible. 
For that, I think we can make it quite scalable too.

>>> It all boils down to the requirements for the protocol.  As I said, I 
>>> think protocol should be able to easily scale to, say, O(10K) 
>>> simultaneous users in a single subnet.  Is there even experience of 
>>> anything even close to that range?
>>
>> not sure what the exact deployment looks like (I'm not in operations), 
>> and certainly there are not 20 million GPRS data connections up at one 
>> time, but this kind of deployment is quite common for GSM operators. The 
>> NETLMM architecture is very similar to GPRS, but the protocol is intended 
>> to be considerably more lightweight.
>
> Note thatI said "in a single subnet".  I'm not familiar with typical GGSN 
> and GPRS designs, but do those indeed use a subnet?
>

Note that even with NETLMM, it is not intended to be a single subnet. The 
LMA could advertise multiple subnet prefixes, exactly like a last hop router 
in IPv6 can, it would just have to advertise the same set of prefixes across 
all ARs in the LMM domain.

As for GPRS, I don't know specifically, but I don't believe the subnet 
structure matters at that level. The routes for all the addresses handled by 
the GGSN are disaggregated down to the address level at the GGSN and 
reaggregated through the overlay network to the SGSNs (like the ARs). The 
GGSN thus must effectively handle looking up routes for all the addresses it 
handles. It is similar to Mobile IP routing in that respect. I don't claim 
to be a GPRS expert, though, so there may be a subtlety here I'm missing (I 
know there are some on the list if they care to comment).

> solution is different (e.g., each host gets basically just addresses for 
> itself and possibly its connected devices, like prefix delegation) so that 
> there is no IP subnet emulation (or it can be decided upon deployment), my 
> concerns from the Internet Protocol scaling point of view are 
> significantly reduced.
>

That is the intent. Follwing Dave Thaler's advice, we are intending to make 
this as little like a multilink subnet as possible.

The other thing to consider here is that some link layer protocols, like the 
cellular protocols and maybe WiMax, might supress the standard IPv6 link 
configuration protocol. I have not looked at the node requirements document 
lately, but I don't believe GPRS nodes do DAD, for example. Most of the 
functions handled by ND are done at the link layer in cellular protocols. 
That doesn't mean that we should not have a good example for noncellular 
protocols (the MN-AR interface document is intended to address that), but it 
does mean the the primary commercial focus is on link protocols that don't 
need most of the link configuration protocol that IPv6 offers.

> optimal multicast distribution when a link layer is shared), but if the 
> design needs to be scaled up a lot, I'm not confident it's going to work 
> very well anymore.
>

Well, we'll see. I think we can do it, but that's an emperical question we 
can answer once we have an initial design.

            jak

---

From: Vijay Devarapalli <[email-masked]>
Date: Tue, 30 May 2006 11:38:49 -0700

hi Jim,

my 2 cents

James Kempf wrote:

> FWIW, I find the boundary to be firmly at d) , depending on the size of 
> the country, with c) being the most likely deployment scenerio.

> million GPRS customers. We have one GGSN to serve all of Japan. Now, I 
> am not sure what the exact deployment looks like (I'm not in 
> operations), and certainly there are not 20 million GPRS data 
> connections up at one time, but this kind of deployment is quite common 
> for GSM operators. The NETLMM architecture is very similar to GPRS, but 
> the protocol is intended to be considerably more lightweight.

IMHO, the above assumes very low level of data usage and very
low number of subscribers using the data services (which is
true today). I think the boundary would be between b) and c).

Vijay

> You also mention Mobile IP scalability. We did an implementation of 
> HMIP, which uses a Mobile IP-like model for LMM, and an implementation 
> of our Edge Mobility Protocol (EMP), which was intended to be a 
> prototype of NETLMM. The implementations were done on Linux. We wanted 
> to compare scalability and performance. The HMIP implementation required 
> approx. 1.6K of memory on the MAP per mobile node, while the EMP 
> implemenation required approx. 600 bytes. This indicates that memory 
> usage scalability for NETLMM is likely to be good. The reason for the 
> difference is that HMIP requires a tunnel interface per mobile node 
> while NETLMM requires a tunnel interface per AR. The HMIP implementation 
> did not include IPsec, if that is included, it will be more. Of course, 
> your milage may vary and on another operating system or with clever 
> implementation tricks, it might be possible to get the HMIP memory usage 
> down some.
> 
> ----- Original Message ----- From: "Pekka Savola" <[email-masked]>
> To: <[email-masked]>
>> mobility management.  We need to realize what "localized" means here.
>>