Subject: [IMRG] The case for an Internet Measurement Protocol

From: jon b <[email-masked]>
Date: Tue, 11 Nov 2003 16:28:39 -0800 (PST)

There have been a number of attempts to start work in the IETF on a
protocol for performing network measurements, but despite the obvious
need, it has been difficult to demonstrate interest in such a protocol.
Partly this is due to the large number of different measurement problem
areas, there are lots of different issues but it is hard to get the
people concerned with all of those issues in one room at the same time.
After much discussion with Vern and Mark it was decided that this forum
would be a good place to try to pull together such as consensus of
interest. The result of which could then be taken to the IETF. 
----------------------------------------------------------------------

The case for an Internet Measurement Protocol
Jon C. R. Bennett

Outline

0)	Setting the stage

1)	Current problems we have measuring the Internet.
1.1)	Who is 'we'?

2)	Do we need to measure?

3)	What do we need to measure?
3.1)	Operational Considerations to measurement.
3.2)  	What do we want to measure in our wildest dreams?

4)	Can we get what we need/want?
4.1)	Non-technical obstacles
4.2)	Technical obstacles

5)	Straw man Protocol

-----------------------------------------------------------------------
0) Setting the stage.

Take a look at many recent 'network measurement' papers and you will
notice that many times they are more about 'measuring networks' than
about 'network measurements'. They are studies in the difficulty of
getting good network measurements as much (or sometimes more) than
they are a study of the actual measurements. But this is not a bad
thing. In order to do scientific/engineering research it is necessary
to know the (in)accuracy of the collected data.

As a result the authors of these papers often are forced to create new
measurement tools or techniques so that they may obtain better
measurements. But with few exceptions these tools or techniques must be
operated from the edge of the network since the core network equipment
is not considered to be modifiable. The few exceptions to this tend to
be at those ISPs with their own researchers but those tools are
unavailable to the 'rest of us', and in any case they can only answer
questions within a single ISP's network not across the whole Internet.

Many reasons are given why this state of affairs is to be expected or
why a solution would never get deployed. Some even argue that it is
philosophically a 'bad thing' to involve the network in the measurement
of the network.

For purposes of this discussion I will assert that as a result of the 
body of network measurement research results, we have a good
understanding of what properties are so hard to measure that the need
for networks to support the taking of such measurements can be well
supported. And I will further assert that the issues of implementation
and deployment can be addressed without significant real difficulty. So
that we should concern ourselves with answering the questions of 

1) Is such a protocol needed?

2) If yes, what should it look like?

and save

3) Is this practical?

If you feel the need to consider issue #3 first you can skip to
section 4, otherwise just continue below.

-----------------------------------------------------------------------
1) Current problems we have measuring the Internet.
    or why ICMP just don't cut it anymore.

a) Too many things we used to be able to measure that in practice we
   no longer can, either due to new deployments (NATs, etc) or new
   management practices (disable everything for security/privacy, etc)

b) We have learned over time many new things we want to measure,
   but we lack 'good' mechanisms for doing so, or;
   "why running pathchar from GigE attached host is not a good idea"

c) Many new technologies in the Internet have had or are in the
   process of having measurement protocols designed for them that are
   in many cases only available to network operators not to end users,
   e,g MPLS/LSP-Ping, pwe3/VCCV, IP Tunnels/GTTP. If we don't come up
   with a measurement protocol which can interact in some manner with
   these and other protocols we will lose useful visibility over more
   and more of the network.

d) Tools which take measurements by using 'brute force' while they may
   be just the thing to answer the question at hand, are inherently
   undeployable on a large scale.

   One grad student = network tomography research

   30 network testing companies = wide scale DDOS attack

   if the difference between 'collecting research measurements' and a
   'crippling DDOS attack' is simply how many people are running the
   tool, then perhaps it is the wrong tool?

e) While many recent developments in network measurement such as
   network tomography are fascinating demonstrations of how much can
   be learned using only the protocols available to us, if the real
   goal is to measure the network itself, would it not be better to
   remove obstacles to making good measurement rather than always
   just working around them?

   This is not to suggest that we will make measurement tools go away,
   but rather that we will make them better, more accurate, faster,
   more capable, etc.

1.1) Who is "we"?

At first glance, we = researchers, but we should remember why we do
research, need to consider that lots
of other people would benefit from being 'we' as well.

a) network operators

b) 'normal' end users

c) 'programs'

the last item is an often forgotten but I think very important one. By
'programs' I don't mean 'apps' like ping, but applications which have
their own needs for determining the state of the network, e.g. p2p,
receiver multicast, CDN/streaming server selection, RONs, etc.

-----------------------------------------------------------------------
2) Do we need to measure?

I'm going to be slightly philosophical for a moment, this will provide
some basis for later discussion of non-technical issues of deployment.

Do we "need" to measure the network? After all there is no point in
creating a protocol to fill a need that does not exist.

Well what does "need" mean?

If the network always "works" do we "need" to measure anything?

Not really.

Does the network always "work".

Well that would depend on the definition of "works" and as mentioned
above, the choice of "we", in general the answer is, hardly ever.

In the current commercial Internet if "not working" means, no
connectivity at all, one might even argue that we (we = paying
customers) have a "right" to measure. One might even go so far as to
say we have a right to measure in a manner that tells us where the
cause of the lack of connectivity is, i.e. who do we yell at for
dropping the packets we paid to have transported. It doesn't matter
that we didn't pay the network that caused the failure, remember we are
being philosophical here, we paid, the packets should get through. 

If you are a really big company paying lots of money for a connection
with an SLA, or any service provided by more than one ISP this should
not be a philosophical issue, it should be a contractual one. If the
ISP wants to be paid extra for better service, insist on your right to
see you actually got it. 

Since the performance or efficiency of any given application may be
effected by the behavior of the network, at some point "low
performance" == "not working". This last point can be used to show a
"need" to measure just about any property of a network. 

Note that we are still talking here about measuring the network, not
measuring the behavior of an application over the network. It is up to
the user/application to infer from the network properties what its
performance will be.

-----------------------------------------------------------------------
3) What do we want to measure

Reachability		i.e. ping
Non-reachability,	where is the path broken.
Path taken  		i.e traceroute
Return Path		includes finding broken return paths
Ownership,		i.e. AS number, Nanog Traceroute
Path Performance	Reordering/Jitter/Delay
Path Properties		MTU/Avail BW/Link Speed/AQM
Arbitrary Paths		the ping 'gateway' option
Paths taken by apps	proxies/redirectors etc
To/From ANY device	get through NATs/ALGs/Firewalls/etc
Non-IP paths/tunnels	provide end user access to LSP-Ping,GTTP,etc
Flow specific paths	work in the presence of flow specific
			     filtering
Reveal Protocol		show the silent POP proxy, web cache
   "Enhancing" Devices	     redirectors, etc

Is that everything? Too many things? Not enough things?

3.1) Operational Considerations to Measurement

1) Negligible performance impact
2) No "control" functions possible (only present an "information risk"
   not a "control risk")
3) Provide way to restrict visibility of information
4) Authentication/verification and DOS prevention
5) Where possible testing flow specific path should not require any
   extra filters or flow state in a router/firewall/NAT/etc, where not
   possible make the extra state needed as little as possible.

Item 1 : Make sure ISPs don't turn IMP off to protect their routers
         from being overloaded

Item 2 : Make sure ISPs/sys admins don't turn IMP off to prevent
         'control' attacks on routers/hosts

Item 3 : Make sure ISPs/sys admins with a network behind their
         Firewall/NAT don't turn IMP off to prevent information about
         their network and/or hosts from leaking.

Item 4 : Don't provide a DDOS anonymiser and/or only allow use by
         approved persons.

Item 5 : Where ever possible make IMP work with existing equipment
         that is not IMP aware, and where it is IMP aware don't force
         it to have twice the flow state, i.e. the filter that matches
         a data flow should match the

3.2) What do we want to measure in our wildest dreams

This is not an invitation to include the "kitchen sink" but an
invitation to think outside the box.

For example;

a) Show Layer 2 hops.... somehow

just because its a layering violation doesn't mean we can't want it....

or a much more utilitarian example;

b) Automatic network error reporting.

   Right now if you are trying to connect to site "A" and you can't
   what do you do?
   a) you run 'ping'... it fails
   b) you run 'traceroute'... if you are lucky it fails at a 
      machine you own or one your ISP owns if not you are basically 
      out of luck
   c) wait and hope it gets fixed quickly or 'fixes' itself, or
   c') if you are a really important customer, see if you can get your
       ISP to get the other ISP to fix it.
   c'') if you are not an important customer, you call the ISP and
      they wish you would stop calling and bothering them.

   Now if you had an Internet Measurement Protocol (IMP) path you could

   a) send an IMP 'ping' to A... fails
   b) do an IMP 'traceroute' (find the longest working path) to A
   c) find the Autonomous System (AS) num/contact info of the owner of
      the last working hop 
   c) send an "I want to report a problem" message (comprised largely
      of the returned IMP packet) to the owner of the last reached hop
   d) since the IMP packet contains enough information to both
      describe what the user is trying to reach, e.g. src/dst
      addr/port, protocol number, DiffServ Code Point (DSCP), and at
      least some components of the path including at least one hop 
      within the ISP (and the TTL value at that hop) it is possible 
      for the ISPto attempt to exactly reproduce the customers 
      failed attempt to reach A, by sending the same packet from any 
      one of the hops indicated.

   the automated system could then perform the *identical* test that
   you performed, sending the same packet the user sent and if it got
   the same result, i.e. can't reach destination, then it could flag
   it for a human to look at, or it could perform more tests to make
   sure the problem really is in it's network. NOTE the packet would be
   identical except for the source address, but if the user were
   allowed to send an IMP packet redirected through the ISP router,
   then the ISP would be able to run the test with the same addresses.

   One could also imagine where you had to make your report to your
   ISP (or to any other 'reputable' entity that you could authenticate
   yourself to) regardless of where the error occurs and they would
   forward it to the appropriate ISP. This way ISPs would only take
   requests from their own customers or from other ISPs that they have
   authenticated.

   There are many different ways the user (or application) could
   determine who to send the trouble report to, and there are many
   different actions the receiver of the trouble report can take with
   it. The key point is that the information contained in the body of
   the IMP packet contains enough information for the users failed
   attempt to reach A to be *automatically* reproduced. This allows
   automatic validation of the reported trouble before taking further
   action, which might include informing a human operator or any other
   corrective action.

-----------------------------------------------------------------------
4) Can we get what we need/want?

4.1) Non-technical issues

a) This is a bad idea, it violates the end to end principle and/or we
all know hop-by-hop anything is bad.

Firstly it doesn't violate end to end anything. We are conducting
'network' measurements not 'host' measurements. The network is the
'end point' that we want to talk to. Secondly it's not hop by hop, one
of the ideas of using a new protocol is to allow it to be handled in
the forwarding path rather than by the router CPU.

b) No ISP will ever run anything like this. 

It is not at all clear this is true given that major ISPs have brought
in drafts such as draft-bhattacharyya-monitoring-deployment-00.txt,
there is clearly interest by ISPs both for their own operational
benefits as well as to be able to demonstrate that they are in fact
providing a high quality service.

c) It will take so long to deploy that it will never be adopted.

The recent run of security problems in the Internet have brought us to
the point where if the device is software/firmware based then
deploying IMP into it is just a patch release away. So getting wide
spread deployment of new 'useful' networking stuff has suddenly become
much easier.

hosts --> windowsupdate.microsoft.com
host firewalls --> update.symantec.com, etc
home routers/NATs (most sw based) --> the sooner we get a protocol the
				      sooner all the not yet sold
				      boxes will have it.
edge routers --> sw/firmware update
core routers --> hope they can reprogram their Network Processors

I am not saying it will be 'easy', but that with a bit of effort one
could have fairly wide deployment in a relatively short time
frame. Also the devices which are hardest to get deployment (the realy
high speed routers) on, are also the fewest in number. They are also
the devices most likely to be recently installed (for any definition
of recent). 

4.2) Technical considerations

a) Too complicated for hardware devices
   The router functionality required is not very involved,
   particularly compared to capability of modern FPGAs/ASICs. Beyond
   that, I will just say that I've been involved with a hardware
   design for the version on an IMP referenced below and really its
   not that hard. (I used proof by assertion to save typing, I am
   happy to discuss it the mater further)

b) Performance impact on software devices
   i) For NATs,Firewalls,etc handling IMP should be much easier than
      the requirements complex statefull protocols impose
   ii) For routers since there is no interaction with packet
       forwarding the work could be done in 'parallel' while waiting
       on the memory accesses for the routing and filtering lookups.

       Another point to consider is does the added 'per packet' load
       of supporting IMP reduce the overall packet load on the router?
       The answer should be, yes.

       Consider a router in the middle of a path being
       tracerouted. With IMP the router would process one IMP
       packet. With traceroute it would have to forward on average
       (path length/2) packets and send 1 TTL exceeded message. So the
       average load should be greatly reduced.

       In the case of a DDOS attack, while there will be lots of
       packets to be dealt with, they will have the advantage that
       each and everyone will carry the full path back to the
       attacker! 

--------------------------------------------------------------------
5) Straw man Protocol

Current protocols and measurement tools provide only a limited subset
of these requirements or restrictions stated above. What we want is an
efficient, mechanism with low processing overhead for getting as much
information as possible in a manner usable by as many
people/applications as possible. 

As a starting point for the Internet Measurement Protocol (IMP) I will
propose the version of IPMP found in 

draft-bennett-ippm-ipmp-01.txt
ftp.rutgers.edu/pub/internet-drafts/draft-bennett-ippm-ipmp-01.txt

This draft was also meant to provide a starting point for discussion
of the protocol, so please ignore the typos, I have recieved lot of
commentary on it, and there are number of changes that have not been
incorporated into the last version. 

NOTE somehow the 'Security Considerations' section went missing, it
can be found in the 00 version of the draft at
ftp.rutgers.edu/pub/internet-drafts/draft-bennett-ippm-ipmp-01.txt

These drafts were based on the original IPMP work by Tony McGregor
(http://www.watersprings.org/pub/id/draft-mcgregor-ipmp-00.txt)

5.1) What next?

Is this good thing?

Are there issues that have not been considered?

Additional requirements? 

Security considerations? 

Is there functionality that should be removed? or added?

Suggestions? Comments?

_______________________________________________
IMRG mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/imrg

---

From: Michael Welzl <[email-masked]>
Date: 12 Nov 2003 08:24:55 +0100

Dear Jon, dear IMRG'ers,

I believe that this is a very good idea, and I would like
to contribute an additional protocol to start with.

My Ph.D. thesis was based on the "Performance Transparency
Protocol" (PTP), which was merely designed to be a means
to retrieve performance related information from a path
as efficiently as possible. I then used the protocol for
congestion control, which is not really related to IMRG.
Anyway, some years ago, I specified it in an I-D.

I really think that PTP is a good match here.

It may lack some features that we'd like to have in an
IMP (e.g., authentication), but on the other hand, it has
some things that I'd sure like to see in an IMP - e.g.,
the separation between the protocol and Content Types
which could be specified in separate documents, much
like the DCCP / CCID-* specs or RTP payload definitions.

The PTP coordinates are: http://www.welzl.at/ptp/

Best regards,
Michael

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 12 Nov 2003 09:19:10 -0500

For what it's worth, the IETF NSIS working group is specifying a generic 
path-associated signaling and data gathering protocol, with a clear 
transport/payload separation. GIMPS is the transport part and there are 
already payloads for QOS and NAT/FW traversal. A data gathering mode has 
been informally specified and implemented, but not been published.

---

From: Michael Welzl <[email-masked]>
Date: 12 Nov 2003 16:23:37 +0100

Hi,

I can't say I'm an expert on the NSIS work, but I'm following
the discussions along and my impression is that this is
going in a different direction. In general, NSIS is
related to setting up and tearing down some kind of
state at nodes (set up QoS, manage QoS ..) while this is
only about (stateless, which I think it should remain!)
retrieval of performance information.

Related, as what would now be the third source to build
upon, but hardly an alternative for an "Internet Measurement
Protocol", it would seem?

Best regards,
Michael

-- 
Michael Welzl <[email-masked]>
University of Innsbruck

---

From: jon b <[email-masked]>
Date: Wed, 12 Nov 2003 08:43:11 -0800 (PST)

> Protocol", it would seem?

Actually IMP would seem to be a good way to address a number of issues
in the NSIS framework draft in;

5.1 IP Routing Interactions 

   The NTLP is responsible for determining the next node to be visited 
   by the signaling protocol.

5.1.1  Load Sharing and Policy-Based Forwarding 

.... The significance of load sharing in 
   the context of NSIS is that, if the load sharing mechanism in use 
   will forward packets on any basis other than the destination
address, 
   routing of signaling messages using end-to-end addressing does not 
   guarantee that the messages will follow the data path.

5.1.2  Route Changes 

...Many route change detection methods can be used, some needing 
   explicit protocol support and some of which are implementation-
   internal. They differ in their speed of reaction and the types of 
   change they can detect. In rough order of increasing applicability, 
   they can be summarized as: 
  a) monitoring changes in local interface state 
....
  f) changed route of an end-to-end addressed signaling packet 
  g) changed route of a specific end-to-end addressed probe packet 

in fact this last one suggests that the NSIS folks would already
consider something like IMP to me the most applicable solution to one o
there problems.

---

From: "Henk Uijterwaal (RIPE-NCC)" <[email-masked]>
Date: Wed, 12 Nov 2003 18:14:52 +0100 (CET)

Jon,

I do think that this is a good idea.  Some comments while reading
your mail.

> 3) What do we want to measure

If a box is reachable for ping (ICMP) but everything (ssh, telnet,
http,...) is block, is the box reachable?

> Ownership,		i.e. AS number, Nanog Traceroute

I think that ownership should be based on the routing tables, not NANOG
traceroute or another IRR.  We recently showed that IRR's are off w.r.t.
reality in about 1/5 cases.

>          a data flow should match the

Item 6: Make IMP such that the owner of the box can control who uses it.
I'm sure that in a network like:

   A ----+---- B
        / \
       /   \
    C +-----+ D

C and D won't like it if A or B can measure the C-D path.

>    Now if you had an Internet Measurement Protocol (IMP) path you could

Yes, when I have an IMP, then everybody will have an IMP.  How does one
avoid that all users start measuringing?  (Or perhaps: how do we publish
results that are of interest to others).

> providing a high quality service.

Support for this draft (and related work) in Vienna was minimal.

> a) Too complicated for hardware devices

I think one has to get a H/W person in this from the start.  If IMP or
some variation thereof on routers isn't going to be picked up by C* and
J*, then it is DOA.

Henk

------------------------------------------------------------------------------
Henk Uijterwaal                             Email: [email-masked]
RIPE Network Coordination Centre            WWW: http://www.ripe.net/home/henk
P.O.Box 10096          Singel 258           Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam    Fax: +31.20.5354445
The Netherlands        The Netherlands      Mobile: +31.6.55861746
------------------------------------------------------------------------------

That problem that we weren't having yesterday, is it better? (Big ISP NOC)

---

From: jon b <[email-masked]>
Date: Wed, 12 Nov 2003 11:17:59 -0800 (PST)

Henk,

> http,...) is block, is the box reachable?

Yes the box is reachable over the network. This is a NETWORK
measurement protocol, that the box might not want to let you through
its firewall is a host issue. The network has demonstrated that it can
transport packets to and from the target host.
> I think that ownership should be based on the routing tables, not
> NANOG traceroute or another IRR. 

I agree that Nanog traceroute or IRRs are not the way to go I gave
 Nanog traceroute as an example of current tools. The idea with IMP is
 that the router will tell you directly which AS it belongs to, rather
 then having to go consult routing table or IRRs.

> C and D won't like it if A or B can measure the C-D path.

If the traffic from A to B is normaly routed over C-D, then C and D
should not object to the path being measured. In several discussions
with people I have made the following point, if my packets cross your
network is there any resasonable justification for not at a minimum
stamping your AS number into the IMP packet at the boundry in and out
of your network? You dont have to let me see anything inside your
network as long as you let me observe the boundry. That way if my
packets are being lost, or the performance is being degraded I will
know in which network it is happening.  

Now if the traffic from A to B does not normaly cross C-D then it is
upto C and D if they will allow you to use the redirection option from
their routers to measure the path C-D or not. 

> Yes, when I have an IMP, then everybody will have an IMP.  How does
> one avoid that all users start measuringing?  (Or perhaps: how do we
> publish results that are of interest to others).

Im afraid I don't understand what you meant here? Could you explain it
differently?

> > It is not at all clear this is true given that major ISPs have
> brought in drafts such as
draft-bhattacharyya-monitoring-deployment-00.txt,
> Support for this draft (and related work) in Vienna was minimal.

I don't think your statement contradicts mine. The draft was brought
in by an ISP, hence ISP support. There was little support in the room
at the IETF, because there are lots of people with measurement
problems to be solved in the IETF but they are all off is some other
room. In many cases the measurement/path probing problem is not
central to their work  (NSIS,RSVP-TE) or it is, but they have a
specific rather than general interest (GTTP,VCCV). The reason for
taking this up in the IRTF is to see if we can't get a discussion
going and draw in a representitive or two from each of these areas to
form a single body of interest in this work.

> I think one has to get a H/W person in this from the start. 

I guess I wasn't clear enough on this. I AM a H/W person and have
looked into implementing this in H/W. If you are also a H/W person and
have specific concerns I would be happy to discuss them. I spent a lot
of time developing my drafts in working on the details of the packet
layout and operations to insure that in fact the functions I wanted to
see in the protocol could be easily implemented in hardware.

So I would like to ask that we avoid getting side tracked with
general/non-specific concerns, at least until we can have some general
dicussion about the protocol first. But if there are any specific
concerns I would be happy to address those at any time.

> If IMP or some variation thereof on routers isn't going to be picked
> up by C* and J*, then it is DOA.

I agree completely, but I think the real issue here is demonstrating
that there will be significant ISP interest. If that is the case then
I think they will build it. 

In general I would like to discuss what features should be included
in the protocol and which groups of people we should be trying to
attract the interest of first since that may alter the details of the
protocol. And then we can discuss the implementation issues that may
arise from those details.

jon

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 12 Nov 2003 14:32:28 -0500

> retrieval of performance information.

NTLP and NSIS in general is designed to work in stateless mode as well. 
It is not restricted in the way you imply. In some cases, having a 
stateful mode helps with measurement applications, e.g., if you need to 
do authentication on the request or set up security associations.

---

From: "Michael Welzl" <[email-masked]>
Date: Wed, 12 Nov 2003 21:19:31 +0100

----- Original Message ----- 
From: "Henning Schulzrinne" <[email-masked]>
To: "Michael Welzl" <[email-masked]>
Cc: <[email-masked]>; "jon b" <[email-masked]>
Sent: Wednesday, November 12, 2003 8:32 PM
Subject: Re: [IMRG] The case for an Internet Measurement Protocol

> do authentication on the request or set up security associations.

I didn't mean that it would be restricted in that way, I meant that
it is "in general about" (= main purpose) setting up and tearing down
state - it is therefore probably overloaded for our purposes. Still, it
might be used as another basis to build upon. That's all I wanted to
say...

Best regards,
Michael

---

From: "Henk Uijterwaal (RIPE-NCC)" <[email-masked]>
Date: Wed, 12 Nov 2003 21:16:28 +0100 (CET)

Jon,

> their routers to measure the path C-D or not.

Exactly, but this should be in the requirements.

> Im afraid I don't understand what you meant here? Could you explain it

Sorry, wasn't too clear here.

Suppose IMP becomes part of the popular O/S's, like ping or traceroute is
today, what will prevent that everybody switches on a few measurement
sessions when they boot their computer, thus probably overloading the poor
routers.

> in by an ISP, hence ISP support.

The research department of 1 ISP, a group that is in the process of
falling apart.  That is not ISP support.

Henk

---

From: stanislav shalunov <[email-masked]>
Date: Wed, 12 Nov 2003 15:47:59 -0500

On Wed, Nov 12, 2003 at 06:14:52PM +0100, Henk Uijterwaal (RIPE-NCC) wrote:
> How does one avoid that all users start measuringing?

Why should this be undesirable?  If the fraction of measurement traffic
is small for each user, then it is small globally.  If the fraction of
measurement traffic is large for a particular user they are wasting
their own network capacity (and are or should be paying for it).

						--Stanislav

---

From: "Henk Uijterwaal (RIPE-NCC)" <[email-masked]>
Date: Wed, 12 Nov 2003 22:26:12 +0100 (CET)

On Wed, 12 Nov 2003, stanislav shalunov wrote:

> > How does one avoid that all users start measuring?
>
> is small for each user, then it is small globally.

But the distribution over the network also matters.

Henk

---

From: "Jin Guojun [DSD]" <[email-masked]>
Date: Wed, 12 Nov 2003 13:41:04 -0800

> their own network capacity (and are or should be paying for it).

Depend on what measurement users doing. Ping or Traceroute like
measurement is OK for this condition. For bandwidth measurement,
this could be a problem.
I am just asked a question that what is the tolerant average probe
traffic when 500 sites do a full mesh measurement twice per day.
I cannot answer it directly. I would expect less than 5% average probe
traffic, probaly 10% is the maximum.
500 sites are not big, but a full mesh probe means 250,000
measurements. One day has 1440 minutes or 86400 seconds.
This tells that to run all 250K measurements in one day, we have to
schedule 2.8935 tests every second. This is about the limit of the
best bandwidth algorithm can do (100 MTUs in 3 RTT).
If run test twice a day, then two probes have to run at the same time,
which is no bad.
If there is no global schedule, it is possible that 100 to 1000 probes run
together during the day.
If every one run network measurement a few times a day, we may expect
every body measures other probe traffic, but not real network traffic.

I would like to measure the bandwidth by using application traffic
instead of injecting a lot of probe traffic.
Yes, it is good if measurement can be done in network layer.

--
------------ Jin Guojun ----------- v --- [email-masked] ---
Distributed Systems Department          http://www.itg.lbl.gov/~jin
M/S 50B-2239                            Ph#:(510) 486-7531 Fax: 486-6363
Lawrence Berkeley National Laboratory,  Berkeley, CA 94720

---

From: jon b <[email-masked]>
Date: Wed, 12 Nov 2003 14:29:10 -0800 (PST)

--- "Henk Uijterwaal (RIPE-NCC)" <[email-masked]> wrote:
> Jon,
> 
> > > Item 6: Make IMP such that the owner of the box can control who
> uses
>

ok, in what form? I don't think it said that support for redirection
was  MUST so that is already at the operators option?  Or do you mean
make the "you MUST at least stamp you AS num at boundy" part?

> sessions when they boot their computer, thus probably overloading the
> poor routers.

well;
1) do people do this now?
2) why would IMP make them start?
3) the idea is that the performance impact of IMP should be zero (for a
HW router) or close to zero (for a SW router) so this should not be an
issue even if they did.

> The research department of 1 ISP...........
>.........  That is not ISP support.

Well thats one of our goals to see if we can demonstrate more ISP
support than that. Hopefuly some operators will chime in at some point.

jon

---

From: Neil Spring <[email-masked]>
Date: Wed, 12 Nov 2003 15:21:48 -0800

Users can already send a whole lot of UDP messages into the network as 
measurements, consuming resources without getting particularly accurate 
results, so excess measurement load is not a new possibility.  The 
problem is that IPMP makes it easier to observe excess measurement 
traffic because a specific protocol is provided.  That is, I'm worried 
more about perception than reality.

It seems that the conclusion to draw is one of:
a) IMP implementations must use ECN, TFRC, or another mechanism (eg. 
"yep, I'm here, but please stop pinging www.cnn.com, mmkay?") to 
self-regulate, or
b) IMP (the protocol) must support queries for values (eg. media type, 
queue length) that are useful but expensive to measure, or
c) IMP is doomed because one day someone will send too much IMP through 
a constrained link and it will be blocked with "bad press" (as 
described in the IPMP draft when describing ICMP).

I think this is a different problem (link bandwidth consumption) than 
what Jon focuses on in S3.1-item 1 (router processing overhead).  
However, unlike UDP-based network probing tools that have to infer when 
two measurements are interfering (I'm thinking of the Backbeat poster 
at SOSP), a new protocol might offer an opportunity to identify 
interfering measurements explicitly.

Are there other possibilities that would ensure that measurement 
traffic is "nice" without restricting access to just a few chosen 
researchers and operators?

> instead of injecting a lot of probe traffic.

I think this is the strategy that performance-sensitive applications 
will adopt, perhaps by downloading from multiple servers at once.  IMP 
would allow applications to be more clever and identify that, low and 
behold, bandwidth is limited by a "56k" modem link, or a lossy wireless 
link, and take one measurement instead of hundreds.  That is, the power 
of the measurement protocol is to let us learn more about the network 
with fewer packets.

I think IPMP would be quite valuable as a measurement and diagnostic 
tool, and the reason I jumped in on this issue is that my ambitious 
goal would be to make it beyond reproach -- not just that it's 
operationally valuable, but that there's no reason at all to disable or 
block it.

-neil

---

From: jon bennett <[email-masked]>
Date: Wed, 12 Nov 2003 19:21:16 -0800

At 12:50 PM 11/12/2003, stanislav shalunov wrote:
>For what it's worth, there's a measurement protocol being worked on in
>the IPPM WG---OWAMP.  The two relevant drafts are
>draft-ietf-ippm-owdp-reqs-06.txt (the requirements document) and
>draft-ietf-ippm-owdp-07.txt (the protocol specification).
>OWAMP implementation is available from http://owamp.internet2.edu/download/
>(the latest version is http://owamp.internet2.edu/download/owamp-1.6c.tar.gz).

OWAMP and IMP have almost totally disjoint design goals and disjoin 
measurement goals.

from draft-ietf-ippm-owdp-07.txt

Additional design goals of OWAMP include being hard to detect and
    manipulate,

from draft-bennett-ippm-01.txt

    The IP Measurement Protocol (IPMP) is based on packet-probes, and is
    designed to allow routers to participate in measurements by the
    insertion of path information as the probe passes between a pair of
    hosts.

OWAMP's goal is to measure the path as a whole, IMP's goal is to measure 
the path as a sequence of hops. OWAMP's goal is to measure in one direction 
(hence its name) IMP's is to measure in both.  There a many other 
differences, but the basic answer is that OWAMP fails to meet almost all of 
the needs which IMP is intended to address.

On a related note, if you were unconcerned with the requirement to "hide" 
from routers so they can't give the measurement packets special treatment 
then IMP would probably be a better way to collect IPPM measurements than 
OWAMP, since it would show you *where* along the path the delay. jitter, 
reordering, etc occurred.

Also if people really think routers can afford the processing effort to 
"cheat" on the measurement packets, then obviously they can easily afford 
the effort to processes them correctly, yes?.

---

From: "Jin Guojun [DSD]" <[email-masked]>
Date: Wed, 12 Nov 2003 16:59:11 -0800

jon bennett wrote:

> ......

Just having some fun story. The synchronous processor clock may be limited
around 100 GHz, which will happen soon. I do not know if router will be
able to use  async clock processors to process packets. If so, how much CPU
power can be left for doing statistics. If not, obviously, there will be no time
left for CPU to enjoy ice cream after chewing packets.
Since processor is inexpensive, vendor can put a second processor to do
the statistics. I would think this is useful thing to do, so applications or
traffic itself can inquery routers to get path status.
Currently, some routers do 1 out of 10,000 packets sampling, which is not
very useful statistics as some study has been done: "It is impossible to invert
this sampling to the original traffic past the router".

---

From: David Moore <[email-masked]>
Date: Wed, 12 Nov 2003 17:03:15 -0800

> the effort to processes them correctly, yes?.  

Not for dropping (and probably not for other things also).  When
queues are full and packets are being dropped, if one can easily
decide to not drop the measurement packets, the network will look
better than it is.  Since the measurement traffic is presumed to
be a tiny fraction of the total traffic, it's easy to not drop it.

Also, imagine that there is a short delay path between A & B, but
that it has smaller bandwidth than the primary path between A & B.
It may be very expensive to "process them correctly", since this
isn't a function of the router, but of the fiber and line cards
and expensive equipment.  However, it may be very cheap to send
packets of a particular type out a different interface than they
would normally go from local routing weights.

I don't think IPMP should be worried about cheating in this way
particularly, but it is the case that cheating could occur.  So
protocols which try to look like normal traffic are still also
needed.

-- david

---

From: Tony McGregor <[email-masked]>
Date: Thu, 13 Nov 2003 15:18:30 +1300 (NZDT)

> would normally go from local routing weights.

If routers cheat in this way, it's pretty easy to catch them at it.
Send some normal traffic and compare the end-to-end characteristics
with the measurement traffic.

> protocols which try to look like normal traffic are still also

Right.  But, it can just be an end-to-end measurement.

Tony

---

From: stanislav shalunov <[email-masked]>
Date: Wed, 12 Nov 2003 21:26:33 -0500

> OWAMP and IMP have almost totally disjoint design goals and disjoin 

Your description of differences between the design goals of OWAMP and IPMP
is quite correct.  The key is that OWAMP wants to be as much as practical
similar to ``normal'' (non-measurement) traffic while IPMP wants to be as
as dissimilar as practical (by using a special protocol number).

Both approaches have their place.  I've already seen in several networks
ping spoofing (where a reply to ping is spoofed from a middlebox)
and ping priority processing (where ICMP Echo Request/Response packets
are queued ahead of other packets at the bottleneck).  These practices
convince me that the user-centric (rather than provider-centric) design
of OWAMP makes sense -- at least in some environments.  On the other hand,
router participation can be immensely useful.

One concern with requiring special treatment from routers is that it
would become *difficult* to be honest and helpful if non-measurement
packets and measurement packets take different forwarding paths.

> the effort to processes them correctly, yes?.  

I doubt that this is correct.  First, the cheating routers and the routers
that lack resources might form disjoint sets (e.g., so far all the cheating
routers I've seen were close to the edge while we should be most concerned
about resources of backbone routers).  Second, cheating in the form of
priority queuing of measurement packets uses features that need to be
in the router anyway, while special processing -- that might very well
be quite a bit more expensive in terms of complexity -- requires new
router functionality that is not currently there.

All that said, we should consider what would be the best ways to ask
routers to co-operate in measurement sessions and provide information
about networks in a configurable way so that providers can decide with
good granularity what information would and would not be exposed.

---

From: Tony McGregor <[email-masked]>
Date: Thu, 13 Nov 2003 15:58:15 +1300 (NZDT)

> (the latest version is http://owamp.internet2.edu/download/owamp-1.6c.tar.gz).

OWAMP has a lot of work in it related to the OWAMP-Control protocol.
IPMP has (deliberately) omitted that.  One possible outcome might be
that a IMP is an alternate OWAMP-Test protocol within the the OWAMP
architecture.  That way the IMP can be used outside that structure but
can also leverage all the work that has been done in that part of
OWAMP.  It might also facilitate the use of OWAMP as an end-to-end
validation protocol against cheating routers.

Tony

---

From: jon bennett <[email-masked]>
Date: Wed, 12 Nov 2003 21:52:23 -0800

>are queued ahead of other packets at the bottleneck).

But in these cases this was not "cheating" but rather these behaviors where 
taken because they identified ICMP as being important and wanted to make 
sure they pings got through.

>packets and measurement packets take different forwarding paths.

well actually the idea is that IMP should not take a different forwarding 
path than regular packets. Also in my draft I have a bit in the path record 
for the router to tell you that it was not able to process the packet in 
the same data path as normal traffic. As well as a flag in the IMP header 
to request the router not process the packet unless it can do so in the 
same data path as normal traffic.

This addresses the issue of knowing if the response you got is "honest" or 
not. So if you want timing measurement you would set the bit and only get 
responses from routers that could handle it in the fast path. If you wanted 
a non-timing measurement (AS num, link speed, etc) you could unset the bit 
to try to get responses from as many routers as possible.

>I doubt that this is correct.

I didn't mean to start a discussion on cheating routers. As I see it the 
solution is real simple, someone would have to build this feature into a 
router, your couldn't keep it secret. Any relevant RFC simply indicates 
that any such behavior is absolutely forbidden. Any vendor found doing this 
gets public ridicule. Any ISP found turning on the "cheat" feature by a 
customer who is trying to verify an SLA that they paid for sues them for 
fraud.

I think this sort of issue is much better addressed non-technically, just 
make it unacceptable behavior to cheat.

>Second, cheating in the form of
>in the router anyway,

This behavior would be very easy to detect. If you have actual data traffic 
flowing between your end points and you intersperse come IMP packets and 
the IMP packets always arrive out of order early it is a dead give away.

Since the idea of cheating is to make you network "look good", unless you 
can do it in an undetected manner, all that will happen is you will be made 
to look very bad instead.

>good granularity what information would and would not be exposed.

Agreed

---

From: stanislav shalunov <[email-masked]>
Date: Wed, 12 Nov 2003 22:31:26 -0500

On Wed, Nov 12, 2003 at 01:41:04PM -0800, Jin Guojun [DSD] wrote:
> [...] 500 sites do a full mesh measurement twice per day.

If a user wants to perform measurements that consume 1Gb/s of network
capacity *and is willing to pay for it*, the provider will be thrilled.
If a user wants to exchange traffic without compensating the provider,
it doesn't matter if the traffic is related to measurement, digital music,
or transport of HTML files: the provider will be unhappy.

(In the case of a provider measuring its own network, the consideration
of ``what if everyone,'' of course, does not apply.)

---

From: stanislav shalunov <[email-masked]>
Date: Wed, 12 Nov 2003 22:48:35 -0500

> If routers cheat in this way, it's pretty easy to catch them at it.

Does the default configuration of the IPMP user tools include tests
for ``put IPMP traffic into the priority queue'' cheating?  Other forms
of cheating?  If this detection is not there now, and you wanted to add
it, how much more traffic would need to be sent?

---

From: stanislav shalunov <[email-masked]>
Date: Wed, 12 Nov 2003 22:55:38 -0500

On Thu, Nov 13, 2003 at 03:58:15PM +1300, Tony McGregor wrote:
> One possible outcome might be that a IMP is an alternate OWAMP-Test
> protocol within the the OWAMP architecture.

Could be worth exploring.

---

From: Tony McGregor <[email-masked]>
Date: Thu, 13 Nov 2003 16:57:33 +1300 (NZDT)

> of cheating?

No.  The current tools look pretty much like a cross between
traceroute and ping.

> If this detection is not there now, and you wanted to add it, how
> much more traffic would need to be sent?

I guess it depends on the details of the measurement.  If you're doing
something like the NLANR AMP then very little.  Just enough packets to
get a statisticlly significant answer to the question: "Is this traffic
within the tollerances reported by the information exchange?"

---

From: stanislav shalunov <[email-masked]>
Date: Wed, 12 Nov 2003 23:11:23 -0500

> sure they pings got through.

We must have different notions of what is cheating.  There's absolutely no
legitimate purpose for ICMP Echo Response spoofing.  The only purpose is
so that PING results look good when user runs PING.  The fact that this
technique is trivial to spot (the way I noticed it first was when my
probes were supposedly traveling faster than light) doesn't seem to
discourage those who use it.

> path than regular packets.

It looks to me like this would require gates in the most critical place.
I'm sure those gates should be more and more available over time.
I haven't looked at IPMP spec from that angle and don't know much about
hardware, but this could require extra cycles as well.  Someone who
programs router ASICs would be quite welcome in this discussion.

> As well as a flag in the IMP header 

Where would this flag be processed?

> I didn't mean to start a discussion on cheating routers.

Then why do so?

> As I see it the 
> router, your couldn't keep it secret.

The necessary feature (the ability to classify traffic based on protocol
and the ability to put traffic thusly classified into a priority queue)
is already in almost every backbone router.

> make it unacceptable behavior to cheat.

We already have cheaters, even before we got to the point where SLAs are
common.  Making cheating trivial (especially while making being honest
difficult) would encourage more cheating.  Cheating is not catastraphic,
but measurement tools need to be prepared to deal with it.

> This behavior would be very easy to detect.

Is it currently detected by your toolset?  What matters is the default
configuration of the end-user tools.  If cheaters can make numbers
produced by whatever succeeds PING look good, that's all they need.

---

From: stanislav shalunov <[email-masked]>
Date: Wed, 12 Nov 2003 23:15:47 -0500

> No.  The current tools look pretty much like a cross between

Do you plan to add these tests?  Why or why not?

> I guess it depends on the details of the measurement.

The most relevant scenario is users running whatever they are supposed
to run instead of PING.

---

From: stanislav shalunov <[email-masked]>
Date: Wed, 12 Nov 2003 23:47:14 -0500

On Thu, Nov 13, 2003 at 05:33:17PM +1300, Matthew Luckie wrote:
> i've written most of the existing tools.  the existing tools basically 
> allow us to use IPMP in an fping-like operation because that's all they 
> were designed to do.

I take it as a ``no.''  (Which, I agree, is a sensible answer.)
This, however, means that normal users when they run normal tools
will not be able to see cheating, right?

> i took a chunk of NLANR AMP data and managed to establish which router 
> interfaces were the beginning point of some kind of priority queueing 
> where ICMP was seemingly relegated to the bottom.

This might be related to a common practice of treating ICMP differently
from TCP and UDP (worse, not better) in response to DoS attacks and worms
that send ICMP, I'd imagine.  Such downgrading of ICMP is sometimes done
by campus networks and other responsible providers.  Upgrading of ICMP
seems to be done by hotel network operators, wireless hotspot operators,
and, to a lesser degree, by the less scrupulous DSL providers (DSL users
who run games know that low ``ping times'' are good).

> this is comparing 
> ICMP echo delay to IPMP echo delay - medians, 90th percentiles - i did 
> this 18 months ago, i can try and dig this up if you're interested.

Sound detection technique.  End users, however, wouldn't have comparable
amounts of data to sift through.  (Also, if cheating detection techniques
are always the same -- perhaps because they are part of a standard toolset
-- one should be able to come up with more clever cheating and get into
an arms race.)

---

From: jon bennett <[email-masked]>
Date: Thu, 13 Nov 2003 00:31:18 -0800

>hardware, but this could require extra cycles as well.

No it does not, there is no interaction between the forwarding operation 
and the stamping of path records. The gates to inspect and stamp the packet 
can be placed anywhere along the data path taken by the packet. They don't 
"require extra cycles", each bit of hardware makes its own "cycles". And 
for the case of Network Processors such as in a Cisco ESR 10K
http://www.cisco.com/en/US/products/hw/routers/ps133/products_white_paper09186a008008902a.shtml
they process the packet in a pipeline of processors so as long as you have 
a free or under utilized processor in the pipeline it can perform the 
operations without reducing the performance. (If I recall when the ESR 10K 
was announced, less that half of the processors in the pipeline were being 
used)

 >  Someone who programs router ASICs would be quite welcome in this 
discussion.

I am not sure how to interpret this? What am I chopped liver?

>Where would this flag be processed?

If a router were IMP aware, but could not perform the processing in the 
fast path, whatever bit of code or software was responsible for pulling IMP 
packets out of the fast path would inspect that flag before pulling the 
packet out. If set it would leave the packet alone and let it continue on 
its way.

> > I didn't mean to start a discussion on cheating routers.

I interpreted "cheating" to mean "playing with the stamp values" not just 
using a high priority service. And I was trying to suggest that after all 
the complaints raised about it being too hard to do, to say it was too hard 
in general, but not too hard struck me as contradictory.

>is already in almost every backbone router.

Again there is nothing which say that you have to treat an ICMP packet the 
same as a data flow, because you can't figure out which data flow you 
should be treating it the same as. As I said before the draft says that you 
process an IMP packet exactly the same way as an non-IMP packet with the 
same src/dst addr/ports, DSCP. If you don't then you have to set the bit 
saying you didn't process this in the same path. So this is not just 
cheating, it is total and deliberate non-compliance with the spec.

Given that OWAMP has taken the goal of making measurements in the presence 
of people who want to cheat is there any reason to take this as a 
requirement for IMP?

As I see it you don't need to have everyone capable of easily detecting 
cheating, once you do find someone doing it you just make sure they get bad 
press for it. When magazines publish there ratings of ISPs based on 
performance measurements they should list those ISPs in the category of 
"ISPs not measured because they configure their routers in violation of RFC 
XXXX"

I suppose you could add the requirement that any device if configured to 
handle IMP packets differently MUST set the bit indicating that they were 
processed in a different forwarding path. That way even if an ISP 
misconfigured their box, the box would in effect let you know that it was 
cheating.

jon

---

From: Matthew Luckie <[email-masked]>
Date: Thu, 13 Nov 2003 17:33:17 +1300

i've done some work in this general area, though.

where ICMP was seemingly relegated to the bottom.  this is comparing 

I've not seen anything in reverse, but it might be interesting to run 
those programs again.

---

From: Tony McGregor <[email-masked]>
Date: Thu, 13 Nov 2003 20:08:00 +1300 (NZDT)

> Do you plan to add these tests?  Why or why not?

It would be possible to provide some options to include "validating
end-to-end" traffic.  We don't currently have any plans to add that,
but mostly for human resource reasons.  (The code is there if anyone
else wants to do it).  You'd need a bunch of options to generate
pretty much any kind of validation traffic.  And you'd probably want
to send at least some of the validation traffic first.

Jon's approach (of having some people watch out for cheaters and
naming them) seems like a good one to me.  If there was a satisfactory
public watchdog out there (say using OWAMP) cheating might never
happen (because the cheters would stand a good chance of being picked
up) and users wouldn't ever need such an option.  

> The most relevant scenario is users running whatever they are supposed

I don't really agree.  There are HEAPS of different reasons to
measure.  Yes users do it as one off tests but network admins do it,
ISPs and operators do it, researchers do it.  Gosh, I guess every one
does it, must be almost as good as ...  ;}

Tony

---

From: Tony McGregor <[email-masked]>
Date: Thu, 13 Nov 2003 20:43:08 +1300 (NZDT)

> Where would this flag be processed?

At the risk of descending into details I think it's worth pointing out
that this (and a bunch of other flags) is one of places that Jon's draft
and ours differ.

IPMP contains a field in the (non-time-critical) information exchange
for a router to report the worst case difference between an IP packet
and an IPMP packet (with a matching five-tuple).  This allows an
honest router to say this is the best I can do.  It dosen't require
any extra silicon or forwarding path cycles.

I'm mentioning this now to make the point that, whether you like the
"fast path bit" or not, it's a fine detail, not something that an IMP
has to have.  Hence, it's probably not something we have to agree on
now.

Tony

---

From: Tony McGregor <[email-masked]>
Date: Thu, 13 Nov 2003 20:54:21 +1300 (NZDT)

> I am not sure how to interpret this? What am I chopped liver?

This point has come up before.  What would be REALLY cool would be to
publish a VLSI design for IPMP.  Obviously it would lack a little
context in terms of the rest of the forwarding path architecture but
it could be pretty close to realistic so I still think it would serve
the point.

I have some people here who could do this but they are assigned to
other tasks.  I've had it in my head for some time that I'd steal some
of their time to do this but it hasn't happened.

Jon, any chance you could do this?  Not only would it put to bed the
"this can't be done in hardware" it would give us real numbers to work
with.

Tony
----------------------------------------------------------------------------
Tony McGregor                   Mail:   [email-masked] 
Department of Computer Science  Phone:  +64 7 838 4651 
Waikato University              Fax:    +64 7 858 5095(w) +64 7 825 5047(h)
Private Bag 3105                Home:   +64 7 825 5040 mobile: (021)313004
Hamilton, New Zealand           www:    http://www.cs.waikato.ac.nz/~tonym
----------------------------------------------------------------------------

---

From: Ratul Mahajan <[email-masked]>
Date: Thu, 13 Nov 2003 01:16:03 -0800

jon b wrote:

> we = researchers 

we should recognize that these entities have different interests and
needs (or, rights, as jon argues below). as researchers we should not 
claim that applications should be able to measure X because *we* need to 
measure X.

> that we didn't pay the network that caused the failure, remember we
> are being philosophical here, we paid, the packets should get
> through.
> 
>  with an SLA, or any service provided by more than one ISP this
> should not be a philosophical issue, it should be a contractual one.
> If the ISP wants to be paid extra for better service, insist on your
> right to see you actually got it.

as a customer, i may have a right to know where my packets are being
dropped, but its not clear to me that i have a right to know the speed
of that link.

> "need" to measure just about any property of a network.

applications may have a right to know what happened to their packets,
and where on the paths they take. but should they be able to easily 
discover the properties of paths/links they never use? my SLAs are 
unlikely to give me that right. where is the line between network 
performance measurement and spying?

seems to me that we (researchers) are better off being honest with 
others (ISPs), and convince them that the only way to improve current 
networks is by measuring them. while we already measure networks (and 
extract phenomenal amount of information with little support), our task 
would be significantly simpler if they supported such a protocol (which 
would not operationally impact their network). in return they will be 
able to understand, diagnose, and improve their network (and would see 
less research attacks from graduate students).

cheers,
	-- ratul

---

From: jon bennett <[email-masked]>
Date: Thu, 13 Nov 2003 14:18:25 -0800

>  This allows an

The flag in question (that gets put in the path record) is static. Either 
you can process in the fast path or you can't. If you can you will just 
have a register which contains the flag values in question

     0                   1
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  TT |F|AT | L |Accuracy |T|E|R|
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    TT = Timestamp Type
    F = Processed in "Fastpath"
    AT = Address Type
    L = Stamping Location
    A = Accuracy
    T = Tunnel [De/En]capsulation Point
    E = Extensions
    R = Reserved

    (in this case everything but T and E are static) and you will just copy 
the register into the packet when writing the path record. There is no 
issue of extra silicon or cycles to set the fast path flag.

If you need to set the T flag, then you are also busy encapsulating or 
decapsulating the packet and setting a bit clearly pales by comparison.

---

From: Tony McGregor <[email-masked]>
Date: Fri, 14 Nov 2003 11:33:50 +1300 (NZDT)

> The flag in question (that gets put in the path record) is static.

That's not actually the flag we were discussing.  

But, you miss the point of my mail which was to say that, whether this
bit is a good idea or not is not something we need to decide on yet.
IMP does not REQUIRE this facility.  It just potentially adds some
value (at some extra cost).  Stanislav's original comment was about
how an honest router may have to deal with a difference between the
forwarding path for the packets of the IMP and other traffic.

Of course, the goal is to make the IPMP traffic so easy to process
that that doesn't happen.  But, if it must, it can be reported through
the information reply.  And that doesn't require any extra silicon or
forwarding path cycles.

Tony

---

From: "Maurizio Molina" <[email-masked]>
Date: Fri, 14 Nov 2003 01:16:13 +0100

> "ISPs not measured because they configure their routers in violation of RFC 

Let me hook to this statement.
This is actually the final goal: having a measuremet methodology (and
protocols)
that both operators and end user can use and trust, and that can be the basis
for
troubleshooting (operator), traffic engineering (operator), automated ISP 
selection (*big* multi-homed user) compliants (*ordinary* user).
But the statement contains also the crucial point: there must be some 
operator's interest on this. Otherwise, no chance to have this RFC, and/or 
no chance that somebody cares about it (i.e. that manufacturers really
undertake the effort of implementing the processing of IMP packets on the same
forwarding path as ordinary packets). Manufacturers will do so only if they
feel operators will ask for it, and operators will ask for it only if they're
convinced it solves (some of their) problems. 
That said, and referring to Jon's question "who are we?", probably everybody
realizes that "we"(->those reading this ML) are researchers (or
research@operator....).
Therefore, beyond discussing about bits and bytes of a specific proposal
(which is wonderful, for researchers...) I think that people involved in 
the IMRG should first also try to get inside the discussion more people.
Starting with research@operator is probably a good idea, but not limiting to
it.
Actually, the ISPMON BOF in Wien failed because there was only 1 *big* operator
supporting it.
Second, before (or in parallel to) concentrating on the details of this single
protocol,  there should be some discussion if this protocol and its associated
measurement approach (it's ACTIVE measurements, after all...) it's the only one
IMRG wants to consider (what about PASSIVE measurements? As somebody already
pointed out in the *to cheat or not to cheat* discussion, passive meas. help at
least in trusting active meas. results).
Third, what's going on in the NSIS WG in this area should be carefully taken
into account. My understanding is that somebody thinks about using NSIS for
*confuguration of passive measurements* (and related result collection).
Probably, it's researcher ideas as well, at this stage, and therefore it is not
something that can be ignored by a group called IMRG, is it?
Also, how much OWAMP is similar/overlaps to this NSIS idea?

Finally, two specific comments to discussed items:
1) I support the view that the *flooding the network* problem is not a primary
concern:
it's already possible to do it without IPMP, and with volume-based charging
schemes there should be 
a $$$-based feedback loop controlling it... 
2) There were some "full mesh" examples, but I think that being in the Internet
everything "heavy tailed"
most of the tests can be concentrated on the more relevant traffic interests.

Clearly, I support Jon's initiative to clustering mesurement's people
interests/efforts.
I'd like also to see some proposal about an IMRG meeting.

Regards,
Maurizio
--

---

From: jon bennett <[email-masked]>
Date: Thu, 13 Nov 2003 19:41:59 -0800

>That's not actually the flag we were discussing.

which flag were you discussing then?

>  But, if it must, it can be reported through

Every time I discuss this with Matt he indicates that you expect the 
Information Request packet to be handled in the line card, which would be 
orders of magnitude more of both complexity and of silicon than setting a 
static flag in the path record. So in this case your statement is simply 
untrue, please do not make this claim this again.

The other possibility is that you expect the Information Request packet to 
be handled in the router CPU. That's fine, and then we can discuss which 
case is more of an impact on the router, and which version is likely to be 
deployed and enabled by the operators.

So I ask that you not make any more assertions about what takes more 
silicon or cycles until you decide where your Information Request packets 
get processed, and then we can compare the two approaches.

jon

---

From: stanislav shalunov <[email-masked]>
Date: Thu, 13 Nov 2003 20:47:31 -0500

On Fri, Nov 14, 2003 at 01:16:13AM +0100, Maurizio Molina wrote:
> [...] how much OWAMP is similar/overlaps to this NSIS idea?

I don't know enough about the NSIS idea to say.  What are the relevant drafts
I could read to compare the approach to OWAMP?

---

From: stanislav shalunov <[email-masked]>
Date: Thu, 13 Nov 2003 20:52:56 -0500

> of people who want to cheat is there any reason to take this as a 

Not as long as users can run OWAMP and IMP with equal ease.

> "ISPs not measured because they configure their routers in violation of RFC 

A lot more users today run ping than read magazines that publish ISP
comparisons.

---

From: "Maurizio Molina" <[email-masked]>
Date: Fri, 14 Nov 2003 03:17:59 +0100

> I could read to compare the approach to OWAMP?

AFAIK there's this individual submission
http://www.ietf.org/internet-drafts/draft-couturier-nsis-measure-00.txt

Furthermore, perhaps Henning could specify whether he was referring to this
while
saying 
"...the IETF NSIS working group is specifying a generic 
path-associated signaling and data gathering protocol, with a clear 
transport/payload separation. GIMPS is the transport part and there are 
already payloads for QOS and NAT/FW traversal. A data gathering mode has 
been informally specified and implemented, but not been published."

Maurizio

--

---

From: stanislav shalunov <[email-masked]>
Date: Thu, 13 Nov 2003 21:28:33 -0500

On Fri, Nov 14, 2003 at 03:17:59AM +0100, Maurizio Molina wrote:
> > I don't know enough about the NSIS idea to say.

I can't find anything about active measurement in this draft after a quick
scan.  OWAMP is entirely about active measurement.

---

From: "Maurizio Molina" <[email-masked]>
Date: Fri, 14 Nov 2003 03:53:17 +0100

> scan.  OWAMP is entirely about active measurement.
Good to know it. 
But passive measurements are also in the scope of this IMRG, isn't it?
Maurizio
--

---

From: jon b <[email-masked]>
Date: Thu, 13 Nov 2003 18:59:47 -0800 (PST)

--- jon bennett <[email-masked]> wrote:
> Information Request packet to be handled in the line card, 

Matt asked me to correct this to make it clear that he said that it
*could* be done in the line card, not that it was *required* to be done
in the line card.

Regardless of this it my question to Tony still stands, which in effect
is  do you expect that the Information packets will be handled in the
line cards or in the router CPU.

---

From: stanislav shalunov <[email-masked]>
Date: Thu, 13 Nov 2003 22:12:09 -0500

> But passive measurements are also in the scope of this IMRG, isn't it?

I believe they are.  (However, neither OWAMP nor IPMP have anything to
do with passive measurements.)

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 13 Nov 2003 22:33:35 -0500

While I should have mentioned this draft, that particular draft appears 
to be specific to QOS measurement. The (as-of-now) unpublished work is 
more in the line of the parameters mentioned. We are currently gathering 
information such as link speed, interface type, interface byte count, 
and similar parameters.

---

From: Tony McGregor <[email-masked]>
Date: Fri, 14 Nov 2003 21:29:57 +1300 (NZDT)

> same data path as normal traffic.

That's different to the one in the path record.  But, this is
seriously going nowhere.  Can we drop it?

> get processed, and then we can compare the two approaches.

I can't imagine Matthew saying this has to be done in the forwarding
path.  The whole point of the information exchange part of the
protocol is to minimise the work done in the packet forwarding path.
That's why the router doesn't have to insert a real time timestamp but
instead timestamp are resolved later through the information exchange.

The router is free to process the information exchange in any way that
is convenient to it.  It is not time critical and doesn't have to be on
the line card.

---

From: Tony McGregor <[email-masked]>
Date: Fri, 14 Nov 2003 22:10:22 +1300 (NZDT)

> line cards or in the router CPU.

IMHO It's not really about the line card vs _the_ CPU.  That implies a
very specific architecture of a router which might not way a
particular router is built, now or in 10 years time.

The key question (and the one relevant to the point I was trying to
make when this came up) is "Must the information exchange be done in
the forwarding path?".  My answer to that is no.

Tony

---

From: Tony McGregor <[email-masked]>
Date: Fri, 14 Nov 2003 23:15:55 +1300 (NZDT)

> selection (*big* multi-homed user) compliants (*ordinary* user).

I'd add researchers to your list.  They are also legitimate users of
measurement tools.

> But the statement contains also the crucial point: there must be
> some operator's interest on this. Otherwise, no chance to have this
> RFC, and/or no chance that somebody cares about it (i.e. that
> manufacturers really undertake the effort of implementing the
> processing of IMP packets on the same forwarding path as ordinary
> packets).

Right.  I think there are three reasons operators might want an IMP.

  1. They need it for their internal work.
  2. Their users want it.  E.G. Sell me an SLA and give me the
     mechanism to measure it.
  3. They want to be measured accurately, not badly.

When I was first thinking about IPMP I attended a CAIDA ISMA.  There
were a bunch of large ISPs there.  Several expressed the view that one
of the worst things that happens to them is some wally at some
Internet magazine does some pings to a bunch of places through some
different ISPs and publishes a league table of how good the ISPs are.
Then the chief engineers find themselves called up before management
to explain why they're not doing their job properly.  These ISPs said
"We'd rather not be measured, but since we will be then we'd like to
be measured well".

Tony

---

From: Mark Allman <[email-masked]>
Date: Fri, 14 Nov 2003 14:34:39 -0500

> > I can't find anything about active measurement in this draft
> > after a quick scan.  OWAMP is entirely about active measurement.
>
> Good to know it. 
>

Yes, passive measurements are within scope.  But, that does not mean
we cannot consider only active measurements in some contexts.

allman
(wearing the lame chair hat)

---

From: Christophe Diot <[email-masked]>
Date: Fri, 14 Nov 2003 18:11:19 +0000

>Actually, the ISPMON BOF in Wien failed because there was only 1 *big* operator
can I disagree? :-)

---

From: jon bennett <[email-masked]>
Date: Sun, 16 Nov 2003 16:57:28 -0800

>it could be pretty close to realistic so I still think it would serve

I don't think this would really address the issue. Giving the VLSI code 
would not prove anything to anyone unless it were basically a fully 
complete, synthesized, laid out, simulated implementation. Aside from not 
being in a position to provide such an answer, it would still be totally 
detached from the rest of the packet handling and as a result still not 
tell you anything. I mean suppose the checksum update takes Z*10,000 gates, 
is that bad? Suppose for only Y*100 extra gates I can reuse the existing 
checksum update logic from the IP header processing function? Well then it 
probably is irrelevant what Z is, isn't it? Suppose it takes 15 cycles to 
do the IMP processing is that bad? Depends on if it takes 5 or 25 cycles to 
do regular packet processing.

>Jon, any chance you could do this?

Not much of a chance for a number of reasons. First I don't have the time 
(nor do I have someone like you do who I can ask to do it). Second, even if 
I had time I don't currently have access to the tools needed to do it. 
Third I don't see what question it would answer, particularly as we still 
have not agreed on the details of the protocol.

>  Not only would it put to bed the "this can't be done in hardware"

I don't see that there is anything that needs to be "put to bed", I am 
still waiting for someone to  raise any specific hardware related technical 
concerns with my draft.

>it would give us real numbers to work with.

It might give us "real" numbers, but it would not give us "meaningful" 
numbers. And I don't think specific numbers are the issue, but rather 
certain properties which are good/bad for hardware implementations.

---

From: jon bennett <[email-masked]>
Date: Sun, 16 Nov 2003 17:16:53 -0800

>particular router is built, now or in 10 years time.

I think that it is about the line card vs the (a?) CPU. It is not an issue 
of implying anything, it is an accurate description of a great many routers 
today, as well as probably a great many in the future. Just because there 
might some day be an architecture which is different does not mean that we 
don't have to address issues with todays architectures.

>the forwarding path?".  My answer to that is no.

Does that mean you believe the information request will not be handled in 
the line card, but will be handled in the/a CPU (of current architecture 
routers)?

The question which might better ask the point I was trying to make when I 
brought this up is; "Must the router have to *reply* to IMP messages as 
well as forward them?" And my answer to that would also be no.

---

From: "Maurizio Molina" <[email-masked]>
Date: Mon, 17 Nov 2003 16:22:20 +0100

> >Actually, the ISPMON BOF in Wien failed because there was only 1 *big*
operator

Of course you can. But what's your view, then? 
My impression was of a good idea not supported by an adequate effort to
convince people (and ISPs) of its advantages.
Maurizio

--

---

From: Tony McGregor <[email-masked]>
Date: Tue, 18 Nov 2003 08:31:23 +1300 (NZDT)

On Sun, 16 Nov 2003, jon bennett wrote:

> complete, synthesized, laid out, simulated implementation. 

If we had a VHDL design it could be synthesized and simulated.

> Aside from not being in a position to provide such an answer, it
> would still be totally detached from the rest of the packet handling
> and as a result still not tell you anything. I mean suppose the
> checksum update takes Z*10,000 gates, is that bad? Suppose for only
> Y*100 extra gates I can reuse the existing checksum update logic
> from the IP header processing function? Well then it probably is
> irrelevant what Z is, isn't it? Suppose it takes 15 cycles to do the
> IMP processing is that bad? Depends on if it takes 5 or 25 cycles to

I guess someone with enough knowledge of router forwarding paths could
tell us how many gates they take, typical clock rates and bus widths
etc for a range of routers today.  W.R.T. reuse of other components
like the checksum generator, a reference design could state what
assumptions it made in this respect.  It doesn't need to be a final
design for a particular router, just an indicative one.

> certain properties which are good/bad for hardware implementations.

There are two questions that come up from time to time in discussion
of an IMP.  The first is: "Can this be easily implemented in hardware
in a router?"  So far we only have assertions that it can be done
"pretty easily".  A reference design would add weight to that
assertion by showing (roughly) how.  It is something that could be
assessed and refined.  It shows that nothing has been forgotten.  And
we can quote things like the number gates and any additional latency
in the packet pipeline.  Certainly, we would need to make some
assumptions about the rest of the packet forwarding path, but those
can be stated and also form a concrete reference point for discussion.

Secondly, there are arguments about what the impact of a certain
feature is on the design.  These would be a lot more concrete if there
was a reference design to compare them against.  We could say
something like "feature X adds Y gates (or CLBs) to the reference
design".

I don't think the fact that the protocol is not totally nailed down is
a bad thing.  We have to do it before the protocol is complete if it's
going to have an impact on the design.

Tony

---

From: Tony McGregor <[email-masked]>
Date: Tue, 18 Nov 2003 09:11:13 +1300 (NZDT)

> don't have to address issues with todays architectures.

Limiting the protocol to a design that has a central CPU and line
cards of a particular architecture would be a huge mistake.

> the line card, but will be handled in the/a CPU (of current architecture 

No, like I said it can be handled any where that's appropriate.  It
might be in a central CPU, it might be on a line card if there are
appropriate resources there.  It could possibly even be on a different
machine all together (although I personally don't think so).

> well as forward them?" And my answer to that would also be no.

Of course a device that processes information request packets has to
reply to them.

Tony

---

From: jon bennett <[email-masked]>
Date: Mon, 17 Nov 2003 17:59:01 -0800

>cards of a particular architecture would be a huge mistake.

I did not say to limit the protocol to this architecture, I said do not 
ignore this architecture when designing the protocol.

>No, like I said it can be handled any where that's appropriate.

Any where that's appropriate based on what?

>Of course a device that processes information request packets has to

I meant does a device that processes echo packets have to reply to 
information packets? I go into this more in the next message.

---

From: jon bennett <[email-masked]>
Date: Mon, 17 Nov 2003 19:09:43 -0800

>If we had a VHDL design it could be synthesized and simulated.

Yes, but why do you need to?  If you really feel that it is important then 
have one the people you said you had who could work on it produce such a 
design.

>"pretty easily".

So who is it that is raising this question anyway?

Is there anyone on this list right now who understands router hardware and 
has any actual concerns about the implementation issues? If there is I 
would ask that they explain their concerns, at which point they can be 
addressed. If there are no concerns more concrete than "gee it looks hard" 
then there is very little to be responded to.

>something like "feature X adds Y gates (or CLBs) to the reference

There is no way that you can have a meaningful evaluation of features in 
this manner, the number of gates or cycles that an implementation takes is 
dependent on so many choices that you could never say "feature X added Y 
gates" because maybe if feature W had been implemented differently you 
could have gotten feature Y for 0 extra gates?

Heck I've known designs where adding "features" *reduced* the gate count! 
In that case the "features" used up the remaining values in a small binary 
number, resulting in there being no "illegal" values to be handled and 
gates saved in not having to handle any out of range values were more than 
the gates added by the features. So this method of evaluation is not as 
straight forward as you think it is.

>We have to do it before the protocol is complete if it's

Many of the changes I made from your original draft where meant to address 
specific concerns I had w.r.t. hardware implementations. One does not need 
a reference design to allow hardware considerations to be reflected in the 
protocol.

With the exception of encryption algorithms I am not aware of any other 
ietf protocols where real in depth hardware evaluation was considered 
necessary. Certainly it does not seem to be warranted in this case, but you 
are obviously free to conduct one if you feel the need.