Subject: [Gen-art] Gen-ART review of MPLS OAM Requirements and Framework Drafts

From: [email-masked]
Date: Mon, 21 Nov 2005 12:24:27 -0500

Background for those who may be unaware of GenART:

GenART is the Area Review Team for the General Area of the IETF.
We advise the General Area Director (i.e. the IETF/IESG chair) by
providing more in depth reviews than he could do himself of documents
that come up for final decision in IESG telechat.  I was selected
as the GenART member to review these documents.  Below is my review,
which was written specifically with an eye to the GenART process, but
since I believe that it will be useful to have these comments more
widely distributed, others outside the GenART group are included.

Review criteria for WG submissions: "Is the document a reasonable
contribution to the area of Internet engineering which it covers? If
not, what changes would make it so?"

These drafts are basically ready for publication, but have nits that
should be fixed before publication.

The requirements draft needs enough attention that a new version
would be a good idea, especially as there may be a technical issue
hidden in Section 4.1's discussion of why ICMP is insufficient for
MPLS OAM purposes.  The conclusion that ICMP is insufficient is
correct, but some of the examples that are used to support the
conclusion may be incorrect about control-vs-data-plane handling
of ICMP by MPLS LSRs.

The framework draft is in much better shape, as the minor issues
identified below appear to be reasonable to address via an RFC
Editor Note.

This email needs prompt attention, as Alex is requesting expedited
publication of these two drafts.

-- draft-ietf-mpls-oam-requirements-06.txt

I found a significant number of typos and English language
editorial cleanup nits that I'll leave to the RFC Editor to
address.  It might be good to alert the RFC Editor that this
draft is likely to require heavy editorial attention.

Section 2.1 Terminology - Both "Head-end Label Switch
Router (LSR)" and "Head-end Label Switching Router (LSR)"
are defined.  Only one of these two definitions should be
used - the other should be removed.

Section 2.2 Acronyms - 

   LSP:  Label Switch Path

   LSR:  Label Switch Router

I suspect those should be "Label Switched Path" and "Label
Switching Router".  In any case, between this and the previous
comment, consistency on whether "LS" in acronyms stands for
"Label Switch" vs. "Label Switched" and/or "Label Switching"
is needed throughout the draft.

Section 3 Motivations - RFC 2119 terminology is not appropriate
in this section, as it does not express requirements.  Change
all occurrences of "MAY" to "may".

The statement that:

   However as of this writing, existing drafts focus on single 
   provider solutions or focus on a single aspect of the MPLS 
   architecture or application of MPLS.

appears to be inconsistent with Section 4.9's discussion of
standard MPLS MIBs, which includes references to the RFCs that
specify them.  In addition, this statement will be dramatically
incorrect when the LSP Ping draft is published.  It should be
rewritten to be more about the drafts that respond to these
requirements rather than a criticism of what has gone before.

Section 4.1 Detection of Label Switched Path Defects

The characterization of "reasonable amount of time" for
detection as "both predictable and consistent" is incomplete.
For example, always detecting defects 47-48 hours after they
manifest in customer traffic is "both predictable and
consistent", but not "reasonable".  A ballpark/approximate
order-of-magnitude discussion should be added covering the
time frames in which LSP defects will likely affect customer
traffic, and the detection timeframes required for operator
detection to precede customers noticing that something is
wrong.

The discussion of ICMP-based ping appears to assume that
ICMP-based ping is an MPLS control plane function, even at
LSRs that only forward an ICMP packet (i.e., not the destination
of the ICMP packet, and not the node at which the TTL hits zero).
If that assumption is correct, it needs to be stated, but it
does not appear to be consistent with RFC 3032's discussion
of ICMP and MPLS, which specifically states that ICMP packets
can be label switched.  If ICMP-based ping goes through the
data plane as long as the TTL has not expired (which is what
I would expect), then this paragraph needs to be reworked.

Section 4.4 Service Level Agreement Measurement

    ... the qualitative 
   aspects of LSPs such as jitter, delay, latency and loss 

I believe "quantitative" should be substituted for "qualitative".

Section 4.5 Frequency of OAM Execution

   ... probe-based detection mechanisms 
   that incur significant mismatches in the probe rate 

That doesn't parse for me - what is mismatched to what?  If
it's the rates of probing by multiple probe-based mechanisms,
the text quoted above should read:

	... probe frequency or rate differences among multiple
	probe-based detection mechanisms

At the end of the section, I understand how synchronizing
probe rates helps address the problem of inconsistent views of
network problem state, but explanation is needed for how
"having the probes self-identify their probe rate" helps.

Section 4.11 Per-LSP Accounting Requirements - ASBR acronym
used without expansion or definition.

Section 4.11.1 Requirements - The use of the word "key" in
(4) invites confusion with security usage of that word.
Please pick another term such as "tag" or "identifier",
unless there is a specific need to use "key".

Section 4.11.2 Scalability - This is not about "scalability"
per se, but rather minimum accounting requirements motivated
by scaling considerations.  It should be retitled to "Location
of Accounting" or something like that.

Section 5 Security Considerations

The term "tools" in the first paragraph should be changed
to "network mechanisms", which is I believe this paragraph
is primarily about (e.g., authorization restrictions on use
of LSP Ping); tools is susceptible to being misread as being
primarily about the software applications that inspect the
network as opposed to the network mechanisms they use.

Sections 6 and 11 on IANA Considerations are duplicative
(but consistent).  Remove one of them.

-- draft-ietf-mpls-oam-frmwk-03.txt

Section 3.1.1 Enumeration and Detection ...

Discontinuities in the MPLS Encapsulation paragraph -
FEC acronym used without prior expansion or definition.

Misordering paragraph - mention of TCP's ability to correct
misordered received packets should also point out that
misordering can cause TCP to retransmit some of the misordered
packets.  This is important in order to not leave the impression
that TCP is insensitive to misordering.

Figure 1 is split across pages.

Section 7 Security Management - please weaken the following wording
in the second paragraph:

    OAM messaging does address existing security concerns with the 
    MPLS architecture. i.e. through rigorous defect handling ...

To:

    OAM messaging can address an existing security concern with the 
    MPLS architecture, as through rigorous defect handling ...

The original text could be misread to imply that all security
concerns can be addressed via OAM messaging, which is definitely
not intended.

Third paragraph - what is "trusted space"?  If this is "a single
provider's network", use that terminology instead.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

_______________________________________________
Gen-art mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/gen-art

---

From: George Swallow <[email-masked]>
Date: Mon, 21 Nov 2005 13:45:49 -0500

> publication of these two drafts.

David, Alex --

Do you know when the next telechat is?  Is there a particular deadline.
I know that the ITU would like a number by December 12.  I'm guessing
that this needs to be done well in advance of that.

...George

========================================================================
George Swallow             Cisco Systems                  (978) 936-1398
                           1414 Massachusetts Avenue
                           Boxborough, MA 01719

---

From: Alex Zinin <[email-masked]>
Date: Mon, 21 Nov 2005 14:19:43 -0800

George,

> that this needs to be done well in advance of that.

The telechat is on Dec 1st. We will need the draft updated, but please
hold the submission till end of this week--I'd like us to incorporate all
comments we can potentially receive.

Thanks.

--
Alex

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Tue, 22 Nov 2005 11:01:12 -0500

> comments we can potentially receive.

	Will do. I am on vacation this week, but will
get to this next Monday and give it top priority to
complete.

	--Tom

---

From: "Thomas D. Nadeau" <[email-masked]>
Date: Sun, 27 Nov 2005 18:53:27 -0500

This response addresses the comments directed at
the OAM Requirements draft.  Please see inline for responses
(or questions if necessary).

> is needed throughout the draft.

	Changed LSR to "Label Switching Router" and LSP to
"Label Switched Path" as per RFC3031 and also globally
made things consistent throughout the document.

> all occurrences of "MAY" to "may".

	Fixed: MAY -> may

> requirements rather than a criticism of what has gone before.

	I think this is short and sweet and no longer has
the conflicts you mentioned:

3.  Motivations

    This document was created in order to provide requirements
    which could be used to create consistent and useful OAM
    functionality that meets operational requirements of those
    service providers who have or are deploying MPLS.

> consistent", but not "reasonable".

	That is true; however, the discussion of how "reasonable"
or "acceptable" is defined, is really based on the provider and
the SLA offered to the customer in question.

> A ballpark/approximate

	I hope this makes things clearer. In Particular, see the last
two sentences:

    4.1 Detection of Label Switched Path Defects

    The ability to detect defects in a broken Label Switched Path
    (LSP) SHOULD not require manual hop-by-hop troubleshooting of
    each LSR used to switch traffic for that LSP. For example,
    it is not desirable to manually visit each LSR along the data
    plane path used to transport an LSP; instead,this function
    SHOULD be automated and able to be performed at some operator
    specified frequency from the origination point of that LSP.
    This implies solutions that are interoperable as to allow for
    such automatic operation. Furthermore, the automation of path
    liveliness is desired in cases where large numbers of LSPs might
    be tested. For example, automated ingress LSR to egress LSR testing
    functionality is desired for some LSPs. The goal is to detect LSP
    path defects before customers do, and this requires detection and
    correction of LSP defects in a manner that is both predictable and
    sufficiently within the constraints of the service level agreement
    under which the service is being offered. Simply put, the sum of
    the time it takes an OAM tool to detect a defect and
    the time needed for an operational support system to react to
    this defect by possibly correcting it or notifying the customer,
    must fall within the bounds of the service level agreement in
    question.

> I would expect), then this paragraph needs to be reworked.

	ICMP-based ping packets are carried within MPLS LSPs
and so are label-switched, but that is not guaranteed. Thus
inconsistencies can result (and be unpredictable).  Here is the
revised text:

    Although ICMP-based ping [RFC792] can be sent through an LSP as
    an IP payload, the use of this tool to verify the defect free  
operation
    of an LSP has the potential for returning erroneous results (both
    positive and negative) for a number of reasons. First, since
    the ICMP traffic is based on legally addressable IP addressing,
    it is possible for ICMP messages that are originally transmitted
    inside of an LSP to "fall out of the LSP" at some point along
    the path. In these cases, since ICMP packets are routable
    a falsely positive response may be returned.  In other cases
    where the data plane of a specific LSP needs to be tested, it
    is difficult to guarantee that traffic based on an ICMP ping
    header is parsed and hashed to the same equal-cost multi-paths
    as the data traffic.

    Any detection mechanisms that depend on receiving status via a
    return path SHOULD provide multiple return options with the
    expectation that one of them will not be impacted by the original
    defect.  An example of a case where a false negative might occur
    would be a mechanism that requires a functional MPLS return path.
    Since MPLS LSPs are unidirectional, it is possible that although
    the forward LSP which is the LSP under test, might be functioning,
    the response from the destination LSR might be lost, thus giving
    the source LSR the false impression that the forward LSP is
    defective. However, if an alternate return path could be
    specified -- say IP for example -- then the source could
    specify this as the return path to the destination, and in
    this case, would receive a response indicating to it that
    the return LSP is defective.

> I believe "quantitative" should be substituted for "qualitative".

	Done.

> 	probe-based detection mechanisms

	How about:

    This includes the frequency of the execution of any OAM
    functions. The capability to synchronize OAM operations is required
    as to to permit consistent measurement of service level agreements.
    To elaborate, there are defect conditions such as misbranching or
    misdirection of traffic for which probe-based detection mechanisms
    that incur significant mismatches in their detection frequency MAY
    result in flapping. This can be addressed either by synchronizing
    the rate or having the probes self-identify their probe rate.
    For example, when the probing mechanisms are bootstrapping,
    they might negotiate and ultimately agree on a probing rate,
    therefore providing a consistent probing frequency and avoiding
    the aforementioned problems.

> "having the probes self-identify their probe rate" helps.

	Gotcha. See last sentence above.

> used without expansion or definition.

	Added ASBR to acronyms section.

> unless there is a specific need to use "key".

       (4) All LSRs that contain LSPs that are being measuremented
           need to have a common identifier to distinguish each LSP.
           The identifier MUST be unique to each LSP, and its mapping to
           LSP SHOULD be provided from whether manual or automatic
           configuration.

> network as opposed to the network mechanisms they use.

	Done. Here is the new text:

5. Security Considerations

    Provisions to any of the network mechanisms designed to satisfy
    the requirements described herin are required to prevent their
    unauthorized use. Likewise, these network mechanisms MUST
    provide a means by which an operator can prevent denial of
    service attacks if those network mechanisms are used in such
    an attack.

    LSP mis-merging has security implications beyond that of simply
    being a network defect. LSP mis-merging can happen due to a number
    of potential sources of failure, some of which (due to MPLS label
    stacking) are new to  MPLS.

    The performance of diagnostic functions and path characterization
    involve extracting a significant amount of information about
    network construction which the network operator MAY consider
    private.

> (but consistent).  Remove one of them.

	Done; removed last one.

	I am attaching the revised version w/o formatting.
I will send this version to the I-D editor with formatting
as soon as I hear back from you that these changes are okay
(hopefully tomorrow -- Monday).

	--Tom

Network Working Group                         Thomas D. Nadeau
Internet Draft                                Monique Morrow
Expires: June 2006                            George Swallow
                                               Cisco Systems, Inc.

                                               David Allan
                                               Nortel Networks

                                               Satoru Matsushima
                                               Japan Telecom

                                               December 2005

            OAM Requirements for MPLS Networks
         draft-ietf-mpls-oam-requirements-07.txt

Status of this Memo

    By submitting this Internet-Draft, each author represents that
    any applicable patent or other IPR claims of which he or she is
    aware have been or will be disclosed, and any of which he or she
    becomes aware will be disclosed, in accordance with Section 6 of
    BCP 79.

    Internet-Drafts are working documents of the Internet Engineering
    Task Force (IETF), its areas, and its working groups. Note that  
other
    groups may also distribute working documents as Internet-Drafts.

    Internet-Drafts are draft documents valid for a maximum of six  
months
    and may be updated, replaced, or obsoleted by other documents at any
    time. It is inappropriate to use Internet-Drafts as reference
    material or to cite them other than as "work in progress."

    The list of current Internet-Drafts can be accessed at
    http://www.ietf.org/1id-abstracts.html

    The list of Internet-Draft Shadow Directories can be accessed at
    http://www.ietf.org/shadow.html.

Abstract

    As transport of diverse traffic types such as voice, frame
    relay, and ATM over MPLS become more common, the ability to detect,
    handle and diagnose control and data plane defects becomes
    critical.

    Detection and specification of how to handle those defects is not
    only important because such defects may not only affect the

    fundamental operation of an Multi-Protocol Label Switching (MPLS)
    network, but also because they MAY impact service level  
specification
    commitments for customers of that network.

    This document describes requirements for user and data
    plane operations and management for MPLS.
    These requirements have been gathered
    from network operators who have extensive experience deploying
    MPLS networks, similarly some of these
    requirements have appeared in other documents. This draft specifies
    Operations and Management requirements for Multi-Protocol Label
    Switching, as well as for applications of Multi-Protocol Label
    Switching such as pseudowire voice and virtual private network
    services. Those interested in specific issues relating to
    instrumenting    MPLS for Operations
    and Management purposes are directed to the Multi-Protocol Label
    Switching Architecture specification.

Table of Contents
      Abstract.....................................................1
1.  Introduction.................................................2
2.  Document Conventions.........................................2
2.1 Terminology..................................................2
2.2 Acronyms.....................................................2
3.  Motivations..................................................2
4.  Requirements.................................................2
5.  Security Considerations.....................................26
6.  IANA considerations.........................................27
7.  References..................................................27
7.1 Normative references........................................27
7.2 Informative references......................................29
8.  Author's Addresses..........................................29
9.  Intellectual Property Notice................................30
10. Full Copyright Statement....................................29

1. Introduction

    This document describes requirements for user and data
    plane operations and management (OAM) for Multi-Protocol
    Label Switching (MPLS). These requirements have been gathered
    from network operators who have extensive experience deploying
    MPLS networks. This draft specifies OAM requirements
    for MPLS, as well as for applications of MPLS.

    No specific mechanisms are proposed to address these
    requirements at this time.  The goal of this draft is to
    identify a commonly applicable set of requirements for MPLS
    OAM at this time. Specifically, a set of requirements that apply
    to the most common set of MPLS networks deployed by service
    provider organizations today. These requirements can then be used
    as a base for network management tool development and to guide
    the evolution of currently specified tools, as well as the
    specification of OAM functions that are intrinsic to protocols
    used in MPLS networks.

    Comments should be made directly to the MPLS mailing list
    at [email-masked].

2. Document Conventions

2.1 Terminology

    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
    document are to be interpreted as described in RFC 2119.

    Defect:   Any error condition that prevents an Label Switched
              Path functioning correctly. For example, loss of an
              IGP path will most likely also result in an Label
              Switched Path not being able to deliver traffic to its
              destination. Another example is the breakage of
              a TE tunnel.  These MAY be due to physical
              circuit failures or failure of switching nodes
              to operate as expected.

              Multi-vendor/multi-provider network operation typically
              requires agreed upon definitions of defects (when it is
              broken and when it is not) such that both recovery
              procedures and service level specification impacts can
              be specified.

    Probe-based-detection: Active measurement using a tool such as
           Label Switched Path Ping.

    Collecting traffic: Passive measurement of network traffic.

    Head-end Label Switching Router (LSR): The beginning of a label
           switched path. A head-end LSR is also referred to as
           an Ingress Label Switching Router.

    Probe-based-detection: Active measurement using a tool such as
           Label Switched Path Ping.

    Collecting traffic: Passive measurement of network traffic.

    propagation latency: delay added by the propagation of the packet
                         through the link (fixed value that depends on
                         the distance of the link and the propagation
                         speed).

    transmission latency: delay added by the transmission of the packey
                          over the link i.e. the time it takes put the
                          packet over the media (value that depends of
                          the link throughput and packet length).

    processing latency: delay added by all the operations related to the
                        switching of labeled packet (value is node
                        implementation specific and may are considered
                        as fixed and constant for a given equipment).

    queuing/buffering latency: delay caused by packet queuing (value is
                               variable since depending on the packet
                               arrival rate in addition to the
                               dependance on the packet length and the
                               link throughput).

    node latency: delay added by the network element resulting from of
                  the sum of the transmission, processing and queuing/
                  buffering latency

    one-hop delay: fixed delay experienced by a packet to reach the next
                   hop reesulting from the of the propagation latency,
                   the transmission latency and the processing latency.

    minimum path latency: sum of the one-hop delays experienced by the
                          packet when travelling from the ingress to the
                          egress LSR.

    variable path latency (jitter): sum of the delays caused by the
                                    queuing latency experienced by the
                                    packet at each node over the path.

2.2 Acronyms

ASBR:   Autonomous System Border Router

    CE:   Customer Edge

    SP:   Service Provider

    ECMP: Equal Cost Multipath

    LSP:  Label Switched Path

    LSP Ping: Label Switched Path Ping

    LSR:  Label Switching Router

    OAM:  Operations and Management

    RSVP: Resource reSerVation Protocol

    LDP:  Label Distribution Protocol

    DoS:  Denial of service

3.  Motivations

    This document was created in order to provide requirements
    which could be used to create consistent and useful OAM
    functionality that meets operational requirements of those
    service providers who have or are deploying MPLS.

4. Requirements

    The following sections enumerate the OAM requirements
    gathered from service providers who have deployed MPLS
    and services based on MPLS networks. Each requirement is
    specified in detail to clarify its applicability.
    Although the requirements specified herein are defined by
    the IETF, they have been harmonized with requirements
    gathered by other standards bodies such as the ITU [Y1710].

    4.1 Detection of Label Switched Path Defects

    The ability to detect defects in a broken Label Switched Path
    (LSP) SHOULD not require manual hop-by-hop troubleshooting of
    each LSR used to switch traffic for that LSP. For example,
    it is not desirable to manually visit each LSR along the data
    plane path used to transport an LSP; instead,this function
    SHOULD be automated and able to be performed at some operator
    specified frequency from the origination point of that LSP.
    This implies solutions that are interoperable as to allow for
    such automatic operation. Furthermore, the automation of path
    liveliness is desired in cases where large numbers of LSPs might
    be tested. For example, automated ingress LSR to egress LSR testing
    functionality is desired for some LSPs. The goal is to detect LSP
    path defects before customers do, and this requires detection and
    correction of LSP defects in a manner that is both predictable and
    sufficiently within the constraints of the service level agreement
    under which the service is being offered. Simply put, the sum of
    the time it takes an OAM tool to detect a defect and
    the time needed for an operational support system to react to
    this defect by possibly correcting it or notifying the customer,
    must fall within the bounds of the service level agreement in
    question.

    Synchronization of detection time bounds by tools used to detect
    broken LSPs is required. Failure to specifying defect detection
    time bounds may result in an ambiguity in test results. If the
    time to detect is known, then automated responses can be specified
    both with respect to and with regard to resiliency and service
    level specification reporting. Further, if synchronization of
    detection time bounds is possible, an operational framework can be
    established that can guide the design and specification of MPLS
    applications.

    Although ICMP-based ping [RFC792] can be sent through an LSP as
    an IP payload, the use of this tool to verify the defect free  
operation
    of an LSP has the potential for returning erroneous results (both
    positive and negative) for a number of reasons. First, since
    the ICMP traffic is based on legally addressable IP addressing,
    it is possible for ICMP messages that are originally transmitted
    inside of an LSP to "fall out of the LSP" at some point along
    the path. In these cases, since ICMP packets are routable
    a falsely positive response may be returned.  In other cases
    where the data plane of a specific LSP needs to be tested, it
    is difficult to guarantee that traffic based on an ICMP ping
    header is parsed and hashed to the same equal-cost multi-paths
    as the data traffic.

    Any detection mechanisms that depend on receiving status via a
    return path SHOULD provide multiple return options with the
    expectation that one of them will not be impacted by the original
    defect.  An example of a case where a false negative might occur
    would be a mechanism that requires a functional MPLS return path.
    Since MPLS LSPs are unidirectional, it is possible that although
    the forward LSP which is the LSP under test, might be functioning,
    the response from the destination LSR might be lost, thus giving
    the source LSR the false impression that the forward LSP is
    defective. However, if an alternate return path could be
    specified -- say IP for example -- then the source could
    specify this as the return path to the destination, and in
    this case, would receive a response indicating to it that
    the return LSP is defective.

    The OAM packet MUST follow exactly the customer data path in order
    to reflect path liveliness used by customer data. Particular cases
    of interest are forwarding mechanisms such as equal cost multipath
    (ECMP) scenarios within the operator's network whereby flows are
    load-shared across parallel (i.e., equal IGP cost) paths. Where
    the customer traffic MAY be spread over multiple paths, what is
    required is to be able to detect failures on any of the path
    permutations.  Where the spreading mechanism is payload specific,
    payloads need to have forwarding that is common with the traffic
    under test. Satisfying these requirements introduces complexity
    into ensuring that ECMP connectivity permutations are exercised,
    and that defect detection occurs in a reasonable amount of time.

   4.2 Diagnosis of a Broken Label Switched Path

    The ability to diagnose a broken LSP and to isolate the failed
    component (i.e., link or node) in the path is required. For
    example, note that specifying recovery actions for misbranching
    defects in an LDP network is a particularly difficult case.
    Diagnosis of defects and isolation of the failed component is
    best accomplished via a path trace function which can return the
    the entire list of LSRs and links used by a certain LSP (or at
    least the set of LSRs/links up to the location of the defect) is
    required. The tracing capability SHOULD include the ability to
    trace recursive paths, such as when nested LSPs are used. This
    path trace function MUST also be capable of diagnosing LSP
    mis-merging by permitting comparison of expected vs. actual
    forwarding behavior at any LSR in the path. The path trace
    capability SHOULD be capable of being executed from both the
    head-end Label Switching Router (LSR) and MAY permit downstream
    path components to be traced from an intermediate mid-point LSR.
    Additionally, the path trace function MUST have the ability to
    support equal cost multipath scenarios described above in
    section 4.1.

   4.3 Path characterization

    The path characterization function is the ability to reveal details
    of LSR forwarding operations. These details can then be compared
    later during subsequent testing relevant to OAM functionality.
    This would include but is not limited to:

      - consistent use of pipe or uniform time to live (TTL) models by
        an LSR [RFC3443].
      - sufficient details that allow the test origin to
        excersize all path permutations related to load spreading
        (e.g. ECMP).
      - stack operations performed by the LSR, such as pushes, pops,
        and TTL propagation at penultimate hop LSRs.

    4.4 Service Level Agreement Measurement

    Mechanisms are required to measure the diverse aspects of Service
    Level Agreements which include:
      - defect free forwarding. The service is considered to be
        available and the other aspects of performance measurement
        listed below have meaning, or the service is unavailable and
        other aspects of performance measurement do not.
      - latency - amount of time required for traffic to transit
        the network
      - packet loss
      - jitter - measurement of latency variation

    Such measurements can be made independently of the user traffic
    or via a hybrid of user traffic measurement and OAM probing.

    At least one mechanism is required to measure the number
    of OAM packets. In addition, the ability to measure the
    quantitative aspects of LSPs such as jitter, delay, latency and
    loss MUST be available in order to determine whether or not the
    traffic for a specific LSP are traveling within the
    operator-specified tollerances.

    Any method considered SHOULD be capable of measuring the latency
    of an LSP with minimal impact on network resources.  See section
    2.1 for definitions of the various quantitative aspects of LSPs.

    4.5 Frequency of OAM Execution

    The operator MUST have the flexibility to configure OAM
    parameters insofaras to meet their specific operational
    requirements.

    This includes the frequency of the execution of any OAM
    functions. The capability to synchronize OAM operations is required
    as to to permit consistent measurement of service level agreements.
    To elaborate, there are defect conditions such as misbranching or
    misdirection of traffic for which probe-based detection mechanisms
    that incur significant mismatches in their detection frequency MAY
    result in flapping. This can be addressed either by synchronizing
    the rate or having the probes self-identify their probe rate.
    For example, when the probing mechanisms are bootstrapping,
    they might negotiate and ultimately agree on a probing rate,
    therefore providing a consistent probing frequency and avoiding
    the aforementioned problems.

    One observation would be that wide-spread deployment of MPLS, common
    implementation of monitoring tools and the need for
    inter-carrier synchronization of defect and service level
    specification handling will drive specification of OAM parameters
    to commonly agreed on values and such values will have to be
    harmonized with the surrounding technologies (e.g. SONET/SDH,
    ATM etc.) in order to be useful. This will become particularly
    important as networks scale and misconfiguration can result in
    churn, alarm flapping etc.

   4.6 Alarm Suppression, Aggregation and Layer Coordination

    Network elements MUST provide alarm suppression functionality that
    prevents the generation of superfluous generation of alarms by
    simply discarding them (or not generating them in the first place),
    or by aggregating them together, and thereby greatly reducing the
    number of notifications emitted.  When viewed in conjuction with
    requirement 4.7 below, this typically requires fault notification
    to the LSP egress that
    MAY have specific time constraints if the application using the LSP
    independently implements path continuity testing (for example ATM
    I.610 Continuity check (CC)[I610]).  These constraints apply to
    LSPs that are monitored. The nature of MPLS applications allows
    for the possibility to have multiple MPLS applications attempt to
    respond to defects simultaneously. For example, layer-3 MPLS VPNs
    that utilize Traffic Engineered tunnels, where a failure occurs on
    the LSP carrying the Traffic Engineered tunnel. This failure would
    affect he VPN traffic that uses the tunnel's LSP. Mechanisms are
    required to coordinate network response to defects.

    4.7 Support for OAM Interworking for Fault Notification

    An LSR supporting the interworking of one or more networking
    technologies over MPLS MUST be able to translate an MPLS defect
    into the native technology's error condition. For example, errors
    occurring over a MPLS transport LSP that supports an emulated
    ATM VC MUST translate errors into native ATM OAM Alarm Indication
    Signal (AIS) cells at the termination points of the LSP. The
    mechanism SHOULD consider possible bounded detection time
    parameters, e.g., a "hold off" function before reacting as to
    synchronize with the OAM functions.
    One goal would be alarm suppression by the upper layer using
    the LSP. As observed in section 4.5, this requires that MPLS
    perform detection in a bounded timeframe in order to initiate
    alarm suppression prior to the upper layer independently
    detecting the defect.

    4.8 Error Detection and Recovery.

    Recovery from a fault by a network element can be facilitated by
    MPLS OAM procedures. These procesures will detect a broader range
    of defects than that of simple link and node failures.
    Since MPLS LSPs may span multiple routing areas and service provider
    domains, fault recovery and error detection should be possible
    in these configuration as well as in the more simplifed
    single-area/domain configurations.

    Recovery from faults SHOULD be automatic. It is a requirement that
    faults SHOULD be detected (and possibly corrected) by the network
    operator prior to customers of the service in question detecting
    them.

    4.9 Standard Management Interfaces

    The wide-spread deployment of MPLS requires common information
    modeling of management and control of OAM functionality. This is
    reflected in the the integration of standard MPLS-related MIBs
    (e.g. [RFC3813][RFC3812][RFC3814]) for fault, statistics and
    configuration management. These standard interfaces provide
    operators with common programmatic interface access to
    operations and management functions and their status.

    4.10  Detection of Denial of Service Attacks

    The ability to detect denial of service (DoS) attacks against the
    data or control planes MUST be part of any security management
    related to MPLS OAM tools or techniques.

    4.11 Per-LSP Accounting Requirements

    In an MPLS network, service providers (SPs) can measure traffic
    from an LSR to the egress of the network using some MPLS related
    MIBs, for example. This means that it is a reasonable to know how
    much traffic is traveling from where to where (i.e., a traffic
    matrix) by analyzing the flow of traffic. Therefore, traffic
    accounting in an MPLS network can be summarized as the following
    three items.

      (1) Collecting information to design network

          Providers and their customers MAY need to verify high-level
          service level specifications, either to continuously
          optimize their networks, or to offer guaranteed bandwidth
          services. Therefore, traffic accounting to monitor MPLS
          applications is required.

      (2) Providing a Service Level Specification

          For the purpose of optimized network design, a service
          provider may offer the traffic informationr. Optimizing
          network design needs this information.

      (3) Inter-AS environment

          Service providers that offer inter-AS services require
          accounting of those services.

      These three motivations need to satisfy the following.

         - In (1) and (2), collection of information on a per-LSP
           basis is a minimum level of granularity of collecting
           accounting information at both of ingress and egress
           of an LSP.

         - In (3), SP's ASBR carry out interconnection functions as an
           intermediate LSR. Therefore, identifying a pair of ingress
           and egress LSRs using each LSP is needed to determine the
           cost of the service that a customer is using.

     4.11.1 Requirements

      Accounting on a per-LSP basis encompasses the following set of
      functions:

       (1) At an ingress LSR accounting of traffic through LSPs
           beginning at each egress in question.

       (2) At an intermediate LSR, accounting of traffic through
           LSPs for each pair of ingress to egress.

       (3) At egress LSR, accounting of traffic through LSPs
           for each ingress.

       (4) All LSRs that contain LSPs that are being measuremented
           need to have a common identifier to distinguish each LSP.
           The identifier MUST be unique to each LSP, and its mapping to
           LSP SHOULD be provided from whether manual or automatic
           configuration.

        In the case of non-merged LSPs, this can be achieved by
        simply reading traffic counters for the label stack associated
        with the LSP at any LSR along its path. However, in order to
        measure merged LSPs, an LSR MUST have a means to distinguish
        the source of each flow so as to disambiguate the statistics.

      4.11.2 Scalability

      It is not realistic to perform the just described operations by
      LSRs in a network on all LSPs that exist in a network.
      At a minimum, per-LSP based accounting SHOULD be performed on the
      edges of the network -- at the edges of both LSPs and the MPLS
      domain.

5. Security Considerations

    Provisions to any of the network mechanisms designed to satisfy
    the requirements described herin are required to prevent their
    unauthorized use. Likewise, these network mechanisms MUST
    provide a means by which an operator can prevent denial of
    service attacks if those network mechanisms are used in such
    an attack.

    LSP mis-merging has security implications beyond that of simply
    being a network defect. LSP mis-merging can happen due to a number
    of potential sources of failure, some of which (due to MPLS label
    stacking) are new to  MPLS.

    The performance of diagnostic functions and path characterization
    involve extracting a significant amount of information about
    network construction which the network operator MAY consider
    private.

6. IANA Considerations
    This document creates no new requirements on IANA namespaces
    [RFC2434].

7. References

7.1 Informative References

    [RFC3812]     Srinivasan, C., Viswanathan, A. and T.
                  Nadeau, "MPLS Traffic Engineering Management
                  Information Base Using SMIv2", RFC3812, June 2004.

    [RFC3813]     Srinivasan, C., Viswanathan, A. and T.
                  Nadeau, "MPLS Label Switch Router Management
                  Information Base Using SMIv2", RFC3813, June 2004.

    [RFC3814]     Nadeau, T., Srinivasan, C., and A.
                  Viswanathan, "Multiprotocol Label Switching
                  (MPLS) FEC-To-NHLFE (FTN) Management
                  Information Base", RFC3814, June 2004.

    [Y1710]       ITU-T Recommendation Y.1710, "Requirements for
                   OAM Functionality In MPLS Networks"

    [I610]      ITU-T Recommendation I.610, "B-ISDN operations and
                maintenance principles and functions", February 1999

    [RFC2434] Narten, T. and H. Alvestrand, "Guidelines for Writing
              an IANA Considerations section in RFCs", BCP 26, RFC
              2434, October 1998.

    [RFC792]  Postel, J., "Internet Control Message Protocol", RFC792,
              September 1981.

    [RFC3443] Agarwal, P, Akyol, B., "Time To Live (TTL) Processing in
              Multi-Protocol Label Switching (MPLS) Networks.", RFC3443,
              January 2003.

8. Authors' Addresses

    Thomas D. Nadeau
    Cisco Systems, Inc.
    300 Beaver Brook Road
    Boxboro, MA 01719
    Phone: +1-978-936-1470
    Email: [email-masked]
    Monique Jeanne Morrow
    Cisco Systems, Inc.
    Glatt-Com, 2nd Floor
    CH-8301
    Switzerland
    Voice:  (0)1 878-9412
    Email: [email-masked]

    George Swallow
    Cisco Systems, Inc.
    300 Beaver Brook Road
    Boxboro, MA 01719
    Voice: +1-978-936-1398
    Email: [email-masked]

    David Allan
    Nortel Networks
    3500 Carling Ave.
    Ottawa, Ontario, CANADA
    Voice: 1-613-763-6362
    Email: [email-masked]

    Satoru Matsushima
    Japan Telecom
    4-7-1, Hatchobori, Chuo-ku
    Tokyo, 104-8508 Japan
    Phone: +81-3-5540-8214
    Email: [email-masked]

9. Intellectual Property Statement

    The IETF takes no position regarding the validity or scope of any
    Intellectual Property Rights or other rights that might be  
claimed to
    pertain to the implementation or use of the technology described in
    this document or the extent to which any license under such rights
    might or might not be available; nor does it represent that it has
    made any independent effort to identify any such rights.   
Information
    on the procedures with respect to rights in RFC documents can be
    found in BCP 78 and BCP 79.

    Copies of IPR disclosures made to the IETF Secretariat and any
    assurances of licenses to be made available, or the result of an
    attempt made to obtain a general license or permission for the  
use of
    such proprietary rights by implementers or users of this
    specification can be obtained from the IETF on-line IPR  
repository at
    http://www.ietf.org/ipr.

    The IETF invites any interested party to bring to its attention any
    copyrights, patents or patent applications, or other proprietary
    rights that may cover technology that may be required to implement
    this standard.  Please address the information to the IETF at ietf-
    [email-masked].

10.  Full Copyright Statement

     Copyright (C) The Internet Society (2005).  This document is
     subject to the rights, licenses and restrictions contained in BCP
     78, and except as set forth therein, the authors retain all their
     rights.

     This document and the information contained herein are provided
     on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
     REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND
     THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES,
     EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT
     THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR
     ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
     PARTICULAR PURPOSE.

11. Acknowledgment

    Funding for the RFC Editor function is currently provided by the
    Internet Society.

    The authors wish to acknowledge and thank the following
    individuals for their valuable comments to this document:
    Adrian Smith, British Telecom; Chou Lan Pok, SBC; Mr.
    Ikejiri, NTT Communications and Mr.Kumaki of KDDI.
    Hari Rakotoranto, Miya Khono, Cisco Systems; Luyuan Fang, AT&T;
    Danny McPherson, TCB; Dr.Ken Nagami, Ikuo Nakagawa, Intec Netcore,
    and David Meyer.