Subject: Proposal for changing how IPv6 Hop-by-Hop Options are processed

From: Bob Hinden <[email-masked]>
Date: Thu, 3 Dec 2020 15:15:45 -0800

Hi,

Gorry and I put together a draft that proposes to change how IPv6 Hop-by-Hop Options are processed.  Links to the draft below.

Abstract:

  This document specifies procedures for how IPv6 Hop-by-Hop options
  are processed.  It modifies the procedures specified in the IPv6
  Protocol Specification (RFC8200) to make processing in routers more
  practical with the goal of making IPv6 Hop-by-Hop options useful to
  deploy in the Internet.  When published, this document updates
  RFC8200.

A very short summary is that there can only be one Hop-by-Hop option in a Hop-by-Hop Option header, and that Hop-by-Hop Options must be process in the fast path.

Please read and comment on the draft (that is, not on just this email).   We think this might make Hop-by-Hop options practical to use in the Internet.

Bob & Gorry

> Begin forwarded message:
> 
> From: [email-masked]
> Subject: New Version Notification for draft-hinden-6man-hbh-processing-00.txt
> Date: December 3, 2020 at 3:04:46 PM PST
> To: "Robert M. Hinden" <[email-masked]>, "Godred Fairhurst" <[email-masked]>, "Robert Hinden" <[email-masked]>, "Gorry Fairhurst" <[email-masked]>
> 
> 
> A new version of I-D, draft-hinden-6man-hbh-processing-00.txt
> has been successfully submitted by Robert M. Hinden and posted to the
> IETF repository.
> 
> Name:		draft-hinden-6man-hbh-processing
> Revision:	00
> Title:		IPv6 Hop-by-Hop Options Processing Procedures
> Document date:	2020-12-03
> Group:		Individual Submission
> Pages:		11
> URL:            https://www.ietf.org/archive/id/draft-hinden-6man-hbh-processing-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-hinden-6man-hbh-processing/
> Html:           https://www.ietf.org/archive/id/draft-hinden-6man-hbh-processing-00.html
> Htmlized:       https://tools.ietf.org/html/draft-hinden-6man-hbh-processing-00
> 
> 
> Abstract:
>   This document specifies procedures for how IPv6 Hop-by-Hop options
>   are processed.  It modifies the procedures specified in the IPv6
>   Protocol Specification (RFC8200) to make processing in routers more
>   practical with the goal of making IPv6 Hop-by-Hop options useful to
>   deploy in the Internet.  When published, this document updates
>   RFC8200.
> 
> 
> 
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat
> 
>

---

From: Tianran Zhou <[email-masked]>
Date: Fri, 4 Dec 2020 01:41:47 +0000

Hi Bob,

" A very short summary is that there can only be one Hop-by-Hop option in a Hop-by-Hop Option header, and that Hop-by-Hop Options must be process in the fast path."

I have two concerns:
1. Only one option may restrict many future applications. I believe there are needs to do multiple actions with the hbh behavior. Moreover, the chip capability will improve year by year. We can predict the future the capability to deal with multiple options.

2. The HbH option "must" be processed in the fast path seems also too strict. That means HbH cannot be processed by slow path. I would like to say it "must be firstly processed in the fast path". That means this option should not be blindly sent to slow path. But it should firstly be considered in the fast path. Then several following options: fast path, slow path, by pass.

Best,
Tianran 

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Bob Hinden
Sent: Friday, December 4, 2020 7:16 AM
To: IPv6 List <[email-masked]>
Cc: Gorry Fairhurst <[email-masked]>; Bob Hinden <[email-masked]>
Subject: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Hi,

Abstract:

Bob & Gorry

> To: "Robert M. Hinden" <[email-masked]>, "Godred Fairhurst" 
> <[email-masked]>, "Robert Hinden" <[email-masked]>, "Gorry 
> Fairhurst" <[email-masked]>
> 
> 
> Please note that it may take a couple of minutes from the time of 
> submission until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat
> 
>

---

From: Tom Herbert <[email-masked]>
Date: Thu, 3 Dec 2020 18:42:47 -0700

Bob,

I understand the motivation for this draft, but I don't believe that
forcibly and retroactively restricting a feature in a well deployed
protocol is the right approach to fix the problem.

>From the draft: "The motivation for this change is to simplify the
processing of Hop-by-Hop options in the fast path.". This change would
not be a simplification for everyone. For devices that properly
support multiple HBH options, including all end hosts, this represents
a complication.

>From the draft: "Nodes creating packets with a Hop-by-Hop option
headers MUST only include a single option in the packet.". The only
nodes that create HBH options are end hosts, so would retroactively be
changing host requirements and hence probably all hosts would be out
of protocol conformance since they implement RFC8200 which allows
multiple HBH options. I suspect that if someone sent a patch to Linux
netdev that enforces this requirement, there would be significant
pushback since it is limiting existing functionality that some users
may be benefitting from without providing any tangible benefits in
itself (other than the rather vague possibility that it _might_ make
HBH more supported in network network devices).

In section 5.3 of RFC8504, there is a description of non-mandatory
limits that may be applied to extension header processing at hosts. I
wonder if similar limits could be applied to routers for hop-by-hop
options. The RFC8504 limits weren't just in terms of number of
options, but also limits for padding and byte length of an options. If
the number of options is limited, then the recommended default is 8.
As pointed out in the reviews of the draft, when we set a recommended
default this was essentially establishing an exception on what hosts
should be expected to process. This is a tradeoff, we need to have a
limit that is less than several hundred which are possible for
feasible implementation, but something large enough to allow
reasonable extensibility.

Tom

On Thu, Dec 3, 2020 at 4:16 PM Bob Hinden <[email-masked]> wrote:
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: Jen Linkova <[email-masked]>
Date: Fri, 4 Dec 2020 12:59:01 +1100

Hi Bob, Gory,

Thanks for writing this and for all your effort to make HbH deployable/useful!
A few comments:
1) If we summarize RFC8200 position on HbH, it basically says smth like:
'a node MAY process HbH or MAY not - depends on the configuration and
the default config is NOT to process'.
Your draft suggests to change it to:
-  MUST process in the fast path.
- MUST NOT process in the slow path.
I feel like the second MUST NOT ("If the IPv6 node can not process the
Hop-by-Hop Option header in the fast path it MUST skip over this
option and continue processing the header normally.") a bit too
restrictive. Is there reason not to use SHOULD here? What if my router
can not process a new HbH option in the fast path yet but I want to
apply a policier to it and get it processed in the control plane?

2) I'm a bit concerned about 'only and only one option per HbH header
only'. Aren't' we limiting the use cases drastically by doing this?
Wouldn't it mean that the first HbH option (let's say the MTU one)
which gets deployed would prevent any other options from being used?
I understand the problem with chaining multiple options and I have a
very dumb qiestion to people who understands the routers hw internals
better that I do: would it help if we have an 'Table of content'
option? Which would list codes for all options present in that header
so a single lookup would be enough to know if there are any options
here we are interested in or do not want to process at all. What if we
have such an option and always put it first? am I crazy?

-- 
SY, Jen Linkova aka Furry

---

From: Haoyu Song <[email-masked]>
Date: Fri, 4 Dec 2020 02:26:17 +0000

I share the same concerns. In my opinion, we should only enforce that routers will not drop packets with HbH options and allow the routers to be configured to process 0, 1, or more HbH options or do it in the best effort fashion. So it basically means, packet with HbH options are guaranteed to be forwarded but the processing of the option is optional.

BR,
Haoyu

-----Original Message-----
From: ipv6 <[email-masked]> On Behalf Of Tianran Zhou
Sent: Thursday, December 3, 2020 5:42 PM
To: Bob Hinden <[email-masked]>; IPv6 List <[email-masked]>
Cc: Gorry Fairhurst <[email-masked]>
Subject: RE: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Hi Bob,

I have two concerns:

Best,
Tianran 

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Bob Hinden
Sent: Friday, December 4, 2020 7:16 AM
To: IPv6 List <[email-masked]>
Cc: Gorry Fairhurst <[email-masked]>; Bob Hinden <[email-masked]>
Subject: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Hi,

Abstract:

Bob & Gorry

> URL:            https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-hinden-6man-hbh-processing-00.txt&amp;data=04%7C01%7Chaoyu.song%40futurewei.com%7C2d02d0b86be0472a572208d897f5ee10%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637426429867138791%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=CUJFqmkmjLLcT0lFWZn2HVpX0Fp8BU0DN9yskwaeBas%3D&amp;reserved=0
> Status:         https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fdraft-hinden-6man-hbh-processing%2F&amp;data=04%7C01%7Chaoyu.song%40futurewei.com%7C2d02d0b86be0472a572208d897f5ee10%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637426429867138791%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=FD9nh8S1xvkoCCYAGuCXv%2FiH%2BdGNRX3IBig1BLz0QMk%3D&amp;reserved=0
> Html:           https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-hinden-6man-hbh-processing-00.html&amp;data=04%7C01%7Chaoyu.song%40futurewei.com%7C2d02d0b86be0472a572208d897f5ee10%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637426429867138791%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=TvBfaQURQuZvIMUFT59VdI4wqP8gNdQF1uGo13CC6dY%3D&amp;reserved=0
> Htmlized:       https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-hinden-6man-hbh-processing-00&amp;data=04%7C01%7Chaoyu.song%40futurewei.com%7C2d02d0b86be0472a572208d897f5ee10%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637426429867138791%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=VBUOGO%2BkrnPCPP64l4CguOTU5SIQyduVSmT%2BLOMnyOU%3D&amp;reserved=0
> 
> 
> Abstract:

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fipv6&amp;data=04%7C01%7Chaoyu.song%40futurewei.com%7C2d02d0b86be0472a572208d897f5ee10%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637426429867138791%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=tWTygpRYLt56POZasHaJY50BnUgtUmOQLQlem47lZSk%3D&amp;reserved=0
--------------------------------------------------------------------

---

From: "Pengshuping (Peng Shuping)" <[email-masked]>
Date: Fri, 4 Dec 2020 02:38:16 +0000

Hi Bob, 

Thank you for the efforts on trying to make Hop-by-Hop options practical to use in the Internet!

A few comments as below, 

1. 
"  The Hop-by-Hop Options Header as defined in [RFC8200] can contain one
   or more Hop-by-Hop options.  This document updates [RFC8200] such
   that there MUST only be one option contained in an Hop-by-Hop Options
   header in a single packet."  

Just wonder what are the reasons for this constraint on the number of HBH options to contain per packet.
If there are multiple options that require the same processing behavior (e.g. processing on the fast path), they should be allowed if capable. If we use MUST, then we will add another new constraint.

2.   
"The motivation for this change is to simplify the processing of Hop-by-Hop options in the fast path."

How to simplify compared with the existing multiple options? Would you like to elaborate more? 

3.  "Nodes processing a packet with a Hop-by-Hop options header MUST
   discard the packet if there is more than one Hop-by-Hop option in
   that header."

This is a very strong constraint. Do we really need to introduce this constraint? If the nodes are capable of processing, the multiple options can be handled. Otherwise, simply ignore and skip. 

4. " IPv6 nodes MUST only process an Hop-by-Hop Options header if it can
   be done in the fast path of the router."

What if it is a RA option to be consumed/processed by the control plane?

5. " If the IPv6 node can not
   process the Hop-by-Hop Option header in the fast path ..."

This is not very clear. How would the device know whether it can process the HBH options header or not? Normally the device will act based on the certain value of a field in the packet. Regarding the processing of the HBH options header, there are two fields that can decide how to process this header, that is, the next header (NH) in the IPv6 header and each option type. The current common implementation is that when the device sees the NH =0, then it will directly send the packet to the slow path/control plane.

In this draft https://tools.ietf.org/html/draft-peng-v6ops-hbh-02#section-7, we have listed some requirements to be met to make the HBH usable considering various factors causing the current situation including the common implementations and the existing deployments etc.. I can see that we are sharing the same goal and aiming to the same direction. It would be good if we could get feedbacks. Thank you!

6. "... it MUST skip over this option and continue processing the header normally "

Is this option a HBH option? Is the header the HBH options header? Or IPv6 header? It would be better to clarify in the text. 

7. " 00 - skip over this option and continue processing the header. "

This is recommended in order to support the interop among multi-vendor scenario. When there are devices from multiple vendors, if one of the boxes in the middle does not recognize this HBH option it is better to skip rather than drop the packet which will impact the service deployment.

8. " It is no longer required to send ICMP Parameter Problem messages. "

Would it be better to keep the ICMP messages because it would know the reasons of the packet drop? Otherwise, it would be a silent drop?

9. " This would allow for a node to quickly determine if an option is supported and can be processed. "

Supported -> Recognized and supported?

10. "If the option is not
   supported, then the node processes it as described in Section 5.2 of
   this document. "

Supported -> Recognized?

Some nits as follows, 

11. "   *  Hop-by-Hop options would be sent to the "slow path".  This could
      could degrade the routers performance and it's ability to process
      important control traffic."

This could could degrade -> This could degrade
it's ability -> its ability

12. "   Nodes creating packets with a Hop-by-Hop option headers MUST only
   include a single option in the packet.  This also requires that all
   Hop-by-Hop Options MUST be 8-octet aligned.  See Section 6 for more
   detail. "

Hop-by-Hop option headers -> Hop-by-Hop option header
more detail -> more details

13. " 5.2.  Hop-by-Hop Headers Processing "
-> " 5.2.  Hop-by-Hop Options Header Processing "

14. " Section 4.2 of [RFC8200] defines an Option Type field in options that
   controls how the option is processed if the IPv6 node processing the
   option header does not recognize the Option Type.  This document
   modifies that behavior for options carried in an Hop-by-Hop Option
   header in two ways."

an Option Type field in options that controls -> 
an Option Type field in options, the highest 2 bits of which control

15. " IPv6 nodes MUST only process an Hop-by-Hop Options header if it can
   be done in the fast path of the router. "

an Hop-by-Hop Options header -> a Hop-by-Hop Options header

16. "If the IPv6 node can not
   process the Hop-by-Hop Option header in the fast path is MUST skip
   over this option and continue processing the header normally."

is MUST -> it MUST

17. "   A possible approach to implementing this is to maintain a lookup
   table based on Option Type of the IPv6 options that are supported in
   the fast path. "

to implementing -> to implement
Option Type of the IPv6 options -> Option Type of the IPv6 Hop-by-Hop options

18. "The following sections list current Hop-by-Hop Options that meet the
   alignment requirement defined here (Section 6.1) or if they need to
   be be deprecated or modified (Section 6.2)."

need to be be -> need to be

Best regards, 
Shuping 

> Gorry and I put together a draft that proposes to change how IPv6
> Hop-by-Hop Options are processed.  Links to the draft below.
> 
> Abstract:
> 
> A very short summary is that there can only be one Hop-by-Hop option in a
> Hop-by-Hop Option header, and that Hop-by-Hop Options must be process in
> the fast path.
> 
> Please read and comment on the draft (that is, not on just this email).   We
> think this might make Hop-by-Hop options practical to use in the Internet.
> 
> Bob & Gorry
> 
> 
> > Begin forwarded message:
> >
> > URL:
> https://www.ietf.org/archive/id/draft-hinden-6man-hbh-processing-00.txt
> > Status:
> https://datatracker.ietf.org/doc/draft-hinden-6man-hbh-processing/
> > Html:
> https://www.ietf.org/archive/id/draft-hinden-6man-hbh-processing-00.html
> > Htmlized:
> https://tools.ietf.org/html/draft-hinden-6man-hbh-processing-00
> >
> >
> > Abstract:

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 4 Dec 2020 09:33:55 +0000

I see lots of thoughts - all of which might help see us through towards 
some conclusions.

Just on (1). Whether one HBH option si enough, might depend on the 
use-case.

The HBH options do not need to be set on every packet. For instance, 
they might be sent on packet that carries a transport-layer control 
payload (with no "data") to solict some feedback from the path. This 
might have good deployment possibilities ... because even if there is 
loss of such a packet because of its EH, this loss can be detected by 
the upper layer protocol, and it does not impact the data transmission.

A node can send multiple control such packets with different HBH options 
to perform different actions. That means even with only one HBH option 
per packet, I could still envisage performing different actions, 
MTU-discovery, etc.

I do expect other use cases, and we should identify and evaluate these!

On (2), I do think we can improve the wording. My own thoughts are that 
we need to find words that set the expectation correctly, to avoid 
people seeing this as just a significant DoS vector.

I hope we find support for something in the near future...

Gorry

---

From: Nick Hilliard <[email-masked]>
Date: Fri, 4 Dec 2020 11:43:41 +0000

> Just on (1). Whether one HBH option si enough, might depend on the 

 From the point of view of steering hbh headers off the slow plane, this 
is probably not a bad idea in the sense of "go big or go home".

How does this proposal interact with srv6 which is also processed on a 
hop-by-hop basis?  I.e. if you have srv6 + traditional hbh, then you're 
already at a multiple inspections, potentially deep.

NPUs aren't mentioned; they probably need to be addressed.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Fri, 4 Dec 2020 08:54:48 -0700

> Just on (1). Whether one HBH option si enough, might depend on the
Gorry,

If one option only is a MUST then that would force all use cases to conform.

> The HBH options do not need to be set on every packet.

Many network devices like uniformity in packets being sent on a flow.
For instance, packets may be routed differently based where there is a
HBH EH as opposed when they don't (for instance, some device does
parse over the EH to find the transport ports for ECMP) so this could
cause OOO or other performance issues. Per the transmit clause of
robustness principle, it's best to make all packets of a flow look
alike as much as possible.

> For instance,
> might have good deployment possibilities ...

That's pushing a lot of complexity to the host stack, especially if it
has to decide which among multiple options need to be sent at every
packet.

> because even if there is

Having a HBH container option that could efficiently encode multiple
HBH options (not in TLVs!) is an interesting idea. The flag-fields
approach used by GRE and GUE allows efficient encoding of some number
of fixed length options that are well ordered and allow random access.
For instance, in thirty bit flags we could encode the presence of all
currently defined HBH options (those that are variable length could be
limited to a few lengths). The flags indicate presence of options and
hardware can process this in array or TCAM lookups, in particular the
serial processing needed for processing options in TLVs goes away.

> people seeing this as just a significant DoS vector.
Please look ar section 5.3 of RFC8504 and that should probably be a
reference because it is addressing the same DoS vector.

Tom

---

From: Ron Bonica <[email-masked]>
Date: Fri, 4 Dec 2020 16:52:23 +0000

Folks,

The following are a few things we can all agree on:

- The maximum length of an HBH Options header is 2,048  bytes.
- A source node can encode hundreds of options in 2,048 bytes.
- Given today's technology, it would be impractical to design an ASIC that can process hundreds of options.
- When an ASIC encounters a packet with more options than it can process, the best thing it can do is to drop the packet

Having established that hundreds of options are too many, we are left with the following questions:

- Should each node determine the maximum number of options that it can process, or should there be a global maximum?
- If there should be a global maximum, what should that maximum value be?

If we leave each node to determine the maximum number of options that it can process, source nodes will need to discover the maximum number of options that can be used on each path. This should remind us of PMTU Discovery. We have all seen that movie. It has  a very sad ending.

If we can agree that each node should support at least a global maximum number of HBH options, we are faced with the following dilemma:

- If we pick a constant that is too small, we may not be able to support future use-cases
- If we pick a constant that is too large, low-cost processors may not be able to support it

I think that even low-cost processors can handle 2 or 3 options without blowing their processing budgets. Maybe we should consider a global maximum value somewhere between 1 and 4?

                                                                                               Ron

Juniper Business Use Only

-----Original Message-----
From: ipv6 <[email-masked]> On Behalf Of Tom Herbert
Sent: Friday, December 4, 2020 10:55 AM
To: Gorry Fairhurst <[email-masked]>
Cc: Haoyu Song <[email-masked]>; IPv6 List <[email-masked]>; Bob Hinden <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

[External Email. Be cautious of content]

On Fri, Dec 4, 2020 at 2:34 AM Gorry Fairhurst <[email-masked]> wrote:
>
> I see lots of thoughts - all of which might help see us through 
> towards some conclusions.
>
Gorry,

-----------------------------------------------------------

---

From: tom petch <[email-masked]>
Date: Fri, 4 Dec 2020 17:03:17 +0000

From: ipv6 <[email-masked]> on behalf of Ron Bonica <rbonica=[email-masked]>
Sent: 04 December 2020 16:52

Folks,

<tp>
On the other hand, I have been surprised at the enthusiasm for signalling MSD.  I do not know what the deployment is like but my sense is that it will become widespread so a different approach to PMTU Discovery might have succeeded, as might  a different approach here, one which does not open up scope for DoS attacks.

Tom Petch

                                                                                               Ron

-----Original Message-----
From: ipv6 <[email-masked]> On Behalf Of Tom Herbert
Sent: Friday, December 4, 2020 10:55 AM
To: Gorry Fairhurst <[email-masked]>
Cc: Haoyu Song <[email-masked]>; IPv6 List <[email-masked]>; Bob Hinden <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Gorry,

---

From: Tom Herbert <[email-masked]>
Date: Fri, 4 Dec 2020 10:08:11 -0700

> I think that even low-cost processors can handle 2 or 3 options without blowing their processing budgets. Maybe we should consider a global maximum value somewhere between 1 and 4?

Ron,

Why not use the value of eight which is recommended default limit for
number of options, if there is one, in RFC8504? When we were working
on that we did poll some vendors that mentioned that was reasonable.

Tom

---

From: Ron Bonica <[email-masked]>
Date: Fri, 4 Dec 2020 17:30:15 +0000

Tom,

When you say MSD, do you mean the SPRING Maximum SID Depth? Or are you referring to Bob's MTU Option?

                                                                            Ron

-----Original Message-----
From: tom petch <[email-masked]> 
Sent: Friday, December 4, 2020 12:03 PM
To: Ron Bonica <[email-masked]>; Tom Herbert <[email-masked]>; Gorry Fairhurst <[email-masked]>
Cc: Haoyu Song <[email-masked]>; Bob Hinden <[email-masked]>; IPv6 List <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Folks,

<tp>

Tom Petch

                                                                                               Ron

-----Original Message-----
From: ipv6 <[email-masked]> On Behalf Of Tom Herbert
Sent: Friday, December 4, 2020 10:55 AM
To: Gorry Fairhurst <[email-masked]>
Cc: Haoyu Song <[email-masked]>; IPv6 List <[email-masked]>; Bob Hinden <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Gorry,

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipv6__;!!NEt6yMaO-gk!Rd8Iqrir3dTnmgearoikghlTU2xxSWui2yCSEkmM856ZZ_DVSF8dkDKtbXMZIKL6$
--------------------------------------------------------------------

---

From: Gyan Mishra <[email-masked]>
Date: Fri, 4 Dec 2020 13:25:34 -0500

Hi Bob & Ron

I can see the benefits of this draft as two fold.

1.  Eliminating hbh ddos vector on the internet that forces operators to
drop hbh.

2. Allows new applications that use hbh functionality - that requires hbh
to be processed in the fast path.  There are networking technologies such
in IPPM WG OAM, BIERv6 etc as well as other applications that can greatly
benefit form hbh processing in the fast path.

The concern I have with existing hbh headers that exist well known or
otherwise that would be broken that use more then one option.

https://www.iana.org/assignments/ipv6-parameters/ipv6-parameters.xml

I can see the byte alignment to 8 bits being valuable for processing and
eliminating of all the pad options.

The verbiage below seems pretty tightly focused on trying to minimize
processing to single hbh option to ensure all router platforms can process
in the fast path.  I don’t think that is practical and used a sledge hammer
to solve a problem that could be solved with a tooth pick.

Maybe a one size fits all model could be detrimental as most internet
backbone routers are higher end, and may not require the stringent 1 option
requirement as that is the problem to be solved by with hbh being dropped
on internet backbone routers.

For the backwards compatibility with existing well known and other hbh with
more then one option could result in the same “dropping” of hbh that occurs
today with operators filtering hbh explicitly to avoid ddos vector.  This
is due to the stringent single hbh option requirement.

   Nodes creating packets with a Hop-by-Hop option headers MUST only
   include a single option in the packet.  This also requires that all
   Hop-by-Hop Options MUST be 8-octet aligned.  See Section 6
<https://tools.ietf.org/html/draft-hinden-6man-hbh-processing-00#section-6>
for more
   detail.

   Nodes processing a packet with a Hop-by-Hop options header MUST
   discard the packet if there is more than one Hop-by-Hop option in
   that header.

We have an application use case for hbh in the BIER WG for BIERv6 draft
below:

https://tools.ietf.org/html/draft-xie-bier-ipv6-encapsulation-08

In this draft today we encode the BIER header in the DOH options and set
the option flag for en route hop by hop processing.

So with that instead of the DOH being processed last it would be processed
after the hbh.

Per RFC 8200 we would set the 3rd highest order bit in the option type so
the options data would change en route.

   Two of the currently defined extension headers specified in this
   document -- the Hop-by-Hop Options header and the Destination Options
   header -- carry a variable number of "options" that are type-length-
   value (TLV) encoded in the following format:

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- - - - - - - - -
      |  Option Type  |  Opt Data Len |  Option Data
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- - - - - - - - -

      Option Type         8-bit identifier of the type of option.

      Opt Data Len        8-bit unsigned integer.  Length of the Option
                          Data field of this option, in octets.

      Option Data         Variable-length field.  Option-Type-specific
                          data.

   The third-highest-order bit of the Option Type specifies whether or
   not the Option Data of that option can change en route to the
   packet's final destination.  When an Authentication header is present

   in the packet, for any option whose data may change en route, its
   entire Option Data field must be treated as zero-valued octets when
   computing or verifying the packet's authenticating value.

       0 - Option Data does not change en route

       1 - Option Data may change en route

So with the 3 highest order bits being reserved above their are only 5 bits
remaining or 32 options remaining user configurable for custom options.
There is not a ton of options to begin with but I think 1 is quite small.

I think the DOH Header had similar identical properties of the hbh header
when the 3rd higher octet is set for in route hop by hop processing
specifying the data will change en route.

Could this specification be extended to DOH as you can see the similarities
for applicability in hop by hop en route processing flag use case that can
be processed in the fast path.

When you do the actual accounting or counting of the number of options
being used for BIERv6 draft, if we decided to change to use hbh would we be
using only 1 option I believe being the 3rd higher order bit flag for en
route hop by hop processing.  Would BIERv6 be in conformance with the
single option setting requirement.  We would ensure that when we encode the
BIER header we have it 8 bit aligned boundary so we don’t require any
padding bits.

Kind Regards

Gyan

On Thu, Dec 3, 2020 at 6:16 PM Bob Hinden <[email-masked]> wrote:

> Please read and comment on the draft (that is, not on just this email).
>  We think this might make Hop-by-Hop options practical to use in the
> Internet.
>
> Bob & Gorry
>
>
> > Begin forwarded message:
> >
> > To: "Robert M. Hinden" <[email-masked]>, "Godred Fairhurst" <
> [email-masked]>, "Robert Hinden" <[email-masked]>, "Gorry
> submission
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*M 301 502-134713101 Columbia Pike *Silver Spring, MD

---

From: Jeff Tantsura <[email-masked]>
Date: Fri, 4 Dec 2020 14:41:14 -0800

Tom,

If this is about SPRING MSD - in case of MPLS - ASIC vendors express their capabilities wrt label imposition in a number of labels the ASIC can impose (push), straightforward on a pizza box, more complex on  a line card based system where the functionality could be divided between ingress and egress ASICs.
IPv6 data plane limitations are more subtle though 

Regards,
Jeff

> On Dec 4, 2020, at 09:31, Ron Bonica <rbonica=[email-masked]> wrote:
> 
> Tom,
>

---

From: Fernando Gont <[email-masked]>
Date: Sat, 5 Dec 2020 03:08:19 -0300

Hello, Bob & Gorry,

I've just read your I-D. -- I see there has been some discussion on the 
mailing-list regarding this document, and I've not yet been able to 
catch up with the discussion. So my apologies if some (or even all!) of 
my comments have been mentioned/raised by others.

Here's my feedback:

1) Section 4:
>  The changes meant that an implementation complied with the IPv6
>  specification even if it did not process hop-by-hop options, and that
>  it was expected that routers would add configuration information to
>  control which hop-by-hop options they would process.
 >
>  Unfortunately, this did not improve the processing of Hop-by-Hop
>  options and they are often not possible to be deployed and used in
>  the Internet.  It merely documented how they were being used in the
>  Internet.

While I'm not I'd say the changes incorporated into RFC8200 *will* make 
a difference, I'm not sure we can say they haven't or won't, either. 
RFC8200 was published mid 2017. I've seen rather trivial (software) 
changes to take a year to test and implement. It would probably take 
longer when the change involves hardware. And then the devices needs to 
be bought and deployed. So in that sense I think it's premature to tell 
whether the corresponding change in RFC8200 has made or will make any 
difference.

2)
>   In hindsight, hop-by-hop options were still not practical to be used
>   widely in the Internet.  Many operational routers are configured to
>   drop all packets containing a hop-by-hop option header. 

At the risk of "soudign our own horn", I think a referecen to RFC7872 is 
missing here.

3)
>    *  Allowing multiple hop-by-hop options in a single packet makes it
>       even more expensive in router resources to process these packets
>       in the fast path.  It adds complexity to the number of
>       permutations that might need to be processed.

As noted in draft-ietf-v6ops-ipv6-ehs-packet-drops, there are other 
implications of EHs, that do not necessarily have to do with the number 
of options. One of them is the option size. Because many (most?) 
implementations have limits when it comes to how deep into a packet they 
can peek. So, allowing a single option in a packet, without contraining 
its size (or at least recommending against long options) has the 
potential of resulting in packets where the IPv6 header chain spans past 
the fast memory that the device employs to do fast processing of the 
packets.  This is a key point, and is discussed at length in 
draft-ietf-v6ops-ipv6-ehs-packet-drops .

4)
> For example [Hendriks]
> states that "dropping all packets with Extension Headers, is a bad
> practice", and that "The share of traffic containing more than one EH
> however, is very small.  For the design of hardware able to handle
> the dynamic nature of EHs, we therefore recommend to support at least
> one EH"

I recently read this paper, after Gorry had pointed to it. When it comes 
to the part you quote, I have two obervations:

   * Dropping all packets with EHs might not be nice. But having your box
     owned or DoS'ed is certainly worse. So, at the end of the day, you
     choose the lesser evil. In that light, the claim that "dropping all
     packets with EHs is bad practice" is unfounded and lacks context.
     One might actually argue the opposite: if *not* dropping packets
     with EHs will open your box to DoS, then, then *not* dropping
     such packets is really bad practice.

   * If anything, what the paper shows is that EHs are scarcely used.
     The measurements in the aforementioned paper show that a neglegible
     amount of traffic employs EHs. And that traffic boils down to:
     fragmented UDP traffic, IPsec, local traffic (mostly MLD), or bogus
     traffic (fragmented TCP, ICMP errors that get fragmented or employ
     HBH options, etc..  Other than IPsec, there's not much of a use case
     for EHs nowadays... so I'm not sure one could venture into "how many
     EHs nodes typically employ", because there's nor really much to be
     measured in the first place.

5)
>    discard the packet if there is more than one Hop-by-Hop option in

Limiting things, in general, can make the allowed subset feasible -- 
rather than target at way too much flexibility that it's very expensive 
to implement, if at all possible.

That said, my question is: why is the number of options seen as a more 
important item than, say, the overall length of the HbH, or the overall 
length of the IPv6 header chain as a whole?

Is the number of options within a HbH really a thing for hardware platforms?

6)
> If the IPv6 node can not
>    over this option and continue processing the header normally.

Typo: s/is/it/

7) Section 5.2:

Isn't this the current behavior, anyway?

e.g., generation of error messages is already rate-limited, and there 
are multiple reasons for generating them (including traceroute). Is this 
specific trigger any special?

OTOH, RFC8200 states that HbH should only be processed if the router has 
a reason to -- which seems like good advice to me.

So I'm not sure what is being changed. It seems that the only thing that 
is changed is that even if there's explicit configuration to process 
HbH, the router wouldn't even process it if that can't be done in the 
fast path. -- but, at the end of the day, if the operator has explicitly 
configured the router to process HbH, it's probably because they have no 
option.

8) Section 6.  Hop-by-Hop Option Header Alignment

Some options might have a size that wouldn't make the *end* of the HbH 
fall into a 8-byte boundary, and hence would *require* padding. Router 
Alert seems to be one of those, and is indeed the most common option 
employed with HbH.  The requirement to drop packets that contain more 
than one option in the HbH would thus cause the packets for the only 
current user of HbH to be droopped.

For example, the pcap for MLD at: 
https://www.cloudshark.org/captures/696d02f3316e  (the first one I found 
when googling for one) has a total of three (not even just two1) 
options: one router alert, and two pad1 options for padding (you might 
argue that they could have used a single PadN... but still, a PadN 
option would have been needed).

In that light, I'm not sure you'd be able to implement this document 
(the constrain on the number of options per HbH) in a way that is 
backwards-compatible.

9) Section 6.2:

You catched my previous comment. Apologies -- I was writing comments 
while reading.
That said, in the light of my previous comment, I wonder if, if you 
really wanted to pursue this idea, whether it would be better to define 
a *new* EH.

OTOH, modulo all the comments I made above, you could probably achive 
the same by simply requiring the router to process the first option, and 
ignore the rest. IN that way, it would seem to me that you'd achieve 
what it seems you want to achieve, in a way that it would still work (*) 
  with current users of HbH.

   (*) there could still be cases where implementations include the Pad 
options *first*, and then the router alert. WHile possibly awkward, such 
approach is still valid as per current standards.

Hope the above are of help.

Thanks!

Regards,
Fernando

-- 
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: Fernando Gont <[email-masked]>
Date: Sat, 5 Dec 2020 03:25:41 -0300

Hi, Ron,

> - If we pick a constant that is too large, low-cost processors may not be able to support it

Agreed on all of the above.

My questions are:

1) Does the number of options really matter?

2) Regardless of your response to #1, aren't we overlooking what's 
probably the bigger elephant in the room, which is the overall length of 
the IPv6 header chain?

Thanks,

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 6 Dec 2020 09:03:49 +1300

> probably the bigger elephant in the room, which is the overall length of 

It's more than that. It's the *nature* of the header chain. It's a linked
list with different logical structures for each component of the chain.
It's something that by its nature can never be completely processed by
predefined ASICs or FPGAs; it cries out for an NPU. Not all routers have
NPUs.

It's well worth reading this:
https://datatracker.ietf.org/meeting/104/materials/slides-104-wgtlgo-forwarding-plane-realities-00

  Brian

---

From: Tom Herbert <[email-masked]>
Date: Sat, 5 Dec 2020 15:15:03 -0700

> predefined ASICs or FPGAs; it cries out for an NPU. Not all routers have

Brian,

I'm not sure how to translate that into protocol requirements. What we
did in RFC8504 and RFC8883 was to define different limits that may be
applied to extension header processing (and in one case of RFC8504,
all headers processing).

>From RFC8540 the possible host limits are:

- A host MAY limit the number of consecutive PAD1 options in
destination options or hop-by-hop options to 7.
- A host MAY limit the number of bytes in a PADN option to be less than 8.
- A host MAY disallow unknown options in destination options or
hop-by- hop options.
- A host MAY impose a limit on the maximum number of non-padding
options allowed in the destination options and hop-by-hop extension
headers. (default if limit is used is 8)
- A host MAY impose a limit on the maximum length of Destination
Options or Hop-by-Hop Options extension headers.

For RFC8883 we defined ICMP errors limits that might be applied by
hosts and host routers (the limits here are super set of those
described in RFC8504:

- "Extension header too big"
- "Option too big" for length or number of consecutive padding options
exceeding a limit.
- "Option too big" for the length of an option exceeding a limit.
- "Too many options in an extension header"
- "Extension header chain too long" headers exceeding a limit.
- "Too many extension headers"
- "Headers too long"

The last one is not specific to extension headers, but can be used if
a router drops a packet because the length of all the headers it needs
to process is greater than its capabilities.

Tom

---

From: Fernando Gont <[email-masked]>
Date: Sun, 6 Dec 2020 02:02:07 -0300

> predefined ASICs or FPGAs; it cries out for an NPU. Not all routers have

Indeed.

I was wondering what the motivation for limiting the number of options 
in a HbH is. You're correct that no matter what, the packet structure is 
what it is.  That aside, the only thing I can think of that could be 
improved (number of bytes from the beginning of the packet that the 
router can peek into very fast) isn't addressed by limiting the number 
of options.  Hence my question.

Thanks!

Regards,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Ron Bonica <[email-masked]>
Date: Sun, 6 Dec 2020 23:27:52 +0000

Fernando,

You have a point. I think that we need two global limits:

- number of options
- two size of the HBH options header.

                                       Ron

-----Original Message-----
From: Fernando Gont <[email-masked]> 
Sent: Saturday, December 5, 2020 1:26 AM
To: Ron Bonica <[email-masked]>; Tom Herbert <[email-masked]>; Gorry Fairhurst <[email-masked]>
Cc: Haoyu Song <[email-masked]>; Bob Hinden <[email-masked]>; IPv6 List <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Hi, Ron,

On 4/12/20 13:52, Ron Bonica wrote:
> - When an ASIC encounters a packet with more options than it can 
> process, the best thing it can do is to drop the packet
>

My questions are:

2) Regardless of your response to #1, aren't we overlooking what's probably the bigger elephant in the room, which is the overall length of the IPv6 header chain?

Thanks,
--
Fernando Gont
e-mail: [email-masked] || [email-masked] PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: Gyan Mishra <[email-masked]>
Date: Sun, 6 Dec 2020 19:16:50 -0500

I think the draft should state how limiting the number of options
translates into forwarding rate improvement in the fast path versus punt to
slow path.

Thanks

Gyan

On Sun, Dec 6, 2020 at 6:28 PM Ron Bonica <rbonica=
[email-masked]> wrote:

> Cc: Haoyu Song <[email-masked]>; Bob Hinden <
> [email-masked]>; IPv6 List <[email-masked]>
> > - Should each node determine the maximum number of options that it can
> process, or should there be a global maximum?
> > If we leave each node to determine the maximum number of options that it
> can process, source nodes will need to discover the maximum number of
> options that can be used on each path. This should remind us of PMTU
> Discovery. We have all seen that movie. It has  a very sad ending.
> >
> > If we can agree that each node should support at least a global maximum
> number of HBH options, we are faced with the following dilemma:
> >
> e-mail: [email-masked] || [email-masked] PGP Fingerprint:
> 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>
>
-- 

*Gyan Mishra*

---

From: Fernando Gont <[email-masked]>
Date: Mon, 7 Dec 2020 03:34:43 -0300

On 6/12/20 21:16, Gyan Mishra wrote:
> translates into forwarding rate improvement in the fast path versus punt 
> to slow path.

+1

Thanks,

---

From: tom petch <[email-masked]>
Date: Mon, 7 Dec 2020 12:45:34 +0000

From: Ron Bonica <[email-masked]>
Sent: 04 December 2020 17:30

Tom,

<tp>
Ron,

Apologies for the ambiguity.  I meant the SR usage as signalled by OSPF, BGP-LS and such like.  I should have remembered how many different meanings that the RFC-Editor finds for this abbreviation!

Tom Petch
                                                                            Ron

-----Original Message-----
From: tom petch <[email-masked]>
Sent: Friday, December 4, 2020 12:03 PM
To: Ron Bonica <[email-masked]>; Tom Herbert <[email-masked]>; Gorry Fairhurst <[email-masked]>
Cc: Haoyu Song <[email-masked]>; Bob Hinden <[email-masked]>; IPv6 List <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Folks,

<tp>

Tom Petch

                                                                                               Ron

-----Original Message-----
From: ipv6 <[email-masked]> On Behalf Of Tom Herbert
Sent: Friday, December 4, 2020 10:55 AM
To: Gorry Fairhurst <[email-masked]>
Cc: Haoyu Song <[email-masked]>; IPv6 List <[email-masked]>; Bob Hinden <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Gorry,

---

From: Gyan Mishra <[email-masked]>
Date: Mon, 7 Dec 2020 09:25:54 -0500

All

Just thinking about the hop by hop processing in the context of history of
routers and processors in the fast path.  I think some context should be
laid down as far as processor evolution.

>From early 2000s fixed function ASIC fixed function processors which
provided line rate fast path processing but were inflexible in deep packet
inspection for traffic punted to slow path and required CoPP policies or
ACLs to protect the control plane.  This unfortunately did lead to
filtering of hbh on the internet that still exists today.

Early 2000 NPUs which provided line rate fast path but also has better
capabilities to shunt to slow path for traffic that could not be processed
in fast path.
This helped and allowed hbh to be shunted to slow path but did not impact
fast path processing of packets without hbh. As this evolution happens
operators being stuck in the mindset of fixed function processors continued
to keep the hbh filters in place even though they could be removed.

As the paradigm evolution has evolved from large chassis LC based
architecture with advent of Merchant silicon and need for SDN and openflow
of the shelf and major price point reduction as compare to proprietary ASIC
based architecture came Broadcom as well as other incumbent vendors
charging forward with Merchant silicon which lead to advent and philosophy
shift to pizza box 1RU 2RU boxes with high speed forwarding plane back to
ASIC FPGA based architecture non fixed function that delivers fast path and
shunt to slow path for packets requiring additional processing.  Also now a
lot of operators as well as manufacturers providing disaggregated software
so you can run any mix of white box hardware with white box software or
even incumbent vendors providing disaggregated software to run on white box
hardware all of which is based on Merchant silicon.

Link that has list of Merchant silicon manufacturers

https://packetpushers.net/virtual-toolbox/list-merchant-silicon-manufacturers-chips/

So I think we have to look at this 20 year history of processors out there
and industry evolution as well as paradigm shifts as they happen in the
marketplace.

Gyan

On Mon, Dec 7, 2020 at 7:46 AM tom petch <[email-masked]> wrote:

> Apologies for the ambiguity.  I meant the SR usage as signalled by OSPF,
> BGP-LS and such like.  I should have remembered how many different meanings
> that the RFC-Editor finds for this abbreviation!
>
> Tom Petch
>
>   Ron
>
>
> - When an ASIC encounters a packet with more options than it can process,
> the best thing it can do is to drop the packet
>
> On the other hand, I have been surprised at the enthusiasm for signalling
> MSD.  I do not know what the deployment is like but my sense is that it
> will become widespread so a different approach to PMTU Discovery might have
> succeeded, as might  a different approach here, one which does not open up
> scope for DoS attacks.
>
> Tom Petch
>
> I think that even low-cost processors can handle 2 or 3 options without
> blowing their processing budgets. Maybe we should consider a global maximum
> value somewhere between 1 and 4?
>
>
>                      Ron
>
>
>
> Administrative Requests:
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipv6__;!!NEt6yMaO-gk!Rd8Iqrir3dTnmgearoikghlTU2xxSWui2yCSEkmM856ZZ_DVSF8dkDKtbXMZIKL6$
-- 

*Gyan Mishra*

---

From: Tom Herbert <[email-masked]>
Date: Mon, 7 Dec 2020 08:54:43 -0700

Ron,

Yes, very much so. HBH and Destination options are encoded in TLVs and
in their nature must be serially processed, and so processing a long
list of TLVs is expensive for any device. Consider that someone could
put over 700 options (zero data length) into a standard 1500 MTU size
packet. That is going to be hard for anyone to process efficiently,
and really the only purpose of doing that is DoS. Limiting the size of
the EH will limit the number of options; like a 128 byte limit would
limit the number of options to 64-- however that is still many more
than would ever seem to be ever be needed and still is hard for low
end devices to process.

> 2) Regardless of your response to #1, aren't we overlooking what's probably the bigger elephant in the room, which is the overall length of the IPv6 header chain?

Yes, that is why we have that limit in addition to the number of
options limit in RFC8504 and ICMP errors for it in RFC8883.The hard
part here is establishing what the exact limit should be. In the UDP
Encapsulation Design Team, we discussed this and polled various
hardware vendors as to what the size of their parsing buffer is (i.e.
the number of bytes of headers that a router can parse in the fast
path). The answers we got we're all over the board, something like 64,
128, 256, or MTU of the packet. We didn't see consensus and didn't
pursue trying to specify a limit at that time.

I'll also point out that not all deployed routers require specialized
hardware, in some use cases line rate can be sustained with a
forwarding path in a commodity CPU (I imagine this is in a lot of home
routers). Efficient processing of options is an issue for them as
well. For such devices, the number of options to processing is a
bigger problem than how many bytes of header there are (i.e. serial
processing in a CPU is more expensive than a few cache misses). So
both limits for number of options as well as length of EH seem to be
needed

Also, routers participating in segment routing need to process
destination options before the routing header also, therefore It makes
sense that the same limits should be applicable to destination options
before the routing header. So routers might process both HBH and
Destination options and destination hosts must process them, and hosts
send them in packets. If there are to be limits, I suggest they should
be applicable across all nodes.

I think the most prudent thing to do is update RFC8504 to make the
limits described in section 5.3 applicable to routers (make the limits
applicable to all nodes). That is, make the default limit of eight
options applicable to routers, and then define a specific default
limit for length of HBH and Destination Otions EH. The limits on
option padding could also be applicable to routers.

Tom

---

From: Ron Bonica <[email-masked]>
Date: Mon, 7 Dec 2020 18:43:35 +0000

Tom,

Signaling MSD is different from signaling the maximum number of options in an HBH header.

SR paths are confined to SR domains that have SR signaling capabilities. By contrast, IPv6 delivery paths can span many domains whose signaling capabilities are unpredictable.

                                       Ron

-----Original Message-----
From: tom petch <[email-masked]> 
Sent: Monday, December 7, 2020 7:46 AM
To: Ron Bonica <[email-masked]>; Tom Herbert <[email-masked]>; Gorry Fairhurst <[email-masked]>
Cc: Haoyu Song <[email-masked]>; Bob Hinden <[email-masked]>; IPv6 List <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Tom,

<tp>
Ron,

Tom Petch
                                                                            Ron

-----Original Message-----
From: tom petch <[email-masked]>
Sent: Friday, December 4, 2020 12:03 PM
To: Ron Bonica <[email-masked]>; Tom Herbert <[email-masked]>; Gorry Fairhurst <[email-masked]>
Cc: Haoyu Song <[email-masked]>; Bob Hinden <[email-masked]>; IPv6 List <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Folks,

<tp>

Tom Petch

                                                                                               Ron

-----Original Message-----
From: ipv6 <[email-masked]> On Behalf Of Tom Herbert
Sent: Friday, December 4, 2020 10:55 AM
To: Gorry Fairhurst <[email-masked]>
Cc: Haoyu Song <[email-masked]>; IPv6 List <[email-masked]>; Bob Hinden <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

Gorry,

---

From: Tom Herbert <[email-masked]>
Date: Mon, 7 Dec 2020 12:19:27 -0700

> Signaling MSD is different from signaling the maximum number of options in an HBH header.
A node can signal back to the source host the maximum number of
options it supports by sending an RFC8883 ICMPv6 error when a packet
is processed that has a number of options that exceed the limit
(Parameter Problem, code 9). This works on the Internet and limited
domains to the extent that ICMP is reliable (albeit, it's probably
more useful in limited domains).

---

From: Ron Bonica <[email-masked]>
Date: Mon, 7 Dec 2020 20:32:51 +0000

Tom,

Do we have any data regarding whether networks filter ICMP Parameter Problem messages?

                                                     Ron

-----Original Message-----
From: Tom Herbert <[email-masked]> 
Sent: Monday, December 7, 2020 2:19 PM
To: Ron Bonica <[email-masked]>
Cc: tom petch <[email-masked]>; Gorry Fairhurst <[email-masked]>; Haoyu Song <[email-masked]>; Bob Hinden <[email-masked]>; IPv6 List <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

> Signaling MSD is different from signaling the maximum number of options in an HBH header.
A node can signal back to the source host the maximum number of options it supports by sending an RFC8883 ICMPv6 error when a packet is processed that has a number of options that exceed the limit (Parameter Problem, code 9). This works on the Internet and limited domains to the extent that ICMP is reliable (albeit, it's probably more useful in limited domains).

> To: Ron Bonica <[email-masked]>; Tom Herbert 
> <[email-masked]>; Gorry Fairhurst <[email-masked]>
> Cc: Haoyu Song <[email-masked]>; Bob Hinden 
> <[email-masked]>; IPv6 List <[email-masked]>
> To: Ron Bonica <[email-masked]>; Tom Herbert 
> <[email-masked]>; Gorry Fairhurst <[email-masked]>
> Cc: Haoyu Song <[email-masked]>; Bob Hinden 
> <[email-masked]>; IPv6 List <[email-masked]>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipv6
> __;!!NEt6yMaO-gk!Rd8Iqrir3dTnmgearoikghlTU2xxSWui2yCSEkmM856ZZ_DVSF8dk
> DKtbXMZIKL6$

---

From: Fernando Gont <[email-masked]>
Date: Mon, 7 Dec 2020 21:22:12 -0300

On 7/12/20 12:54, Tom Herbert wrote:
[...]
> Yes, very much so.

I'm not saying that the limit doesn't make sense -- clearly limits are 
generally a good thing, and for a software router, it's clear that some 
limit has benefits.

However, my question is whether this particular limit would make a 
difference by the devices that are currently droping packets with HbH.

>  HBH and Destination options are encoded in TLVs and

Please note that if one is to limit the number of options in a HbH, one 
should also limit the number of HbHs. Otherwise, as an attacker I could 
simply send packets with multiple HbHs, which the current standard 
recommends nodes to process.

> part here is establishing what the exact limit should be.

Right now, we have no limit, but we also don't know the least common 
denominator. If it was me, I'd say that having something along the lines 
of "MUST support IPv6 header chains of at least 64 bytes, and SHOULD 
support IPv6 header chains of up to the link MTU" or the like would be a 
good start.

  -- Yes, some could claim that requiring EH lengths of 64 bytes. OTOH, 
I'd say that getting to "hey buddies, enough is enough, you should 
really be supporting at least X" would really be a step forward.

Note: I'm not a super fan of EHs, and I'm also skeptical about the 
extent to which something like this would make a difference. But if I 
was asked to do something to help improve the current situation, that's 
probably what I'd try.

> In the UDP

How about sticking to the least common denominator?

> processing in a CPU is more expensive than a few cache misses).

Agreed on this last bit.

> be applicable across all nodes.

FWIW, a limit on the number of EH instances of the same type should also 
be applied if we are going to go this way.

Thanks,

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 8 Dec 2020 02:42:23 -0500

On Mon, Dec 7, 2020 at 10:55 AM Tom Herbert <[email-masked]> wrote:

> > > If we leave each node to determine the maximum number of options that
> it can process, source nodes will need to discover the maximum number of
> > > If we can agree that each node should support at least a global
> maximum number of HBH options, we are faced with the following dilemma:
> > >

    If someone put 700 options with 0 data length is that almost like an
ddos attack signature.  Is that something to flag on to drop 0 data length
hbh.

Do hbh options have to be processed serially?

-- 

*Gyan Mishra*

---

From: "Eric Vyncke (evyncke)" <[email-masked]>
Date: Tue, 8 Dec 2020 13:36:11 +0000

Bob and Gorry,

[Wearing no special hat except the 'belated reply' one...]

As you know, I am really in favor of extension headers to provide evolution but not at the expense of scalability or security.

Section 5.1, "only one option", is very limitative with little performance gain. The complexity to configure source node and intermediate routers to handle this is probably huge. I would prefer a text around "routers MAY process 0, 1, or more options depending on their local configuration / state". Opinion shared by many ;-)

Section 5.2, "only fast path", as long as it is a local policy and not mandatory, then I am fine with it as it is rather common sense. OTOH, rate limiting those to the slow path to send ICMP back to the source could be really useful for network operators / app developpers.

Section 6 & 7, "alignment" is the logical consequence of "only one option" and could be a good thing even if I still believe that having multiple options should be allowed.

Like others, this RFC 8200 update will be of little help until fully deployed over the Internet... 

In 2020, it is not only between generic CPU and ASIC but also NPU that are more flexible and quite agile in the 'fast path'

Thank you both for trying to improve the usability of HbH and not trying to kill it ;-)

Regards

-éric

PS: minor nits, in section 4, first item in the 2nd bullet list: s/fast path/slow path/ ?

PS2: other nots, in section 5.2, " continue processing the header normally" which header ?

-----Original Message-----
From: ipv6 <[email-masked]> on behalf of Bob Hinden <[email-masked]>
Date: Friday, 4 December 2020 at 00:16
To: IPv6 List <[email-masked]>
Cc: Gorry Fairhurst <[email-masked]>, Bob Hinden <[email-masked]>
Subject: Proposal for changing how IPv6 Hop-by-Hop Options are processed

    Hi,

    Abstract:

    Bob & Gorry

---

From: Tom Herbert <[email-masked]>
Date: Tue, 8 Dec 2020 09:20:11 -0700

On Tue, Dec 8, 2020 at 12:42 AM Gyan Mishra <[email-masked]> wrote:

>> > > - Should each node determine the maximum number of options that it
>> can process, or should there be a global maximum?
> Gyan,

No, options with zero data length might be useful. A limit on the number of
options to process should be sufficient to thwart this DoS attack.

> Do hbh options have to be processed serially?

Yes, per RFC8200: "The sequence of options within a header must be
processed strictly in the order they appear in the header". However, an
interesting observation can be made with respect to that: as long as the
external visible behavior of the processing is the same as what strictly
serialize processing would output, an implementation could process options
any way it sees fit as and still satisfy the requirement. For instance,
since the PMTU option and IOAM option are independent they could be
processed in any order regardless of their relative ordering when both are
present in HBH EH. This observation also allows the possibility that
options could be processed in parallel (I believe this affords
opportunities to create more parallelism in serial processing).

Note that before a TLV is processed, it needs to be identified. i.e.
parsed, in a packet. In order to parse the N'th TLV we must parse the first
N-1 TLVs, and it's really difficult to do this in any other manner than
parsing each TLV one at a time from the beginning of the list. One possible
technique to make this more efficient would be to input the whole TLV list
into a hardware TCAM that could output the set of TLVs, their offsets, and
their lengths. Ignoring for a moment that you'd need a very wide TCAM to
parse a HBH EH this way, the technique is quickly rendered impractical
because the number of TCAM entries needed.  The number of possible TLV
arrangements in a list is combinatorial based on the number of possible TLV
types and the possible lengths of TLVs. The combinatorial nature arises
from the fact that TLVs are unordered, so for N different TLV types there
are approximately e*N! possible arrangements of TLVs that would require an
equivalent number of entries in a TCAM (assuming TLVs are fixed length and
appear at most once in the list). If TLVs were strictly ordered then the
number of possible arrangements of N TLVs is 2^N which starts to come to a
practical implementation in TCAM. Flag-fields, like in GRE and GUE, employ
ordered options, enforce an option appears at most once, fixed length
options, and move the types in a bitmap so that option parsing for those is
feasible in a TCAM.

Tom

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 8 Dec 2020 14:36:12 -0500

Thanks Tom that really helps understand TLV processing characteristics.

See in-line

On Tue, Dec 8, 2020 at 11:20 AM Tom Herbert <[email-masked]> wrote:

>>> > > - Given today's technology, it would be impractical to design an
>>> ASIC that can process hundreds of options.
>>> > > If we leave each node to determine the maximum number of options
>>> that it can process, source nodes will need to discover the maximum number
>>> of options that can be used on each path. This should remind us of PMTU
>>> > > - If we pick a constant that is too large, low-cost processors may
>>> not
>>> > > be able to support it
>>> >
> No, options with zero data length might be useful. A limit on the number
> of options to process should be sufficient to thwart this DoS attack.
>
>

   Gyan> My thoughts are that modern current generation high end
distributed LC based platforms with NPU or pizza box with ASIC FPGA as long
as you have multiple cores that parallel TLV processing can occur to assist
with fast path TLV processing.  Also on the older generation fixed function
ASIC FPGA based architecture that were inflexible in parsing extended
headers are still the main gap that is driving the hbh filtering mindset.
Also I think even after hardware is upgraded to NPU based that can properly
shunt hbh TLV processing to slow path, operators are still continuing the
old mindset of continuing to filter hbh.  So maybe education on the
operators side from v6ops perspective to stop filtering hbh on modern high
end routers.

> parsed, in a packet. In order to parse the N'th TLV we must parse the
> first N-1 TLVs, and it's really difficult to do this in any other manner
> than parsing each TLV one at a time from the beginning of the list. One
> possible technique to make this more efficient would be to input the whole
> TLV list into a hardware TCAM that could output the set of TLVs, their
> offsets, and their lengths. Ignoring for a moment that you'd need a very
> wide TCAM to parse a HBH EH this way, the technique is quickly rendered
> impractical because the number of TCAM entries needed.  The number of
> possible TLV arrangements in a list is combinatorial based on the number
> of possible TLV types and the possible lengths of TLVs. The combinatorial
> nature arises from the fact that TLVs are unordered, so for N different TLV
> types there are approximately e*N! possible arrangements of TLVs that
> would require an equivalent number of entries in a TCAM (assuming TLVs
> are fixed length and appear at most once in the list). If TLVs were
> strictly ordered then the number of possible arrangements of N TLVs is
> 2^N which starts to come to a practical implementation in TCAM.
> Flag-fields, like in GRE and GUE, employ ordered options, enforce an option
> appears at most once, fixed length options, and move the types in a bitmap
> so that option parsing for those is feasible in a TCAM.
>

    Gyan> Do you have any research articles or publications that is
publicly available that has approaches for TLV processing techniques for
hbh fast path processing.

Outside of  the IANA list of TLVs below,  does there really exist many
unknown TLVs that could  be processed due to local configuration which the
default per RFC 8054 is to process them.   Since there is a processor hit
for trying to process unknown  TLVs and if so maybe having the default be
to disallow them so are drop silently.

I guess if there are 1000s of valid valuable unknown options that we would
not want to change the default accept and process of unknown options, but
does unknown options imply or open door to a possible ddos vector.

>>> > e-mail: [email-masked] || [email-masked] PGP
>>> Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>>> >
>>> >
>>>
>> *M 301 502-134713101 Columbia Pike
>> <https://www.google.com/maps/search/13101+Columbia+Pike%C2%A0+Silver+Spring,+MD?entry=gmail&source=g>*Silver
>> Spring, MD
>> <https://www.google.com/maps/search/13101+Columbia+Pike%C2%A0+Silver+Spring,+MD?entry=gmail&source=g>
>>
>> --

*Gyan Mishra*

---

From: "Xiejingrong (Jingrong)" <[email-masked]>
Date: Tue, 15 Dec 2020 03:40:14 +0000

Hi!
Thanks Gorry and Bob for this inspiring work !
I think the proposal of using one Option in HBH is really a perfect idea to make the HBH useful, practical, and survived.
For the worries about one HBH option limit, I think the option that expect to work nicely in HBH could/should be designed practical, compact, and also extensible.
For example, one could design the option with a basic option data part, and a flag field to indicate some "optional" parts, and the option length to include the basic part and the optional parts.
The "optional" part itself could even be some kind of sub-option-tlv of the option. 
It could even be an existing HBH Option TLV ---- directly embedded in another option to be used as the "only" HBH option of a packet.

Thanks
Jingrong

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Gorry Fairhurst
Sent: Friday, December 4, 2020 5:34 PM
To: Haoyu Song <[email-masked]>; Tianran Zhou <[email-masked]>; Bob Hinden <[email-masked]>; IPv6 List <[email-masked]>
Subject: Re: Proposal for changing how IPv6 Hop-by-Hop Options are processed

I see lots of thoughts - all of which might help see us through towards some conclusions.

Just on (1). Whether one HBH option si enough, might depend on the use-case.

The HBH options do not need to be set on every packet. For instance, they might be sent on packet that carries a transport-layer control payload (with no "data") to solict some feedback from the path. This might have good deployment possibilities ... because even if there is loss of such a packet because of its EH, this loss can be detected by the upper layer protocol, and it does not impact the data transmission.

A node can send multiple control such packets with different HBH options to perform different actions. That means even with only one HBH option per packet, I could still envisage performing different actions, MTU-discovery, etc.

On (2), I do think we can improve the wording. My own thoughts are that we need to find words that set the expectation correctly, to avoid people seeing this as just a significant DoS vector.

Gorry

On 04/12/2020 02:26, Haoyu Song wrote:
>> <[email-masked]>, "Robert Hinden" <[email-masked]>, 
>> "Gorry Fairhurst" <[email-masked]>
>>
>>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> ietf.org%2Fmailman%2Flistinfo%2Fipv6&amp;data=04%7C01%7Chaoyu.song%40f
> uturewei.com%7C2d02d0b86be0472a572208d897f5ee10%7C0fee8ff2a3b240189c75
> 3a1d5591fedc%7C1%7C0%7C637426429867138791%7CUnknown%7CTWFpbGZsb3d8eyJW
> IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&
> amp;sdata=tWTygpRYLt56POZasHaJY50BnUgtUmOQLQlem47lZSk%3D&amp;reserved=
> 0