Subject: Re: [Last-Call] [v6ops] Tsvart last call review of draft-ietf-v6ops-ipv6-ehs-packet-drops-05

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 7 Apr 2021 14:22:19 +0000

Hi Tom, Fernando,

I would like to send this document to the IESG.

Have we managed to reach a conclusion on this issue?

Tom, my reading of this draft is as Fernando describes, in that it
only intends to describe the implications and avoids making any
recommendations.

I was going to suggest that a sentence be added to the introduction to
make this clear, but on checking again I see that the document
already contains a pretty clear disclaimer in section 2.
 
2.  Disclaimer

   This document analyzes the operational challenges represented by
   packets that employ IPv6 Extension Headers, and documents some of the
   operational reasons why these packets are often dropped in the public
   Internet.  This document is not a recommendation to drop such
   packets, but rather an analysis of why they are dropped.

Is this sufficient?  

I guess that the authors could consider adding a sentence that it also doesn't provide any recommendation on how end hosts make use of extension headers, but that might be a bit incongruous in the sense that the document doesn't appear to talk about end host behaviour at all ...

Regards,
Rob

> -----Original Message-----
> From: v6ops <[email-masked]> On Behalf Of Tom Herbert
> Sent: 10 March 2021 02:03
> To: Fernando Gont <[email-masked]>
> Cc: Gorry Fairhurst <[email-masked]>; IPv6 Operations
> <[email-masked]>; [email-masked];
> [email-masked]; [email-masked]
> Subject: Re: [v6ops] [Last-Call] Tsvart last call review of draft-ietf-
> v6ops-ipv6-ehs-packet-drops-05
> 
> On Tue, Mar 9, 2021 at 4:03 PM Fernando Gont <[email-masked]>
> wrote:
> >
> > On 9/3/21 19:07, Tom Herbert wrote:
> > [...]
> > >
> > > Yes, ACLs on transport layer ports are common requirements, however
> > > the problem arises from related requirements that arise due to the
> > > limitations of routers to be able to locate the transport layer
> > > information in a packet. An example of such an implied requirement
> > > from this draft is "don't send packets with IPv6 header chains that
> > > are too long because some routers can't parse deep enough into packets
> > > to find the transport layer ports due to implementation constraints
> > > (like limited size parsing buffer)".
> >
> > You seem to be reading more from the document than what we actually said
> > in the document.
> >
> > There are no requirements in this document. We simply explain things
> > operators need to do, what are the associated limitations in real-world
> > devices, and what's the likely outcome.
> >
> > That's not an implied requirement, but simply a description of facts.
> >
> It's obvious that the implied or at least inferred requirement is that
> if a host wants to increase the probability of packets making it to
> the destination then they should not make header chains too long. This
> would also be an obvious interoperability requirement, i.e. if I make
> my header chains too long then packets will be dropped and my host
> stack is not interoperable with some elements in the network.
> 
> >
> >
> > > While the rationale for the
> > > requirement may make sense, the problem, at least from the host stack
> > > perspective of trying to send packets with low probability they'll be
> > > dropped, is that a requirement that "don't IPv6 header chains that are
> > > too long" is is useless without any quantification as exactly to what
> > > "too long" might be.
> >
> > "too long" for the processing device(s). You don't know what devices
> > will process your packets, hence cannot even guess what "too long" might
> > mean.
> >
> > What you know for sure is that the longer the chain, the lower the
> > chances of your packets surviving -- as per RFC7872.
> >
> That seems to me more like an assumption than a proven fact. To prove
> it we'd need the data that correlates the length of the chain with
> probability of drop, or alternatively, one could survey common router
> implementations' capabilities and similarly extrapolate the
> correlation. If we had this data then we could derive a meaningful
> quantified requirement for both what routers are expected to process
> and what hosts can expect. RFC7872 doesn't really have sufficient data
> to make this correlation, and besides that it is not current.
> 
> In any case, this draft qualitatively describes why routers are
> droppings. Which I suppose is good, but, given that information, I
> don't see much that helps host developers that are sending packets in
> the network and are trying to go beyond sending packets that conform
> to the least common denominator of plain TCP/IP.
> 
> Tom
> 
> > Thanks,
> > --
> > Fernando Gont
> > SI6 Networks
> > e-mail: [email-masked]
> > PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
> >
> >
> >
> >
> 
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 7 Apr 2021 15:58:52 +0100

Rob Wilton (rwilton) wrote on 07/04/2021 15:22:
> I guess that the authors could consider adding a sentence that it
> also doesn't provide any recommendation on how end hosts make use of
> extension headers, but that might be a bit incongruous in the sense
> that the document doesn't appear to talk about end host behaviour at
> all ...
well, this is kinda the concern that we as authors have.  The document 
is completely clear that it's descriptive, and not prescriptive, and 
there is an entire disclaimer section which points this out.  We get 
Tom's interest in having a document that tells forwarding stack software 
authors what to aim towards, but this isn't that document and we've got 
well-defined reasons for not wanting to move in that direction.

If Tom or others find that the idea of a descriptive-only document 
doesn't match what they would want, well ok, I guess that's 
understandable from the point of view of a software author - but 
consensus isn't unanimity either.  As you suggest, adding an extra 
sentence as a further disclaimer seems incongruous and I'm not sure that 
it would turn the document into a better version of itself.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Wed, 7 Apr 2021 08:20:14 -0700

> I guess that the authors could consider adding a sentence that it also doesn't provide any recommendation on how end hosts make use of extension headers, but that might be a bit incongruous in the…

Rob,

Given that hosts are the ones creating extensions headers and other
packet formats, hosts have a vested interest in how routers are
dealing with their packets. Even before this document was created, we
have long known that extensions headers might be dropped and have been
working on mitigations to reduce the number of drops which are already
addressing some of the reasons that packets with EH. For instance,
consider draft-hinden-6man-hbh-processing-00; this is a proposal to
limit the number of HBH options to exactly one. The idea is that
routers will make it feasible for routers packets that have HBH
options, with the trade off of specifically limiting the extensibility
of the protocol. The problem is there is no data that indicates this
proposal would have the desired effect; we don't if routers would
start accepting packets that are limited to one HBH option.

So my fundamental concern with this draft is that it is an entirely
qualitative description of a well known problem, however a qualitative
analysis is insufficient input for moving extension headers forward.
In the draft, there are several reasons suggested as to why routers
might drop packets, however there is no indication of the relative
occurrence frequency of these. Also, there are parameterizations
mentioned such as in the state that routers might drop if the chain is
"too long", there is no analysis on exactly what "too long" commonly
is (a couple of sizes for parsing buffers are mentioned but without
reference which is another frustration of mine with this draft). A
quantified analysis of the problem would delve into implementations
and deployment thereby providing actionable data. Note this is not the
same as making recommendations, I am just asking for the operational
data as part of the analysis from which we could derive guidance or
new protocol requirements.

Tom

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 8 Apr 2021 10:34:38 +1200

On 08-Apr-21 03:20, Tom Herbert wrote:
...
> occurrence frequency of these.

That seems to call for a fairly major measurement project by an
organisation like CAIDA or RIPE Labs, with collaborative ISPs.
While that is a perfectly good idea, it would presumably take
a couple of years to get data. I personally don't see it as a
valid reason to hold up this draft. Maybe the authors should
add a note about the need for data.

> Also, there are parameterizations

Again, I don't see how that can be done without a major and organised
effort. The issue of buffer sizes may also involve proprietary
information, which is another difficulty. Again, it is neither quick
nor easy to get data.

Regards
    Brian

---

From: Fernando Gont <[email-masked]>
Date: Wed, 7 Apr 2021 20:49:19 -0300

Hi,

> it would turn the document into a better version of itself.

Tom seems to be willing to have a document that proposes a path forward 
to improve usage of EHs. As noted in the very disclaimer, this is not 
that document. That's subject to a different document that interested 
parties should submit.

So, in my opinon, this document is ready tp ship to the IESG.

Thanks!

Cheers,

---

From: Fernando Gont <[email-masked]>
Date: Wed, 7 Apr 2021 21:03:26 -0300

Hi, Tom,

On 7/4/21 12:20, Tom Herbert wrote:
[....]
> start accepting packets that are limited to one HBH option.

What does that proposal have to do with this document?

> qualitative description of a well known problem, however a qualitative

No. It is not a well known problem. If you look at 
draft-hinden-6man-hbh-processing, itś clear that their assumption is 
that limiting the number of EHs or options solves the problem. Whereas 
our document essentially notes that to a large extent the problem has to 
do with the overall EH-chain length -- it doesn't matter if the 
EH-chain: it doesn matter whether you have one long EH, multiple small 
ones, one large EH with one large option, one large EH with many small 
options, or any combination of them.

The fact that youŕe raising this issue and that thereś a belief that 
there'ś a clear and easy way to make EHs work probes that itś certainly 
not a well known problem.

I can also say that I have consulted for different operators, and they 
were not even aware about this issue.

> analysis is insufficient input for moving extension headers forward.

Please read the Abstract:
    This document summarizes the operational implications of IPv6
    extension headers specified in the IPv6 protocol specification
    (RFC8200), and attempts to analyze reasons why packets with IPv6
    extension headers are often dropped in the public Internet.

and the disclaimer:

2.  Disclaimer

If you want to embark in the project of "moving Ehs forward", getting 
whatever data you need for that, etc., thatś totally fine.

> "too long", there is no analysis on exactly what "too long" commonly

We discussed this one to death: Thatś impossible to tell. Itś 
implementation dependent, and the information may not even be public.

> new protocol requirements.

You are asking for a different document. If you want that data, I 
encourage you to start the experiment, and submit an I-D with the results.

Some of us did that homework for the data that we considered useful, and 
provided it to the community in RFC7872. If that data is not enough, I 
certainly encourage you to work on that. But thatś a project thatś 
totally unrelated to this document.

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Wed, 7 Apr 2021 17:05:55 -0700

> information, which is another difficulty. Again, it is neither quick

Brian,

Maybe so, but in the absence of meaningful data or guidance that could
be derived from such data, host stacks have to fall back to reverse
engineering what is actually implemented in the network as opposed to
getting recommendations from IETF or a consortium of vendors, which is
pretty much where we are anyway. The situation seems contradictory to
the whole intent of defining standard inoperable protocols, but I
digress...

Tom

---

From: Tom Herbert <[email-masked]>
Date: Wed, 7 Apr 2021 17:30:57 -0700

> What does that proposal have to do with this document?

Because that proposal is ostensibly addressing a perceived reason for
packets being dropped. If it's not really a problem, then by all means
please chime in on 6man list where the draft is under discussion.

> our document essentially notes that to a large extent the problem has to

What exactly does "large extent " mean? Does that mean that at least
50% or some greater than that percentage of drops of packets with EH
were dropped precisely because the header chains were too long?

> options, or any combination of them.

Perhaps, but again I ask for either data or references from vendors to
verify that supposition.

> not a well known problem.
That is not the only issue that is being addressed. There are also the
suggested limits in RFC8504 for host processing, the mitigation in
RFC8200 to allow intermediate nodes to ignore HBH options, and ICMP
errors in RFC8883 that intermediate nodes drop packets including if
header chains are too long.

---

From: Fernando Gont <[email-masked]>
Date: Wed, 7 Apr 2021 23:46:04 -0300

On 7/4/21 21:30, Tom Herbert wrote:
> packets being dropped. 

That doesn't change what this document is doing *at all*. We're 
explaining what operators do at the time of this writing.

> If it's not really a problem, then by all means

I did raise my concerns during the last 6man minutes. My comments should 
be in the meeting minutes.

> were dropped precisely because the header chains were too long?

That's not the topic that this document is addressing.

> Perhaps, but again I ask for either data or references from vendors to

Our document contains a reference to the very IEPG presentation that was 
done by a vendor.

> header chains are too long.

Again: what does has to do with this document.

Please read the Abstract and the Disclaimer. That's what this document 
is about.

You want this document to be something that it has never been, and has 
never been meaning to be.

Thanks,

---

From: Tom Herbert <[email-masked]>
Date: Wed, 7 Apr 2021 20:29:29 -0700

> That's not the topic that this document is addressing.
Fernando,

But you said "Whereas our document essentially notes that to a large
extent the problem has to do with the overall EH-chain length" so it
would appear this is indeed within the scope of the document. Clearly
that is a conclusion you've reached in writing the doc, I'm asking
that you quantify what you mean by "large extent" and provide the
basis for that quantification. If we knew the relative frequencies of
the various reasons that are mentioned, then that is useful input to
work for both send hosts and protocol development trying to mitigate
drops.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Wed, 7 Apr 2021 20:51:25 -0700

Fernando,

I will also point that the data of RFC7872 is not consistent with
header length being the top reason for drops. If I'm reading the data
correctly, the HBH options are drop at about 3-4 time that rate that
Destination options in that analysis. Since there both the same size
(8 bytes), it seems that the majotiy of HBH options are being dropped
because they are hop-by-hop options, not because the header chain is
too long. This makes sense, because of the original requirement that
all intermediate nodes must process hop-by-hop options, and we know
many vendors didn't implement then. That requirement was relaxed in
RFC8200, and so we'd expect drop rates to drop as nodes move to
ignoring HBH options instead of blindly dropping them.

Tom

---

From: Fernando Gont <[email-masked]>
Date: Thu, 8 Apr 2021 02:16:33 -0300

Hi, Tom,

> ignoring HBH options instead of blindly dropping them.

Again: this document is not trying to explain the numbers in RFC7872. 
PLease do read the abstract and the disclaimer, because you keep trying 
to have this document do something that it's not meant to do.

You should also note that the possible impact on devices is dependent on 
the EH chain length. However, the decision to drop packets need not be 
based on the EH length -- among other reasons, because such information 
might not be readily available.

Folks running a network can't be bothered digging into how many bytes of 
EHs each device they use can handle, and then be expected to figure out 
if there'ś any way to let only packets of shorter EH-chain lengths 
through. There are other problems to tackle day-to-day -- and this one 
is not even a problem: If I have no practical requirement to support 
them, and on the other hand there'ś risk in supporting them, I don't 
need to explain what my decision will be. You'll have a hard time 
explaining your manager or CEO why your network was brought down for 
supporting something you didn't need to support ("didn't need" == "the 
folk that pays the bill didn't ask for or care about it")..

Thanks,
-- 
Fernando Gont
e-mail: [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Thu, 8 Apr 2021 09:32:13 +0000

Hi Tom,

Thank you for your comments.

Regarding the header sizes, I asked the authors to remove reference to specific products because I believe that information is only illustrative rather than definitive.  Even within the same product family there are often different linecards that are based on different silicon families.  Those silicon families will likely have different characteristics/optimizations, and those characteristics vary by different generations of those product families.  Whether a router will forward a particular packet with extension headers might be due to the size of the packet header that is made available to the forwarding ucode, or it might be due to ucode instruction limits, which may change depending on software version as the ucode gets extended to support more features.  It may also depend on what features are configured by the network operator.  All of which makes me feel that providing some illustrative information about the size of header processing for some devices is helpful, but we should not regard that information as being definitive unless it has both been confirmed by the vendors as being accurate, and the precise scenarios under which it was been tested have been documented.

I am generally supportive of efforts to gather extra empirical test data about how extensions headers are being treated on the Internet, but I agree with Brian's subsequent comments that gathering such data in a meaningful way would take a significant amount of time and effort.

Getting more information from the router vendors may also be helpful, but I'm not sure how forthcoming that information will be, and it may depend on how operators decide to configure those devices.

On a more positive note, I see that various technologies currently being standardized in the IETF (SRv6 and IOAM) that make use of IPv6 extension headers, so we may start seeing greater extension header support in new revisions of hardware and software.

Whilst I entirely understand your desire for end hosts to have better information about how extension headers may be used, I don't think that is the purpose of this document, nor am I willing to send it back to the WG to ask them to rework it into something that would be a significantly larger scope, and would take a long time to collect.

I'm satisfied that this issue has been discussed and that the document represents rough consensus.  I will include a note regarding this discussion in my IESG writeup so that the other ADs are aware of it when they ballot on the document.

Regards,
Rob

> From: Tom Herbert <[email-masked]>
> Sent: 07 April 2021 16:20
> To: Rob Wilton (rwilton) <[email-masked]>
> Cc: Fernando Gont <[email-masked]>; Gorry Fairhurst
> <[email-masked]>; IPv6 Operations <[email-masked]>; draft-ietf-
> [email-masked]; [email-masked]; tsv-
> [email-masked]
> > I guess that the authors could consider adding a sentence that it also
> doesn't provide any recommendation on how end hosts make use of extension
> headers, but that might be a bit incongruous in the sense that the
> document doesn't appear to talk about end host behaviour at all ...
> 
> Rob,
>

---

From: Tom Herbert <[email-masked]>
Date: Thu, 8 Apr 2021 07:21:16 -0700

Fernando,

>From the draft:  "The Hop-by-Hop Options header has been particularly
challenging since in most circumstances, the corresponding packet is
punted to the control plane for processing.  As a result, operators
usually drop  IPv6 packets containing this extension header."

"usually" implies that most routers are dropping IPv6 packets with
Hop-by-Hop Options. However, no evidence for that is offered and in
fact the best data available, RFC7872,  shows the drop rate is less
than 50% six years ago well before the mitigation in RFC8200. IMO,
this wording is potentially a mischaracterization and an exaggeration
of the problem.

> folk that pays the bill didn't ask for or care about it")..

Sure, tell me, as a host stack developer, what at reasonable minimum
length is for an IP header chain and I'll fix the packets I'm sending
so that the operators don't need to be bothered with this.

Tom

---

From: "Timothy J. Salo" <[email-masked]>
Date: Thu, 8 Apr 2021 09:33:27 -0500

> so that the operators don't need to be bothered with this.

No one really knows.  Moreover, there is no reason to believe that the
value won't change in the future.  Furthermore, this is well beyond the
control of the IETF: it is driven by decisions made by router vendors
and ISPs.

I think that this has been explained several times, has it not?

-tjs

---

From: Tom Herbert <[email-masked]>
Date: Thu, 8 Apr 2021 08:00:59 -0700

On Thu, Apr 8, 2021 at 7:33 AM Timothy J. Salo <[email-masked]> wrote:
> No one really knows.

That presumes there is no data, and that it is impossible to get the
data. I don't believe that is true. For instance, RFC7872 does show
that between 80-90% of eight byte Destination Options make it through
the Internet. That's a pretty high percentage, so that suggests that a
minimal recommendation for EH length could be at least eight.

> Moreover, there is no reason to believe that the
The ADs can correct me if I'm wrong, but I believe it is within the
scope of IETF to make recommendations and suggest best practices
concerning the use of standard protocols and those recommendations can
and should take operational data into account.

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 9 Apr 2021 08:49:15 +1200

On 09-Apr-21 03:13, Fernando Gont wrote:
...
> JUst as a data point, this was posted yesterday: 
> https://blog.quarkslab.com/analysis-of-a-windows-ipv6-fragmentation-vulnerability-cve-2021-24086.html

Everybody should read that post. It's masterful, and also very funny in parts.

   Brian

---

From: Fernando Gont <[email-masked]>
Date: Thu, 8 Apr 2021 15:13:38 +0000

Hi, Tom,

> minimal recommendation for EH length could be at least eight.

You seem to keep misinterpreting things:

1) This document does not make recommendations.
If you want recommendations, please do submit a draft. (I will note
that RFC7112 at the time wanted to restrict the IPv6 header chain to
1280 bytes, and even *that* didn't fly. -- SO, good luck with that
project....) 

2) As noted by many, many times, the effect of e.g. EH length on
processing describes on so many factors (vendor, model, deployment,
etc.) that trying to come up with a "minimal recommendation" is simply
guesswork.

3) There's no reason to believe that packets that are currently let
through do not or could not cause issues.

4) And no, I don't think that you can expect an operator to make a
survey of every single device they use, and try to find out what's the
ipv6 header chain length that wouldn't screw the network, and work hard
to support it. As noted before, you're going to have a hard time
explaining your CEO or manager why you've decided to unnecessarily
accept risk that you do not need to accept. 

> concerning the use of standard protocols and those recommendations
> can and should take operational data into account.

Indeed. But this document is not a recommendation, and was never meant
to.  Isn't the Disclaimer clear enough on this point?

Thanks,
-- 
Fernando Gont
Director of Information Security
EdgeUno, Inc.
PGP Fingerprint: DFBD 63E3 B248 AE79 C598 AF23 EBAE DA03 0644 1531