Subject: Re: [arch-d] I-D Action: draft-iab-protocol-maintenance-00.txt

From: Brian E Carpenter <[email-masked]>
Date: Thu, 10 May 2018 14:05:20 +1200

Hi,

I have very mixed feelings about this draft.

Firstly, let me clarify that although I was the editor of
RFC 1958, the wording about the robustness principle was either
drafted by or wordsmithed by Jon Postel himself.

That being said, it isn't sacred. But the present draft uses
language that describes it as a failure, which is far from true.
Before this becomes a community document, IMHO the tone *really*
needs to change, starting with the title. How about
"Some Unintended Consequences of the Robustness Principle"
or even
"When to Apply the Robustness Principle"?

I regret that we didn't discuss this topic in RFC 6709
("Design Considerations for Protocol Extensions").
Interestingly, a poster child for extensibility problems
in that RFC is TLS, the first problem case mentioned in the
present draft. Just read this extract:

> It has turned out that many SSLv3 and TLS implementations that do not
> support TLS extensions did not ignore the unknown extensions, as
> required by the protocol specifications, but instead failed to
> establish connections.  Since several of the implementations behaving
> in this manner are used by high-profile Internet sites, such as
> online banking sites, this has caused a significant delay in the
> deployment of clients supporting TLS extensions, and several of the
> clients that have enabled support are using heuristics that allow
> them to disable the functionality when they detect a problem.

In this case, heuristics are needed to get round implementations
that don't respect the robustness principle.

In general I think the present draft doesn't pay enough
attention to extensions, which are after all the second
main reason for maintaining protocols, along with bug fixes.

The draft is seriously unbalanced, it seems to me, by not
containing an explicit description of the benefits of the
robustness principle and of cases where it is necessary.
Frankly I am surprised to see it as an IAB draft in its
current state.

> 5.  Active Protocol Maintenance
> 
>    The robustness principle is best suited to safeguarding against flaws
>    in a specification that is intended to remain unchanged for an
>    extended period of time.

I don't think that's true. Firstly, very few people publish protocol
designs that they intend to be updated after a short period of time.
I'd argue that at the time of publication, all designs are intended
to remain unchanged indefinitely. Secondly, I always understood that
the robustness principle was primarily a safeguard against flaws
in *implementations*. That's why we normally specify MBZ bits as
MUST be zero on transmission and MUST be ignored on reception, etc.

If we *expect* a protocol to be extended by adding options or message
types, the robustness principle leads to very different text, like:

> If an unrecognized MESSAGE_TYPE is received in a unicast message, an
> Invalid message (Section 2.8.12) MAY be returned.  Otherwise the
> message MAY be logged and MUST be discarded.  If an unrecognized
> MESSAGE_TYPE is received in a multicast message, it MAY be logged and
> MUST be silently discarded.

In this case the Postel principle allows nodes that support different
but overlapping sets of message types to communicate robustly. That's
not a problem; it's a solution.

(As TLS shows, this approach may not be acceptable for a security
protocol where downgrade attacks are a problem. But that's a subset
of all protocols.)

I think the draft in its present unbalanced form is misleading.
A balanced discussion of when the Postel principle is useful,
and when it's harmful, would be more what I'd expect from the IAB.

   Brian

---

From: Yaakov Stein <[email-masked]>
Date: Thu, 10 May 2018 14:42:30 +0000

I have a meta-comment on this draft, or alternatively a proposal for a different draft.

I'm sure a lot of you have heard Scott Shenker's talk "The future of network and the past of protocols",
in which he makes the case for SDN.

His basic tenet is that computer science has perfected methods for implementing software very quickly
while protocols take a tremendously long time to design, perfect, and implement.
So, he concludes, throw out protocols and run everything on a centralized server.

Putting aside the obvious fact that he overlooks that protocols between multiple network elements need to be standardized
so that every implementation can interwork with every other one (while an algorithm running on a single CPU doesn't require standardization),
he makes a good point.

A lot of the speed of programming comes from use of "abstractions".
If every time one coded an app for a smartphone one needed to return to opcodes and bits, 
then it would indeed be a lengthy error-prone process.
But instead programmers stand on the shoulders of those who came before them.

It works something like this. Say someone solves some problem.
The solution is initially made into a routine or function that can be open-sourced and reused. No-one need write that routine again.
If the solution is really successful it can be made into a library with various initializations and options that can be loaded when needed. 
You needn't know how the library works, and if someone comes up with a better implementation it can refactored without the programmer knowing about it.
If it becomes even more successful it can be turned into an object that can be inherited and extended and even changed without looking at the source code.
If it is wildly successful, the functionality becomes part of the next generation of programming languages, and everyone has access to it without giving it any thought.

The SDN theorists claim that networking has no abstractions (or at least that they don't know of any),
and suggest a very strong abstraction - the "whitebox" switch (which can be any network element - an Ethernet bridge, an IP router, a firewall, etc).

That jump in logic aside, they have a point. Every time we design a protocol we go back to the basics.
We usually make a bit diagram (in ASCII art no less). We don't formally inherit functionality expect by riding over an existing protocol
(e.g., running a protocol over TCP can give it reliability).

As a concrete example, we often obtain protocol extensibility by utilizing TLVs.
Now, the designers of LDP came up with a nice idea to handle the case of an implementation not knowing a particular type - the U bit.
If the U bit is set it signifies that this unknown type can be gracefully ignored. Otherwise an error message must be returned.
Were we to work like computer scientists, once this idea was implemented it should be automatically inherited by every future protocol using TLVs.
Automatically. Not - if you heard about it put a U bit into your bit diagram.

A while back the IEEE 802 people tried doing something like this when they design the Generic Attribute Registration Protocol.
GVRP and GMRP inherited it, but nothing else.

What can we do in the IETF to turn our ideas into abstractions that are inherited?

Y(J)S

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 May 2018 17:48:38 +1000

Hi Brian,

Thanks for your note.  This is a very valuable contribution.  I suspect
that we might have a fundamental disagreement, but it's definitely worth
exploring the issues you raise a little more.

Firstly, this draft was intended to be a comprehensive rebuttal of Postel's
principle.  The tone is intentional, largely as a means of reinforcing the
message that the robustness principle does more harm than good.  It's a big
idea, and pervasive, and I believe that a strong response is justified.
There are almost certainly ways in which that could be softened, but I'm
not in favour of changing the underlying message in the ways you suggest.

It's possible that our disagreement stems from a disconnect regarding the
scope of the application of the robustness principle.  I think that the
"When in doubt" part clearly limits its applicability to situations where
there is insufficient or unclear specification.  It seems like you might
have a broader notion of the idea.  It might pay to push on that notion a
little.  I've done a little of that below.

I've staged a few changes to the document based on your feedback.  There
are definitely a few places where the emphasis was wrong (and an outright
mistake on my part).  Others separately made points about extensibility,
which suggests that it wasn't paid sufficient attention.  Preview here:

https://martinthomson.github.io/postel-was-wrong/implementation-robustness/draft-iab-protocol-maintenance.html

Of course I don't consider this to be the end of the discussion.  More
inline.

On Thu, May 10, 2018 at 12:05 PM Brian E Carpenter <
[email-masked]> wrote:
> How about

This would be a mistake, I think.  The answer to the question is in my
opinion: "as little as possible, ideally never".

> present draft.

I'm familiar with the many of the ways in which TLS has failed over the
years.  As you say, it has long history of being a bad example.
Separately, my hope is that it can now be the source of a few good examples
in addition to the many bad ones.

I personally attribute the bad examples to its age, plus the extended
period of relative neglect it saw.  In many ways, the protocol hasn't had
any significant maintenance until recently, when we started work on TLS
1.3.  TLS 1.2 is fundamentally the same protocol as SSL 3.

> that don't respect the robustness principle.

That's a conclusion I might have once agreed to, but that was before I
started working on TLS.

The extension mechanism in TLS relied on a hole in the original
specification of SSL 3.  So yes, the robustness principle was applied and
it was in fact critical in enabling extensions.  TLS extensions rely on SSL
3 implementations consistently applying the robustness principle when
dealing with the inner and outer length of the ClientHello message.  Then,
when it was clear that some implementations were stricter in their parsing,
the principle was again used in applying heuristics to work around the
problem.  In this case, the short term impact was interoperation.  Of
course, those were undocumented workarounds which made it hard to build an
interoperable client, but that's a minor drawback in the grand scheme.

The long term effect was devastating.  The most common technique clients
used to work around extension intolerance was to make another connection
without extensions.  Some clients still include the code that does that,
and it has since been repurposed many times to work around other forms of
server intolerance.  One of the consequences of that has been that certain
classes of errors have been tolerated for many years.  Clients could be
relied on to try again with a different configuration, so there is less
pressure on servers to implement features correctly.  This added
significant delays to the deployment of new TLS versions, and was a huge
problem in the design of TLS 1.3.

Once particularly nasty consequence was the application of this retry to
version intolerance.  That is, there was a strong tendency of servers to
fail to connect if a client offered a newer protocol version.  For example,
we saw astonishingly high rates of intolerance to TLS 1.3 before the
version negotiation mechanism there was redesigned.  Some tests saw around
20% failure rates.  That resulted in TLS - as deployed - being vulnerable
to a range of trivial downgrade attacks, some of which are listed in RFC
7457.

> main reason for maintaining protocols, along with bug fixes.

This is a fair criticism.  We might draw different conclusions here, but
here's my take, which expands on the text that I've staged.

Regarding extensibility as the application of the robustness principle is a
mistake.  There are examples, such as with TLS, where extensibility was
added to a protocol by relying on the tolerance of implementations to
unexpected inputs.  However, this is a poor strategy for extensibility.
That particular case went disastrously, as RFC 6709 explains.

Planning for new uses of a protocol means laying down expectations for what
implementations will do when presented with unspecified inputs.

Your MBZ bit example is a nice demonstration of how that works: you specify
semantics for zero (do what this specification says), explicitly decline to
define semantics for one, and then say two things that hopefully make it
possible to start using one in future: "don't send one unless you know what
it means" (an obvious thing really, but somehow still necessary), and the
more important "if you don't know what one means, don't change your
behaviour".

There are also alternatives like "if you don't know what one means, ignore
this much of the message" and "generate an error of the following type",
but examples like the M bit in L2TP or the critical extensions in X.509
show that these options are chosen less often.

Critically, none of this results in any doubt about what to do.  So the
recipient never has cause to contemplate the advice in "When in doubt,
discard faulty input silently, [...]".

There is still room for doubt even with a good design unfortunately, but
that's the subject of a different draft, which you can read here:
https://martinthomson.github.io/use-it-or-lose-it/draft-thomson-use-it-or-lose-it.html

> I don't think that's true.

I agree.  With the perspective you add, it's clear that this is not a good
formulation.  The edit that I shared above is an attempt a fixing this.

> Firstly, very few people publish protocol
> to remain unchanged indefinitely.

I disagree with this.  This is not the stance I've seen - at least in the
groups I've been participating in.  There, the recognition that the
combination of an imperfect specification and the vicissitudes of time mean
that there is never any expectation that the protocol will be viable in
perpetuity.  Yes, the intent is to provide something that is fit for
purpose, which means deployment lifetimes measured in years, but that never
means forever.  For the gap between specification and perfection, there is
the responsibility to tend to the ecosystem.

QUIC might have over-rotated here, I'll admit.  There's talk of shipping
two versions on day 1, or having version 2 within a period as short as 6
months.  But we had the same discussions with HTTP/2 and we while QUIC
means less pressure to start on HTTP/3, it's still been a while now.

> Secondly, I always understood that
> in *implementations*.

Yes, that's the mistake here.  I agree.

> not a problem; it's a solution.

As I said above, if the extensibility is well-defined (invoke 6709!), then
the robustness principle never gets to play.

I couldn't pick out this example easily, so without knowing the
expectations around the handling of Invalid messages, I don't know if this
is indeed a case where robustness would help.  That said, if it would, then
I would say that the specification needs to be improved rather than advice
reliance on robustness.

> protocol where downgrade attacks are a problem. But that's a subset

People often point at TLS and say "oh, that's special", but I don't think
that's accurate at all.  At least not any more.  The bulk of TLS is fairly
mundane protocol engineering, with lessons that apply to many other
protocols.  Hence it being the perennial poster child.  Of course, that's
more true now than it was historically, so the misconception is
understandable.

> and when it's harmful, would be more what I'd expect from the IAB.

My apologies, but that made me think "Fair and balanced," which I'm sure
was not at all your intent :)

In any case, if we disagree here, I think that's OK.  However, my hope is
that discussing the point about the scope of the statement will help.

---

From: Martin Thomson <[email-masked]>
Date: Fri, 11 May 2018 17:59:42 +1000

Thanks for the message Yaakov.

It's an interesting thought, but I definitely think that it's a different
draft.  If you get the time, I'd be interested in reading it.  Especially
if you have ideas about how to go about solving the problem without every
protocol having precisely one implementation.  That's a failure mode we've
lived already and it wasn't pleasant.

The IAB has in the past attempted to create guidance on patterns for
protocols that work.  I agree that's not materially the same as a directly
usable protocol component though.

It's possible that the new waist of the Internet hourglass -
https://people.eecs.berkeley.edu/~alig/papers/http.pdf  - is the reusable
component you are talking about.  We're getting a lot of traction with
reuse there.  Of course, that's also a version of a horrible nightmare for
many.

Cheers,

> Putting aside the obvious fact that he overlooks that protocols between
multiple network elements need to be standardized
> so that every implementation can interwork with every other one (while an
algorithm running on a single CPU doesn't require standardization),

> The solution is initially made into a routine or function that can be
open-sourced and reused. No-one need write that routine again.
> If the solution is really successful it can be made into a library with
various initializations and options that can be loaded when needed.
> You needn't know how the library works, and if someone comes up with a
better implementation it can refactored without the programmer knowing
about it.
> If it becomes even more successful it can be turned into an object that
can be inherited and extended and even changed without looking at the
source code.
> If it is wildly successful, the functionality becomes part of the next
generation of programming languages, and everyone has access to it without
giving it any thought.

> and suggest a very strong abstraction - the "whitebox" switch (which can
be any network element - an Ethernet bridge, an IP router, a firewall, etc).

> We usually make a bit diagram (in ASCII art no less). We don't formally
inherit functionality expect by riding over an existing protocol

> Now, the designers of LDP came up with a nice idea to handle the case of
an implementation not knowing a particular type - the U bit.
> If the U bit is set it signifies that this unknown type can be gracefully
ignored. Otherwise an error message must be returned.
> Were we to work like computer scientists, once this idea was implemented
it should be automatically inherited by every future protocol using TLVs.

> A while back the IEEE 802 people tried doing something like this when
they design the Generic Attribute Registration Protocol.

> _______________________________________________
> Architecture-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/architecture-discuss

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 17 May 2018 15:43:02 +1200

Yaakov,

> What can we do in the IETF to turn our ideas into abstractions that are inherited?

At the level of blocks of protocol design, I don't have an answer,
unless we specify the protocols in a suitable OO language. That
would be an IRTF topic for a good number of years, I suspect.

At the level of components of PDUs, isn't that an argument for
using CBOR instead of TLVs? CBOR isn't OO as such but it can
be used to construct abstractions and re-useable components.

Regards
   Brian

---

From: Yaakov Stein <[email-masked]>
Date: Thu, 17 May 2018 07:36:57 +0000

Brian, 

CBOR is a nice platform for some types of work, if the goals are small code/message size and extensibility without version negotiation.
If you want object orientated extensibility, CORBA achieved that, and had mappings to all useful programming languages.

But, what I was talking about was (at least initially) a mindset.

In TICTOC we now have a draft on timestamp formats.
Ideally, it would give options with some standardized nomenclature, and then all future work would refer to this document,
thus achieving basic inheritance (including a certain degree of options).

I can imagine a set of abstract RFCs specifying how to format timestamps, addresses, QoS levels, hop counts, etc.
Then a set of abstract RFCs for procedures - discovery, handshaking, option negotiation, versioning (if not doing something like CBOR), error recovery, resilience,  etc.

The next step would be standard Open Source implementations of all of these RFCs.

Maybe we should start with defining what elements/procedures need to be defined,
and try to write formalized versions of some of the important protocols.

Y(J)S

-----Original Message-----
From: Brian Carpenter [mailto:[email-masked]] On Behalf Of Brian E Carpenter
Sent: 17/05/2018 06:43
To: Yaakov Stein <[email-masked]>; [email-masked]
Subject: Inheriting protocol abstractions [I-D Action: draft-iab-protocol-maintenance-00.txt]

Yaakov,

> What can we do in the IETF to turn our ideas into abstractions that are inherited?

At the level of blocks of protocol design, I don't have an answer, unless we specify the protocols in a suitable OO language. That would be an IRTF topic for a good number of years, I suspect.

At the level of components of PDUs, isn't that an argument for using CBOR instead of TLVs? CBOR isn't OO as such but it can be used to construct abstractions and re-useable components.

Regards
   Brian

On 11/05/2018 02:42, Yaakov Stein wrote:
> I'm sure a lot of you have heard Scott Shenker's talk "The future of 
> network and the past of protocols", in which he makes the case for SDN.
> 
> His basic tenet is that computer science has perfected methods for 
> implementing software very quickly while protocols take a tremendously long time to design, perfect, and implement.
> Putting aside the obvious fact that he overlooks that protocols 
> between multiple network elements need to be standardized so that 
> every implementation can interwork with every other one (while an algorithm running on a single CPU doesn't require standardization), he makes a good point.
> 
> If every time one coded an app for a smartphone one needed to return 
> to opcodes and bits, then it would indeed be a lengthy error-prone process.
> The SDN theorists claim that networking has no abstractions (or at 
> least that they don't know of any), and suggest a very strong abstraction - the "whitebox" switch (which can be any network element - an Ethernet bridge, an IP router, a firewall, etc).
> 
> We usually make a bit diagram (in ASCII art no less). We don't 
> formally inherit functionality expect by riding over an existing protocol (e.g., running a protocol over TCP can give it reliability).
> 
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> .ietf.org%2Fmailman%2Flistinfo%2Farchitecture-discuss&data=01%7C01%7Cy
> aakov_s%40rad.com%7C969c4364bef04d6c64d408d5bba84a55%7Cf9047108cc2c4e4
> 897a343fad1b3bf9d%7C1&sdata=JI768aSLAPEM97h98bmfmDEwfwsU%2BdyC3R7Z9yJl
> 9Ho%3D&reserved=0
>

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 19 May 2018 08:46:56 +1200

> CBOR is a nice platform for some types of work, if the goals are small code/message size and extensibility without version negotiation.

(Which doesn't imply that CBOR prevents version negotiation. It just isn't
particularly necessary.)

What types of work are not suitable for CBOR? We switched the design
of GRASP from TLV to CBOR, and it was certainly an improvement.

> If you want object orientated extensibility, CORBA achieved that, and had mappings to all useful programming languages.

Hmm. I think some of my worst nightmares 15 to 20 years ago were about
CORBA. I guess we are a bit less scared of complexity now than then.

> But, what I was talking about was (at least initially) a mindset.

Well, re-use in general is a mindset.

> thus achieving basic inheritance (including a certain degree of options).

Yes. This is something you can do with CDDL in the case of CBOR.

> and try to write formalized versions of some of the important protocols.

Sounds like an IRTF project.

Regards
   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 19 May 2018 13:53:14 +1200

Martin,

Comments in line...

> not in favour of changing the underlying message in the ways you suggest.

Clearly you're entitled to that view. But equally clearly it can't
become a view promulgated by the IAB without a wide debate. (RFC1958
was signed by the IAB *and* IESG and was openly discussed on [email-masked],
so it's clear that any significant update deserves the same.)

If we don't hear from other voices here, that isn't OK.

> little.  I've done a little of that below.

I think there are several aspects.

#1. The robustness principle should indirectly guide protocol design.
Firstly there should be no unspecified parts of any PDU. Secondly,
there should be no unspecified responses to any valid or invalid parts
of any PDU. If you do that correctly, the robustness principle isn't
needed in any implementation.

But we know that in real life either specifications are imperfect,
or implementations are imperfect, or both. Therefore:

#2. Test or reference implementations should verify that all parts
of all PDUs are valid. Invalid items should be logged and reported.
(Whether test implementations should also generate invalid PDUs
in order to test the other end is a separate question.)

I hope we agree so far. Then we get to:

#3. Production implementations should ...

where we apparently disagree. The fact is that production
implementations have to make the best of what they are given,
which is typically a mixture of design errors in the protocol
and implementation errors in other implementations. What is
an implementer going to do except tolerate and recover from
errors? Users expect apps to work, whatever happens. So that's
what implementers will do, regardless of RFCs. The only thing
that's changed since the robustness principle was formulated
is that there are thousands of times more users and they are less
tolerant than ever of failure modes.

> https://martinthomson.github.io/postel-was-wrong/implementation-robustness/draft-iab-protocol-maintenance.html

That's 404, unfortunately. Synch problem??

> opinion: "as little as possible, ideally never".

Again: your opinion under your name is fine. But under the IAB's
name I expect a community view. I know there's no IETF consensus
rule for the IAB, but this sort of topic really does need some level
of consensus, as we did for RFC1958 and, iirc, was done for RFC3439
(not even an IAB document), RFC5218 and the like.

> 1.3.  TLS 1.2 is fundamentally the same protocol as SSL 3.

And also it didn't even start out as an IETF consensus proposal.
 
> to a range of trivial downgrade attacks, some of which are listed in RFC

Yes. But all of this seems to be due to weak design in SSL; blame that,
not the subsequent 3rd party efforts to fix that design, as the
root cause.

> That particular case went disastrously, as RFC 6709 explains.

I agree. That's why I believe we should have discussed the robustness
principle in RFC6709, to point out that it is not intended to replace
proper design for extensibility. A lot of protocols do specify the
behaviour for unknown extensions, but it can still be problematic (see
RFC7045 and RFC7872, for example).

> implementations will do when presented with unspecified inputs.

We agree violently on that.

> https://martinthomson.github.io/use-it-or-lose-it/draft-thomson-use-it-or-lose-it.html

Yes, that's all good stuff.
 
> groups I've been participating in.  

Fair enough. It does indeed depend on the experience of the designers
involved. With experience, we all learn that change is inevitable.
s/all designs/some designs/ would be more accurate.

> There, the recognition that the

Yes, that's how I see it too (except when there are faulty implementations).

> I would say that the specification needs to be improved rather than advice

FYI, the protocol's already in the RFC queue:
https://datatracker.ietf.org/doc/draft-ietf-anima-grasp/

We believe its extension mechanism will work, but your point about
exercising it is a good one. 
 
> more true now than it was historically, so the misconception is

Fair enough, but downgrade attacks are mainly a problem for security
protocols, I think. Using an older version is normally benign for
applications, except for missing features.

> was not at all your intent :)

No. But I still like "When to Apply the Robustness Principle", or if
you prefer "When Not to Apply the Robustness Principle".
 
> that discussing the point about the scope of the statement will help.

Yes, and that's why I don't think the document title should be hard
over.

Regards
   Brian

---

From: Christian Huitema <[email-masked]>
Date: Sat, 19 May 2018 08:33:52 -0700

> On May 18, 2018, at 6:53 PM, Brian E Carpenter <[email-masked]> wrote:
...
> tolerant than ever of failure modes.

That's indeed the core of the debate. On one hand, tolerance maximizes interoperability in the short term. On the other hand, intolerance causes production applications to work as the "protocol police". This harnesses network effects to block unsanctioned extensions. Martin's argument is that this leads to a healthier ecosystem.

Brian's proposal place the burden of maintaining this healthy environment on reference implementations and test suites. That's probably the weakest part of his argument. Experience shows that offline tests have no teeth.

My worry with "policing in production" is that it places a lot of power in the dominant implementations, and can lead to big barriers of entry for competing new implementations. We sort of see that today in the browser market.

And then there is ossification. Each of the approaches carries its own risks.

-- Christian Huitema

---

From: Bernard Aboba <[email-masked]>
Date: Sat, 19 May 2018 09:48:56 -0700

Martin --

While the document largely focuses on implementation flaws or differences
in interpretation, I would suggest that the arguments could be made
stronger by consideration of non-conformance more severe than Jon Postel's
maxim was intended to address - the deployment of implementations so far
removed from the standard that they constitute "protocol dialects" or
"uncoordinated protocol development".   In the face of such a large scale
deviation from the standard, "liberal in what you accept" implies
development and maintenance of multiple code paths. This can be highly
burdensome not just for development but also for testing, particularly if
the dialects are undocumented or poorly documented.

One example that comes to mind was the mass self-allocation of
standards-space RADIUS code points by Ascend during the 1990s, establishing
a widely deployed proprietary and non-interoperable variant of RADIUS.
Today, more than 20 years after Ascend shipped their RADIUS dialect, the
IETF has still not completed the task of fully documenting (as
informational RFCs) all aspects of Ascend's RADIUS dialect.  The "dialect"
problem has been particularly prevalent in Web standards over the years, so
that could also provide a rich set of examples.

The IAB has in the past recognized the harm caused by protocol dialects.
While RFC 5407 "Uncoordinated Protocol Development Considered Harmful"
focuses on lack of coordination between SDOs, the arguments are equally
valid if applied to vendor "dialects":

   The uncoordinated adaptation of a protocol, parameter, or code-point
   ... either through the allocation of a code-point without following the

   formal registration procedures or by unilaterally modifying the semantics

   of the protocol or intended use of the code-point itself, poses a risk of

   harm to the Internet [RFC4775 <https://tools.ietf.org/html/rfc4775>].

Nowhere in RFC 5407 is Postel's maxim mentioned - which I interpret as a
recognition that it does not apply to these situations.

---

From: Joe Touch <[email-masked]>
Date: Sat, 19 May 2018 10:48:55 -0700

> On May 19, 2018, at 9:48 AM, Bernard Aboba <[email-masked]> wrote:
> 
> The IAB has in the past recognized the harm caused by protocol dialects.  While RFC 5407 "Uncoordinated Protocol Development Considered Harmful" focuses on lack of coordination between SDOs, the arguments are equally valid if applied to vendor "dialects": 
> 
> 
> Nowhere in RFC 5407 is Postel's maxim mentioned - which I interpret as a recognition that it does not apply to these situations. 
> 

I agree that adapting to such behavior is NOT an application of the Postel principle. IMO, that principle applies to “allowed but unexpected” behaviors, not behaviors that vary completely outside the envelope of what is defined.

Here’s an example, IMO:
	-  (expected) TCP can optimize for packets arriving in order
	- (Postel principle - allowed but unexpected) TCP must always accept packets out of order that arrive in-window
	- (outside envelope)  TCP *must never* attempt to interpret *any* information in packets arriving out of the window

Put more directly, IMO, there are three principles that work *together* (and that protocol designers and implementers confuse or ignore at their peril):

	COMPLIANCE:
	1) a protocol is defined not only by what it is, but by what it is not
		frankly, that’s true for *any* definition	
		the rules of a protocol define what is “allowed” and “not allowed”

	OPTIMIZATION:
	2) protocols are often optimized for what is “expected"
		- both sides must still avoid the “unallowed”, otherwise the protocol is undefined

	ROBUSTNESS:
	3) the Postel principle recognizes that:
		- the sending side of a protocol should prefer to behave as “expected” rather than “unexpected, but allowed”
		- the receiving side of a protocol should allow both “expected” and “unexpected, but allowed”

Joe

---

From: Russ Housley <[email-masked]>
Date: Wed, 23 May 2018 16:22:25 -0400

> On May 19, 2018, at 11:33 AM, Christian Huitema <[email-masked]> wrote:
> 
> ....

In security protocols, accepting things that does not completely conform to the specification does increase interoperability, but it might be done at the expense of actual security.

Russ

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 24 May 2018 08:55:44 +1200

> In security protocols, accepting things that does not completely conform to the specification does increase interoperability, but it might be done at the expense of actual security.

Exactly. That's why I'm reasonably sympathetic to Martin's argument for
security protocols, but less so for "normal" protocols.

While I'm here, I'll respond to Christian's point that reference
implementations and test suites have no teeth. It's certainly true;
we learnt that from OSI. Still, I think they have value - include
a slow, very picky implementation with lots of diagnostics in an
interop hackathon and you will improve the quality of the other
implementations.

     Brian

---

From: Joe Touch <[email-masked]>
Date: Wed, 23 May 2018 17:10:03 -0400

On 2018-05-23 16:22, Russ Housley wrote:.

> In security protocols, accepting things that does not completely conform to the specification does increase interoperability, but it might be done at the expense of actual security.

Being generous in what you receive does not imply being indiscriminately
accomodating or naive. 

It's more (IMO) about not being over-restrictive, i.e., not letting the
"expected case" drive the protocol to its minimum permitted behaviors. 

I.e., no magic or special cases are needed for security protocols. In
all cases, being generous still means playing inside the known lines. 

Joe

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 23 May 2018 17:15:32 -0400

I think this discussion would benefit from citing specific (historical)
examples. My hunch is that all the contributors to this discussion have
specific cases or issues in mind that may actually be non-overlapping and
may illustrate different problems. For example, some may be ambiguities
where the text omits some restriction that the ABNF or more formal
specification includes, and then implementations follow the text.

Henning

On Wed, May 23, 2018 at 5:10 PM, Joe Touch <[email-masked]> wrote:

> In security protocols, accepting things that does not completely conform
> to the specification does increase interoperability, but it might be done
> at the expense of actual security.
>
>
> I.e., no magic or special cases are needed for security protocols. In all
> cases, being generous still means playing inside the known lines.
>
> Joe
>

---

From: [email-masked]
Date: Wed, 23 May 2018 14:25:59 -0700

> interop hackathon and you will improve the quality of the other

While they do add value, the real teeth happen during the installation process, where one implementation has to interoperate with another implementation, in a particular configuration, for the first time.

The reality is that the customer always holds the newly installed equipment responsible. It does not matter one whit what the spec says or that the older equipment is completely insane. If the newly installed equipment can interoperate, then it stays. If not, then the vendor loses.

Experienced vendors come to revere Jon’s wisdom.

While I certainly agree that protocol maintenance is an eventual necessity, toleration of variations in syntax or sequence that do not violate the semantics of the protocol can and should be tolerated.

The discussion however, is effectively moot: the market will not accept implementations that do not interoperate. If you choose to be strict in what you accept, then no one will use your implementation. And Darwin will win again.

Tony

---

From: [email-masked]
Date: Wed, 23 May 2018 14:34:04 -0700

> I think this discussion would benefit from citing specific (historical) examples. My hunch is that all the contributors to this discussion have specific cases or issues in mind that may actually be non-overlapping and may illustrate different problems. For example, some may be ambiguities where the text omits some restriction that the ABNF or more formal specification includes, and then implementations follow the text.

If it helps, sure, but names are withheld to protect the guilty…

Once upon a time Vendor X specified and implemented protocol P.  Somehow, they mis-implemented one particular field, using the most significant bit in the field as a flag, where the spec had used the least significant bit.  Vendor X got to market first. Vendor X was notified of the bug, but to date has chosen not to repair it as their implementation would simply become non-interoperable with previous releases.

Today, all other vendors use the exact same bit as Vendor X. The spec has NOT been maintained to reflect Vendor X’s reality (a bug, IMHO), and Vendors A, B, C, D, etc. all make use of the most significant bit.

The points are simple:
- The market rules. The market wants interoperability.
- What the spec says is pretty much irrelevant. The market doesn’t care about the spec.
- The market is not affected by idealistic ramblings about how we should be more pedantic.
- Postel’s Principle helps interoperability. A lot.
- This is reality. Accept it.

Tony

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 May 2018 12:12:29 +1000

Hi Brian,

Thanks for articulating your views here.  I think that we have different
notions of what this thing is still...

> If we don't hear from other voices here, that isn't OK.

Work in progress :)

> needed in any implementation.

Hmm, perhaps we see this differently.  I see this more as a case of "well,
we know that implementations are going to screw this up if we don't specify
things, what can we do to avoid that?".  That mostly involves specifying
semantics for everything.  Describing what it means to send every valid
value and describing what to do with every other value.

I don't see this as being informed by the robustness principle as much as
it is just writing down a complete specification.

> tolerant than ever of failure modes.

Yes.  We clearly disagree.  Maybe there's a time sequence that we need to
be sensitive to here though.

The first implementation, A, contains a mistake.  It allows value X, which
is maybe not specified clearly enough, to be interpreted as a Y.

The second implementation, B, sends X sometimes.  This is also a mistake,
but it doesn't get caught because A does that.

The third implementation, C, rejects messages with X.  Now we have an
interop issue sometimes when B talks to C.

So you say that the robustness principle leaps to the rescue.  C can
discover the problem and adjust to tolerate X somehow.

But C just kicked the can down the road.  How does C discover that Y is the
right way to interpret X?  What if they choose Z?  Things might appear to
continue to work, but the difference between Y and Z mean that you have new
problems until the next fix.

The application of the robustness principle assumes that C discovers the
problem first (which is likely, see also Langley's law of the internet:
"blame attaches to the last thing that changed" --
https://www.imperialviolet.org/2016/05/16/agility.html ).  It also assumes
that A and B won't or can't change, and that the spec can't be fixed.

That sounds like more work with limited prospects for success.  But I think
that's a little defeatist.  The apathy that its application implies, as
much as the principle itself, is the target I'm aiming at here.  It is less
possible now than it ever was to release software (or protocols) onto the
Internet and walk away, so the odds of finding and engaging with other
implementers is pretty good.

> That's 404, unfortunately. Synch problem??

Name change problem.  We have an intarchboard repo on github, and we
recently moved this there.

https://intarchboard.github.io/protocol-maintenance/implementation-robustness/draft-iab-protocol-maintenance.html

> not the subsequent 3rd party efforts to fix that design, as the

This is a good point to latch on to.  We might like to believe that a good
design is the answer, but I don't think that SSL is fundamentally a bad
design, mostly it's just some choices, some of which might be made
differently by different people.  Yes, failing to allow for extensibility
is a fairly bad mistake, and there were others that were worse.  I don't
think that we can so easily dismiss the example because the protocol was so
bad.  As RFC 5218 established, we don't really get to choose which
protocols become successful, and a good design is no prerequisite.

We might extract different wisdom from the lesson, but I see this an
application of maintenance.  TLS extensions are effectively 100% reliable
today.  And mainly because the community collectively has maintained both
the specification and the implementations.  If you like, think of it as
pigs flying given sufficient thrust; which is perhaps a good analogy for
the lessons of RFC 5218.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 May 2018 12:39:12 +1000

On Sun, May 20, 2018 at 1:34 AM Christian Huitema <[email-masked]>
wrote:
> My worry with "policing in production" is that it places a lot of power
in the dominant implementations, and can lead to big barriers of entry for
competing new implementations. We sort of see that today in the browser
market.

Yes, we've seen the effect that a need to interoperate with big players
causes.  That's something I'm quite sensitive to, and as a browser I know
we try hard to avoid using the power we don't want to have.  But we don't
always succeed.

What I've observed there is that ensuring that specifications match reality
has helped new implementations considerably.  If you need to be bug-for-bug
compatible in a particular way, documenting that as an interoperability
requirement in a specification means that new implementations have less of
a barrier to entry.  The dominant implementation might have all the
existing code, but they also suffer from the other problems of incumbency
(more users, more use cases, more legacy code, etc...).  It's not perfect,
but were specifications not maintained, the costs for entry would be hard
to understand.

In the end, if big implementations match the spec, then it's probably OK.
That means the spec remains relevant.

This was the lesson from deploying HTTP/2.  If existing implementations
took steps to exclude implementations that violated rules of the
specification, then we got fewer non-conforming implementations.  That was
a little rocky early on for some errors, but those are now almost purely
historical.  We generate zero INADEQUATE_SECURITY errors now, where it was
a little difficult early one (I'm looking at a sample of 4 billion that
claim).  That same attitude has served us well with that protocol thus far.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 24 May 2018 12:48:06 +1000

Yes, it's a concern I share, but there are two problems in there.

The first is what 5407 tackles: the use of protocols that are ostensibly
the same, but that have different semantics attached to the same
codepoints.  As you say, that's a disaster.

The second is protocol mitosis.  The emergence of divergent communities of
use for the same protocol that find that they no longer share the ability
to communicate.  I think that's possible even without applying the
robustness principle, just with the accretion of disjoint sets of protocol
extensions that eventually become critical to applications.

The IAB has stated a desire to address small issues in its recent work, so
I'm a little reluctant to increase scope here.  I can appreciate that being
tolerant of inputs can lead you further into a situation where you thought
that you had one protocol when there were really two, but without a
concrete example, I'm not sure how useful saying that would be.

I'd prefer to work separately on the general protocol mitosis problem.

I'm not sure that I'd try to read too much into it.  5407 hits a pretty
easy target, I'm not sure that robustness necessarily entered into
consideration.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 24 May 2018 08:27:38 -0400

Watching this discussion, I think that there are two points that most 
folks agree on, and that may help the community if they are stated

1) When writing specs, try our best to be clear and specific.  leaving 
behavior unspecified tends to lead to interoperability failures down the 
road.  When we are sure that it does not matter, we can say that.

1') Do make sure that restrictions we put in specs actually affect 
interoperability, even if it is future interoperability.  Whenever 
possible, explain the implications so folks do not ignore the restriction.

2) As implementors, do not permit behaviors that violate the spec.  The 
restrictions are there for reason.  The Postel principle is NOT an 
instruction to violate the spec.

Having those two things, there is one additional aspect that I think we 
need to keep in mind.  I don't know whether folks agree with

3) We can not cover every possible case in a spec.  In order to improve 
interoperability, implementors should assume for processing incoming 
messages that all behaviors not prohibited are permitted.  At the same 
time, implementors should hew as close to the explicit text of the spec 
when generating messages.

---

From: Joe Touch <[email-masked]>
Date: Thu, 24 May 2018 09:52:04 -0700

Hi, Joel, 

On 2018-05-24 05:27, Joel M. Halpern wrote:

> ...
> Having those two things, there is one additional aspect that I think we need to keep in mind.  I don't know whether folks agree with
> 
> 3) We can not cover every possible case in a spec.  In order to improve interoperability, implementors should assume for processing incoming messages that all behaviors not prohibited are permitted.  At the same time, implementors should hew as close to the explicit text of the spec when generating messages.

Agreed - but there may be an additional nuance here. 

When you say "permitted", it could mean many things: 
- act on as actively as possible (inferring intended behavior) 
- act on as benignly as possible (e.g., silently ignore or repeat the
last ACK in TCP) 
- some combination of the two above (e.g., act where possible, ignore
where not) 

I think there are some basic dimensions of a reasonable approach that
should be included: 
- do not infer malice where ignorance or benign behavior could suffice 
- be agnostic about content that is undefined (don't over-infer) 
- the same holds on transmission, focusing on implying rather than
inferring  

I call these, in summary: 

     on sending, do not IMPLY 
     on receiving, do not INFER 

Joe

---

From: Robert Sparks <[email-masked]>
Date: Thu, 24 May 2018 13:09:44 -0500

On 5/23/18 9:39 PM, Martin Thomson wrote:
> has helped new implementations considerably.
So, there was a case in SIP where an influential vendor had a bug in 
their parser
that caused them to fail if they received a message formatted correctly 
according
to RFC3261 (this is the lr; vs lr=true; in Via header field values issue 
- you can
find a lot about it digging through the sip-implementors archives). It 
took, IIRC,
more than a year before their heavily deployed application was updated 
to accept
what the specification said. During that year (and for a few years 
after) all
implementations spoke the lr=true; variant. After the bug was fixed, and 
a couple
of years passed, the implementation base universally went back to lr;

If I read what you suggest correctly, you would have preferred that the 
IETF have
updated 3261 shortly after it was clear that this variant was pervasive 
in deployment,
and updated it again after the variant went away? Or do I misunderstand 
the time-scope
of ensuring that specifications match reality above? If I have it right, 
note that there is
a frog-boiling problem on when to decide to reflect the variant in the 
specifications.
For the high-order vs low-order bit variant discussed earlier, when 
would it have been
reasonable for the specs to be changed?

Also, please reread the above paragraph substituting "bug" for "variant" 
and see if it
changes anything.

Then reread it thinking "sips:".

RjS
>   If you need to be bug-for-bug

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 24 May 2018 14:43:52 -0400

The reason I was asking for examples is exactly that: from my experience,
the case of implementations not accepting valid input and then forcing
everybody else, by market dominance or avoiding blame, to implement either
restricted or wrong behavior. Common cases I have seen:
* the spec says case-insensitive, but applications are case-sensitive;
* the spec doesn't say ASCII or even explicitly allows binary or UTF-8
content, but the implementation barfs on anything but ASCII;
* the spec has extension mechanisms that demand that unknown elements
(headers) get passed unmodified, but intermediate entities don't ("for
security reasons");
* the spec doesn't impose a length limit (or a very liberal one), but the
implementation does.

This seems more common in text-based protocols as these often seem to get
implemented by looking at the examples only.

Beyond the sips: case, there's the well-known SIP multi-part payload case.

This occurs even if the "intolerant" application is in the minority, not
just for market-dominant implementations. See
https://medium.com/incerto/the-most-intolerant-wins-the-dictatorship-of-the-small-minority-3f1f83ce4e15
for one take on this.

One good example of where protocols have to "reign in" applications might
be RFC 2822, Section 4 and A.6

 Earlier versions of this standard allowed for different (usually more
   liberal) syntax than is allowed in this version.  Also, there have
   been syntactic elements used in messages on the Internet whose
   interpretation have never been documented.

The following are examples of obsolete (that is, the "MUST NOT
generate") syntactic elements described in section 4 of this document.

Henning

On Thu, May 24, 2018 at 2:09 PM, Robert Sparks <[email-masked]> wrote:

> So, there was a case in SIP where an influential vendor had a bug in their
> parser
> to RFC3261 (this is the lr; vs lr=true; in Via header field values issue -
> you can
> more than a year before their heavily deployed application was updated to
> accept
> what the specification said. During that year (and for a few years after)
> all
> implementations spoke the lr=true; variant. After the bug was fixed, and a
> couple
> updated 3261 shortly after it was clear that this variant was pervasive in
> deployment,
> For the high-order vs low-order bit variant discussed earlier, when would
> it have been

---

From: Joe Touch <[email-masked]>
Date: Thu, 24 May 2018 11:53:02 -0700

Hi, Henning, 

These cases are somewhat different, IMO, as noted below. 

On 2018-05-24 11:43, Henning Schulzrinne wrote:

> The reason I was asking for examples is exactly that: from my experience, the case of implementations not accepting valid input and then forcing everybody else, by market dominance or avoiding blame, to implement either restricted or wrong behavior. Common cases I have seen: 

I don't think it's being "generous" to go outside explicit spec
requirements. The implementation is in violation of the spec. 

> * the spec doesn't say ASCII or even explicitly allows binary or UTF-8 content, but the implementation barfs on anything but ASCII;

That implementation is, IMO, too restrictive. We can't say it's in
violation of the spec, but clearly in violation of the Postal Principle
- and a good example of where it is important, IMO. 

> * the spec has extension mechanisms that demand that unknown elements (headers) get passed unmodified, but intermediate entities don't ("for security reasons");

Those intermediate entities are in violation of the spec. I don't care
what their reasons are - if they want to block a protocol for"security
reasons", fine, but don't half-block and half-allow... 

> * the spec doesn't impose a length limit (or a very liberal one), but the implementation does.

Strictly, the implementation should allow fairly large sizes to be as
generous as possible (Postel Principle), but it's reasonable to have a
limit to how generous it can be (e.g., resource limitations). Being
generous doesn't mean shooting yourself in the foot. 

---- 

> This seems more common in text-based protocols as these often seem to get implemented by looking at the examples only. 
> 
> This occurs even if the "intolerant" application is in the minority, not just for market-dominant implementations. See 
> One good example of where protocols have to "reign in" applications might be RFC 2822, Section 4 and A.6 
> 
> The following are examples of obsolete (that is, the "MUST NOT generate") syntactic elements described in section 4 of this document.
> 
> Henning 
> 
> My worry with "policing in production" is that it places a lot of power in the dominant implementations, and can lead to big barriers of entry for

RjS

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 24 May 2018 15:09:26 -0400

Indeed, we are agreeing and I wasn't clear on my point. What I'm saying is
that most cases (except the egregious "protocol dialects" mentioned
earlier), the problem doesn't seem to be the Postel principle, but overly
restrictive implementations that then force other applications to "dumb
down" the spec that can be implemented or work around the limitations.

This is why is was asking for examples where "liberal acceptance" caused
problems as I'm having a hard time coming up with examples, with the
exception of the 2822 gray areas mentioned (which seem to fit into the "it
made sense at one time" or "we didn't really mean to allow this").

What I'm looking for is where an overly liberal implementation forced
others to conform to their errant ways. I can think of lots of overly
conservative examples forcing implementations to change their behavior, as
per my list. (Temptation to draw political analogies resisted.)

Henning

On Thu, May 24, 2018 at 2:53 PM, Joe Touch <[email-masked]> wrote:

> violation of the spec, but clearly in violation of the Postal Principle -
> and a good example of where it is important, IMO.
>
>
> https://medium.com/incerto/the-most-intolerant-wins-the-
> dictatorship-of-the-small-minority-3f1f83ce4e15
> for one take on this.
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 24 May 2018 12:18:46 -0700

On 2018-05-24 12:09, Henning Schulzrinne wrote:

> Indeed, we are agreeing and I wasn't clear on my point. What I'm saying is that most cases (except the egregious "protocol dialects" mentioned earlier), the problem doesn't seem to be the Postel principle, but overly restrictive implementations that then force other applications to "dumb down" the spec that can be implemented or work around the limitations.

Generally, I would agree - I saw this in the constant stream of TCP
"additions" based on what was deployed that I felt only served to make
TCP more fragile by violating the Postal Principle. I pushed to have
those deviations called "bugs", not "de-facto deployment features". 

> This is why is was asking for examples where "liberal acceptance" caused problems as I'm having a hard time coming up with examples, with the exception of the 2822 gray areas mentioned (which seem to fit into the "it made sense at one time" or "we didn't really mean to allow this").

We could argue that accepting RSTs anywhere in the TCP window was
over-application of the Postal Principle caused the vulnerabilities
described in RFC 4953, and RFC 5961 closed that gap. 

> What I'm looking for is where an overly liberal implementation forced others to conform to their errant ways.

See above, but it's a tough call. I tend to think that one is a case of
robustness in the face of reboot which now might require an exchange to
clean up, though. 

> I can think of lots of overly conservative examples forcing implementations to change their behavior, as per my list. (Temptation to draw political analogies resisted.) 
> 
> Henning
> 
> I don't think it's being "generous" to go outside explicit spec requirements. The implementation is in violation of the spec. 
> 
> That implementation is, IMO, too restrictive. We can't say it's in violation of the spec, but clearly in violation of the Postal Principle - and a good example of where it is important, IMO. 
> 
> Those intermediate entities are in violation of the spec. I don't care what their reasons are - if they want to block a protocol for"security reasons", fine, but don't half-block and half-allow... 
> 
> Strictly, the implementation should allow fairly large sizes to be as generous as possible (Postel Principle), but it's reasonable to have a limit to how generous it can be (e.g., resource limitations). Being generous doesn't mean shooting yourself in the foot. 
> ---- 
>

---

From: Russ Housley <[email-masked]>
Date: Thu, 24 May 2018 16:56:52 -0400

> I think this discussion would benefit from citing specific (historical) examples. My hunch is that all the contributors to this discussion have specific cases or issues in mind that may actually be…

Sometimes the specification authors make the mistake, and when the mistake is fixed implementation and other specifications are still broken.  Here is one that keeps causing trouble for me.

Too many specifications allow two encodings for ASN.1 AlgorithmIdentifier parameters.  The two alternatives arise from the loss of the OPTIONAL associated with the algorithm identifier parameters when the 1988 syntax for AlgorithmIdentifier was translated into the 1997 syntax.  Later, the OPTIONAL was recovered via a defect report, but by then many people thought that algorithm parameters were mandatory.  Because of this history, some implementations encode parameters as a NULL element while others omit them entirely.  The correct encoding is to omit the parameters field; however, when some uses of these algorithms were defined, it was done using the NULL parameters rather than absent parameters.  For example, PKCS#1 [RFC3447] requires that the padding used for RSA signatures MUST use SHA2 AlgorithmIdentifiers with NULL parameters.

Russ

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 24 May 2018 17:12:52 -0400

Interesting example. Sounds like we could use a checklist of common problem
areas, some very specific (such as Russ' example) and some more generic
(e.g., cautions about case-sensitivity and character sets).

Henning

On Thu, May 24, 2018 at 4:56 PM, Russ Housley <[email-masked]> wrote:

> Sometimes the specification authors make the mistake, and when the mistake
> is fixed implementation and other specifications are still broken.  Here is
> one that keeps causing trouble for me.
>
> Too many specifications allow two encodings for ASN.1 AlgorithmIdentifier
> parameters.  The two alternatives arise from the loss of the OPTIONAL
> associated with the algorithm identifier parameters when the 1988 syntax
> for AlgorithmIdentifier was translated into the 1997 syntax.  Later, the
> OPTIONAL was recovered via a defect report, but by then many people thought
> that algorithm parameters were mandatory.  Because of this history, some
> implementations encode parameters as a NULL element while others omit them
> entirely.  The correct encoding is to omit the parameters field; however,
> when some uses of these algorithms were defined, it was done using the NULL
> parameters rather than absent parameters.  For example, PKCS#1 [RFC3447]
> requires that the padding used for RSA signatures MUST use SHA2
> AlgorithmIdentifiers with NULL parameters.
>
> Russ
>
>

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 25 May 2018 10:33:49 +1200

> (e.g., cautions about case-sensitivity and character sets).

Yes. Whatever position the eventual form of the document takes, the
more concrete problem cases we have the better.

Also, when we think about what the IETF+IAB can actually do, rather than
merely say, it seems more relevant to tease out recommendations for
protocol *specifications* to be less prone to problems that will invoke
the robustness principle. As Tony Li eloquently said, implementers
will in any case do what they need to do for interoperability.

    Brian

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 24 May 2018 18:42:33 -0400

One mechanism that I think could help and maybe be mentioned as a concrete
step is the RFC errata. This can be a good way to highlight areas of
ambiguity or "frequently made mistakes".

In the SIP world, the SIPit events have proven to be quite useful, but they
require an organization that can make them happen regularly and some
liaison who can report interoperability issues back to the IETF, acting as
institutional memory.

Publishing test cases and torture tests is also probably a useful activity.

I agree that just berating developers is unlikely to make much difference,
but removing excuses for laziness could.

As an IETF example, the working group could maintain a "resources for
developers" page for key protocols that highlight resources that facilitate
interoperability, such as implementor mailing lists (the SIP implementors
list was mentioned and was/is quite helpful, in my view; developers may not
care about protocol discussions, so overloading the regular IETF list is
not usually a good idea, even if the list persists after the WG shuts
down.).

---

From: Martin Thomson <[email-masked]>
Date: Mon, 28 May 2018 17:11:18 +1000

Top-posting here because I think that this highlights a common error we
make in our building of standards.

Henning gave a concise example of another such example as well: "the spec
says case-insensitive, but applications are case-sensitive" (and several
others, all of which I've seen). [1]

The standard exists to record an agreement, and if it is a poor match for
reality, it isn't worth much.  If we consider the RFC to be a declaration
by fiat that this is what shall be, then if that agreement is demonstrably
different because the code says otherwise, what then?

So, to answer Robert's question:

On Fri, May 25, 2018 at 4:10 AM Robert Sparks <[email-masked]> wrote:
> IETF have updated 3261 shortly after it was clear that this variant was
pervasive
> in deployment, and updated it again after the variant went away?

More or less.  If you are maintaining the protocol specification, you need
to reach an agreement with protocol participants.  If the agreement was
that the clearly bad implementation needed to comply, then that's it.  But
that really only works if the infringing party was willing to change - or
other implementations were willing to exclude that (popular) application.

Given that particular situation, which involved a non-trivial period of
non-compliance, I might have suggested that documenting the variant would
be help, but frankly a year isn't that long.  As long as the bug was
well-known, distributing knowledge might have been enough.  Publishing an
RFC might have taken longer than fixing the bug.  (That's another sad
story, but not for here.)

The article that Henning cited was a great read.  One thing in the thesis
there is that change (both good and bad) is primarily driven by a minority,
and that the primary precondition for success is that the minority is
intolerant.  I still think that there is value in virtuous intolerance - an
earlier version of this draft advocated for it - the challenge being
two-fold: first in deciding whether one is being virtuous, and second in
managing the cost of your own exclusion for being so intolerant.

[1] As an aside, SIP (also, SMTP and HTTP before it) invited this error by
allowing messages to be case-insensitive, then providing nicely consistent
casing in examples.  When you consider the value provided by this "feature"
of these protocols, the cost has over time far outweighed the benefits.
Programmers are well used to getting case right, but the number of problems
from case variations has been high.  I consider the case-insensitivity in
these protocols to be a misfeature.  We were lucky to be able to remove
that feature for HTTP header fields with HTTP/2, but that was the result of
taking advantage of a rare opportunity.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 28 May 2018 17:29:01 +1000

> Generally, I would agree - I saw this in the constant stream of TCP
"additions" based on what was deployed that I felt only served to make TCP
more fragile by violating the Postal Principle. I pushed to have those
deviations called "bugs", not "de-facto deployment features".

That's just another case of uncoordinated standards development.  The
failure there exists in the gap between what is specified/documented and
what is practiced.  If what was practiced was in your view poor, then
that's sad, but it doesn't change the fact that it was the protocol that
was deployed and the specifications weren't serving the community well.  I
understand that the situation with TCP continues to improve, in both
specification fidelity and implementation quality.

The TCP RFC is very clear on this point, so I'd chalk this up to a simple
spec bug.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 28 May 2018 17:46:37 +1000

On Fri, May 25, 2018 at 5:10 AM Henning Schulzrinne <[email-masked]>
wrote:
> Indeed, we are agreeing and I wasn't clear on my point.. What I'm saying
is that most cases (except the egregious "protocol dialects" mentioned
earlier), the problem doesn't seem to be the Postel principle, but overly
restrictive implementations that then force other applications to "dumb
down" the spec that can be implemented or work around the limitations.

There are two suggested actions, and yeah, most of our examples of bad
appear to fall on the "be conservative in what you send" side.

I'm not sure about "dumbing down" being bad.  If our specifications can't
be implemented easily, without errors, can't we claim some of the fault?

We've learned the value of defensive design with crypto: yes it's possible
to use RSA without creating a side channel, but it's shockingly hard.
Absent AES-NI, the same can be said for AES-GCM.  Those sorts of sharp
edges have mostly been filed off for all of the recent primitives.

> What I'm looking for is where an overly liberal implementation forced
others to conform to their errant ways.

Not an example of a forcing function, because Google don't seem to care,
but perhaps relevant:
https://www.theregister.co.uk/2018/04/10/gmail_netflix_phishing_vector/

I confess, I don't have any such examples ready-to-hand other than the
above.  I agree, it would be interesting if tolerance wasn't a problem
here, but the tolerant rarely make a fuss, so their voices are harder to
hear.

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 28 May 2018 13:21:50 +0000

I think this adds up to a slightly different recommendation: "liberal"
specifications that allow lots of variability in syntax or behavior are a
bad idea.

With hindsight, I think our text-based protocols would be better off
without case-insensitivity, with very limited white space, no line folding
and other restrictions. (Escaping is probably another common source of
interoperability problems at the syntax level.)

This partially explains the recent move (back) towards binary-style
protocols.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 29 May 2018 11:15:46 +1200

>> If we don't hear from other voices here, that isn't OK.

Indeed. I paused for a few days to see what other people thought.

>>> It's possible that our disagreement stems from a disconnect regarding

There's an issiue right there: sloppy specification of line wrapping
in email, plus the way it interacts with "> " quoting and/or CRLF
vs LF alone, not to mention some UAs that appear to generate only
html, all leads to shattered text that needs human or artificial
intelligence to process properly.

> it is just writing down a complete specification.

My argument is really that designers need to bear in mind that if they
leave loose ends of any kind, it is very probable that implementers
*will* deal with those loose ends via the robustness principle, for reasons
that Tony Li stated. If you argue that this is a bad thing, I won't disagree,
but my conclusion is not that the robustness principle is a mistake - it's
that specs should not leave loose ends.

> that A and B won't or can't change, 

That's right. And if A and B have 100% of the market when C arrives, then
C has little choice. It can't even carve out a niche for itself, and it has
no commercial choice but to hack whatever needs to be hacked to interoperate
with both of the others. As the conversation has showed, this isn't a
theoretical argument.

> and that the spec can't be fixed.

Even if we do fix the spec, it's very unclear that A or B will change,
because it will cost them money and will only help C.

> much as the principle itself, is the target I'm aiming at here.  

I don't think it is apathy - it's in the commercial self-interest of
A and B to keep C out of the market. Open source might help here, of
course, since it changes the incentives and the motivations of the
players.

> It is less

I think that still varies enormously from case to case. Basically that's
why I believe the draft is unbalanced.
 
> https://intarchboard.github.io/protocol-maintenance/implementation-robustness/draft-iab-protocol-maintenance.html

Thanks. The proposed changes are all improvements. All the same, I still
think the tone (including the title) is wrong. I think we agree that the
robustness principle isn't a get-out-of-jail card; it's more of a survival
tactic. The draft doesn't recognise that in many situations it's unavoidable.
Minimising those situations is a very valid goal.
> protocols become successful, and a good design is no prerequisite.

Also, of course, "good" is always a judgment. But postulating that SSL is
fundamentally a good design, it clearly had defects too, and it still
remains to be seen whether TLS has finally fixed them.
 
> pigs flying given sufficient thrust; which is perhaps a good analogy for

The other thing about SSL/TLS is that it was, and is, terribly important:
a viable approach to e2e security for any arbitrary pair of endpoints,
unlike IPsec. That amplifies the effect of its defects.

Regards
    Brian

---

From: Martin Thomson <[email-masked]>
Date: Tue, 29 May 2018 10:50:47 +1000

Yes, that too :)

Actually I was going to use SDP as an example of this.  It's not
case-sensitive, nor is it supposed to be tolerant of arbitrary whitespace.
You read the RFCs and you could be forgiven from getting the impression
that the usual separator is a single space (not tab nor zero-width
non-breaking space).

However, it turns out that it is a great example of over-tolerance being
imposed on others.  I don't know when this started, but SDP implementations
have to accept HTAB in practice.  We saw this when we were beating the SDP
parser we were bequeathed into shape.  It's one of the originals, and it
seems like this is simply a result of being overly tolerant.

---

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 29 May 2018 01:05:41 +0000

Thanks for the example. I don't know how this started, but I suspect that
this is one way how "culture" beats back attempts to reign in older bad
ideas. Since all (or almost all) other IETF text protocols accept HTAB as
separator, implementors may just assume that they should, too.

The only advice I can deduce: if your spec deviates from long-held cultural
assumptions, shout out the deviation (and motivate it), rather than letting
the ABNF whisper.

---

From: Melinda Shore <[email-masked]>
Date: Mon, 28 May 2018 17:34:54 -0800

> be implemented easily, without errors, can't we claim some of the fault?

I think so.  It's been my experience that for general guidance
for implementers tends to be appreciated primarily by the authors
of the guidance, and implementers of IETF protocols will tend not to
root around in IETF documents not directly related to the protocol
spec.  So I think that if there's anything productive to be done
here it's probably best written for an IETF audience (esp. for those
people who tend to do document reviews).  At this point I think we
have a pretty good handle on what sorts of implementation errors
are made most often, and figuring out how to specify defensively
seems like a worthwhile effort.  I'm not sure that revisiting
the Postel rule will, in and of itself, change much.

That is to say that I can see the value in a document along these
lines, but I think that framing it in terms of revising the
Postel rule will tend not to reach quite the right audience for
fixing known problems, and not incidentally has all sorts of
cultural baggage associated with it that I'm not sure would
be productive to deal with.

Melinda

-- 
Software longa, hardware brevis

PGP fingerprint: 4F68 2D93 2A17 96F8 20F2
                 34C0 DFB8 9172 9A76 DB8F

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 29 May 2018 09:32:32 -0500

I'm watching this discussion with quiet interest. On one (of many
excellent) point(s) ...

> be productive to deal with.

This is actually kind of a hot button for me, because it's really tempting
when a standards-track protocol specification is going through IESG
Evaluation, for ADs to ask, "could this non-normative text about why things
are the way they are, etc. be moved to an appendix? Or to an informational
draft? Or just kept in the working group's wiki?", and that kind of falls
down Melinda's rabbit hole of who will see the guidance they need to see,
as it gets further and further from the mechanisms it's explaining.

So I think we need to be pretty selective about what background and
guidance we provide, but I think we need to provide the most useful
guidance we can provide in places where people who need to see it, will see
it.

But this is more for me to think about.

Spencer