Subject: [tcpm] Thoughts on EXP vs. PS in TCPM

From: "Scharf, Michael" <[email-masked]>
Date: Tue, 19 Nov 2019 12:08:37 +0000

[chair hat off]

In TCPM we repeatedly run into the question whether to publish specs as PS or as EXP. We also have several worked examples in which TCPM first published a EXP RFC and later a PS.

This somehow implies that TCPM often follows a process with three maturity levels (EXP, PS, and Internet Standard), while the IETF actually has two maturity levels (see RFC 6410). Formal definitions are:
* EXP in https://tools.ietf.org/html/rfc2026#section-4.2.1
* PS in https://tools.ietf.org/html/rfc7127#section-3.1 
* Internet Standard in https://tools.ietf.org/html/rfc6410

As questions on EXP and PS emerges repeatedly, I think it would be useful to discuss two questions:

1/ What is the minimum requirement for publishing an EXP RFC in TCPM?

2/ What is the minimum requirement for publishing an PS RFC in TCPM?

I have been thinking myself about these questions quite a bit, most notably since I have recently also worked in working groups outside TSV (such as RTG and OPS). As far as I can tell, other IETF working groups don't make use of EXP as frequently as TCPM. In many working groups PS is the norm.

I start to get worried by that. At the end of the day, the IETF is one standardization organization and TCPM documents matter also to other areas of the IETF, as well as the networking industry as a whole. So, TCPM does not operate in isolation.

Of course, I am partly accountable for that, given that I have run several consensus calls on the target status of RFCs in the last years. I think the community feedback was pretty clear in all these cases, typically in favor of EXP instead of PS. Yet, after more insight in the rest of the IETF, I wonder if maybe TCPM sometimes decides to early to go for EXP if PS is actually possible?

For instance, as individual contributor, I believe that IW10 and TLP could fulfill the PS requirements now. And, in retrospective, I think they were not far away from PS at the time of publication. So, in some sense, TCPM has been very careful and conservative when publishing these RFCs. This now implies that we would have to reissue a PS to actually clean up the RFC services.

Given that other working groups apparently operate differently regarding PS as target, do we need to rethink our minimum bars?

I believe that it would be useful to discuss this question, and I'd be in particular interested in the opinions of implementers.

As a starting point, I offer my own understanding of how to answer the two questions - with chair hat off. So, feel free to disagree... Of course, a case-by-case analysis is always needed, so this is not really a checklist, but I guess we should have some general understanding of what "maturity" really implies.

Michael's thought on question 1/:

In order to start a WGLC in TCPM for EXP status, there SHOULD be one implementation in the main source code tree of an important TCP/IP stack; a default to "off" is perfectly acceptable. Alternatively, there SHOULD be at least one implementation outside the main tree and a certain level of operational experience e.g. in very controlled environments. Comprehensive testing by simulation can also be understood as operational experience if the simulation results are verified independently. (As example, the converter specification meets this requirement in my point of view.)

Note that this requirement is significantly higher than what is written in RFC 2026 - in principle, thousands of research papers on TCP could meet the EXP requirement of RFC 2026. But, at least to me, there is little value in documenting an experiment without proof that it is actually implementable in a TCP stack.

Michael's thought on question 2/:

The requirement follows from RFC 7127, which strongly suggests implementation and operational experience, but it is not strictly required. So, there is some room for interpretation. The bar for PS is _not_ widespread deployment in the Internet. For instance, to me a WGLC for an PS could be started if there are two independent implementations, at least one being deployed in an important TCP stack (i.e., one does not have to be production quality), and some operational experience over a subset of the Internet. For instance, such operational experience can also be tests, it does not require enabling by default. (As example, RACK easily meets these requirements as far as I understand.)

Note that with two independent implementations one gets already relatively close to the requirements of an Internet Standard, but in that case also widespread deployment and some other conditions are required (see RFC 6410). So, the bar for PS must be lower than that.

Please feel free to comment and/or disagree...

Michael

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 20 Nov 2019 00:14:06 +0800

Hi Michael,

while I agree with the two broad acceptance criterial you laid out for
EXP and PS, unless my recollection is wrong, at least for IW10 and TLS
at the time I wouldn't have seen them to pass the PS criteria.

However, as both of these mechansims were quite well understood, with a
good amount of discussion of unindended consequences (IW10), and a
limited impact if they would (unexpectedly) fail to meet their goal
(TLS), I believe the criterial for PS should actually slightly wider...

A change that is in a reasonably well defined space, with no
(forseeable) dramatic consequences to the operation of the tcp protocol
itself, could be allowed for PS even before multiple independent
implementations are available...

Alternatively (which is, what came to pass if I remeber correctly), the
status may need to be desided only very late in the process - if
additional implementations were in the works once the technical details
of a draft settled down.

Stiking a new balance in tcpm for the intended status seems to be in order.

Regards,
   Richard

Am 19.11.2019 um 13:08 schrieb Scharf, Michael:
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: "Scharf, Michael" <[email-masked]>
Date: Tue, 19 Nov 2019 19:35:21 +0000

> at the time I wouldn't have seen them to pass the PS criteria.

Yes, the IETF consensus was that IW10 and TFO (which I wanted to write) didn't pass the PS criteria. I don't want to revisit this. My point is what to do in future in an IETF that publishes many PS documents...
 
> implementations are available...

I agree that for relatively straightforward TCP changes we could possibly publish a PS even with a single implementation. It may be a case-by-case decision, and possibly we would have to precisely define for which well-defined space such a PS is applicable. But, the IETF mandates two independent implementations for Internet Standard, not for PS. So, in principle nothing prevents us from at least trying PS status if there is a way to reason why the requirements in RFC 7127 are fulfilled.
 
> additional implementations were in the works once the technical details

Indeed, we discussed this variant in a number of cases.

In TCPM, our default was often EXP as target, in particular if there were unknowns at WG adoption time. But IMHO we should ask ourselves if we should more frequently _try_ to produce a PS. Of course, if that is not possible or not possible in a timely fashion, downgrade to EXP later in the process is always possible. But at the end of the day IETF is a standardization organization - so we are assumed to write standards, no?

Whether more PS specs are doable in TCPM is not a trivial question. Yet, I believe we should think about that.

Michael

> > In TCPM we repeatedly run into the question whether to publish specs as PS
> or as EXP. We also have several worked examples in which TCPM first
> published a EXP RFC and later a PS.
> >
> > This somehow implies that TCPM often follows a process with three maturity
> levels (EXP, PS, and Internet Standard), while the IETF actually has two maturity
> levels (see RFC 6410). Formal definitions are:
> > I have been thinking myself about these questions quite a bit, most notably
> since I have recently also worked in working groups outside TSV (such as RTG
> and OPS). As far as I can tell, other IETF working groups don't make use of EXP
> as frequently as TCPM. In many working groups PS is the norm.
> >
> > I start to get worried by that. At the end of the day, the IETF is one
> standardization organization and TCPM documents matter also to other areas
> of the IETF, as well as the networking industry as a whole. So, TCPM does not
> operate in isolation.
> >
> > Of course, I am partly accountable for that, given that I have run several
> consensus calls on the target status of RFCs in the last years. I think the
> community feedback was pretty clear in all these cases, typically in favor of
> EXP instead of PS. Yet, after more insight in the rest of the IETF, I wonder if
> maybe TCPM sometimes decides to early to go for EXP if PS is actually
> possible?
> >
> > For instance, as individual contributor, I believe that IW10 and TLP could
> fulfill the PS requirements now. And, in retrospective, I think they were not far
> away from PS at the time of publication. So, in some sense, TCPM has been
> very careful and conservative when publishing these RFCs. This now implies
> that we would have to reissue a PS to actually clean up the RFC services.
> >
> > Given that other working groups apparently operate differently regarding PS
> as target, do we need to rethink our minimum bars?
> >
> > I believe that it would be useful to discuss this question, and I'd be in
> particular interested in the opinions of implementers.
> >
> > As a starting point, I offer my own understanding of how to answer the two
> questions - with chair hat off. So, feel free to disagree... Of course, a case-by-
> case analysis is always needed, so this is not really a checklist, but I guess we
> should have some general understanding of what "maturity" really implies.
> >
> > In order to start a WGLC in TCPM for EXP status, there SHOULD be one
> implementation in the main source code tree of an important TCP/IP stack; a
> default to "off" is perfectly acceptable. Alternatively, there SHOULD be at least
> one implementation outside the main tree and a certain level of operational
> experience e.g. in very controlled environments. Comprehensive testing by
> simulation can also be understood as operational experience if the simulation
> results are verified independently. (As example, the converter specification
> meets this requirement in my point of view.)
> >
> > Note that this requirement is significantly higher than what is written in RFC
> 2026 - in principle, thousands of research papers on TCP could meet the EXP
> requirement of RFC 2026. But, at least to me, there is little value in
> documenting an experiment without proof that it is actually implementable in a
> TCP stack.
> >
> > The requirement follows from RFC 7127, which strongly suggests
> implementation and operational experience, but it is not strictly required. So,
> there is some room for interpretation. The bar for PS is _not_ widespread
> deployment in the Internet. For instance, to me a WGLC for an PS could be
> started if there are two independent implementations, at least one being
> deployed in an important TCP stack (i.e., one does not have to be production
> quality), and some operational experience over a subset of the Internet. For
> instance, such operational experience can also be tests, it does not require
> enabling by default. (As example, RACK easily meets these requirements as far
> as I understand.)
> >
> > Note that with two independent implementations one gets already relatively
> close to the requirements of an Internet Standard, but in that case also
> widespread deployment and some other conditions are required (see RFC
> 6410). So, the bar for PS must be lower than that.
> >

---

From: "Mark Allman" <[email-masked]>
Date: Wed, 20 Nov 2019 11:44:20 -0500

Hi Michael!

Good questions.  Thanks for starting the thread.

I agree that TCPM is basically too conservative.  We have been too
conservative for a long time.  I probably contributed my fair share
to that.  But, I'd put a finer point on it: Have we ever issued
something at EXP (or PS for that matter!) that had to be pulled back
because it was too aggressive?  I am not conjuring anything at the
moment.

TCPM has long used EXP as a crutch.  We don't have the guts to label
something a PS, yet we still want to publish it.  And, no amount of
non-answers to "what's the experiment?" garners rough consensus for
issuing as PS.

And, my take is that the world doesn't much buy there to be a
distinction.  I.e., EXP doesn't dissuade general use of something.
And, I bet at least part of this is that we're just too conservative
with both kinds of RFCs.  If we occasionally did publish an EXP that
caused TCP to fall on its face then maybe people would be a bit more
careful.

That said, it's sort of hard to come up with good rules for
delineating.  For instance, consider this ...

> In order to start a WGLC in TCPM for EXP status, there SHOULD be
> one implementation in the main source code tree of an important
> TCP/IP stack; a default to "off" is perfectly
> acceptable.

Well, there are a number of implementations of IW=15 or IW=100, it's
just that these are off by default.  I.e., something like the
initial window is just a constant somewhere so what does it mean
that we have 'one implementation' of something like this?

I think in the end TCPM standardizes a wide variety of stuff and a
one-size rule may not apply.  E.g.,

  - We standardize actual protocol mechanisms (a la TFO) and those
    need some sort of interoperability kinds of criteria.

  - We standardize algorithms and those need some sort of behavioral
    and safety criteria (a la RACK).

  - We standardize constants and those need safety criteria (a la
    IW or RTO).

Further, my observation is that for the latter two in particular the
WG cares about 'one impl' vs. 'two impls' much less than we care
about the overall evidence that something works---or, where it falls
down.

I wish I had some idea about how to synthesize this better than
'perhaps we should relax and be less conservative', but I don't.
I'll think on it.

allman

--
https://www.icir.org/mallman/
@mallman_icsi

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 20 Nov 2019 09:49:10 -0800

> that we have 'one implementation' of something like this?

I like Mark's remarks on # of implementations. This might be too easy
to game: I and my friend each implemented X in our toy stacks. It
really should be WG's consensus if the proposed change has been
well-tested (or else WG is very confident it works).

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 21 Nov 2019 04:57:15 +0800

Hi Mark,

>> caused TCP to fall on its face then maybe people would be a bit more

Well, in my world there certainly is a distinction between an RFC in EXP
vs. PS vs STD status... I agree, for those of us who deal with drafts
and RFCs in the transport area, what you state here is correct.

However, I've made the observation, that less well initiated people, in
particular decision makers who can not make a call on the technical
merit, will often simply look at procedural things like the status of a
draft or RFC, when funding or not funding resources to implement and
test them.

This is where I would wish the TCPM would actually have a proper follow
up process, where long-time experimental drafts, which are found to be
safe and beneficial would get re-issued from time to time as PS (and
ultimately STD), and failed experiments get cleaned out. For the latter,
we recently had one such major revisit of the state of various RFC (eg.
rfc6247 and rfc8311 for rfc3540). But IIRC, we very rarely promote RFCs
up from experimental - one recent example with good expirience would be
DSACK RFC3708 here - also due to the many dependencies on it (with one
PS document depending on an EXP document :). RFC793bis is the one
currently in progress here.

IMHO, perhaps whenever the TS Area has two new chairs, that could be
made a more routine maintenance task...