Subject: Re: [tcpm] Fwd: New Version Notification for draft-gont-tcpm-urgent-data-00

From: Fernando Gont <[email-masked]>
Date: Sun, 02 Nov 2008 22:43:32 -0300

Hello, Joe,

Comments in-line....

> > So we have to issues:
> >
> > * Deprecate recv(2)/MSG_OOB and encourage SO_OOBINLINE. (with which you
> > seem to agree)
>
>That sounds like a Unix man page update, not an RFC.

RFC 793 specifies a minimalistic/generic API for TCP. Most (if not 
all) systems have misunderstood that API. And that's what I think 
should be clarified. recv(2)/MSG_OOB is simply an example of how the 
TCP API was misunderstood.

> > * Update RFC 793/RFC1122 wrt where the UP points to, to reflect what
> > everybody does and has been doing. (with which I don't know whether you
> > agree or not).
>
>Either this is an implementation error or it updates our definition of PUSH.

Push? Why Push?

And, well, you may take it as an "implementation error". But 
everybody has been ding this for... how long?... twenty years? If you 
were to push TCP implementations to implement the RFC1122-mandated 
semantics of the urgent pointer, that would simply break existing 
applciations, as that change is not backwards-compatible.

After twenty years of doing so, and with every stack doing it, it's 
completely unrealistic to expect any real system to move to the 
RFC1122 semantics of the Urgent Pointer (i.e., "the UP points to the 
last byte of urgent data" instead of "the urgent pointer points to 
the byte following the last byte of urgent data).

Keeping things "as they are" simply means that anybody trying to 
analyze real traffic and interpret it according to the existing 
specs, will be puzzled.

(FWIW, the same day I posted the announcement of our I-D, a TCP 
implementor of one of the major free OSes sent me an off-list note 
mentioning that he found our draft just in time, as he was in trouble 
trying to figure out what everybody was doing wrt TCP urgent data)

>If the former, it can be noted in the next version of the TCP
>implementation errors doc. If the latter, it would update 1122 - NOT
>793. As Dave noted, 1122 supercedes 793 on this issue.

Agreed. It would update RFC1122.

>I don't like the idea of continuing to document implementations that do
>not comply with current specs without either stating
>         a) this is BAD and should be fixed
>         b) this is GOOD and we're changing our spec

We are aiming at "b", if you want. It's not really that it is "good". 
It simply is. It's the real world. Ignoring the real world simply 
helps to have the specs become irrelevant.

>FWIW, so far, this looks to me like an implementation error, not a
>reason to change the spec.

At some point, the specs (RFC 793) were ambiguous. So implementations 
simply picked one of the possible intrepretations. The IETF later 
mandated the other possible interpretation. But nobody changed their 
stacks. And nobody will. At some point it may have been an 
implementation error. Nowadays... who cares? it's the de-facto standard.

What do others think about all this?

Thanks!

Kind regards,

--
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Nov 2008 13:37:37 -0800

> Hello, Joe,
...
> implementation error. Nowadays... who cares? it's the de-facto standard.

FWIW, there was an errata in RFC961 in 1985, that indicates that page 17
of RFC793 is incorrect, and that the pointer is to the last octet of
urgent data (not the first octet of non-urgent data).

That information is reiterated in RFC1122 in 1989, and is very specific
in overriding RFC793.  There is no remaining ambiguity. There is only
the difference between what is implemented and what the requirements state.

That does NOT make it a 'de-facto standard'; it can also be a 'de-facto
bug'.

The key question is why RFC1122 chose the "points to last URG byte"
interpretation, and why we think that should be changed. Simply
describing it as "what has been deployed" is insufficient justification
to call this anything other than a bug without further rationale.

If that rationale exists, it would be useful to provide it.

Joe

---

From: Fernando Gont <[email-masked]>
Date: Fri, 07 Nov 2008 22:32:10 -0300

>urgent data (not the first octet of non-urgent data).

Agreed.

>That does NOT make it a 'de-facto standard'; it can also be a 'de-facto

Does this change anything? Do you expect anybody to change their 
implementations at this point in time? They would not even do it even 
if they wanted to (it would break existing apps) C'mon Joe...let's be 
realistic, please.

>interpretation, and why we think that should be changed.

Because nobody has ever implemented that, there's not benefit in the 
RFC1122 behavior vs. the deployed behavior, and nobody is going to 
change their stacks in this respect at this point in time.

>Simply

And... what would that change? Okay, I can answer this one for you: 
More and more people would look somewhere else to find out how the 
protocols work in practice.

P.S.: What was it called? "rough consensus and.... "?

---

From: Joe Touch <[email-masked]>
Date: Fri, 07 Nov 2008 17:47:53 -0800

Fernando Gont wrote:
...
> And... what would that change? Okay, I can answer this one for you: More
> and more people would look somewhere else to find out how the protocols
> work in practice.
> 

It's not "running code." either.

We are not here to write documents that describe only what has been
implemented. We need to decide what the target is.

If you want merely to document "what is", that's a fine description for
a man page, but it doesn't provide utility in a standards body.

Joe

---

From: Fernando Gont <[email-masked]>
Date: Fri, 07 Nov 2008 22:51:53 -0300

>implemented. We need to decide what the target is.

Again: Do you really expect anybody to change their stacks so that 
they become RFC1122-compliant in this respect???

>a man page, but it doesn't provide utility in a standards body.

Well, I guess that depends whether you want the specs to be useful 
for implementers, or not. I just hope that some of the kernel hackers 
that send comments off-list post something in this respect on-list.

FWIW, I guess we have to agree to disagree....

Kind regards,

---

From: Joe Touch <[email-masked]>
Date: Sat, 08 Nov 2008 09:51:22 -0800

Note to the chairs:
This is an important topic for this WG, and I hope we will have time to
discuss it in Minneapolis.

Fernando Gont wrote:
> Again: Do you really expect anybody to change their stacks so that they
> become RFC1122-compliant in this respect???

I expect that, even if the IETF had a compliance process, the specs are
always out of sync with what is deployed, and represent what
implementations should try to achieve.

> Well, I guess that depends whether you want the specs to be useful for
> implementers, or not. I just hope that some of the kernel hackers that
> send comments off-list post something in this respect on-list.

Implementers should design protocols to specs; they should design
applications to specs together with man pages and errata that discuss
where the two diverge.

IMO, the specs cannot constantly be downgraded to represent what is
implemented solely for that reason; downgrading can - and should - occur
if we, as a community, decide to change what we *want* the protocol to do.

Joe

---

From: Fernando Gont <[email-masked]>
Date: Sat, 08 Nov 2008 19:54:31 -0300

>implementations should try to achieve.

I understand what you mean, but I believe that there's a difference 
between simply being out of sync (i.e., the specs simply being ahead 
of time), and the specs specifying stuff that the implementations 
will never even try to implement.

>applications to specs together with man pages and errata that discuss

Man pages? We are talking about what is sent on the wire. We're 
talking about the semantics of the urgent pointer. Why should, e.g. a 
UNIX man page state whether the urgent pointer points to the last 
byte of urgent data vs. the byte following the last byte of urgent data??

>if we, as a community, decide to change what we *want* the protocol to do.

Joe, why do you imply that simply updating RFC1122 such that it 
states that the UP points to "the byte following the last byte of 
urgent data" is a downgrade? Could you please elaborate a little bit 
on why you think it would be a downgrade?

Kind regards,

---

From: Joe Touch <[email-masked]>
Date: Sat, 08 Nov 2008 16:10:57 -0800

Fernando Gont wrote:
> between simply being out of sync (i.e., the specs simply being ahead of
> time), and the specs specifying stuff that the implementations will
> never even try to implement.

> Man pages? We are talking about what is sent on the wire. We're talking
> about the semantics of the urgent pointer. Why should, e.g. a UNIX man
> page state whether the urgent pointer points to the last byte of urgent
> data vs. the byte following the last byte of urgent data??

The UNIX man page should explain where its implementation differs from
the spec. The BSD source code does, FWIW.

> Joe, why do you imply that simply updating RFC1122 such that it states
> that the UP points to "the byte following the last byte of urgent data"
> is a downgrade? Could you please elaborate a little bit on why you think
> it would be a downgrade?

I'm trying to have a general discussion about a series of I-Ds that you
have generated along these lines. In this case, we're talking about
downgrading from 1122 (and some earlier RFCs) to allow the ambiguity
from 793.

Joe

---

From: Fernando Gont <[email-masked]>
Date: Sat, 08 Nov 2008 21:18:35 -0300

At 09:10 p.m. 08/11/2008, Joe Touch wrote:

>have generated along these lines.

I don't understand what you mean by "along these lines". I don't 
think this document is in line with any other document I have published.

FWIW, I had not even planned to publish this "urgent data" draft. But 
while analyzing some stuff (including packet traces) with Andrew, we 
realized that the specs had nothing to do with what has actually been 
implemented.

It's a pain for implementers to find out that in many areas the 
specifications have nothing to do with what current impementations do 
(and, in this case, I'd say "with what implementations have always 
done, and still do").

>In this case, we're talking about

No. We are talking about updating RFC 1122 so that the semantics of 
the UP are officially updated so that the specs (theory) agree with 
implementations (practice). We're not "downgrading" anything.

FWIW, I have already asked you to clarify why you think this is a "downgrade".

Kind regards,

---

From: Stefanos Harhalakis <[email-masked]>
Date: Sun, 9 Nov 2008 15:29:20 +0200

> the spec. The BSD source code does, FWIW.

What Fernando says (as far as i understand) is that we have a reality here 
where everyone does the X thing (where X may be "wrong"). This is a reality 
and we have to accept it.

After accepting that this actually happens, we must either (a) persuade 
everyone to change their implementation or (b) update the RFC.

It seems that (a) isn't possible since it will introduce a backward 
incompatibility and practically invalidate the urgent pointer as a whole.

Since everyone does the X thing we can easily update the RFC without 
practically changing the actual meaning (or interpretation since everyone 
does it already the other way) of UP.

RFCs MUST be at sync with the real world. Having RFCs that don't represent 
reality makes RFCs as a whole (or at least TCP related) less valuable and 
less respectable. This will also save time and pain from all current and 
future TCP implementors.
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Fernando Gont <[email-masked]>
Date: Sun, 09 Nov 2008 10:36:10 -0300

>less respectable. This will also save time and pain from all current and

Exactly!

Kind regards,

---

From: Andrew Yourtchenko <[email-masked]>
Date: Mon, 10 Nov 2008 01:47:28 +0100 (CET)

> implementations should try to achieve.

I think you did not mention one step before that - that the creation of 
the specs in the first place is to ensure the interoperability, not just 
for the sake of the specs themselves.

If the interoperability is already achieved by other means ("the running 
code" deployed everywhere), the specs become irrelevant. If then complying 
to the specs breaks that interoperability, they become outright harmful.

> applications to specs together with man pages and errata that discuss

After reading the set of what I could find - the only conclusion 
regarding TCP Urgent that I could come to if I was designing an 
application - "Don't use it. At all."

On a related note, my understanding from reading the 
http://msdn.microsoft.com/en-us/library/ms740102(VS.85).aspx is that when 
the callers use SO_OOBINLINE in an attempt to be the "RFC-compliant", the 
urgent flag and pointer are pretty much a no-op altogether - maybe someone 
could correct me if I missed something:

"Note  When the SO_OOBINLINE socket option is set, the SIOCATMARK IOCTL 
always returns TRUE, and OOB data is returned to the user as normal 
data."

> if we, as a community, decide to change what we *want* the protocol to do.

This is a good point - so I will take a risk of being burned in-place for 
heresy :) - and will put a few open questions of a broader scope to the 
community:

1) Given the state of the matters and practical experience, are the OOB 
notifications (which TCP Urgent boils down to?) utilised enough in the 
today's (and tomorrow's) world to have them as part of the "core" protocol 
header as opposed to being an option ?

(searching suggests this has already been on the table back in '94 as one 
of the "small changes" within the tcpng bounds - but I was not able to 
find any discussions on the topic - I presume they happened offline - so 
would very much appreciate an enlightenment from someone who happened to 
be part of those discussions).

2) Would we want to more effectively utilise these 16 bits in the TCP 
header (assuming the URG flag rests at "0"), rather than transmitting two 
zero bytes for 99.99% of the cases ?

thanks,
andrew

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Sun, 9 Nov 2008 20:11:19 -0800

<snip>
 
> Fernando Gont wrote:
> ...
> >> Simply
> >> describing it as "what has been deployed" is insufficient 
> >> justification to call this anything other than a bug 
> without further rationale.
> > 
> We are not here to write documents that describe only what 
> has been implemented. We need to decide what the target is.
> 
> If you want merely to document "what is", that's a fine 
> description for a man page, but it doesn't provide utility in 
> a standards body.

Why wouldn't BCP help in this case ? Esp. if you are trying to document
what is widely being done and also has agreement with the standards.

-Anantha

---

From: "Eddy, Wesley M. (GRC-RCN0)[VZ]" <[email-masked]>
Date: Sun, 9 Nov 2008 22:47:19 -0600

Yes, it's on the meeting agenda, currently at the very end.
 
I have a question for the authors.  The survey you did of different OSes
and how they handle urgent data is good; did you do any similar study
of applications and which ones attempt to use urgent data?

________________________________

From: [email-masked] on behalf of Joe Touch
Sent: Sat 11/8/2008 11:51 AM
To: Fernando Gont
Cc: [email-masked]; [email-masked]; David Borman; [email-masked]
Subject: Re: [tcpm] Fwd: New Version Notification fordraft-gont-tcpm-urgent-data-00

Note to the chairs:

---

From: Fernando Gont <[email-masked]>
Date: Mon, 10 Nov 2008 01:57:07 -0300

>of applications and which ones attempt to use urgent data?

Not yet.

For open source ones I guess we could simply grep lots and lots of 
source files. For others it might be a little bit more difficult. I 
recall being told that e.g., MS SQL uses urgent data... and there may 
be many others...

Kind regards,

---

From: Andrew Yourtchenko <[email-masked]>
Date: Mon, 10 Nov 2008 06:39:27 +0100 (CET)

On Mon, 10 Nov 2008, Fernando Gont wrote:

> For open source ones I guess we could simply grep lots and lots of source 
> files. For others it might be a little bit more difficult. I recall being 
> told that e.g., MS SQL uses urgent data... and there may be many others...

I think it was oracle SQL*Net:
http://www.stanford.edu/dept/itss/docs/oracle/10g/network.101/b10776/sqlnet.htm

Default is to "use urgent data". I'll try to find an oracle install to see 
what the "urgent data" constitutes there.

One other possible candidate (to be verified) might include Citrix 
terminal server traffic (the anecdotal evidence taken from snort-users 
archive, e.g.: http://archives.neohapsis.com/archives/snort/2001-08/0787.html)

However, both are "closed domain" applications - i.e., usually both 
ends of them are under single administrative domain, so regardless of 
which behaviour they use, standards compliant or not, it can be solved 
with configuration.

The bigger problem is for internet-wide applications, e.g. for FTP, whose 
ABOR command appears to utilise the TCP Urgent as well. Ironically, the 
standard command-line ftp client on linux is BSD, so all the 
"standards-compliant" servers are effectively "broken" by default.

There could be some more proprietary apps as well, since the searchable 
discussion forums on the internet include quite a lot of discussions on 
this topic from the puzzled application programmers.

thanks,
andrew

---

From: Joe Touch <[email-masked]>
Date: Wed, 12 Nov 2008 08:38:32 -0800

Andrew Yourtchenko wrote:
...
> application - "Don't use it. At all."

I'm wondering what the current uses are; if they misuse urgent, then
fixing the specs won't help. Maybe we should instead deprecate the
function altogether?

> today's (and tomorrow's) world to have them as part of the "core"
> protocol header as opposed to being an option ?
> 
> (searching suggests this has already been on the table back in '94 as
> one of the "small changes" within the tcpng bounds - but I was not able
> to find any discussions on the topic - I presume they happened offline -
> so would very much appreciate an enlightenment from someone who happened
> to be part of those discussions).
> 
> header (assuming the URG flag rests at "0"), rather than transmitting
> two zero bytes for 99.99% of the cases ?

If we can't get the community to implement Urgent as per spec, we
probably can't reassign those bits either.

Joe

---

From: Fernando Gont <[email-masked]>
Date: Wed, 12 Nov 2008 19:06:15 -0300

At 01:38 p.m. 12/11/2008, Joe Touch wrote:

>I'm wondering what the current uses are;

As far as I have been able to check (not that much), "urgent data" is 
used in what you'd call legacy applications (telnet, login, ftp) and 
some databases (I had been told MS SQL used it).

>if they misuse urgent, then
>fixing the specs won't help.

FWIW, one could argue that they do misuses urgent.

Basically, there are two things every app does wrt "urgent data"

1) They interpret the semantics of the UP incorrectly. (they 
interpret the UP as pointing to the byte following the last byte of 
urgent data, rather than pointing to the last byte of urgent data).
2) They read urgent data OOB.

#1 can be easily "accommodated" by updating RFC1122. It's not that I 
"really love the idea", but I'm being pragmatic: it doesn't hurt, 
doesn't break anything, and keeps the specs relevant.

While one could argue that #2 is simply broken, after all the 
end-result is the same: you discard everything up to the urgent mark. 
Also, as there are not that many applications using urgent data, it 
would not be crazy to expect/pretend that they get fixed (by having 
hem set SO_OOBINLINE).

>Maybe we should instead deprecate the function altogether?

Honestly speaking, this is what Andrew and me really and in mind, but 
we thought the proposal would sound controversial to many. :-) So we 
decided to trigger discussion on these issues, and propose (in the 
very first version of the draft) just to "fix" (for some meaning of 
"fix) urgent data, rather than deprecating it.

I'll be putting on-line a rev of the urgent data soon, that better 
reflects the options we have on the table, the feedback we have got 
so far, etc., and probably even some survey of the apps that use 
urgent data. I guess that might help to decide which specific path to pursue.

How does this sound?

>probably can't reassign those bits either.

I guess we could find some backwards-compatible approach to reassign 
the URG bits and the UP.....

Thanks!

Kind regards,

---

From: Joe Touch <[email-masked]>
Date: Wed, 12 Nov 2008 15:15:35 -0800

Fernando Gont wrote:
...
> reflects the options we have on the table, the feedback we have got so
> far, etc., and probably even some survey of the apps that use urgent
> data. I guess that might help to decide which specific path to pursue.
> 
> How does this sound?

Good idea...

Joe