Subject: Re: TCP end-to-end Semantics

From: Anil Agarwal <[email-masked]>
Date: Tue, 9 Jan 2001 10:51:47 -0500 (EST)

All,

Here is another way to look at whether "TCP-spoofing" causes violation
of end-to-end TCP semantics. This is similar to Hussein's analysis.

1. In a purely, theoretical sense, the answer is YES, as stated by many
   authors. The spoofer acks packets to the sender that have not been 
   delivered to the destination host.
   
2. On a more practical plane, things are a little different.

   Most well-written TCP applications do not assume that if the sender
   TCP has accepted user data, then the data will get delivered to the 
   receiver with absolute guarantee. To verify end-to-end data delivery,
   such applications use one of several methods -

   1. Close the socket, with the LINGER option enabled, and wait for TCP 
      to finish its FIN echange and to return success/failure indication
      to the user.

   2. Do an application-to-application message exchange to verify whether
      that all data has been received. This may also be done at intermediate
      points in the session.

   3. Inquire of TCP as to how much of its data has been acknowledged by
      the peer and use that info. to verify whether data has been delivered to
      the receiver.
      It is theoretically possible for applications to inquire of TCP as to
      how much of its data has been acknowledged by the peer. The TCP RFC 
      does NOT mention this as one of the interface primitives and I don't
      think there are any standard socket libraries that allow this.

   4. Simply close the connection, with the assumption that the data
      will be successfully delivered with a fairly high probability.

Let us assume that the TCP spoofer uses a reliable protocol to deliver
packets to the final destination.  Let us also assume that the TCP spoofer
does not spoof the FIN exchange, i.e., the the spoofer ensures
that the FIN exchange finishes successfully only if all sent data
is successfully delivered to and acknowledged by both ends.

 With such a spoofer, one can say that applications that use methods 1 and 2
 mentioned above, will not see any difference in "semantics" or end-results,
 when operating with an intermediate spoofer.

 Applications that use method 3 will of course be negatively affected. One can
 presume that there are not many such applications.

 Applications that use method 4 will not experience any significant change
 in probability of success or failure, assuming that the spoofer itself
 has a low probability of failure. Network or destination host failures
 will affect the probability of failure by the same amount, whether spoofing
 is used or not.

In my humble opinion, TCP spoofing is not as bad as it sometimes sounds!

As an aside, there are many other things that a good TCP-spoofer has to do,
such as faithfully transfer the values of the some of the TCP header fields
such as PUSH, URGENT, checksum and TCP options. We have developed
and sell a TCP-spoofer product for use over satellite links.

Regards,
Anil Agarwal
LMGT
22300 Comsat Dr
Clarksburg, MD 20871
(301)428-4655
[email-masked]

---

From: Craig Partridge <[email-masked]>
Date: Tue, 09 Jan 2001 12:28:12 -0500

In message <[email-masked]>, Anil Agarwal wr

>      to finish its FIN echange and to return success/failure indication

Which may fail when spoofing is done.  If the RTT between the sender and
the spoofing node is sharply less (and less variable) than the RTT between
the sender and the receiver, it is conceivable (esp. if the real RTT
includes a satellite delay) that the sender will multiply retransmit
and finally time out the FIN.

The end result is a transfer that has occurred, but due to spoofing the
sender deems to have failed and may retry (many times!).

>      that all data has been received. This may also be done at intermediate

This works, but is also subject to the timeout problem above.

> ...

I think I just illustrated a case where this is not true.

> will affect the probability of failure by the same amount, whether spoofing

By this logic if my chance of failure is originally .001 and I add a spoofer
with a chance of failure of 0.001, I should not be distressed that my chance
of failure has roughly doubled to (0.002)?

Craig

---

From: Mingyan Liu <[email-masked]>
Date: Tue, 9 Jan 2001 17:51:44 -0500 (EST)

> of failure has roughly doubled to (0.002)?

I have a question: if by using the spoofer (or other type of PEPs that
basically violate the e-t-e semantics) the performance is improved under 
certain conditions (e.g., increased link utilization etc. in the case of
satellite), then should it be left to the application/user to decide
whether they would rather use the spoofer and be exposed to higher failure
probability, or just play safe and bypass the spoofer (assume that the use
of a spoofer is not mandatory)? or is it that the e-t-e semantics should
always supersede performance gain? 

-mingyan

---

From: Fred Baker <[email-masked]>
Date: Wed, 10 Jan 2001 07:18:10 +0800

>    delivered to the destination host.

this is what makes me wonder "what is the objective here"?

If what we're trying to do is make the sender put additional data in 
flight, we don't have to acknowledge the data that is outstanding. We only 
need to increase the window, and do so in a manner that triggers additional 
transmissions. For example,

         Receiver <-> Spoofer <-> Sender

if Receiver says that it will accept a window of N octets starting with 
octet n, Spoofer can increment the window to tell Sender that Destination 
is willing to receive N+M octets starting at octet n. Doing this will go 
through standard slow-start, and so will not actually increase Sender's 
transmission rate until several RTTs have gone by. If Spoofer is trying to 
get Sender to send more quickly NOW, he must increment the window in 
several successive messages: he can send a message every 10 milliseconds 
(say) which says Receiver will accept a window of N+m*MSS octets starting 
at octet n, for m = 1, 2, ..., M/MSS. That way, Sender will be provoked to 
send the next segment every 10 ms, achieving a high rate quickly, and 
Receiver will be none the wiser. We don't actually acknowledge anything 
until Receiver acknowledges it, so there is not a question of things being 
lost in flight. As long as there is no loss, this works just fine.

If window size is an issue (if we are trying to move more than 64K bytes 
per RTT), someone who can add to the offered window can insert a window 
scaling option as well. And selective ack while he's at it.

The problem, of course, is when a loss or massive reordering event occurs. 
Receiver receives messages (and acks them) up until the loss point, and 
then receives as much as its window allows after that, but discards all the 
stuff that Sender has sent over that. Sender will use the fast 
retransmission algorithm to get moving again, but will be in the fast 
retransmit phase (essentially sending one segment per RTT) for a really 
long period. The only way around that is for Spoofer to take responsibility 
for his actions and retransmit on Sender's behalf. Spoofer has to buffer 
Sender's data and retransmit it in a manner that appears normal to Destination.

The shared-fate issue is still there - if Spoofer dies an untimely death, 
Destination and Sender have no chance of figuring out what he did, and will 
do better to tear down the connection and try again.

In all, the best approach from my humble vantage point is to actually 
terminate and reopen the TCP connection, and (at least conceptually) copy 
data between them. We break the end to end semantic in the sense that an 
untimely loss of Spoofer leaves Reciever and Sender unable to recover, but 
everything else still works without having to wrap ourselves around an axle.

What is the objective in this case? Is it to increase transfer rate? If so, 
why is pre-acknowledging data on the table?

Joe mentioned the problem of having > 2 billion bytes outstanding so that 
the window wraps. While this is a theoretical problem, until we have 
terabit links with RTTs on the order of seconds, this is not a practical issue.

---

From: Fred Baker <[email-masked]>
Date: Wed, 10 Jan 2001 09:09:22 +0800

At 05:51 PM 1/9/01 -0500, Mingyan Liu wrote:
>it be left to the application/user to decide
>of a spoofer is not mandatory)?

In general, I would agree with that. Now tell me this: do you know that the 
spoofer is there? How do you evade it?

The cases that come quickly to mind are transparent and non-transparent web 
caches, Packeteer-style QoS control boxes which fiddle with TCP headers, 
Arrowpoint-etc devices which front-end sets of web servers, and so on. I 
can get around the non-transparent caches (SQuID etc) readily enough, but I 
may not be able to get around the others, and may not even know they are there.

In such cases, the statement above is a great sentiment, one I would 
wholeheartedly support, but doesn't seem very practical.

---

From: Craig Partridge <[email-masked]>
Date: Wed, 10 Jan 2001 08:59:34 -0500

How would you suppose the application discovers the spoofing?  The whole idea
of spoofing is that it is invisible.

Craig

In message <[email-masked]>, M

---

From: Manish Karir <[email-masked]>
Date: Wed, 10 Jan 2001 09:50:50 -0500 (EST)

> spoofer is there? How do you evade it?
well I believe the question is more along the lines of: if ISP X uses
spoofer or caches or whatever to give you better performance(thruput or 
security or whatever you are measuring performance as) then you will
continue to use them.  If their service to you is unacceptable then 
you as the user can choose to switch to ISP Y.  The bottom line is 
will the free market support for intelligence in the middle applications
be sufficient for them to survive...
if ISP X notices 10 big customers switching to someone else believe me he 
will quickly drop spoofing.

if spoofing is invisible to end users and the users dont
get any data losses or problems to complain about...then does it matter
whether spoofing is used or not....is'nt the whole point of spoofing 
to get more customers by offering a network with (hopefully) superior
performance than an competitor....

if customers do notice data losses degraded throughput hung connections..
whatever....then they are free to switch to another ISP.

so it is correct to say that the end-users..(okay not applications) do 
in a way get to decided whether spoofing is good or not...whether they do
this actively("..I need to call my ISP and find out what they do...") or
passivly("aa...I dont know why my netscape browser hangs every 4th
click.....I hate it...next month I'm switching")

manish karir

> caches, Packeteer-style QoS control boxes which fiddle with TCP
> headers, Arrowpoint-etc devices which front-end sets of web servers,
> and so on. I can get around the non-transparent caches (SQuID etc)
> readily enough, but I may not be able to get around the others, and
> may not even know they are there.
>

---

From: Craig Partridge <[email-masked]>
Date: Wed, 10 Jan 2001 09:58:56 -0500

In message <[email-masked]>, Manish 

>will quickly drop spoofing.

This assumes that the customer understand the source of error.

One of the issues that has cropped up is that customers have expectations
of their ISP.  The introduction of hidden services like spoofers that
may not conform with customer expectations but are not made visible to
the customer means the customer may not get the service he/she expects but
not know the cause.  (E.g., the customer may switch and get another ISP
that does the same hidden thing, and see the same problems, and blame the
technology not the ISPs).

Craig

---

From: Mingyan Liu <[email-masked]>
Date: Wed, 10 Jan 2001 11:15:49 -0500 (EST)

How about we use an IP option indicating "spoofing (not) allowed",              
similar to "no fragmentation", and require that the spoofer conform to
it, will that make sense?  (This then will not require user's knowledge     
but the application writers'...)

Mingyan

---

From: Luigi Rizzo <[email-masked]>
Date: Wed, 10 Jan 2001 17:38:46 +0100 (CET)

> of failure has roughly doubled to (0.002)?

The problem is that it might increase by several orders of magnitude.

I would say that "failure" here is a false positive, i.e. data not
being delivered to a well-behaved receiver (i.e. one which does
all the read()s it is supposed to do) but the sender does not
realize -- basically the sender even  sees the last ack, but the
receiving endpoint crashes before passing all data to the application.
If the receiving protocol endpoint and the application are on the
same physical hw, then P(failure) can easily be several orders of
magnitude below 1.
 
When you split the receiving endpoint from the application (by
means of a proxy somewhere), depending on what you have in the
middle and how you implement the proxy, the P(failure) of the
connection between the proxy and the receiving application can
be much higher (or more vulnerable to attacks).
 
E.g. in the config below, if the proxy buffers everything and
even generates the last ack, the sender will be able to terminate
and it is easy to think that the connection on path <2> might be
teared down with a significantly higher probability than the
P(failure) above. Much better if the proxy retains the last
ack until it gets one from the remote end (this way you can
accommodate chains of proxies) -- modulo the timeout issues that
someone mentioned.

	[sender]----<1>-----[proxy]-----<2>-----[receiver]

	cheers
	luigi

----------------------------------+-----------------------------------------
 Luigi RIZZO, [email-masked]  . ACIRI/ICSI (on leave from Univ. di Pisa)
 http://www.iet.unipi.it/~luigi/  . 1947 Center St, Berkeley CA 94704
 Phone (510) 666 2927             .
----------------------------------+-----------------------------------------

---

From: Lloyd Wood <[email-masked]>
Date: Fri, 12 Jan 2001 14:28:40 +0000 (GMT)

> but the application writers'...)

You can always spoof such an option, making it worthless.

L.

<[email-masked]>PGP<http://www.ee.surrey.ac.uk/Personal/L.Wood/>

---

From: Alhussein Abouzeid <[email-masked]>
Date: Fri, 12 Jan 2001 20:54:16 -0800 (PST)

I think it is a good idea assuming the spoofer will respect it (which
would be expected from a 'good' spoofer).

-Hussein.

---

From: Fred Baker <[email-masked]>
Date: Sat, 13 Jan 2001 15:25:17 +0800

>would be expected from a 'good' spoofer).

practically speaking, it would be a terrible idea. Options don't work well 
with a largeish number of the routers out there. The presence of an option 
kicks the message out of the fast path.

---

From: Alhussein Abouzeid <[email-masked]>
Date: Sat, 13 Jan 2001 00:16:14 -0800 (PST)

> kicks the message out of the fast path.

I don't see that point coming up with a lot of option proposals, like,
say, ECN?

---

From: Fred Baker <[email-masked]>
Date: Sat, 13 Jan 2001 16:30:44 +0800

>I don't see that point coming up with a lot of option proposals, like,

ECN has no options. It has two bits in the IP header and two bits in the 
TCP header, which are already there and are not used.

And yes, it comes up about every time someone decides they want an option. 
You can define them all you want, but plan on messages containing options 
getting dropped by popular ISPs.

---

From: Alhussein Abouzeid <[email-masked]>
Date: Sat, 13 Jan 2001 01:34:54 -0800 (PST)

> TCP header, which are already there and are not used.

Why not use one of them for the spoofing/no-spoofing signal:) Is ECN more
important? You bet it is.

ECN *is* an option. TCP may or may not use it. It requires support from
the routers. ENS (Explicitly No Spoofing) can also be a similar option?

> And yes, it comes up about every time someone decides they want an option.

True.
 
> getting dropped by popular ISPs.
I don't think this is the issue. A lot of popular ISP's have not
implemented ECN anyway, but there is no doubt ECN will soon enough
become a standard. 

The issue is whether there are considerable gains that call for defining
and using such a bit within the IETF standards.

And to drive this point home, I really don't think that the reluctance of
such "ISP's" to improve their "IS" should ever be a factor in the
evolution of protocols.

-Hussein.

---

From: Fred Baker <[email-masked]>
Date: Sat, 13 Jan 2001 17:56:05 +0800

At 01:34 AM 1/13/01 -0800, Alhussein Abouzeid wrote:
>ECN *is* an option. TCP may or may not use it.

we are somehow using entirely different definitions of the word "option". I 
am using it in the sense the word is used in RFC 791 and 793; you are using 
it, I think, in the sense "something that is optional".

I'm not convinced I know what bits you're going to find in the IP or TCP 
headers that are left for doing "something that is optional", but if you 
can, be my guest.

That said, IP Options are a problem. Really.

---

From: Manish Karir <[email-masked]>
Date: Mon, 15 Jan 2001 13:51:04 -0500 (EST)

> headers that are left for doing "something that is optional", but if you 

actually there can be any number of ways of doing this as long as the 
consensus and approval of the ietf is there...
for example:

-the protocol field could be used to identify the packets to be spoofed.
so we would have TCP, or TCP-SPOOF to distinguish between the two.
(I personally think this is a good idea...if what spoofing does is 
break traditional TCP then why call it TCP..just call it something
else...)

-a particular combination of the flags: say for example DF combined with
SYN combined with FIN but no ACK will mean -dont spoof
(bad idea I know..but once again....feasible..)

-or how about a certain combination of bits in the TOS field...
(after all is'nt the purpose of the ToS field to be able to identify 
flows of different types for processing different....so we can specify
that say ToS value of x shall mean the flow should not be spoofed or
should be spoofed)

anyways.....just some random thoughts...

manish

---

From: Manish Karir <[email-masked]>
Date: Mon, 15 Jan 2001 13:37:38 -0500 (EST)

I think there is probably enough support on this list and real
implementation experience to generate some sort of a spoofing 
standard?

if nothing else an informational document which summarizes the 
most "correct" way of implementing this.  I know there have been 
papers and descriptions of this before, but something which has the
blessings of more than just the authors would be quite useful...that way 
we would atleast have a common base to argue about :)

does anyone else agree on the need for such a document??

manish karir

---

From: Fred Baker <[email-masked]>
Date: Tue, 16 Jan 2001 08:26:00 +0800

>-or how about a certain combination of bits in the TOS field...

that would be the diff-serv code point, no?

---

From: Dr G Fairhurst <[email-masked]>
Date: Tue, 16 Jan 2001 17:00:43 +0000

> consensus and approval of the ietf is there...

IP Options are the most awkward (and probably less suited) of these.

> break traditional TCP then why call it TCP..just call it something

That's not very good for inter-working with other user stacks...

> (bad idea I know..but once again....feasible..)

- bad and especially so using a combination of bits across different 
protocol layers.

> that say ToS value of x shall mean the flow should not be spoofed or

I believe these are already allocated to possible ECN and to DSCPs. DSCP
values are not preserved unmodified end-to-end either, so these are not
really 
candidates for this use.

> anyways.....just some random thoughts...

TCP OPTION?

If you can find out why you want to do it, and what benefit you get, why
not consider a TCP option?  I haven't heard of anyone (yet?) trying
these spoofing ideas for UDP (???).  If your target is TCP, and your spoofer
**must** demultiplex individual flows and interpret TCP header semantics,
why not negotiate "willingness to spoof" on TCP connection setup? Failure
to see this option, could indicate unwillingness (including encryption? old
protocol stack? lazy application?) - I wonder whether this has more
potential?

> manish

It seems to me that it is not just a matter of whether
hosts mark packets with the correct identifier if they do (or do not)
know they need spoofing - but also how TCP stacks find out that their
parent application needs this facility - will the API do this (???),
I would guess applications must explicity do this by understanding 
(???) how they plan to use a connection.  Can this
reasonably be expected to be implemented network wide?

Another issue is whether there could be some consensus on **what**
sort of spoofing a connection tolerate. I'm skeptical, on trying
to standardise if there are a diverse set of techniques that are only used
in some types of network... ) see the PILC WG PEP document. 

Best wisjhes,

Gorry

-- 
------------------------------
http://www.erg.abdn.ac.uk/users/gorry

---

From: John Border <[email-masked]>
Date: Tue, 16 Jan 2001 12:51:24 -0500

> protocol stack? lazy application?) - I wonder whether this has more

This is an interesting idea.  Something like the following...

The initiator adds a "PEP" option to the SYN.  If there is no PEP in the path,
the receiver sends no option in the SYN ACK, letting the initiator know the
connection is end to end.  If there is a PEP in the path, the PEP adds a "PEP"
option when it sends a SYN ACK to initiator and when it forwards or sends a
SYN to the receiver.  (The receiver could even, in theory, tell the PEP to get
out of the way by not including the "PEP" option in its own SYN ACK response.)

I see a problem with getting people to put support for such an option in their
TCP stacks, though.  They would have to be convinced of the value of doing
so...

John

---

From: Dr G Fairhurst <[email-masked]>
Date: Tue, 16 Jan 2001 18:06:58 +0000

> TCP stacks, though.  They would have to be convinced of the value of doing

Yes, that sounds encouraging (to me).

I think I still need to be convinced that we **should** do it, but at
least 
something at the TCP level seems only to interact with TCP transport
entities... and also give reasonable functionality, rather than trying to
"overload" extra semantics to existing mechanisms...

Gorry

---

From: Lloyd Wood <[email-masked]>
Date: Tue, 16 Jan 2001 20:06:40 +0000 (GMT)

[most of this thread has little to do with satellites, so I've cc'd
tsvwg, and suggest discussion moves there.]

On Tue, 16 Jan 2001, Dr G Fairhurst wrote:

> not consider a TCP option?

Because there's no space left.

The problem with TCP options is that there's limited space for them,
and what with SACK and timestamps, the 40 available bytes for options
soon fill up.

Has anyone given any thought to negotiating accepted size of the TCP
options field when a connection is initiated, or, slightly more crufty
but perhaps better from a legacy viewpoint, negotiating use/acceptance
of an option field indicating that further option information can be
carried elsewhere in the TCP segment, either following the options
field directly (ease of processing) or after any data in the segment?

(The latter is possibly more backwards-compatible for old code that
might assume a limit on option size, but since checksums have to be
adjusted anyway that's probably a moot point. Possibly better for
spoofing/NAT implementations en-route, though.)

This would allow more SACK information to be provided - useful in
itself. So you might have either:

a) optional option at TCP connection setup indicating maximum number
of TCP options sender is willing to process. Default (no given options
limit advertised) is four as present; both ends can set their max
tolerance. Weeny wireless receivers that know they're on
header-compressed links might demand a limit of one or two, etc.

Defining an explicit option size acceptance limit in initial
negotiation and using the null terminator - that's what it's for! -
strikes me as slightly more in the spirit of RFC1263 than the more
baroque alternative below; all a spoofer has to do is not pass the
option on during the initial exchange, so it's backwards-compatible.

b) negotiated use of willingness to accept options 'extension header'
at startup, then in a packet with a lot of options, you'd have:

option field:     timestamp
                  window size
                  optional option indicating use of optional options
optional options: sack
                  sack
                  sack
                  [..]
                  sack. Loads and loads of sack.
                  everything else we can think of.
data.

Yes, MSS payload computation has to be adjusted, but this strikes me
as an easy win for cases where most of your data transfer is one-way
and you're just sending acks where you can piggyback a lot of useful
option information. Nothing wrong with a hundreds-byte ack packet if
it contains useful information that the other end can process.

And then we can worry less about sender heuristics guessing SACK state
from the miserly three records available, too.

Suggestion b) strikes me as vaguely analogous to use of GRE; it's a
generic get-out-of-closed-space be-it-protocol-bits-or-option-bytes
extension-header fix.

L.

hey, if T/TCP with SACK ever takes off, we'll need all the option
space we can get.

---

From: Mingyan Liu <[email-masked]>
Date: Tue, 16 Jan 2001 20:59:34 -0500 (EST)

> out of the way by not including the "PEP" option in its own SYN ACK response.)

I can see why it makes sense to do so, but this would mean the spoofer
will need to check the TCP header of all passing (TCP) packets, even 
those belonging to a connection that did not choose to use the PEP? 
I guess this probably still beats using an IP "option", which gets the 
packet kicked out of the fast lane...? 

-mingyan

---

From: Dr G Fairhurst <[email-masked]>
Date: Thu, 18 Jan 2001 13:01:15 +0000

> packet kicked out of the fast lane...?

Sure it does.

The TCP option allows fast path forwarding by all routers along path from
the source to the destination, except the PEP itself. The IP option,
means the spoofer has to look at all IP packets (slow path), THEN
look at any TCP packet which it MAY spoof. Furthermore each router
along the TCP PATH also looks at the IP OPTIONS.

The PEP can choose to do as much work as it likes. It can look at all
TCP packets - or it can implement a TCP flow cache, based on a list
of established connections. and there known use of PEP or not.