Subject: [Tsvwg] Seeking comments on draft-heffner-frag-harmful-00

From: John Heffner <[email-masked]>
Date: Mon, 03 Apr 2006 16:12:33 -0400

We're moving this doc through the AD-sponsored informational track. 
Though it's not a working group item, we'd like to solicit feedback 
(from both tsvwg and int-area).  Please note that the purpose of this 
document is only to describe a known problem, not a solution.

Abstract:
    IPv4 fragmentation is not sufficiently robust for general use in
    today's Internet.  The 16-bit IP identification field is not large
    enough to prevent frequent incorrectly assembled IP fragments, and
    the TCP and UDP checksums are insufficient to prevent the resulting
    corrupted datagrams from being delivered to higher protocol layers.
    This note describes some easily reproduced experiments demonstrating
    the problem, and discusses some of the operational implications of
    these observations.

Currently available at:
http://www.ietf.org/internet-drafts/draft-heffner-frag-harmful-00.txt

Thanks,
   -John

---

From: Rick Jones <[email-masked]>
Date: Mon, 03 Apr 2006 13:36:49 -0700

> http://www.ietf.org/internet-drafts/draft-heffner-frag-harmful-00.txt

Ah, Frankengrams :) Fun things.  I remember we had "issues" wrt back in 
'91 or so even on 10 Mbit/s Ethernet.

> The case of particular concern occurs when the first fragment of a
> datagram is lost in the network.  The remaining fragments will be
> stored in the fragment reassembly buffer.  When the sender wraps the
> ID field, the first fragment of the new datagram will be mis-
> associated with the rest of the old datagram.  Assuming the fragments
> are delivered in order, the rest of the new datagram will be now be
> incomplete (since it is missing the first fragment), so they will be
> saved in the fragment reassembly buffer, forming a cycle that repeats
> every 65536 datagrams.  It is possible to have a number of
> simultaneous cycles, bounded by the size of the fragment reassembly
> buffer.

Does it really matter if it is the first fragment?  I would think that 
cycling is possible so long as (any of) the lost fragments are simply 
not the last fragment when one assumes the fragments are flowing in-order.

Also, to ass-u-me that fragments flow through the network in-order is 
probably not a good thing?  Load-balanced link aggregates may send 
fragments of one datagram across more than one link in the aggregate, 
and I know of at least one NIC, albeit old and likely unused today, that 
would send the first fragment last as part of accumulating an offloaded 
Checksum (HP-PB FDDI NIC).

A while back, over in the "netdev" mailing list, the we were discussing 
ways to minimize the risk of frankengrams springing to life based on 
ass-u-me-ing :) that if one saw N or more datagrams beyond the one(s) 
being reassembled, the likelihood of those datagrams successfully 
reassembling was epsilon. While the specifics of that may be more 
"linux-centric" than an RFC should be, I suspect the discussion might 
provide a starting point for a useful, additional section in the RFC 
discussing ways transport stacks might minimize the likelihood of 
Frankengrams.

I'd probably add to that list what we put into HP-UX 8.07 and what may 
be in several stacks today - drivers accepting "packet trains" whereby 
the driver either queues the entire train, or none of it.  That was 
implemented to solve the problem of transmit queues filling part way 
through the enqueing of the fragments of a given datagram.  Without that 
behaviour, it is possible to send nothing but unreassemblable/potential 
Frankengram datagrams.  At least for the duration of a burst of sending 
faster than the link.  An intermediate device doing fragmentation might 
want to employ that as well.

sincerely,

rick jones

---

From: John Heffner <[email-masked]>
Date: Mon, 03 Apr 2006 18:09:06 -0400

> not the last fragment when one assumes the fragments are flowing in-order.

It could matter.  If it's a middle fragment, you'll receive some 
duplicate earlier ones before the datagram can be completed.  If the OS 
has a consistency check here (that the data is the same), it could toss 
the whole thing when it fails.  I'm not aware of any implementation that 
does this, but if you're just talking about the first fragment, it's 
more clear that it's a problem.

> Checksum (HP-PB FDDI NIC).

However, data is often delivered in order, so it serves as a good 
example. :)  Assuming in-order delivery is partially for the sake of 
clarity.  Any order (reverse, first-packet-last) that's always repeated 
has the same property.  A randomized delivery order is not as bad as the 
worst case.

> discussing ways transport stacks might minimize the likelihood of 

It would be a good place to start for a *future* draft on the subject. 
We decided it is out of scope for this one.

Thanks,
   -John

---

From: Rick Jones <[email-masked]>
Date: Mon, 03 Apr 2006 15:24:32 -0700

> more clear that it's a problem.

Given the overhead such a consistency check would add, I doubt it would 
ever be implemented?  And the conssitency check rather depends on the 
transmission order remaining the same right?

>> and I know of at least one NIC, albeit old and likely unused today, 
>> that would send the first fragment last as part of accumulating an 
>> offloaded Checksum (HP-PB FDDI NIC).
> 
> 

But it is still bad, and I worry (fear, uncertainty and doubt, oh my) 
that someone reading the draft will say "oh, good, my  stack does things 
in a different order so it must not be so bad"

Maybe just some "while we use in-order as an example the problem is 
still present and bad with other packet orderings" text.

>> A while back, over in the "netdev" mailing list, the we were 
>> discussing ways to minimize the risk of frankengrams springing to life 
>> based on ass-u-me-ing :) that if one saw N or more datagrams beyond 
>> the one(s) being reassembled, the likelihood of those datagrams 
>> successfully reassembling was epsilon. While the specifics of that may 
>> be more "linux-centric" than an RFC should be, I suspect the 
>> discussion might provide a starting point for a useful, additional 
>> section in the RFC discussing ways transport stacks might minimize the 
>> likelihood of Frankengrams.
> 
> 

Maybe it is drifting a bit, but isn't it rather like announcing a 
vulnerabilty without announcing how to mitigate it?

rick jones

---

From: John Heffner <[email-masked]>
Date: Mon, 03 Apr 2006 18:44:48 -0400

> still present and bad with other packet orderings" text.

Will try to make this a little more clear.  Thanks for the input.

>>> discussing ways to minimize the risk of frankengrams springing to 
>>> life based on ass-u-me-ing :) that if one saw N or more datagrams 
>>> beyond the one(s) being reassembled, the likelihood of those 
>>> datagrams successfully reassembling was epsilon. While the specifics 
>>> of that may be more "linux-centric" than an RFC should be, I suspect 
>>> the discussion might provide a starting point for a useful, 
>>> additional section in the RFC discussing ways transport stacks might 
>>> minimize the likelihood of Frankengrams.
>>
>>

I'd say it's more like when a bridge is out, first putting up a warning 
sign, then worrying about how to fix it. ;)

Seriously, we considered writing a section like this.  But on careful 
thought, it was not obvious how to mitigate the problem robustly.  The 
Linux solution works pretty well under specific conditions, but writing 
a BCP on how to safely use fragmentation for high bandwidth apps could 
get thorny.  The safest thing to do is just not use fragmentation for 
these apps, or if you absolutely must, at least add protection (like IPsec).

Thanks,
   -John

---

From: Rick Jones <[email-masked]>
Date: Mon, 03 Apr 2006 16:11:57 -0700

> sign, then worrying about how to fix it. ;)

Well, most poeple already know how to mitigate the problems from a 
bridge being out :)

And since you've basically told the script kiddies "Hey, here is 
something else for you to try..."

> these apps, or if you absolutely must, at least add protection (like 

True enough.  But as you point-out in the RFC, there are cases where 
applications are indeed trying to avoid the use of fragmentation, but 
those intermediate devices are hindering it.

Ain't life grand when the fundamental assumptions shift?-)

rick jones

---

From: Erblichs <[email-masked]>
Date: Mon, 03 Apr 2006 14:53:56 -0700

John Heffner,

	Interesting..

	A few Short comments.. Possible fixes in #1, #4, #5, and #6.

	1) Does your 26 Mbps flow rate in section 2, almost imply that
	a connection should be multiplexed so that each sub connection
	has a bps rate of 26Mbps to remove this issue in IPv4 if
	fragmentation is an issue?

	2) How does xmit of say a 9k jumbo segments effect this?
	 Wasn't their work into accepting 9k size segments a few
	 years back?

	3) Does the use of say a 3/4 multiplier for the actual transmit
	size decrease the linklyhood of a fragment? Could we have tacked
	an additional header that caused the fragmentation, that this
	would fix?

	4) If a fragment does occur and out-of-order segments are
	seen should a algor like the fast-xmit used be implimented
	to drop the partial set of recv'd fragments. TCP will force the
	entire set to be rexmit'ed, right?

	5) Some datagrams being delivered to a system are comprised
	   of distinct objects which if modified are detected.
	    
          - compressed files.

	   I assume that the distinct items being created is not in the
	   scope of this memo, but decrease the chance of a TCP checksum
	   being accepted at a higher layer.

	   However, if the above is true, would it be wise on some object
	   types to do a non-lossy compression before xmit knowing that the
	   expand would detect any abnormalities????

	6) It sounds like some type of SACK ack that states
	   fragmented segment dropped, redo your path MTU and decrease
	   segment size; is NEEDED.

	Mitchell Erblich
	------------------------

---

From: John Heffner <[email-masked]>
Date: Mon, 03 Apr 2006 20:49:31 -0400

> 	fragmentation is an issue?

The problem occurs independently of flows.  It only depends on src,dst 
address pair.

> 	 Wasn't their work into accepting 9k size segments a few

Using 9k frames instead of 1500 as in the examples scale the rate at 
which the problem can occur up by a factor of six.

> 	an additional header that caused the fragmentation, that this

There are many ways fragments can be created, not just from an extra 
tunnel header.  For example, it's natural to want to send 
disk-block-sized datagrams.  If the block size is >PMTU, then you end up 
fragmenting.

> 	entire set to be rexmit'ed, right?

What you're describing is like what Linux does now.  It works pretty 
well under most realistic conditions.  Aside from some theoretical 
corner cases, a couple practical issues:

1) The sender has no way of knowing that the receiver is implementing this.

2) NATs.

> 	   expand would detect any abnormalities????

You could...  Cryptographic hashes would be better yet.

> 	   fragmented segment dropped, redo your path MTU and decrease

When MTU discovery is working, and the DF bit is not ignored, it should 
be adequate.

Thanks,
   -John

---

From: Fred Baker <[email-masked]>
Date: Mon, 3 Apr 2006 18:04:50 -0700

> something else for you to try..."

I could be completely misinformed, but I thought fragmentation  
attacks weren't news? Saying there are issues in an RFC isn't  
anywhere near as bad as naming the vendor in the discussion of the  
vulnerability...

http://www-src.lip6.fr/homepages/Fabrice.Legond-Aubry/www.ouah.org/ 
fragma.html

---

From: Rick Jones <[email-masked]>
Date: Mon, 03 Apr 2006 18:12:44 -0700

> 	 Wasn't their work into accepting 9k size segments a few

While the IETF may have said something wrt "jumbo frames" I don't think 
the IEEE has sanctioned them.  I'm not sure if they ever will.

rick jones

---

From: Rick Jones <[email-masked]>
Date: Mon, 03 Apr 2006 18:13:57 -0700

Fred Baker wrote:
> I could be completely misinformed, but I thought fragmentation  attacks 
> weren't news? Saying there are issues in an RFC isn't  anywhere near as 
> bad as naming the vendor in the discussion of the  vulnerability...
> 

I suspect they are no more or less news than what the RFC is describing 
to begin with since the issue goes back at least 15 years if not more :)

rick jones

---

From: Mark Butler <[email-masked]>
Date: Mon, 03 Apr 2006 19:42:14 -0600

The draft states:

In the case of SSH, any stream corruption results in immediate termination of the connection.

While RFC4253 certainly allows this behavior, I don't believe it 
requires it.  Terminating a connection after a single MAC error is a 
little excessive considering the transports used.

As far as possible solutions go, wouldn't it make sense to have 
fragmenting devices (to the degree they are necessary at all) wrap 
fragments in a new header with a 32 bit identification field and a 32 
bit CRC that covers the contents of the entire datagram?

 - Mark B.

---

From: John Heffner <[email-masked]>
Date: Mon, 03 Apr 2006 22:35:02 -0400

> little excessive considering the transports used.

I actually got the citation wrong.  From RFC4251, Section 9.3.5:

    This protocol is designed to be used over a reliable transport.  If
    transmission errors or message manipulation occur, the connection is
    closed.  The connection SHOULD be re-established if this occurs.

OpenSSH terminates with extreme prejudice.  (I expreienced this myself 
when I had a NIC with a bad DMA engine that did TCP checksum 
offloading.)  I'm not aware of any mechanism in the SSH protocol that 
could repair a corrupted connection.

> bit CRC that covers the contents of the entire datagram?

You mean SCTP over IPv6? ;)

Thanks,
   -John

---

From: Mark Butler <[email-masked]>
Date: Tue, 04 Apr 2006 00:20:29 -0600

>    closed.  The connection SHOULD be re-established if this occurs.

Thanks for the reference.  Hypothetically speaking, SSH could be run 
over a transport that allows ULPs to veto messages before they are 
acked, allowing the connection to be preserved. 

> You mean SCTP over IPv6? ;)

Sort of. I was thinking more of the precedent set by IPSEC over IPv4.  
e.g. taking the IPv6 fragment header, assigning a new protocol number, 
adding 32 bits for the CRC, and inserting it after the IPv4 header.  
Support would be negotiated at the transport layer and signalled to 
fragmenting devices via the Reserved bit next to the DF bit in the IPv4 
header.

- Mark B.

---

From: Fred Baker <[email-masked]>
Date: Tue, 4 Apr 2006 07:52:06 -0700

On Apr 3, 2006, at 6:42 PM, Mark Butler wrote:
> The draft states:
> In the case of SSH, any stream corruption results in immediate  
> termination of the connection.While RFC4253 certainly allows this  
> behavior, I don't believe it requires it.  Terminating a connection  
> after a single MAC error is a little excessive considering the  
> transports used.
>
> fragments in a new header with a 32 bit identification field and a  
> 32 bit CRC that covers the contents of the entire datagram?
>
>  - Mark B.

>    closed.  The connection SHOULD be re-established if this occurs.

This suggests, at least to me, that the data stream being looked at  
by SSH is the TCP data stream, not the MAC data stream. The MAC - and  
IP - remain best effort, which is to say that they can drop,  
duplicate, or reorder anything they pleaase. It is the output of the  
*transport*, which is above IP, that is deemed reliable and  
unreliability in which would close a connection with prejudice.

Right?

Tell me we haven't changed the definition of the word "transport" to  
put it somewhere at or below IP?

---

From: John Heffner <[email-masked]>
Date: Tue, 04 Apr 2006 11:06:57 -0400

Fred Baker wrote:
> This suggests, at least to me, that the data stream being looked at by 
> SSH is the TCP data stream, not the MAC data stream. The MAC - and IP - 
> remain best effort, which is to say that they can drop, duplicate, or 
> reorder anything they pleaase. It is the output of the *transport*, 
> which is above IP, that is deemed reliable and unreliability in which 
> would close a connection with prejudice.

There's a bit of a terminology overload here. :)  MAC in this context is 
Message Authentication Codes, which SSH uses to ensure data integrity 
(above the reliable transport).

   -John

---

From: John Heffner <[email-masked]>
Date: Tue, 25 Apr 2006 08:32:50 -0400

I've updated the draft reflecting the discussion on the list.  Are there 
any other comments?

The new draft can be found at:
http://www.ietf.org/internet-drafts/draft-heffner-frag-harmful-01.txt

A "changebar" version can be found at:
http://www.psc.edu/~jheffner/drafts/draft-heffner-frag-harmful-01.txt.bar

Thanks,
   -John