Subject: [rddp] comments on draft-black-rdma-concerns-00.txt

From: Bill Sommerfeld <[email-masked]>
Date: Thu, 18 Jul 2002 14:39:14 +0900

[aka [email-masked], sending from a different address for now..]

    DDP and RDMA specifications MUST contain mechanisms to prevent
    overwrites from impairing system integrity and to isolate the
    effect of overwrites so that interference among otherwise
    unrelated applications is prevented.

One possible interpretation of this text is that it is not concerned
about application integrity (i.e., applications may crash due to
overwrites but systems remain up).  Some discussion at the RDDP
meeting in Yokohama indicates that this was not the intent of the
words "system integrity"; some clarification is therefore called for..

In particular, my main concern here is that there may be a new class
of overwrite attacks for applications which check a received message
for integrity, then trundle on assuming that it will remain intact and
well formed.  If a malicious peer does a carefully-timed overwrite, it
may then trigger some sort of stack/buffer overrun and take over the
application.

To this end, I believe RDDP/RDMA mechanisms will need to track buffer
ownership very carefully, in particular prohibiting remote writes to a
buffer once an application has been notified that it has been
filled..

				- Bill

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 18 Jul 2002 12:48:32 -0500

On 7/18/02, Bill Sommerfeld wrote:

>   DDP and RDMA specifications MUST contain mechanisms to
>   prevent overwrites from impairing system integrity and 
>   to isolate the effect of overwrites so that interference
>   among otherwise unrelated applications is prevented.
>
>One possible interpretation of this text is that it is not
>concerned about application integrity (i.e., applications
>may crash due to overwrites but systems remain up).  Some
>discussion at the RDDP meeting in Yokohama indicates that
>this was not the intent of the words "system integrity";
>some clarification is therefore called for..
>
>In particular, my main concern here is that there may be a
>new class of overwrite attacks for applications which
>check a received message for integrity, then trundle on
>assuming that it will remain intact and well formed.  If a
>malicious peer does a carefully-timed overwrite, it may
>then trigger some sort of stack/buffer overrun and take
>over the application.
>
>To this end, I believe RDDP/RDMA mechanisms will need to
>track buffer ownership very carefully, in particular
>prohibiting remote writes to a buffer once an application
>has been notified that it has been filled..
>

This is very difficult to deal with unless you standardize
the local interface. There are numerous reasons to avoid
doing so. I'll just toss off two of them:

-- Any 'standard' local interface is going to be heavily
   influenced by the Unix/Windows process model: virtual
   memory, multiple processes, etc.
   
   These assumptions may not be appropriate for embedded
   devices. This is recognized in InfiniBand. Verbs are
   defined for HCAs (and largely incorporate the classic
   Unix/Windows assumptions) but not for TCAs.
   
-- There are major benefits to defining a local interface
   that is compatible with InfiniBand.
   
In the spirit of minimalism, I would propose the following:

-- RDDP may only access ULP buffers to the extent that it
   has been enabled to do so by local interactions with
   the ULP.
   
-- Peers MAY NOT make use of STags except as explicitly 
   provided by the ULP.
   
-- A peer MAY NOT advertise a buffer that it has not enabled
   through its local interface.
   
If you follow these rules you do not need to know what
degree of enforcement your peer has. That is a local matter.
If a given node wishes to use cheaper hardware that exposes
entire pages, because more RAM is cheaper than more
real-time checking, that is a valid local optimization.
It does not affect interoperability, and hence should not
be part of a wire protocol.

Applications can judge whether their local interface
provides adequate protection for their memory, and select
their interface provide based on their needs.

In any event, the benefit of "automatically" expiring an
STag is very slight.

Many applications, especially high performance ones, do not
inform the NIC of each bufer advertisement anyway. Instead
they create a single large STag and advertise buffers within
via ULP to ULP messaging. They would never use such a
feature.

On the contrary side, there are many protocols where
creating an STag for each transaction is a natural solution.
DAFS is an example of a ULP where a buffer advertisement
authorizes the peer to use the supplied buffer only until it
responds to the request.

The question is whether there is a benefit in teaching the
RDDP layers (and presumbly the hardware itself) to
automatically shut down such an STag immediately, rather
than waiting for the ULP to do so itself.

It obviously cannot be done until the ULP notification
conditions are met anyway (generally the Send following the
RDDP puts). There is already an "attack window" concurrent
with the intended  delivery. There is no way that anyone
will design hardware to ensure that each byte in an
advertised buffer is filled at most once.

So the benefit of a one-shot STag is that it reduces the
window of vunerability from the time period when RDDP puts
are going on through the anonymous put, to the same period
plus the time it takes the ULP to respond and turn off the
availability itself.

A local interface could easily authorize an STag to expire
at the next completion on the connection. As a wire protocol
benefit we are talking about the very short period from the
last RDDP put and the anonymous put. Given the total length
of the transaction this is a very narrow time window. I
don't see the justification for complicating the handling of
RDDP puts. It certainly can be done (a "final" bit in the
RDDP put header), but I think keeping tagged RDDP puts as
clean of notification/completion logic as possible is a good
idea.

Caitlin Bestler
Asomi Network Technoogies

---

From: Brent Callaghan <[email-masked]>
Date: Thu, 18 Jul 2002 11:37:42 -0700

Bill Sommerfeld wrote:
> <snip>

But the RDDP/RDMA layer doesn't know when the buffer is filled.
The ULP may utilize several RDMA ops to fill a buffer.

Wouldn't it be better for the ULP just to deregister the buffer
before it checks its integrity ?

	Brent

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 18 Jul 2002 18:03:59 -0500

On 7/18/02, Brent Callaghan wrote:

>Bill Sommerfeld wrote:
>> <snip>
>> In particular, my main concern here is that there may be
>> a new class of overwrite attacks for applications which
>> assuming that it will remain intact and well formed.  If
>> a malicious peer does a carefully-timed overwrite, it
>> may then trigger some sort of stack/buffer overrun and
>> take over the application.
>> 
>> prohibiting remote writes to a buffer once an
>> application has been notified that it has been filled..
>
>But the RDDP/RDMA layer doesn't know when the buffer is
>filled. The ULP may utilize several RDMA ops to fill a
>buffer.
>
>Wouldn't it be better for the ULP just to deregister the
>buffer before it checks its integrity ?
>
>   Brent
>

If such a feature were implemented, the Data Source ULP
would flag the RDDP put as 'final', meaning that this
connection had no further use of the supplied STag.

Theoretically this allows an ever so slightly earlier
cleanup of the STag, but it does so by making the RDDP layer
aware of STag maintenance. I believe it is cleaner to leave
those issues to the RDMA layer, or preferably the ULP.

Keep in mind that this 'final' flag cannot be simple and
accomplish the goal of enhanching security. In particular it
would be contrary to proper security for the Data Sink to be
forced to reveal to the Data Source whether the STag
supplied was in use on any other connection. Therefore the
'final' flag can do no more than release *this* connection's
claim on the STag. The hardware implementation would have to
countdown the uses before releasing the STag.

All of which favors the broader point that its better to
keep the RDDP layer as simple as possible and leave
application specific logic to the ULP. However, the feature
can be implemented with the ULP retaining full control over
when the STag is released.

---

From: Michael Krause <[email-masked]>
Date: Sun, 21 Jul 2002 07:56:52 -0700

> >> take over the application.

If one is concerned about malicious attacks, then security should be 
provided such as IPSec which is not a RDMA protocol issue.

> >buffer before it checks its integrity ?

Absolutely not.  Bind / unbind of a window is not necessarily cheap though 
optimizations can be made.  Anything dealing with allocation / 
de-allocation windows or registration / deregistration of regions would be 
prohibitive.

In general, one should only receive a message once from the transport - see 
the requirements document.  The transport prevents duplicates from 
overwriting the memory.  For non-duplicates, there is no method at the RDMA 
/ DDP layers to distinguish good / bad packets other than what is provided 
by the transport and lower layers.  Hence, if there is a malicious attack 
concern, the lower layer implementation should provide a security solution 
such as IPSec.

My general comment is the document does not highlight new information not 
already stated within the current requirements document.  Perhaps there are 
areas that should be tightened within the requirements document if there is 
any ambiguity which might have been the motivation for this new document 
but I'd prefer to keep the number of documents to a minimum and focus on 
getting them correct from the start if possible.

Mike

---

From: Bill Sommerfeld <[email-masked]>
Date: Sat, 27 Jul 2002 06:03:57 -0400

> provided such as IPSec which is not a RDMA protocol issue.

This represents a fundamental misunderstanding.  IPsec provides a
mechanism to allow for confidentiality and integrity.  It is not
"magic pixie dust" which protects applications against buffer
overruns, etc.,

The mere fact that a peer may be authenticated and even authorized to
talk with an application does not mean that the peer is authorized to
hijack the application.

> de-allocation windows or registration / deregistration of regions would be 

The WG charter says:

   The working group will ensure that the resulting technology will be
   secure and will not enable new attacks on systems supporting RDDP.

An overwrite race condition sounds to me like a "new attack".

						- Bill

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 27 Jul 2002 11:19:25 -0500

On 7/27/02, Bill Sommerfeld wrote:

>The WG charter says:
>
> The working group will ensure that the resulting
> technology will be secure and will not enable new attacks
> on systems supporting RDDP.
>

This is a local interface issue, not a protocol issue.

Fundamentally, the Data Source does not need, nor should
it have, the complete semantics of the STag that is part
of the buffer advertisement it received.

The only knowledge it requires is the extent of permission
given to it by the ULP. DAFS is probably typical of the
rules most ULPs will establish:

-- the client sends a request to an anonymous buffer that
   contains a buffer advertisement and a request for some
   work to be done using that buffer.
   
-- the server performs that work.

-- the server sends a reply back to client, also to an 
   anonymous buffer,  thereby terminating its rights to
   the advertised buffer.

The ULP Data Sink / Client MUST ensure that it has used
local interfaces to enable RDDP layer memory access with
sufficient rights to match the advertised buffers.

There is no *protocol* justification for mandating that all
Data Sinks MUST protect the rest of their memory. The
protocol MUST NOT *prevent* them from doing so.

Nothing in any of the iWARP/ROI/RDDP proposals has ever done
anything to prevent more precise local protection of memory.

Even if such local  mechanisms are available, it  would be
inappropriate to make  them mandatory. There are common
practices with RDMA Writes which are incompatible with
deregistering the STag automatically:

-- Enabling a single large region specifically  for the 
    client, and then advertising buffers within that STag.
    This avoids having to create and distribute STags for
    each interaction. This model can be very attractive
    when the peers have long lasting connections  and
    many interactions, such as for an RDBMS.
-- Enabling a large buffer that is filled on an ongoing
    basis, but with each delivery being a notified event.
    This can be useful for delivery a series of variable
    sized data, especially for streaming media.
    
So ultimately, disabling of an STag is both a ULP and local
interface question. The question is whether to mandate a
specific, and potentially expensive behavior to close a
"vunerability window" between the time the message is
received that would have automatically disabled an STag and
when the ULP would have responded to the notification and
disabled the STag itself.

But keep in mind that the same buffers were vunerable to
unauthorized overwrites from the very same sources during
the entire period *prior* to the completing message.

Now there may be some applications where additional
complexity on the NIC is justifiable if it reduces
vunerabilty  in *ANY* way. They would probably be better
served by sticking with non-RDDP messages with
authenticating signatures to every update, but conceivably
they could decide that automatic STag release was a valuable
feature for them. However, mandating that *all* Data Sinks
MUST exercise the *same* data protection policies is simply
not called for.