Subject: Re: [rddp] comments on draft-black-rdma-concerns-00.txt

From: "H.K. Jerry Chu" <[email-masked]>
Date: Mon, 29 Jul 2002 19:07:18 -0700 (PDT)

>On 7/27/02, Bill Sommerfeld wrote:
>
>
>>The WG charter says:
>>
>> The working group will ensure that the resulting
>> technology will be secure and will not enable new attacks
>> on systems supporting RDDP.
>>
>>An overwrite race condition sounds to me like a "new
>>attack".
>
>This is a local interface issue, not a protocol issue.
>
>Fundamentally, the Data Source does not need, nor should
>it have, the complete semantics of the STag that is part
>of the buffer advertisement it received.

Yes and no. On one extreme the minimal semantic of the STag
the data source needs to know is simply its validity wrt the
given RDMA transaction where the tag is exposed. THe rest is
all left to the implementation. E.g. in an environment where
malicious attacks are possible, implementations must
invalidate/deregister STag at the end of every RDMA transaction.
In other environment and/or application paradigm where attacks
are not an issue, implementations may choose to defer tag
invalidation or reuse/remap tags in order to attain higher
performance. These different levels of protection can be
negotiated upfront.

But the above more liberal approach has the drawback of
putting too much risk and burden on the users of RDDP to "do
the right thing". A better approach might be to define
explicitly the scope of each STag and convey that as well
as the STag to the other side. The scope can be a single
RDMA transaction, or higher (such as a SCTP/TCP session).
Implementations must invalide a STag when the transport
state machine runs beyond the scope of the STag. This
approach may further complicate the RDMA protocol, but seems
to provide much more flexibility to ULPs.

Jerry

>
>The only knowledge it requires is the extent of permission
>given to it by the ULP. DAFS is probably typical of the
>rules most ULPs will establish:
>
>-- the client sends a request to an anonymous buffer that
>   contains a buffer advertisement and a request for some
>   work to be done using that buffer.
>   
>-- the server performs that work.
>
>-- the server sends a reply back to client, also to an 
>   anonymous buffer,  thereby terminating its rights to
>   the advertised buffer.
>
>The ULP Data Sink / Client MUST ensure that it has used
>local interfaces to enable RDDP layer memory access with
>sufficient rights to match the advertised buffers.
>
>There is no *protocol* justification for mandating that all
>Data Sinks MUST protect the rest of their memory. The
>protocol MUST NOT *prevent* them from doing so.
>
>Nothing in any of the iWARP/ROI/RDDP proposals has ever done
>anything to prevent more precise local protection of memory.
>
>
>
>
>
>Even if such local  mechanisms are available, it  would be
>inappropriate to make  them mandatory. There are common
>practices with RDMA Writes which are incompatible with
>deregistering the STag automatically:
>
>-- Enabling a single large region specifically  for the 
>    client, and then advertising buffers within that STag.
>    This avoids having to create and distribute STags for
>    each interaction. This model can be very attractive
>    when the peers have long lasting connections  and
>    many interactions, such as for an RDBMS.
>-- Enabling a large buffer that is filled on an ongoing
>    basis, but with each delivery being a notified event.
>    This can be useful for delivery a series of variable
>    sized data, especially for streaming media.
>    
>So ultimately, disabling of an STag is both a ULP and local
>interface question. The question is whether to mandate a
>specific, and potentially expensive behavior to close a
>"vunerability window" between the time the message is
>received that would have automatically disabled an STag and
>when the ULP would have responded to the notification and
>disabled the STag itself.
>
>But keep in mind that the same buffers were vunerable to
>unauthorized overwrites from the very same sources during
>the entire period *prior* to the completing message.
>
>Now there may be some applications where additional
>complexity on the NIC is justifiable if it reduces
>vunerabilty  in *ANY* way. They would probably be better
>served by sticking with non-RDDP messages with
>authenticating signatures to every update, but conceivably
>they could decide that automatic STag release was a valuable
>feature for them. However, mandating that *all* Data Sinks
>MUST exercise the *same* data protection policies is simply
>not called for.
> 
>
>_______________________________________________
>rddp mailing list
>[email-masked]
>https://www1.ietf.org/mailman/listinfo/rddp

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 29 Jul 2002 21:38:22 -0500

On 7/29/02, H.K. Jerry Chu wrote:

>Yes and no. On one extreme the minimal semantic of the
>STag the data source needs to know is simply its validity
>wrt the given RDMA transaction where the tag is exposed.
>THe rest is all left to the implementation. E.g. in an
>environment where malicious attacks are possible,
>implementations must invalidate/deregister STag at the end
>of every RDMA transaction. In other environment and/or
>application paradigm where attacks are not an issue,
>implementations may choose to defer tag invalidation or
>reuse/remap tags in order to attain higher performance.
>These different levels of protection can be negotiated
>upfront.
>
>putting too much risk and burden on the users of RDDP to
>"do the right thing". A better approach might be to define
>approach may further complicate the RDMA protocol, but
>seems to provide much more flexibility to ULPs.
>

The scope of each STag is explicitly defined, and it is
explicitly conveyed to the other side. But that can be done
by the ULP without complicating the RDDP layer.

Further, by keeping it a pure ULP issue, we preserve the
right of the Data Sink to tell the Data Source that the STag
is  only valid for a transaction, but then to be lazy about
deregistering it.

How can it enhance security to require the Data Sink to
publish the exact scope of its exposure? Suppose the Data
Sink's NIC has a limited number of available STags, and
therefore it must use a single STag for multiple buffer
advertisements? Of what benefit is it to inform the Data
Source?

IMHO, this is akin to posting "Speed Limit 55", and then
adding a sign stating "No officers available after 6:00 PM,
please be extra vigilant in monitoring your own speed."

Whether the same STag is used for other buffers, or even on
different connections, is not relevant to the Data Source.
The Data Source must be informed of what *it* is being given
permission to do with the STag -- but that is a ULP issue.
DAFS, for example, delivers buffer advertisements in
requests that are only valid until the reply is sent.

All of this can be handled strictly with local interfaces.
We can suggest local interfaces, but definitely should not
mandate them. InfiniBand, for example, only defines verbs
for those Channel Adapters that choose to classify
themselves as Host Channel Adapters. There is no attempt to
specificy the local interface for specialized executatiaon
environments such as embedded devices (they are Target
Channel Adapters, or TCAs). Any processing environment that
finds the resource models for HCAs can simply declare itself
to have a TCA instead. Any RDDP local interface needs a
similar escape clause.

---

From: Bill Sommerfeld <[email-masked]>
Date: Tue, 30 Jul 2002 03:17:04 -0400

> performance. These different levels of protection can be

So, in thinking further about this, maliciousness is not actually
required for overwrites to be problematic.

Next hypothetical:

Remember that IP networks may duplicate packets.  

1) application receives packet using DDP
2) application gets notified that data is available.
3) application believes it is in a safe and trusting environment, so
RDDP mapping remains valid (neither explicit invalidation nor implicit
invalidate-on-rx takes place)
4) application modifies received data in place
5) duplicate packet arrives, overwriting buffer.
6) application does some processing which depends on data not
being overwritten after (4)

						- Bill

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 30 Jul 2002 10:08:00 -0500

On 7/30/02, Bill Sommerfeld wrote:

>> In other environment and/or application paradigm where
>> attacks are not an issue, implementations may choose to
>> defer tag invalidation or reuse/remap tags in order to
>> attain higher performance. These different levels of
>> protection can be negotiated upfront.
>
>So, in thinking further about this, maliciousness is not
>actually required for overwrites to be problematic.
>
>Next hypothetical:
>
>1) application receives packet using DDP 2) application
>gets notified that data is available. 3) application
>believes it is in a safe and trusting environment, so RDDP
>mapping remains valid (neither explicit invalidation nor
>implicit invalidate-on-rx takes place) 4) application
>modifies received data in place 5) duplicate packet
>arrives, overwriting buffer. 6) application does some
>processing which depends on data not being overwritten
>after (4)
>

Paul Culley's reply already dealt with the immediate
solution to this problem.

A more applicable problem, which ironically demonstrates the
limitations to automatic STag deregistration, deals more
with the classic problem of stale pointers.

The intended sequence is as follows...

SEND-request A with buffer STag-A,Address-A,Len-A
SEND-request B with buffer STag-B,Address-B,Len-B

        RDMA-Write to portion of STag-A
        RDMA-Write to portion of STag-A
        SEND-reply A
        RDMA-Write to portion of STag-B
        RDMA-Write to portion of STag-B
        RDMA-Write to portion of STag-B
        SEND-reply B

Only the respondent code has a bug (gasp), and accidentally
uses the old remote pointer in processing the first portion
of the second command. This would be as follows...

        RDMA-Write to portion of STag-A
        RDMA-Write to portion of STag-A
        SEND-reply A
        RDMA-Write to portion of STag-A <<--- BUG!!!!
        RDMA-Write to portion of STag-B
        RDMA-Write to portion of STag-B
        SEND-reply B

At first glance this seems like the poster-boy example
justifying automatic tag release on receive. After all, the
erroneous RDMA Write will almost certainly be received more
quickly than the ULP will be able to deregister STag A.

But in fact it shows that automatic deregistration is
futile. Consider the following sequence:

        RDMA-Write to portion of STag-A
        RDMA-Write to portion of STag-A
        << lost message >> SEND-reply A
        RDMA-Write to portion of STag-A <<--- BUG!!!!
        RDMA-Write to portion of STag-B
        RDMA-Write to portion of STag-B
        SEND-reply B    
        << retransmit>> SEND-reply A
        
There is no way to determine that the erroneous RDMA-Write
to STag-A is invalid. The missing message could have been an
RMDA Write, rather than a SEND.

The bottom line is that upon notification of the SEND, the
only thing the ULP knows is that the priorr RDMA Writes have
completed. It cannot know that there have been no
*additional* writes.

[Aside:  To prevent that you must add a "at-once-once" rule
for each advertised buffer (each byte within the advertised
buffer will be delivered at most once). I developed one
in-chassis fabric that used exactly such a rule, and there
are major advantages to it. However it requires that disk
drives complete their never have sent a complete packet with
content that has not completedits media-level CRC check.
Internaly we thought this was a valid trade-off. The one
time I mentioned this approach at an iWARP meeting every
iSCSI vendor in ear-reach disagreed. In any event the rule
would require special hand-shaking to disable it for
circular buffers and publishing of statistical data.]

Sticking with RDMA Writes as they have been implemented to
date, the Data Sink ULP must deregister the STag itself
after it receives notification of the reply. Only THEN does
it know that the buffer is stable, whereupon it can validate
it via whatever ULP specific means deemed appropriate.

It is NICE if the STag deregistration can be automatically
applied at the earliest possible moment. But it does not
close any security holes. The buffers are exposed until the
trailing SEND *and* the deregisration have completed.
Consolidating those two completions is a *local*
optimization, it has no interoperabilty implications.

It is A REALLY GOOD IDEA to use STags exactly once, and it
is also A REALLY GOOD IDEA to deregister them as prompty as
possible. It is also A REALLY GOOD IDEA to have the fastest
processor possible and a few spare GB of DDRAM for your host
processor. Redundant power supplies are nice too.

The point is that how many STags are available, how quickly
they must be reused and how efficiently they can be reused
are all economic tradeoffs. The market should be allowed to
determine those answers, not the IETF.

Caitlin Bestler
http://www.asomi.com/CaitlinBestler.html