Subject: RE: [rddp] Alignment Commentary

From: "Williams, Jim" <[email-masked]>
Date: Tue, 14 Jan 2003 11:09:08 -0500

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Monday, January 13, 2003 8:10 PM
> To: [email-masked]
> Subject: [rddp] Alignment Commentary
> 
> [...]
> 
> The key implications seem to involve receiver behavior when the
> sender is not modified (and in particular winds up sending unaligned
> due to factors beyond its control), as Jim Williams asked:
> 
> > What is the required performance of an RDMA receiver when
> > receiving data from a transmitter using an unmodified TCP/IP stack?
> 
> This is somewhat subtle, as I believe the bulk of the answer
> is to be found in what I originally wrote after commenting on the
> divergence of views on this topic:
> 
> > Fortunately, the WG does not need to
> > settle this issue - implementers
> > are always free to optimize whatever
> > cases they believe are important.
> 
> or in Paul Culley's words:
> 
> > Of course all issues surrounding
> > performance are really for 
> > the market to decide.

This is a little tricky.  I certainly agree that implementers
are and should be free to make performance tradeoffs.  It
is not for this group to mandate performance.  My use of
lower case "must" and "required" were not intended to
suggest the upper case equivalent.

On the other hand, this group does need to make the 
protocol definition decisions that feed directly
into these implementation performance issues.
It's easy to say "all of the above; let the market
sort it out."  But there are a couple issues to
keep in mind.  It makes it a lot harder for an
implementation to hit its own performance objective
if it can't assume anything about the remote end
of its connection (i.e. whether the remote end
will align.)

Also, it doesn't make sense to over generalize the
protocol to cover cases that make no sense.  I am
quite skeptical about running RDDP in a mode that
results in both ends having performance no greater
than conventional TCP.  Why use RDDP at all in that
case?

One also ought not to treat too lightly the issue
of performance interoperability.  Consider the
case where Vendor A
has a high performance product, and vendor B
also has a high performance product.  But each
vendor's product is only high performance when
connected to a like product.  When vendor A
talks to vendor B, the performance is
terrible (but functionally correct).
This may not be the worst thing in the
world, and the market will ultimately
sort it out.  But if this is common, it
constitutes a failure of this WG to do its
job properly.  The market will ultimately
create a more restrictive de facto standard
than the official IETF standard, but this
is not a good thing to plan on.

So I am left with the following thoughts:

1.  RDDP over unmodified TCP is important,
    and will result in unaligned PDUs being
    common.

2.  Senders may align PDUs, but there is no
    MUST or SHOULD associated with alignment.

3.  Receivers may choose that only when talking to
    an aligning sender will high performance
    (better than non-RDDP TCP) be achieved.
    In this case the receiver should not even
    bother to negotiate use of RDDP when talking
    to RDDP over an unmodified TCP.

    There is no compelling reason to require
    a receiver that depends on sender alignment
    for performance to negotiate use of RDDP
    with a non-aligning sender.
    It should be acceptable
    for such a receiver to simply refuse
    to negotiate use of RDDP and use conventional
    non-RDDP connection in that case.

4.  Other receiver implementations may choose
    to achieve high performance with either
    aligning or non-aligning senders.

5.  Some method needs to be created for a sender
    to notify the receiver of ability to align.
    There are strong arguments that this should
    be done at the transport, not the RDDP layer.
    (i.e. a new TCP option on the SYN segment).

============

This line or reasoning quickly leads to the
realization that three modes of RDDP would be
supported:  unaligned TCP, aligned TCP, and
aligned SCTP.

Not surprisingly, my own preference would be
to limit this to two: unaligned TCP and
aligned SCTP.  However three modes 
seems like a reasonable compromise, if
that is how the WG wants to proceed.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 14 Jan 2003 15:54:10 -0600

On 1/14/03, Williams, Jim wrote:

>3.  Receivers may choose that only when talking 
>    to an aligning sender will high performance

DDP over unaligned MPA/TCP still has two major
benefits over the ULP using raw TCP:

1) Content can be shipped/fetched in an order
   optimized for actual physical storage, rather
   than in a precise byte order dictated by its
   nominal ULP form.
   
   Storage servers being able to ship/fetch
   disk sectors in the order they are reached
   is a major advantage, even when both ends
   of the connection are unaligned.
   
   Examples: disk striping, file fragmentation,
   logical files, database joins.
   
2) An unaligned DDP/MPA/TCP receiver still needs 
   fewer system buffers than a plain TCP receiver.
   
   With plain TCP, the number of buffers required
   can be N-1 (where N is the maximum # of buffers
   in flight).
   
   With alignment *totally* destroyed, on a
   persistent basis, an unaligned receiver will
   be able to start placement when at least half
   of the in-flight segment have been received,
   and probably well before that.

It is also important to remember that "non aligned
senders" are probably best described as
"non-alignment guaranteed senders". A DDP/MPA
stack working on top of a conventional unmodified
TCP stack will still be able to produce aligned
segments *most* of the time.

This all leads to a handful of decisions for the
MPA draft:

1) There are major benefits to RDDP for both 
   alignment and prompt transmittal of untagged 
   messages. Does this justify the MPA draft
   overriding some default TCP "SHOULDs"?
   For example:an MPA-aware sender MAY or SHOULD
   use alignment.
   
   I believe there is a strong consensus for yes
   on this point.
   
2) Should the current MPA draft be modified with
   enhancements, such as the split CRC, which 
   would benefit unaligned connections even though
   it would produce no benefits to aligned
   connections and all connections would have to
   pay the cost of the extra feature?
   
   I do not know if this has reached consensus
   yet. Some of the stated opposition was based
   upon a presumption that the second CRC would be 
   too complex. That is clearly not the case.
   
   It is feasible, and would be of benefit for
   unaligned connections. My estimation on the
   portion of connections that would benefit
   from the feature, however, is so low that
   I do not consider it worthwhile.
   
   Someone needs to present a scenario where a
   high performance server would see a substantial
   number of unaligned packets. I cannot come up
   with one.
   
3) Is there a need to configure a connection as 
   being aligned or unaligned, or should all
   connections be prepared to deal with unaligned
   packets at any time? 
   
   I can see allocating my resources based upon a
   statistical model on the number of aligned
   versus unaligned packets. If that model is
   wrong, my performance will suffer. The same
   sort of engineering tradeoff is done with any
   application that wants to predict its bandwidth
   over a media that can require retransmissions
   because of errors.
   
   But just as I would expect to be ready to 
   detect an error in *any* packet, no matter how
   reliable a particular connection has been
   historically, I would expect any connection
   to be ready to handle unaligned packets.
   
   Does anyone have a justification on why a
   receiver would benefit from a "heads up" on
   whether a connection will be aligned or not?
   
4) Are MPA markers sufficiently complex for
   a pure software stack to implement that it is
   worth negotiation of a "no-marker" mode?
   
   IMHO, Markers are truly ugly but ultimately
   not that hard to implement. It is not worth
   the complexity of supporting multiple modes
   and negotiating their use.
   
   However, there may be some merit in trying to 
   synergize interval markers with iSCSI. But
   I wouldn't make that an objective, unless
   the Area Directors jump in to the contrary.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Somesh Gupta" <[email-masked]>
Date: Wed, 15 Jan 2003 10:06:35 -0800

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Caitlin Bestler
> Sent: Tuesday, January 14, 2003 1:54 PM
> To: Williams, Jim; [email-masked]
> Subject: RE: [rddp] Alignment Commentary
> 
> 

  This capability has always existed, but I wonder
  how many storage servers actually implement this.
  Last time I asked this question, the unanimous
  answer seemed to be that even though it is an
  interesting capability, it is not implemented (maybe
  too complex overall for the benefit ??).

  The real benefit may be that upper layer protocols
  (such as storage over RDDP or whatever) are common
  for aligned or unaligned systems -- rather than have
  having two different ULPs.