Subject: RE: [rddp] Some Marker Comments

From: [email-masked]
Date: Thu, 23 Jan 2003 19:50:50 -0500

Caitlin,

(WG chair hat off)

> >provided that we had markers that were reasonable for
> >software to cope with (IMHO, MPA's current 512-byte fixed
> >interval isn't).
>
> I do not see any undue burden on a software solution from
> markers, at whatever interval, unless the software makes the
> misktake of physically assembling the DDP Segment in a
> consecutive sequence of memory bytes.

A copy is a way of completely fouling up, but one can get
into trouble long before then, although "undue" is in the
eye of the beholder.

> The first step is to identify the DDP segment. With markers
> embedded, the identified location consists of portions of
> one or more received IP datagrams. If you want to be even
> more nitty gritty, that is generally within one more
> received Ethernet Frames (but there are exceptions).
> 
> The DDP implementation has to do two things with the
> received DDP segment: interpret the DDP header and place the
> DDP contents. Both can be accomplished with a scatter/gather
> list. Unless you attempt to interpret the DDP header while
> it is has a marker imbedded in the middle of it there really
> isnt' that much of a problem for the software.

The "scatter/gather list" is the opportunity to get into
trouble.  Suppose we're dealing with an 8k data block in
a jumbo frame (can be made to work in the data center).
A 512-byte marker interval yields over 30 entries in the
scatter/gather list, whereas a larger marker interval
could yield 3.  That sort of order of magnitude overhead
difference tends to have visible costs in both software
and hardware, doubly so if the hardware can't cope with
a 30-entry list and needs to be redriven by software to
get the job done.  For a user-mode implementation on an
unmodified stack, those are syscalls rather than scatter/
gather list entries if the syscall interface is one
that doesn't support scatter/gather.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: John Hufferd <[email-masked]>
Date: Fri, 24 Jan 2003 10:34:48 -0800

Jim,
I do not see the world the same as you do.  "Political" considerations as
you call it is process that one follows while dealing with various
different belief structures.  In this case, the gray beards, as you call
them, have had years of work that has helped form their beliefs.  It is
silly to simply ignore them and useful to find a path that considers them.
Doing this may be "political", but that is not necessary a bad thing, and
is the usual way groups of people work together.  You use the term as if
"political" has the meaning of "wrong", and in so doing negate the years of
experience that is represented by the beliefs that are involved.

As I understand the issue, TCP has a lot of implementations and we would
like to have a way for as many of them as possible be able to exploit RDMA.
Also, in some ways, the complexity of TCP and the various implementations
of TCP  -- which may have pushed the envelop, but are never the less are in
wide use -- cause additional caution.  Therefore, as I understand it, the
ADs have asked us to show how unmodified TCP implementations can operate as
a sender or receiver of RDMA PDUs without having a significant problems.  I
think this is in our interest to do so.  And yes, transparency on the
links, also goes a long way to prevent unforeseen side effects.

Clearly one way to do this is to negotiate a parameter or two in which both
sides understand what the other is capable of doing and then do the right
thing.

The ULP is always able to make its own decision on how it wants to react to
the network configuration, I think that is a normal response, and does not
have a lot to do with the RDMA protocol itself.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

"Williams, Jim" <[email-masked]>@ietf.org on 01/24/2003 09:26:24
AM

Sent by:    [email-masked]

To:    "[email-masked]'" <[email-masked]>, [email-masked]
cc:
Subject:    RE: [rddp] Some Marker Comments

David Black wrote:
> Also, a while back, Jim commented:
>
> > 5.  Some method needs to be created for a sender
> >     to notify the receiver of ability to align.
> >     There are strong arguments that this should
> >     be done at the transport, not the RDDP layer.
> >     (i.e. a new TCP option on the SYN segment).
>
> IMHO (WG chair hat still off), the odds of a new TCP
> option for this purpose getting through tsvwg are close
> to zero (I've been involved in somewhat analogous attempts,
> and my ears are still ringing ...).

I hate to see important technical issues decided by
fear of being yelled at by people with gray beards. :)
On the other hand some valuable experience
comes along with the gray beards, and that should be factored
into the discussion.  If the fundamental objection is to
aligning TCP (as opposed to using a TCP option to do it)
then trying to sneak it by tsvwg by negotiating in-band
and pretending not to modify TCP seems like bad protocol
design as well as questionable politics.

> It would be better
> to pursue an inband notification mechanism that could also
> satisfy the "ensure no confusion about bytestream contents"
> requirement.  Getting that design and the underlying rationale,
> weaknesses, tradeoffs, etc. are worked out and well-described
> in an Internet-Draft ought to be done before considering
> whether the result makes a sufficient case for a new TCP
> option.

The fundamental choice is between:

 A.  "TCP is a byte stream protocol and thou shalt not attempt
      to use it otherwise."
and

 B.  "Let's treat TCP as a sequence of segments which the sender
      can construct, the network will preserve, and the receiver
      can exploit."

If the answer is A, then it's a waste of time to talk about either
TCP options or in-band notification of alignment.

If the answer is B, then in-band notification versus TCP option
is a legitimate debate.

I do object to the notion of, "Let's do B, but be sneaky enough
about it so that nobody looking at the wire can determine that
we aren't doing A, therefore tsvwg has no right to object."
It results in a bad and overly complex protocol motivated
by political rather than technical considerations.

---

From: "Williams, Jim" <[email-masked]>
Date: Fri, 24 Jan 2003 14:43:47 -0500

John,

> -----Original Message-----
> From: John Hufferd [mailto:[email-masked]]
> 
> Jim,

That's fair.  I was perhaps being a bit too flip, which
often doesn't work well over email, so I apologize.

I should have substituted "misguided political" for 
"political".  I don't mean to imply that all political
considerations are misguided.  But in some cases
it is fair to argue that they are.

If anything I meant to underscore the value of the
beliefs that have come from years of experience,
but that value comes from fully understanding the
reasons behind those beliefs.  The danger is
in treating those beliefs too superficially,
which constitutes "misguided politics" in 
my opinion.
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Fri, 24 Jan 2003 15:39:14 -0700

David,

One thing markers enable is placement of most of the data that arrives out of order even in the absence of alignment. The marker can be used to find the start of the frame. If markers are placed every 512 bytes, then any DDP segment starting more than 512 bytes after the start of a gap in the TCP stream will have a marker pointing to it. The larger the marker spacing, the more data one has to hold rather than place while waiting for a gap to be filled in.

Pat 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Thursday, January 23, 2003 4:51 PM
To: [email-masked]; [email-masked]
Subject: RE: [rddp] Some Marker Comments

Caitlin,

(WG chair hat off)

Thanks,
--David

---

From: [email-masked]
Date: Fri, 24 Jan 2003 19:49:19 -0500

Jim,

> design as well as questionable politics.

I believe the "fundamental objection" in this case is to
creation of a TCP option for something (alignment) whose
benefits are yet to be convincingly shown.  As I wrote:

> > weaknesses, tradeoffs, etc. worked out and well-described

It's a bit more nuanced than that.  Those who need 100% undiluted
"choice B" need to go use SCTP.  If the answer were 100% undiluted
"choice A", there would not be a TCP work item on the RDDP charter.
So, the answer is somewhere in between.  Among the tasks of the RDDP
WG is to figure out where that right place is and back that choice
with solid design rationale.

Thanks,
--David