Subject: [rddp] Re: iSCSI/iWARP drafts and flow control

From: Mike Ko <[email-masked]>
Date: Thu, 7 Aug 2003 10:52:46 -0700

Caitlin, since MaxCmdSN is used by the target to limit the number of 
commands it can receive from the initiator, your suggestion would work for 
limiting the number of "fringe" messages the initiator can send to the 
target.  However, for "fringe" messages from the target to the initiator, 
such as the Aysnchronous Message, the target cannot simply raise MaxCmdSN 
and expect the initiator to increase the number of receive buffers for 
"fringe" messages correspondingly.

Mallikarjun is correct in pointing out that some form of new wire protocol 
for flow control is required if we need to flow control the messages not 
regulated by CmdSN.

Mike Ko
IBM Almaden Research
[email-masked]

Sent by:        [email-masked]
To:     "Mallikarjun C." <[email-masked]>
cc:     <[email-masked]>, <[email-masked]> 
Subject:        Re: [rddp] Re: iSCSI/iWARP drafts and flow control

On Thursday, July 31, 2003, at 07:43 PM, Mallikarjun C. wrote:

>> No new wire protocol is required.
>
> Please explain to me how credit can be replenished
> in the following example - without new positive flow
> control protocol.

I thought of at least one algorithm overnight. This is
not presented as a definitive proposal, merely to show
a strategy that can allow iSER to provide flow control
of *all* untagged messages.

iSER untagged messages are divided into a predictable
portion regulated by CmdSNs in a very direct fashion
and asynchronous messages. The latter category can be
characterized as having a low sustained rate compared
to the CmdSN-related untagged messages, but that the
traffic is very bursty. That is, the peak rate can be
high.

That suggests adapting a classic "leaky bucket" credit
system of the type typically used to regulate burst
traffic over rate controlled networks. The key
difference is that the "clock" is the Max CmdSN.

What is required is the following:

The sender maintains a asynch-credit counter. It is
initialized to a known value. This value could be
negotiated if it is believed that there is enough
variation to warrant negotiation, otherwise it would
be fixed. That caveat applies to all other "constants"
cited in this algorithm.

When the ULP desires to send an untagged message using
one of these credits:

The credit count is brought up to date: The
current Max CmdSN is compared with the value when
the prior "fringe" send was performed. The delta
is used to grant new credits, however there is a
maximum number of credits that may be accumulated.

If there are enough credits: the credit count is
reduced and the untagged message may be sent.

If there are not enough credits: the untagged
message must be delayed until the Max CmdSN is
advanced. After a configurable delay, iSER
SHOULD send some form of NOP command to cause
Max CmdSN to advance.

Note that there are NO iSER defined messages for
the purpose of flow control.  Untagged messages
may be delayed, but that could happen over TCP
just as easily. The only change is that *only*
untagged messages will be delayed, never tagged
messages, and *where* they are delayed (iSER
versus TCP).

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 7 Aug 2003 14:42:46 -0500

> for flow control is required if we need to flow control the messages 

To be more precise, no general purpose wire protocol is required
that must act as a filter on every untagged message. The procedure
I outlined deals with "fringe" messages from the initiator. It is
logic that is never needed more often than per-fringe-message, and
generally requires no additional messages.

If fail to see why that cannot be worked out for each category
of message that iSCSI requires. All that is required is to identify
some acknowledgment that flows in the desired direction. Under the
worst case you might be forced to create a message for the sole
purpose of restoring credits.

You cannot claim that SCSI functionality over RDMA is incompatible
with flow control. SCSI over TCP/IP is flow controlled by TCP,
iSER flow control would be *less* restrictive than TCP flow control,
but it does have to be identified and specified. Additionally
SRP is fully defined to work over RDMA with flow control. The
problem has a solution. There is no justification for exempting
iSER from the responsibility to provide ULP flow control.

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 7 Aug 2003 20:27:16 -0500

> for flow control is required if we need to flow control the messages 

The target may always send untagged messages when they are
a response to an initiator request. Additionally the target
may have N extra outstanding untagged messages, where N is
a protocol constant or negotiated for the session.

A non-responding target message is "outstanding" until
one of the following occurs:

	The initiator responds to a "Nop Out Request".

	The initiator sends a command with CmdSN that
	was only enabled after the non-responding target
	message.

The fact that a prior non-responding target untagged
message is no longer "outstanding" merely means that
whatever buffers are allocated for this purpose have
been replaced. It does not guarantee that the initiator
has completed any associated processing, that logic
is left totally to the ULP.