Subject: RE: [Tsvwg] SCTP Checksum Change to the IESG

From: "Douglas Otis" <[email-masked]>
Date: Fri, 3 May 2002 17:17:21 -0700

Jonathan,

If there is a situation where hardware may fail to catch an error, then
replicating the check in software can be used to ensure critical
applications are less affected and are afforded end-to-end treatments.  The
current recommendation, however, places the packet at risk when specifically
recommending modification of the header during a hardware checking process.
This recommendation furthers the problem by then requiring modified software
to then re-verify the packet.  The quality of the audit and the data is
diminished with this recommendation.  High speed hardware will not modify
the packet and a status bit indicating a hardware CRC check condition is
equivalent to any number of registers that must be compared for equality it
terms of providing a robust indicator.

My experience found esoteric conditions related to the bus to qualify a
discard of potentially affected packets.  Unfortunately, no particular item
can be mandated to ensure fewer hardware/system related errors nor will
these recommendations extend universally to all implementations.  RFC 2119
makes it clear no implementation recommendation carries a MUST mandate if
unrelated to interchange.  There is no silver bullet, but perhaps the
Hippocratic Oath says it best - [First do no harm.] "I will follow that
system of regimen which, according to my ability and judgment, I consider
for the benefit of my client, and abstain from whatever is deleterious and
mischievous."  I doubt this belongs in the draft however. : )

 -Doug

On May 3, 2002 2:02 PM Jonathan Stone ([email-masked]) wrote:
>
> Dear all,
>
> Michael Tuexen and I have discussed the contentious text offline at
> some length.  One example we addressed is a NIC with TCP checksum
> assist in hardware.  The checksum assist in that NIC will occasionally
> deliver incorrect TCP checksum indications for packets smaller than
> the minimum Ethernet segment.
>
> I hesitate to say such bugs are commonplace, but they certainly
> aren't' rare: both in current, shipping server-class hardware and in the
> previous generation of bugs. The open-source community device drivers
> show the history of first enabling TCP checksum assist, then disabling,
> then finding workarounds or diagnostic tests which are cheaper than
> the alternative: deciding the checksum-assist hardware is busted,
> and ignoring it altogether.
>
> That is what the contentious text is saying. It is not saying
> *anything* about how a hardware CRC assist should compute the CRC.  It
> *is* saying that hardware assists for SCTP must provide some
> additional ``audit-trail'' information on what the hardware really
> did.  We can call that information ``CRC assist framing'': since it
> leaves a record of where the CRC hw took its checksum value; the
> length of the CRC computation; and the final result.
>
> That said, here's an attempt at some text which explains when, and
> where, and why, that audit information is necessary:
>
> -----
>    Errors in NICs and in outboard checksum acceleration have,
>    historically, been commonplace. This document requires that
>    hardware assists for the SCTP CRC provide more information than
>    a single CRC-GOOD/CRC-NOT-GOOD bit.  Suppose that some particular
>    NIC (or model of NIC) will occasionally give incorrect `CRC-GOOD'
>    status for packets that are in fact damaged. In this scenario,
>    GOOD/NOT-GOOD implementations give the SCTP software no recourse but
>    to ignore the CRC assist entirely, and recompute the CRC of
>    every packet in software.
>
>    This extra ``CRC-assist framing'' information  gives
>    software the option to apply a cheaper audit of the CRC assist
>    result based on the framing, before resorting to a software CRC.
>    Such checks are commonplace in the installed base of TCP stacks
>    using hardware- assisted TCP checksum.
>
>    Moreover, the CRC hardware assist, its associated NIC,
>    and the NIC  device-driver will  in practice often
>    be provided by a different vendor than is the SCTP stack.
>    To ensure that the desired CRC-assist audit trail crosses the
>    NIC-vendor/SCTP-stack-vendor boundary, this additional
>    information is [currently] specified as a MUST.
>
> -----
>
> That isn't quite up to snuff for an I-D, but it will be soon:) and it
> gets the idea across.  I hope Michael Tuexen will repost (or give me
> permission to repost) an excerpt of our discussion.  I'm hoping that
> together, those will resolve most of the contentiousness.
>
> An anecdote: when I gave a talk on the background data at SIGCOMM 2000
> in Stockholm, it was a hardware person (not a software guy) who
> commented that there was a `right' way and a `wrong' way to do
> checksum-offload assists; and that the one-bit GOOD/NOT-GOOD result
> was a bad way which should be discouraged.  For pretty much the
> reasons I give here, and of which I walked through an explicit example
> with Michael.  (To be fair, this was in a context of presenting data
> showing that some 1 in 10,00 frame in the Internet pass their Ethernet
> CRC, but fail the TCP checksum -- often in circumstances which point
> squarely at bugs in NICs.)
>
> Even so, I'm surprised that this has been so contentious.
>
> --Jonathan
>

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Jonathan Stone <[email-masked]>
Date: Fri, 03 May 2002 14:01:35 -0700

Dear all,

-----

-----

--Jonathan

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 4 May 2002 20:00:54 +0200

Doug,

it was pointed out on the list the your HW can leave the packet 
untouched.
Leaving it untouched is covered by the ID.

Best regards
Michael

>> Even so, I'm surprised that this has been so contentious.
[email-masked]

---

From: "Douglas Otis" <[email-masked]>
Date: Sat, 4 May 2002 11:39:45 -0700

Michael,

The added hardware text is extremely bad advise and inappropriately MANDATES
the return of the 'final CRC value'.  This mandate does not reduce
undetected hardware errors and the recommendation *will* increase undetected
hardware errors and weaken the integrity of a software check.  To suggest
this returned information is simply various states associated with hardware
assist makes this mandate ludicrous.  It then fails to indicate what and
why.  This brings us back to mandating the Hippocratic Oath. ; )

We both agree this is an important problem.  Don't ask for a hardware
interface that makes the problem worse!  The hardware SHOULD allow the
entire unaltered packet to be examined by software.  The states associated
with the hardware assist SHOULD be available to software to allow software
patches for hardware related faults.

 -Doug

On May 4, 2002 11:01 AM Michael Tuexen ([email-masked])
wrote:
> > If there is a situation where hardware may fail to catch an error,
> > then replicating the check in software can be used to ensure critical
> > applications are less affected and are afforded end-to-end
> > treatments. The current recommendation, however, places the packet
> > at risk when specifically recommending modification of the header
> > during a hardware checking process. This recommendation furthers the
> > problem by then requiring modified software to then re-verify the
> > packet.  The quality of the audit and the data is diminished with
> > this recommendation.  High speed hardware will not modify the packet
> > and a status bit indicating a hardware CRC check condition is
> > equivalent to any number of registers that must be compared for
> > equality in terms of providing a robust indicator.
> >
> > discard of potentially affected packets.  Unfortunately, no particular
> > item can be mandated to ensure fewer hardware/system related errors nor
> > will these recommendations extend universally to all implementations.
> > RFC 2119 makes it clear no implementation recommendation carries a MUST
> > mandate if unrelated to interchange.  There is no silver bullet, but
> > perhaps the Hippocratic Oath says it best - [First do no harm.] "I will
> > follow that system of regimen which, according to my ability and
> > judgment, I consider for the benefit of my client, and abstain from
> > whatever is deleterious and mischievous."  I doubt this belongs in the
> > draft however. : )
> >
> >  -Doug
> >
> >

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 4 May 2002 13:39:28 -0500

Michael,

Not according to Qiaobing's list; nor, the MUST in the draft.

--brian

On Sat, 04 May 2002, Michael Tuexen wrote:

> > equality it
> > item

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Randall Stewart <[email-masked]>
Date: Sat, 04 May 2002 16:03:55 -0500

Doug:

Please focus on the part that you most object to..

If I changed the wording from

" a recommended approach"

to 

"an approach"

would you not still be un-happy with the real issue for
you.. i.e. returning more than a single
bit.

After all the "recommended" approache is NOT a MUST and thus
has no coorelation to RFC2119..

Lets stick to the issue you dis-like that is a requirement and
not this "recommended" approach.. 

It seems we will be doing one more draft here anyway..

Hell I will take out recommended...

Now that is off the table... Are you happy or do
you still have an issue as I suspect?

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Randall Stewart <[email-masked]>
Date: Sat, 04 May 2002 16:06:33 -0500

Brian:

I don't see any MUST that prohibts Doug from not
zeroing a field... its not there..

There is one that requires Doug to return:

1) The csum used for comparison (the one taken from the packet)
2) The yes/no value
3) The csum calculated (I think.. but I bet Jonathan would be willing
                        to drop this requirement).

That is all.. These extra things besides a GO/NOGO are really
what Doug dislikes.. all the other stuff is 
Fluf.. and it is "recommended" NOT A MUST.

R

---

From: Randall Stewart <[email-masked]>
Date: Sat, 04 May 2002 16:11:17 -0500

Doug:

The document ALLOWS YOU to have an UNALTERED packet and csum it
and then return a go no-go.

The ONLY requirement in it is that you must return 2 other values
too..or is it three.. come to think of it .. it is the:

-length of the calcuation
-Csum you calculated
-Csum you read from the packet.

There IS NO REQUIREMENT that prevents you from using an unaltered
packet method... THE ONLY REQUIREMENT is that you return the above
thee pieces of information.

Please lets stay focused on the issue and not get sidetracked
by the "recommended" approach..

I think Jonathan may agree to get rid of the caculated value.. Jonathan
will you agree to this?

If so then you only must return what YOU SAW in the UNALTERED PACKET
and what LENGTH you did your check on..

What, please tell me, is wrong with just that.. I don't want to
hear about the "recommended" method.. I will STRIKE recommended from
the draft.. so all it says is a method of doing it..

What is wrong with returning the length and what you read from
the csum field .. along with a go-nogo?

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 4 May 2002 16:45:06 -0500

Randall,

>                         to drop this requirement).

The above is the "MUST" I was referring to, and the "No!!!" in 
Qiaobing's table, and the passage that Doug objects to.  Remove
this requirement and we're done.

--brian

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 4 May 2002 16:47:08 -0500

Randall,

> thee pieces of information.

This is the requirement to which objections have been raised.

--brian

---

From: "Douglas Otis" <[email-masked]>
Date: Sun, 5 May 2002 00:39:36 -0700

Randall,

The following is the current added hardware implementation notes:

-----
"Any implementation using hardware assists for checksum computation
 MUST make the final CRC value available to the software portion of
 the stack.  Such implementations SHOULD also make the buffer size
 used by the hardware unit available to software.

 NOTE: this requirement explicitly forbids hardware-acceleration
 implementations which provide only a one-bit indication, ``checksum
 valid'' or ``checksum invalid'. This requirement is to permit
 software to audit the hardware assist: to re-compute the SCTP
 checksum over the identical input seen by the outboard hardware
 accelerator, and make a comparison of the software-computed and
 hardware-assisted values.

 This requirement is motivated by experience with a variety of TCP/IP
 checksum assists. Bugs in outboard checksum-assist implementations
 (which may occur only under certain edge conditions) but which did
 not provide the additional information specified here, leave
 protocol implementors with little choice but to completely disable
 the checksum assist.

 An implementation using hardware assist is encouraged to follow the
 exact procedure specified above.  Whatever implementation strategy
 an implementation chooses, it MUST arrange to provide both the
 initial value of the checksum field, and the value computed for the
 SCTP checksum, to the software stack.

 IMPLEMENTATION NOTE: One recommended implementation strategy is:

 * save the contents of the checksum field;
 * zero out the checksum field;
 * compute the checksum;
 * the checksum implementation then provides both the
   checksum computed over the zeroed-out field, and the initial
   value of that zeroed-out field to software.

  This strategy is believed to be robust against errors in an
  outboard acceleration unit, and against communication errors (or
  memory-addressing errors) between an outboard CRC acceleration
  unit and the software stack.  It also provides a clean path for
  retrofitting hardware checksum acceleration into a previously
  `software-only' SCTP stack."
-----

These notes should change to:

-----
"Any implementation using hardware assists for checksum
 computation SHOULD NOT alter content of the packet during the
 checking process to ensure a hardware error does not corrupt
 packet content.  During checking, the checksum field is logically
 zeroed rather than saved and modified.  In addition, by not
 altering the packet, standard software checking routines can
 re-verify the integrity of the packet without code modification.

 It is recommended the checksum hardware assist provides a
 reasonable amount of state information to the software stack
 to afford an opportunity to repair defective processes detected
 by anomalous state.  Critical applications SHOULD re-verify packet
 integrity using software, in addition to the hardware assist.

 This strategy is believed to be robust against errors in a
 checksum hardware assist unit, and against communication or
 memory-addressing errors.  It also provides a clean path for
 retrofitting checksum hardware acceleration into previously
 'software-only' stacks and provides for an audit mechanism."
-----

 -Doug

On May 4, 2002 2:11 PM Randall Stewart ([email-masked])
wrote:
>
> Douglas Otis wrote:
> >
> > Michael,
> >
> > The added hardware text is extremely bad advise and
> > inappropriately MANDATES the return of the 'final CRC
> > value'.  This mandate does not reduce undetected
> > hardware errors and the recommendation *will* increase
> > undetected hardware errors and weaken the integrity of
> > a software check.  To suggest this returned information
> > is simply various states associated with hardware assist
> > makes this mandate ludicrous.  It then fails to indicate
> > what and why.  This brings us back to mandating the
> > Hippocratic Oath. ; )
> >
> > We both agree this is an important problem.  Don't ask
> > for a hardware interface that makes the problem worse!
> > The hardware SHOULD allow the entire unaltered packet
> > to be examined by software.  The states associated
> > with the hardware assist SHOULD be available to
> > software to allow software patches for hardware related
> > faults.
>
>
> Doug:
>
> > > > If there is a situation where hardware may fail to catch an
> > > > error, then replicating the check in software can be used to
> > > > ensure critical applications are less affected and are
> > > > afforded end-to-end treatments. The current recommendation,
> > > > however, places the packet at risk when specifically
> > > > recommending modification of the header during a hardware
> > > > checking process. This recommendation furthers the problem
> > > > by then requiring modified software to then re-verify the
> > > > packet.  The quality of the audit and the data is diminished
> > > > with this recommendation.  High speed hardware will not modify
> > > > the packet and a status bit indicating a hardware CRC check
> > > > condition is equivalent to any number of registers that must
> > > > be compared for equality in terms of providing a robust
> > > > indicator.
> > > >
> > > > My experience found esoteric conditions related to the bus
> > > > to qualify a discard of potentially affected packets.
> > > > Unfortunately, no particular item can be mandated to ensure
> > > > fewer hardware/system related errors nor will these
> > > > recommendations extend universally to all implementations.
> > > > RFC 2119 makes it clear no implementation recommendation
> > > > carries a MUST mandate if unrelated to interchange.  There
> > > > is no silver bullet, but perhaps the Hippocratic Oath says
> > > > it best - [First do no harm.] "I will follow that system of
> > > > regimen which, according to my ability and judgment, I
> > > > consider for the benefit of my client, and abstain from
> > > > whatever is deleterious and mischievous."  I doubt this
> > > > belongs in the draft however. : )
> > > >
> > > >  -Doug
> > > >
> > > >
> > > >> Michael Tuexen and I have discussed the contentious text
> > > >> offline at some length.  One example we addressed is a NIC
> > > >> with TCP checksum assist in hardware.  The checksum assist
> > > >> in that NIC will occasionally deliver incorrect TCP checksum
> > > >> indications for packets smaller than the minimum Ethernet
> > > >> segment.
> > > >>
> > > >> I hesitate to say such bugs are commonplace, but they
> > > >> certainly aren't' rare: both in current, shipping
> > > >> server-class hardware and in the previous generation
> > > >> of bugs. The open-source community device drivers
> > > >> show the history of first enabling TCP checksum assist,
> > > >> then disabling, then finding workarounds or diagnostic
> > > >> tests which are cheaper than the alternative: deciding
> > > >> the checksum-assist hardware is busted, and ignoring it
> > > >> altogether.
> > > >>
> > > >> That is what the contentious text is saying. It is not
> > > >> saying *anything* about how a hardware CRC assist should
> > > >> compute the CRC.  It *is* saying that hardware assists
> > > >> for SCTP must provide some additional ``audit-trail''
> > > >> information on what the hardware really did.  We can
> > > >> call that information ``CRC assist framing'': since it
> > > >> leaves a record of where the CRC hw took its checksum
> > > >> value; the length of the CRC computation; and the final
> > > >> result.
> > > >>
> > > >> That said, here's an attempt at some text which explains
> > > >> when, and where, and why, that audit information is
> > > >> necessary:
> > > >>
> > > >> -----
> > > >>    hardware assists for the SCTP CRC provide more information
> > > >>    than a single CRC-GOOD/CRC-NOT-GOOD bit.  Suppose that some
> > > >>    particular NIC (or model of NIC) will occasionally give
> > > >>    incorrect `CRC-GOOD' status for packets that are in fact
> > > >>    damaged. In this scenario, GOOD/NOT-GOOD implementations
> > > >>    give the SCTP software no recourse but to ignore the CRC
> > > >>    assist entirely, and recompute the CRC of every packet in
> > > >>    software.
> > > >>
> > > >>    software the option to apply a cheaper audit of the CRC
> > > >>    assist result based on the framing, before resorting to a
> > > >>    software CRC. Such checks are commonplace in the installed
> > > >>    base of TCP stacks using hardware- assisted TCP checksum.
> > > >>
> > > >>    To ensure that the desired CRC-assist audit trail crosses
> > > >>    the NIC-vendor/SCTP-stack-vendor boundary, this additional
> > > >> That isn't quite up to snuff for an I-D, but it will be
> > > >> soon:) and it gets the idea across.  I hope Michael
> > > >> Tuexen will repost (or give me permission to repost) an
> > > >> excerpt of our discussion.  I'm hoping that together,
> > > >> those will resolve most of the contentiousness.
> > > >>
> > > >> An anecdote: when I gave a talk on the background data at
> > > >> SIGCOMM 2000 in Stockholm, it was a hardware person (not
> > > >> a software guy) who commented that there was a `right'
> > > >> way and a `wrong' way to do checksum-offload assists; and
> > > >> that the one-bit GOOD/NOT-GOOD result was a bad way which
> > > >> should be discouraged.  For pretty much the reasons I give
> > > >> here, and of which I walked through an explicit example
> > > >> with Michael.  (To be fair, this was in a context of
> > > >> presenting data showing that some 1 in 10,00 frame in the
> > > >> Internet pass their Ethernet CRC, but fail the TCP checksum
> > > >> -- often in circumstances which point squarely at bugs in
> > > >> NICs.)
> > > >>