Subject: [Tsvwg] Another view of the SCTP error check problem

From: Robert Snively <[email-masked]>
Date: Mon, 25 Jun 2001 17:15:07 -0700

Let's run it backwards.  These numbers have not been
cross-checked, so I may be an order of magnitude off 
here and there, but the principle seems clear.

I have a goal of less than one undetected error per
million years.  There are pi * (10**7) seconds per year,
so I can tolerate one error per 3 * (10**13) seconds.
At 10 Gbits = 1.2 GBytes, that means that I can tolerate 
roughly one undetected error in 3.6 * (10**22) bytes.  

Now let us assume that we are running on hardware that
creates the nastiest case identified by Partridge and
Stone, which showed one undetected error on TCP/IP per
16 million packets.  Since undetected packets are passed
with a probability of approximately 1 in 2**16 
(or 1.5 * (10**-5)), that means that roughly one packet in
250 has an error.  The largest packet is likely to be 
an Ethernet packet of about 1500 bytes, which means that
roughly one significant error per 3.7 * (10**5) bytes
occurs, or 3,200 failures per second occur.

So that leaves the following questions:

1) How do we get from here (1 error / 3.7*(10**5) bytes)
   to where we want to be (1 error / 3.6 * (10**22) bytes)?

2) What good is a 32-bit check of any kind (which reduces
   the failure rate by roughly 1 / 2**32 or 2.3 * (10**-10))
   going to do?  That only gets us to one error in 
   1.6 * (10**15 bytes), some 7 orders of magnitude worse
   than we need to achieve.  I am pretty sure a CRC does not
   catch 100% of stuck faults unless they effect only
   a certain number of bits, preferably closely spaced.

3) If it takes 1 msec of latency in recovery every time an
   an error is detected, that means for every second of
   transfer, you will be spending 3.2 seconds in failure
   recovery, and I bet that 1 msec is a low number.  The
   problem with error recovery is that it is not dependent
   on link speed, but rather on round-trip link latency. How
   many people will want a 10 Gbit network that only runs
   2.4 Gbits?

Some of the answers are pretty obvious.  

1) Any product that you have some chance of selling in
   an enterprise environment will have an error rate through
   all hops some seven orders of magnitude better than an 
   ungroomed campus internet link.  

2) Checking will be required at lower levels to achieve that
   improvement, not so much to correct errors, but to isolate
   devices and links that need to be replaced because they
   are throwing too many errors.  That checking
   will be included both in the links and in the boxes.

3) Any application that really cares won't trust even those
   probabilities and will provide additional verification at the
   application level, principally to identify suspect
   links and systems that have failed to perform their checks
   correctly.

Once you have accepted the three answers above, why is
CRC any better than a TCP-like 32-bit sum?

p.s. 
   At 10 Gbits, CRCs are typically calculated 32 or 64 bits
   wide, which makes them a smidgeon more expensive than
   32 register spaces and a few XORs.  Of course, 32 or 64 bit
   adder/accumulators are not free either, but at least they
   handle padding a bit more cleanly. 

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Robert Snively <[email-masked]>
Date: Tue, 26 Jun 2001 15:09:03 -0700

David,

My point is that even with CRC, you are some seven orders of
magnitude from a suitable reliability environment for
enterprise storage applications.  You will be depending
on some combination of lower level checks and higher level
checks to achieve your goals.  SCTP, with CRC or with
a decent sum check, will not meet the requirements.

In iSCSI you have recognized that problem and provided additional
checking at the higher level.

Bob

>  -----Original Message-----
>  From: [email-masked] [mailto:[email-masked]]
>  Sent: Tuesday, June 26, 2001 6:52 AM
>  To: [email-masked]; [email-masked]
>  Subject: RE: [Tsvwg] Another view of the SCTP error check problem
>  
>  
>  Unfortunately, all three answers are debatable and probably wrong:
>  
>  (1) We care about environments other than the enterprise environment.
>  (2) Not all links can be relied upon to have that level of 
>  error detection.
>  (3) Application level checks should be a last resort for two reasons.
>  	a) The transport's retransmit algorithm is better engineered and
>  		more efficient - it should be the primary retransmit
>  algorithm,
>  		and leads to better network behavior when it is.
>  	b) If the transport can be relied upon to have a 
>  sufficient level of
>  		error detection underlying its retransmit, 
>  applications can
>  be
>  		significantly simpler.  If the response to the 
>  check failing
>  		can be designed under a worst-case assumption that the
>  		check will essentially never fail, then simple 
>  brute-force
>  		mechanisms (e.g., close the connection, try 
>  again) are ok,
>  		and lousy performance is acceptable.  Otherwise the
>  		response to a failed check has to be designed under a
>  		common-case assumption that its performance matters,
>  		and the result can be significantly more complex.
>  
>  Beyond this, we know that the existing Adler-32 checksum
>  is broken for short packets, and there's an overwhelming
>  need for the fix to this to be permanent.  I would suggest
>  that this need for permanence significantly outweighs
>  the risk/costs of a CRC being overkill - based on the
>  respective track records of CRCs and checksums, a CRC
>  is the obvious choice to make sure that we never
>  have to revisit this issue again.  The computational
>  overhead argument against CRCs has been refuted on
>  this list, so I fail to see any compelling technical
>  objection to use of CRCs that remains.
>  
>  IMHO, we can put a CRC in now and put this problem behind
>  us, or continue to debate the merits of various forms of
>  checksums to SCTP's detriment.
>  
>  --David
>  
>  ---------------------------------------------------
>  David L. Black, Senior Technologist
>  EMC Corporation, 42 South St., Hopkinton, MA  01748
>  +1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
>  [email-masked]       Mobile: +1 (978) 394-7754
>  ---------------------------------------------------
>  
>

---

From: "Adams, Richard Anthony (Richard)" <[email-masked]>
Date: Fri, 29 Jun 2001 13:38:29 +0100

Alternatively...

> roughly one undetected error in 3.6 * (10**22) bytes. 
[RAA] Why this strict ? Surely one error per SCTP lifetime would be good
enough ?
Anyway, taking the SS7 SCTP requirements, which is one undetected error per
10**10 messages (or 10**9 for ANSI), at a SIF length for narrow band MTP of
272 octets, we have a requirement of 1 undetected error in ~3 * 10**12 (or
10**11) bytes. [raa]
> occurs, or 3,200 failures per second occur.
	[RAA] In our SS7 case, we would have 1 significant error per ~7.4 *
10**4 bytes. [raa]
>    a certain number of bits, preferably closely spaced.
	[RAA] Supposing that the 32 bit check *does* reduce the failure rate
by 2.3 * 10**-10,
	 (which I believe rules out Adler 32), this gives SS7 a rate of 1
byte in 3.2 * 10**14 with a significant error. The requirement is 1 byte in
~3 * 10**12 (at its strictest, based on the MTP service offered to its
users).  So we are up 2 orders of magnitude. [raa]
>    many people will want a 10 Gbit network that only runs
	[RAA] This isn't a requirement as such on how good we want error
detection to be, but how good we want the signalling path to be. [raa]
> CRC any better than a TCP-like 32-bit sum?
[RAA] As long as it lets through no more than 1 in 2**32 errors, for the
networks we want to use SCTP over, and for the packets we want to send in
those networks, a 32 bit check sum would be good enough. But Adler 32 isn't,
so the SCTP checksum should change. And the properties of CRC 32 are good
enough. [raa] 

With kind regards,
Richard A. Adams,
[email-masked], tel : +44 1666 83 2503
fax: + 44 1666 83 2712
Lucent Tech.,
Swindon Road,
Malmesbury,
Wiltshire SN16 9NA.

---

From: Robert Snively <[email-masked]>
Date: Fri, 29 Jun 2001 11:57:53 -0700

Doug,

While it is clear that the iSCSI recovery stack would be
exercised less often with an SCTP plus CRC transport than with a TCP
plus 16-bit Check Sum transport, the number of errors will
be the same.  None of these transport mechanisms support ECC and
all use similar retransmission mechanisms. I am 
assuming that the transport layer recovery
algorithms are about as expensive as the application
layer recovery, so the impact on performance is approximately
the same in both cases.  The difference in detection is
on the order of 10**5 when comparing CRC with a 16 bit check sum
and 10**0 when comparing CRC with a mathematically correct
32-bit check sum.

So, if you want to run real storage applications, you will be
grooming the network to find links with moderate or low
error rates and that have checking at lower levels.  Of course
read-only storage containing video data or web data probably
won't require that, but independent storage providers carrying
customer data bases surely will.

FCIP, FC, and FCP provide several levels of high order checking, 
so that realitively bad TCP links will still be able to provide data
that is accurate.  However, the TCP performance of such links will
quickly encourage the improvement of the error rates.
FCIP is also a rather special purpose link in a Fibre Channel
SAN, so the majority of the data will flow over
Fibre Channel hops with data integrity better than that promised by
WAN environments.

>  
>  Even if you view any transport as needing additional error 
>  checking, if you
>  examine the iSCSI protocol you will discovery a messy 
>  situation involved in
>  attempting to recover for the transport.  Due the relative 
>  weakness of this
>  transport summation scheme that fails at a far higher rate 
>  than the 2^n size
>  of the summation, when used over the Internet an additional 
>  application
>  mechanism may be invoked many times a day at 10 GE.  If the 
>  transport is
>  protected by CRC-32, then for the Internet this additional 
>  mechanism may be
>  invoked at a yearly rate where this lower rate allows simpler schemes
>  without a notable reduction in performance.  An alternative 
>  to using the
>  Internet could be a proactively maintained network however 
>  where the lower
>  failure rate also allows this scheme to be practical.
>  
>  How are you expecting to recover from an error missed by the 
>  transport when
>  using FCIP as example?
>  
>  Doug
>  
>  
>  > My assumption has been that what you are really after is
>  > a transport mechanism that will allow iSCSI and other framed
>  > activity typical of storage interfaces to be carried with
>  > a minimum of higher level checking.  Storage is not satisfied
>  > with any undetected errors, since they are revealed as
>  > undetected data integrity failures in the returned data.
>  > Such failures can have undetermined and perhaps catastrophic
>  > effects to applications.
>  >
>  > If you say that you are guaranteeing 1 undetected error in 10**11
>  > or 1 error in 10**14 bytes, storage folk will say that is all very
>  > nice, now send me the name of a real storage transport.  
>  Alternatively,
>  > the storage folk will say that they will do the checking 
>  themselves and
>  > TCP will do very nicely as a transport mechanism.  That is the
>  > path that iSCSI is presently taking.
>  >
>  > If merely meeting the SS7 error numbers is your goal, then what
>  > I have said is not relevant and may safely be ignored.
>  >
>  > Bob Snively                        e-mail:    [email-masked]
>  > Brocade Communications Systems     phone:  408 487 8135
>  > 1745 Technology Drive
>  > San Jose, CA 95110
>  >
>  >
>  > >  From: Adams, Richard Anthony (Richard) 
>  [mailto:[email-masked]]
>  > >  Sent: Friday, June 29, 2001 5:38 AM
>  > >  To: TSV WG list