Subject: [nfsv4] Extensions to NFSv4 for Checksums

From: Alok Aggarwal <[email-masked]>
Date: Thu, 09 Feb 2006 16:56:28 -0700 (MST)

I would like to propose additional operations to implement checksums
for NFSv4.

Since NFSv4 employs TCP and Ethernet as it's underlying transport 
mechanism, it relies on these protocols to provide data integrity 
in the absence of krb5i.

The standard TCP checksum is arguably weak in that it's not resilient 
to certain single bit errors as well as data re-arrangement. The 
standard Ethernet checksum is strong but is not end-to-end and is 
recomputed at every hop. Given that the underlying transports don't 
provide enough protection, the NFSv4 protocol (when running without 
krb5i) needs to provide it's own data integrity mechanism.

A draft proposal that provides the motivation behind checksums for
NFSv4 and describes extensions to NFSv4 to enable such a capability 
can be found here -

http://www.ietf.org/internet-drafts/draft-aggarwal-nfsv4-cksum-00.txt

Discussion and suggestions for improvements are requested.

Alok

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Eric Werme USG <[email-masked]>
Date: Thu, 09 Feb 2006 20:57:18 -0500

The standard TCP checksum is arguably weak in that it's not resilient 
   to certain single bit errors as well as data re-arrangement.

What single bit errors does it miss?  In its simplest form, it's a
sum of 16 bit values with the carry folded back in.  A single
bit error will have values that are powers of two between 0x0001 and 0x8000,
but the checksum requires that 0xffff be added or subtracted to fail.

Now, if a 0x0000 were replaced with 0xffff, the checksum would fail, but
that's a 16 bit error (albeit off by 1).  If 0x0001 were replaced
with 0x0000 _and_ if another 16 bit value was incremented, that would
also look like a 0xffff addition and the checksum would fail. Again, that
would be a multiple bit error.

The intent of the IP checksum was to catch software corruption in what
should be a clean environment, one protected by parity or ECC hardware.
The purpose of the Ethernet CRC and others is to check for errors in noisy
environment.

Your proposal moves a test outside of TCP accelerators and extends the
distance that data is protected.  One of the problems with TCP
accelerators is that they allow corruption that occurs in the OS
or DMA to be missed.  Until people come up with NFSv4 accelerators,
you'll have protected the message longer.

In my experience, the cases of undetected checksum failures have been
"Frankengrams", UDP messages reassembled from fragments of more than
one message, and a case of a router vendor whose hardware changed
a field of a NFS request and recomputed the checksum.  NFS V4 uses
TCP, so fragments shouldn't be an issue, but you might still catch a flaky
client, router, or server.  I guess that's the goal.

That flaky router I ran into didn't affect the data, but something
in the RPC/NFS header before the data.  I suppose adding a checksum
to COMPOUND would at least cover more of the message....

Overall, I'm not convinced there a problem that needs fixing.

	-Ric (yes, I'm still around!) Werme
-- 
Eric (Ric) Werme         |  [email-masked]
Hewlett-Packard Co.      |  http://werme.8m.net/

---

From: David Robinson <[email-masked]>
Date: Thu, 9 Feb 2006 23:54:45 -0600

> but the checksum requires that 0xffff be added or subtracted to fail.

See http://citeseer.ist.psu.edu/stone00when.html

To quote the abstract:
"Traces of Internet packets from the past two years show that between
1 packet in 1,100 and 1 packet in 32,000 fails the TCP checksum"

In a Sun lab, we spent a few weeks tracking down a bad 10Mbps hub that
was triggering exactly the case that this paper describes. Only
caught by SCCS's stronger CRC.
> The purpose of the Ethernet CRC and others is to check for errors  

While protection at different layers is useful, it is not sufficient.
Pushing checksums as close to the ends as possible is the only way
to insure correctness. Early testers of Sun's new ZFS filesystem
that includes integrity checksums are getting surprised just how
often the disks return bad data, even though there are various
checks in drives and transports.

> in the RPC/NFS header before the data.

Math says that NFS is on shaky grounds, and real world experience
has confirmed that for me.
> Overall, I'm not convinced there a problem that needs fixing.

Well we disagree.
> 	-Ric (yes, I'm still around!) Werme

And still using a DEC address! :-)

	-David

---

From: Mike Eisler <[email-masked]>
Date: Thu, 9 Feb 2006 23:08:03 -0800 (PST)

(I first sent this directly to Alok, but meant to send it to the WG).

---

From: Mike Eisler <[email-masked]>
Date: Thu, 9 Feb 2006 23:16:25 -0800 (PST)

> Discussion and suggestions for improvements are requested.

The i-d discusses applying checksumming to just READ and WRITE.

It appears that the intent is to leverage any checksums maintained in
the server's local file system or storage subsystem. It might be
be desirable to have a scheme that cleanly supports both check sums of all
requests and responses, across the entire RPC and NFS request, 
and leverages any file/storage system check sums without double
computation. On the other hand, just as with iSCSI digest, without
hardware acceleration, a general purpose checksum algorithm is unlikely to be
deployed in software, and unlike iSCSI, I doubt there will be offload engines
to accelerate NFSv4 checksums. So the compromise being expressed in the
i-d is pragmatic, if limited to certain local file systems and storage systems.

The CKINFO description says:

"The CKINFO operation is added in the mount compound by the client
      to gather a list of checksum algorithm preferences that the server
      may have. 

NFSv4 has no mount operation. My guess is that the intent is the server returns the
checksum info for the file system specified by the current file handle in the COMPOUND
request. The problem with that is that the client can cross file systme boundaries, and so
if the mounted file system starts off as say a ZFS file systems that use FLETCHER4 checksums,
the client could navigate into another file system type and not know that the algorithm
has changed. It would be better to make CKINFO a new, per-file system, attribute that GETATTR
can request.

CKSUM either precedes or proceeds the operation depending on whether it is a write or
read. I think it would be better to add PUT_CKSUM and GET_CKSUM operations so that it is
clearer. This would be in the NFSv4 spirit (re: PUTFH and GETFH). 
Note that this way, one could potentially
apply checksums to other operations besides READ and WRITE. It would also let one, for example,
checksum READ arguments, and receive checksums on READ results.  

CKSUM applies to the data arguments or results of READ or WRITE. It would be better if
there was a way to have PUT_CKSUM and GET_CKSUM indicate which parts of the
operation checksumming applies to. The rpc-rdma i-d has a similar problem
in indicating which parts of an rpc request or responsed are to transfered with RDMA.
It uses "chunk lists" to indicate this. Something similar could be done PUT_CKSUM and
GET_CKSUM.
 
That all stated, I wonder if it is too late to add this to the NFSv4.1 feature list.

---

From: Julian Satran <[email-masked]>
Date: Fri, 10 Feb 2006 10:42:21 +0200

With the large body of information we have collected for iSCSI and decided 
to go for a CRC I agree that relying on the TCP checksum is not wise.
However if  you are looking for a check that covers users data from origin 
through disk to consumer that may involve more changes than you might be 
ready to accept. And if disk errors are that frequent they should be 
included in the "check-path".

Julo

David Robinson <[email-masked]> 
Sent by: [email-masked]
10/02/06 07:54

To
Eric Werme USG <[email-masked]>
cc
Alok Aggarwal <[email-masked]>, [email-masked]
Subject
Re: [nfsv4] Extensions to NFSv4 for Checksums

To quote the abstract:

Well we disagree.

                 -David

---

From: Julian Satran <[email-masked]>
Date: Fri, 10 Feb 2006 10:47:36 +0200

You may be surprised on how inexpensive a CRC can be even without 
accelerators. CRCs are linear functions and can be computed incrementally 
(an can be done in the "shadow" of a data copy). A recent paper presented 
by some Intel Colleagues at the 2005 FAST conference may give an idea.

Julo

Mike Eisler <[email-masked]> 
Sent by: [email-masked]
10/02/06 09:16
Please respond to
[email-masked]

To
[email-masked]
cc

Subject

It appears that the intent is to leverage any checksums maintained in
the server's local file system or storage subsystem. It might be
be desirable to have a scheme that cleanly supports both check sums of all
requests and responses, across the entire RPC and NFS request, 
and leverages any file/storage system check sums without double
computation. On the other hand, just as with iSCSI digest, without
hardware acceleration, a general purpose checksum algorithm is unlikely to 
be
deployed in software, and unlike iSCSI, I doubt there will be offload 
engines
to accelerate NFSv4 checksums. So the compromise being expressed in the
i-d is pragmatic, if limited to certain local file systems and storage 
systems.

NFSv4 has no mount operation. My guess is that the intent is the server 
returns the
checksum info for the file system specified by the current file handle in 
the COMPOUND
request. The problem with that is that the client can cross file systme 
boundaries, and so
if the mounted file system starts off as say a ZFS file systems that use 
FLETCHER4 checksums,
the client could navigate into another file system type and not know that 
the algorithm
has changed. It would be better to make CKINFO a new, per-file system, 
attribute that GETATTR
can request.

CKSUM either precedes or proceeds the operation depending on whether it is 
a write or
read. I think it would be better to add PUT_CKSUM and GET_CKSUM operations 
so that it is
clearer. This would be in the NFSv4 spirit (re: PUTFH and GETFH). 
Note that this way, one could potentially
apply checksums to other operations besides READ and WRITE. It would also 
let one, for example,
checksum READ arguments, and receive checksums on READ results. 

CKSUM applies to the data arguments or results of READ or WRITE. It would 
be better if
there was a way to have PUT_CKSUM and GET_CKSUM indicate which parts of 
the
operation checksumming applies to. The rpc-rdma i-d has a similar problem
in indicating which parts of an rpc request or responsed are to transfered 
with RDMA.
It uses "chunk lists" to indicate this. Something similar could be done 
PUT_CKSUM and
GET_CKSUM.
 
That all stated, I wonder if it is too late to add this to the NFSv4.1 
feature list.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 10 Feb 2006 06:51:55 -0800 (PST)

> by some Intel Colleagues at the 2005 FAST conference may give an idea.

Julian,

That doesn't surprise me. But if my client or server already has zero copy,
and zero to light cost IP checksums (because the NIC checksums in bound
and out bound data, see http://people.freebsd.org/~jlemon/csum.txt) it doesn't 
help  because there's no shadow to leverage.

As iSCSI achieves world domination it would not be surprising if semi-dumb
NICs will emerge that do the iSCSI digest on in bound and out bound data.
And if such NICs emerge, NFS could take advantage of it if presented
an interface similar to what http://people.freebsd.org/~jlemon/csum.txt is.

---

From: Julian Satran <[email-masked]>
Date: Fri, 10 Feb 2006 17:03:53 +0200

Mike - you still have not covered the disk. As for the 0 copy - the newer 
RDMA mechanisms have left this to the transport - and that may be good 
enough if the transport embeds IPsec. Disk is not covered here also. I 
think that covering the consumer-2-consumer span including disk would be a 
big leap forward.
It looks like block storage is moving there and if NFS storage is to be at 
level with block it will have to handle the issue.
With the current structures it is a challenging though no unsolvable 
issue.

Julo

10/02/06 16:51
Please respond to
[email-masked]

To
Julian Satran/Haifa/IBM@IBMIL
cc
[email-masked]
Subject
Re: [nfsv4] Extensions to NFSv4 for Checksums

Julian,

That doesn't surprise me. But if my client or server already has zero 
copy,
and zero to light cost IP checksums (because the NIC checksums in bound
and out bound data, see http://people.freebsd.org/~jlemon/csum.txt) it 
doesn't 
help  because there's no shadow to leverage.

As iSCSI achieves world domination it would not be surprising if semi-dumb
NICs will emerge that do the iSCSI digest on in bound and out bound data.
And if such NICs emerge, NFS could take advantage of it if presented
an interface similar to what http://people.freebsd.org/~jlemon/csum.txt 
is.

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 10 Feb 2006 11:20:25 -0600

On Thu, Mike Eisler wrote:

<discussion of specific proposal removed>

> That all stated, I wonder if it is too late to add this to the NFSv4.1 feature list.

Just to be clear (and to solicit other input), I assume the above
is saying that this proposal should not be included in 4.1 given the
timing?

Regardless, I am sure that Alok would appreciate continued comments
to determine what form of this should be included in some minor version.

Spencer

---

From: Brent Callaghan <[email-masked]>
Date: Fri, 10 Feb 2006 11:14:54 -0800

NFS already has a framework you can leverage for additional data  
integrity: RPCSEC_GSS.

The draft makes reference to krb5i which is limited to Kerberos, but  
then goes
on to propose a scheme to protect only NFSv4 reads and writes.

An RPCSEC_GSS mechanism like krb5i protects the *entire* NFS  
transaction.
Bits can be flipped and cause havoc an any part of a compound call or  
reply,
so why not protect the entire message ?  If Kerberos deployment is  
perceived
as problematic, then it would be relatively simple to come up with a new
RPCSEC_GSS plug-in that provides a basic level of fast, non- 
cryptographic
protection without the overhead of authentication, key deployment etc.

RPCSEC_GSS and the SECINFO op already provide a way for client
and server to negotiate data integrity, and ultimately to defer  
automatically
to hardware based protection via CCM.

	Brent

On Feb 9, 2006, at 3:56 PM, Alok Aggarwal wrote:

> The standard TCP checksum is arguably weak in that it's not  
> resilient to certain single bit errors as well as data re- 
> arrangement. The standard Ethernet checksum is strong but is not  
> end-to-end and is recomputed at every hop. Given that the  
> underlying transports don't provide enough protection, the NFSv4  
> protocol (when running without krb5i) needs to provide it's own  
> data integrity mechanism.
>

---

From: Alok Aggarwal <[email-masked]>
Date: Fri, 10 Feb 2006 12:18:46 -0700 (MST)

On Thu, 9 Feb 2006, Mike Eisler wrote:

> has changed.

It is my expectation that as the client crosses file system boundaries,
it will recieve an NFS4ERR_WRONGALG via the CKSUM operation. It will
then have to go back and renegotiate the checksum algorithm preferences
via CKINFO. Sort of like what happens with the security negotiation
but not exactly the same.

> It would be better to make CKINFO a new, per-file system, attribute that GETATTR
> can request.

Since CKINFO won't map to a "real" file system attribute, there would need
to be a way to create a file handle for that attribute. And depending on
how the file handle is stored, the ckinfo file handle namespace would
need to be managed as well.

> It uses "chunk lists" to indicate this. Something similar could be done PUT_CKSUM and

I don't have a strong opinion about the naming of the operation. I'll take a 
look at what rpc over rdma has to offer in the way of applying checksumming
to parts of an operation.

> That all stated, I wonder if it is too late to add this to the NFSv4.1 feature list.

Like Spencer said, I'd like to solicit input on whether there's value in
adding this to 4.1

Alok

---

From: Eric Werme USG <[email-masked]>
Date: Fri, 10 Feb 2006 16:41:07 -0500

Typos:
it's -> its (for possessives, e.g. its list, not the contraction of "it is")
incase -> in case 

      If a server implementation supports finer grained checksumming
      such as per-file based checksums, the preferences may often be
      invalid.  In such cases, the CKSUM operation (See Section 3.4)
      will allow for the flexibility to enable per-file based checksums.

How would that work, and wouldn't that be called coarse grained?  Would the
READ or WRITE have to cover the entire file?

      On a READ or a WRITE, if the client believes it wants to read the
      data or write it regardless of the capabilities of the server, it
      may specify that by specifying the "none" algorithm in it's
      preferences.  The server should honour this by successfully
      reading or writing the data without checksumming.

Perhaps "without returning checksum data"?  You're implying that if the
server keeps checksum data as part of the file's metadata, then in this
case the server would not do its own checks.  (I assume a checksum error
found within the file system turns into an I/O error in all cases.)

Wouldn't it be simpler to allow clients to send READ and not follow it
with CKSUM?

	-Ric Werme

---

From: Alok Aggarwal <[email-masked]>
Date: Fri, 10 Feb 2006 16:53:32 -0700 (MST)

> it's -> its (for possessives, e.g. its list, not the contraction of "it is")

Thanks :)

> READ or WRITE have to cover the entire file?

If checksum preferences are set on a per file system basis, per-file
checksums (and checksum preferences) would probably be considered
finer grained. The READ/WRITE operations wouldn't necessarily have 
to cover the entire file, they can cover partial portions just like 
today. And, how an implementation converts entire file checksums into 
partial file checksums would depend upon the operating system capabilities.

> Wouldn't it be simpler to allow clients to send READ and not follow it

The client could optionally not follow the READ with a CKSUM. However,
it might find it useful to add a CKSUM with a READ with the side-effect
in mind that the server can indicate it's checksum algorithm preferences
in the results.

Alok

---

From: Mike Eisler <[email-masked]>
Date: Sat, 11 Feb 2006 10:33:49 -0800 (PST)

Julian,

What requirements are we trying to solve:

- corruption caused by storage devices
- corruption caused by processors (memory, bus, bad cpus,...)
- corruption caused by networks (wires, radio interference, switches, routers)

?

> RDMA mechanisms have left this to the transport - and that may be good 

NFS needs to run over non-RDMA mechanisms too. 

> enough if the transport embeds IPsec. Disk is not covered here also. I 

We've been hearing for years that ubiquitous IPsec is imminent.
That IPsec does this didn't prevent the IPS WG from making
a separate iSCSI checksum (digest) mandatory to implement (as well as
IPsec).

> It looks like block storage is moving there and if NFS storage is to be at 

So iSCSI is moving away from digest to block check sums?

> level with block it will have to handle the issue.

Why does NFS have to implement protection from disk corruption? Why is that
not a problem for the server's storage subsystem?  

I agree that if we implement protection from network corruption, and if the
storage system has disk corruption protection, then we ought to consider
leveraging it (but every vendor's storaage subsystem implements
its own checksum ... what are the odds we can agree on the checksum,
and what are the odds every  client will support all the
checksums?). But much of NFS, even more so in NFSv4, is
metadata: ACLs, string-based identifiers, lock state, session control, etc.
If that stuff is worth protecting, then the attention should be on
protecting all bits in the NFS.

If so, as Brent Callaghan has pointed out, we can easily create a GSS plug in that
does all this. That part is easy and obvious. The main reason why no one
has stated the obvious before is that checksums kill performance, unless
there is hardware to accelerate them. This is why no one uses
iSCSI digest. And if the ethernet NIC vendors haven't see fit to
accelerate iSCSI digest (yet, I hope, with idealism) I'm skeptical 
they'll accelerate NFSv4.1 digest. Especially if we pick one that is different
from iSCSI.

> With the current structures it is a challenging though no unsolvable 

I think Alok, you, and others need to draw the road map that gets us to
a zero or neglible cost checksum that covers all or a good subset of the NFS
traffic for all clients and servers.

---

From: Julian Satran <[email-masked]>
Date: Sun, 12 Feb 2006 08:50:46 +0200

Mike,

I think that corruption from disks and bad middleboxes has to be covered 
(the writer and reader machine are assumed OK - they may corrupt data in 
many other ways!).

SCSI - the block storage standards - are moving towards a end-2-end check 
(that will cover all transports and the media).
The SCSI stuff will presumably cover data from writer-memory to 
reader-memory.
Whetter this will make the transport checks obsolete is debatable.

NFS is in the enviable position in that it can do it  without regards to 
"legacy" and it may declare that transport is covered enough.
However it has to do it in a fashion that is usable for most applications 
- and IMHO this excludes a single checksum/file.
Any mechanism that will guarantee data correctness in the client 
block-cache would be fine. That may imply checksums available only for 
block aligned transfer at read and partial checksums for writes (provided 
the checksum function is a linear function).

Julo

11/02/06 20:33
Please respond to
[email-masked]

To
Julian Satran/Haifa/IBM@IBMIL, [email-masked]
cc
[email-masked]
Subject
Re: [nfsv4] Extensions to NFSv4 for Checksums

Julian,

- corruption caused by storage devices
- corruption caused by processors (memory, bus, bad cpus,...)
- corruption caused by networks (wires, radio interference, switches, 
routers)

?

> level with block it will have to handle the issue.

Why does NFS have to implement protection from disk corruption? Why is 
that
not a problem for the server's storage subsystem? 

I agree that if we implement protection from network corruption, and if 
the
storage system has disk corruption protection, then we ought to consider
leveraging it (but every vendor's storaage subsystem implements
its own checksum ... what are the odds we can agree on the checksum,
and what are the odds every  client will support all the
checksums?). But much of NFS, even more so in NFSv4, is
metadata: ACLs, string-based identifiers, lock state, session control, 
etc.
If that stuff is worth protecting, then the attention should be on
protecting all bits in the NFS.

If so, as Brent Callaghan has pointed out, we can easily create a GSS plug 
in that
does all this. That part is easy and obvious. The main reason why no one
has stated the obvious before is that checksums kill performance, unless
there is hardware to accelerate them. This is why no one uses
iSCSI digest. And if the ethernet NIC vendors haven't see fit to
accelerate iSCSI digest (yet, I hope, with idealism) I'm skeptical 
they'll accelerate NFSv4.1 digest. Especially if we pick one that is 
different
from iSCSI.

I think Alok, you, and others need to draw the road map that gets us to
a zero or neglible cost checksum that covers all or a good subset of the 
NFS
traffic for all clients and servers.

---

From: Alok Aggarwal <[email-masked]>
Date: Mon, 13 Feb 2006 19:51:55 -0700 (MST)

> - corruption caused by networks (wires, radio interference, switches, routers)

NFS started out by relying on the underlying transports
to gaurantee protection against network corruption but
they haven't held up well in all cases.

So, I believe, corruption caused by networks is what we're 
trying to protect NFS against. Corruption caused by the 
storage sub-system isn't something a network protocol (NFS) 
should be held accountable for IMHO - it's the responsibility
of the local file system/storage sub-system to make sure the 
data it wrote to the platter is the data it's serving up to it's 
consumers (eg: NFS).

However, if NFS, in any way, can integrate with the integrity
protection scheme used by the local file system/storage sub-system,
that would be a great goal to strive for. It would enable better
"end-to-end" integrity protection with the end being the NFS
client memory.

[ other rdma/iscsi comments elided ]

> not a problem for the server's storage subsystem?

I agree.

> checksums?).

We don't necessarily need to agree on what checksums a vendor's 
storage sub-system implements as long as we can agree on what
checksums go over-the-wire.

> But much of NFS, even more so in NFSv4, is
> does all this. That part is easy and obvious.

I agree that there's value in pursuing RPC level checksums -
there needs to be a proposal that looks into an RPCSEC_GSS plugin.

However, at the same time, there needs to be a way to enable a
more end-to-end checksum (NFS level checksum).

As an aside, I would like to make the observation that RPC level 
checksums are hard to integrate with the underlying file system/
storage sub-system. For example, say on a write, the server
would have to verify the checksum at the RPC level; recompute
the checksum for the write portion of the compound and send that
checksum down to the underlying storage sub-system alongwith the
write data. Checksum recomputation breaks down the end-to-end nature
of this scheme and it's vulnerable to loss of integrity.

> The main reason why no one

iSCSI started out with implementing Adler32 but they eventually
settled on CRC32. There's no reason why NFS can't pick CRC32 as well.

> traffic for all clients and servers.

I think, the performance overhead of checksums can be alleviated 
by making checksums optional to use based on the discretion of 
the end user. The security mechanism choice is very similar to this.

Alok

---

From: Mike Eisler <[email-masked]>
Date: Mon, 13 Feb 2006 19:46:09 -0800 (PST)

> checksums go over-the-wire.

OK, so any approach has to consider the case where there are no storage
sub-system checksums, and then how to do it without killing performance.

> more end-to-end checksum (NFS level checksum).

A GSS plugin that just does say, CRC32C, will cover all the RPC header plus
its arguments and results, which includes all of the NFS arguments and results.
How would that not be an end-to-end checksum?

> of this scheme and it's vulnerable to loss of integrity.

Chunking the data into multiple checksums the way RPC/RDMA chucks RPC
arguments and results would solve this. 

> the end user. The security mechanism choice is very similar to this.

In other words, as practically useful as krb5i and krb5p today. 

Or CCM for that matter. :-)

---

From: Julian Satran <[email-masked]>
Date: Tue, 14 Feb 2006 08:37:37 +0200

I don't see how the RPC CRC will cover the disk portion - it covers only 
the transport and assumes the storage is safe (which according to this 
thread is not allways true). Julo

14/02/06 05:46
Please respond to
[email-masked]

To
Alok Aggarwal <[email-masked]>, [email-masked]
cc

Subject

> more end-to-end checksum (NFS level checksum).

A GSS plugin that just does say, CRC32C, will cover all the RPC header 
plus
its arguments and results, which includes all of the NFS arguments and 
results.
How would that not be an end-to-end checksum?

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 14 Feb 2006 01:34:05 -0600

> thread is not allways true). Julo

Well, the filesystem on the fileserver is free to do its own integrity
protection.  I think Alok is saying that protection against faults
between the ONC RPC / NFSv4 layers and the filesystem is desirable;
end-to-end integrity protection where the ends are: client memory and
storage hardware on the fileserver.

Nico
--

---

From: Julian Satran <[email-masked]>
Date: Tue, 14 Feb 2006 09:56:33 +0200

The point that is apprently ignored is that you can't have end-2-end check 
made out of two disparate items (the server check and the transport 
check).
With a storage device end-2-end means client-2-client. I am aware that 
this is a difficult thing but just dismissing it won't help.

Julo

Nicolas Williams <[email-masked]> 
14/02/06 09:34

To
Julian Satran/Haifa/IBM@IBMIL
cc
[email-masked], [email-masked], Alok Aggarwal 
<[email-masked]>, [email-masked]
Subject
Re: [nfsv4] Extensions to NFSv4 for Checksums

Nico
--

---

From: Eric Werme USG <[email-masked]>
Date: Tue, 14 Feb 2006 08:04:09 -0500

With a little bit of cleverness that may extend more into the RPC/network
layer than we'd like, I think you could transmit the bad disk data with
the correct checksum.  (This includes disk data corrupted between DMA into
a buffer and buffer's transmission.)

The disk data comes with a precomputed checksum.  If new checksumming is
done over the RPC portion, using zeros for the disk data, that can be
combined with the disk data checksum to produce the complete checksum.

I think the term is "linear checksum", but only because I saw that in
the exchange.  You can certainly do this with the IP checksum, it's
trivial.  From what I recall from 30 years ago, I think you can do
essentially the same with CRCs.  If you think of the message as a long
stream of bits, it's just a big number, call it M. The CRC is just the
remainder of a modified division that does repeated XORs instead of
subtractions, so it's M % CRC.

If you take a RPC message that has a header (Mh), file data (Fd), and a
trailer (Mt) and take the bit lengths of them as Lh, Ld, and Lt, then
the computation is (Mh*2^(Ld+Lt) + Fd*2^Lt + Mt) % CRC.  So what
I'm saying is a checksum that lets you:

1) Compute Fd*2^Lt % CRC given just Fd % CRC from the file system
2) Compute (Mh*2^(Ld+Lt) + Mt) % CRC
3) Combine the two remainders to derive (Mh*2^(Ld+Lt) + Fd*2^Lt + Mt) % CRC

Will give you coverage all the way to the disk if you don't want the
file system to take time to check for errors.

I haven't read about the other checksums people have mentioned in the
spec or elsewhere, and I confess to never being super comfortable with
CRC's math, so feel free to let me know how many feet I have in my mouth.

Tru64 has some OS support for stuff like this where transmitted messages
have a chain of mbufs, each which may include the IP checksum for its
data.  That data is used instead of scanning the data to compute the
checksum.  That was done mainly to support incoming data from a NIC
with IP checksum computation, but it was easy (easier?) to do things
on the transmit side too.

	-Ric Werme

---

From: Benny Halevy <[email-masked]>
Date: Tue, 14 Feb 2006 17:09:52 +0200

I think that what Julo meant by end-to-end checksums that include the 
storage subsystem (or at least my interpretation of that) is checksums 
computed by the client over the transmitted data, sent over the wire, 
verified by the server, and finally stored persistently (without 
recomputation) along side the data.  On the read path, the checksums are 
read off of storage and returned to the client, again without being 
recomputed (possibly verified but not recomputed).  It is the 
responsibility of the client to verify these checksums against the data 
it received.

The main problem I see with such a method when checksums are opaque for 
the server is what to do when the client is doing non-aligned I/O .  If 
not opaque, and the server can recompute the checksums it could update 
them on partial-block rewrites (ideally by using only the stored 
checksum and the delta between the modified data and the stored data) 
and it can compute a checksum on a partial-block read (again, it better 
use the stored checksum and "subtract" from it the portion not sent over 
the wire, e.g. by reading the whole block and the checksum off the disk 
and sending a modified checksum equal to the checksum it would have 
recomputed if the client wrote zeros on the part of the block not being 
read).

Benny

---

From: Mike Eisler <[email-masked]>
Date: Tue, 14 Feb 2006 07:38:08 -0800 (PST)

> this is a difficult thing but just dismissing it won't help.

Debate won't be very productive until we have consensus on the requirements. E.g.
Alok and you apparently disagree. Anyway ...

Just as RPC/RDMA chunk-izes data, so can an RPC/NFS checksum. A given COMPOUND
consists of an RPC header (which if we use RPCSEC_GSS is checksummed separately
anyway), and arguments/results. The arguments/resuts are split into an array of
operations. Each operation in turn consists of a set of scalars and one or more
arrays of stuff. In the case of READ results and WRITE arguments, we have
an offset, length, and an array of data. A separate checksum can be calculated
on the data. The checksum on the data may not be the same algorithm as that
used in the storage subsystem, but that shouldn't matter. As long as the same
offset and length is checksummed over the network as is checksummed in
the storage subsystem we are fine. We are fine even if the offset and length are
different as long the network length is a multiple of the length of the
unit of data checksummed in the storage system, and similarly the alignment of
the network offset is a multiple of the alignment of the offset of the data checksummed
in the storage system.

The only other requirement is that the checksum algorithm C is defined such:

   C(X, Y) = f(C(X), C(Y))

where 
  X and Y are two consecutive octet patterns in a message,
  f is a function related to C such that f can combine the checksums of two
     messages, X and Y, and get the same result as C(X, Y). 

But all this ignores the basic reality: there's no practical way to do this
today. That's why it has never been proposed before. It isn't because
we are collectively ignorant of the weakness of IP checksums.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 14 Feb 2006 07:52:28 -0800 (PST)

> recomputation) along side the data.  On the read path, the checksums are 

We can't expect that clients are going to support 100s of different
storage subsystem checksums, especially if the checksum defined
for the transport is accelerated by the NIC. So the server will need
to re-calculate the checksum anyway. It is already doing this today,
so there's no performance impact. 

I think supporting existing storage server checksum algorithms is a non-requirement.

As long as the client offers a checksum with an offset and length compatible
with what server's storage subsystem uses we are fine. This isn't
a big requirement. Clients that issue I/Os with prime numbers for offsets
and length lose anyway.

---

From: David Robinson <[email-masked]>
Date: Tue, 14 Feb 2006 13:09:01 -0600

> responsibility of the client to verify these checksums against the data 

I think we have had a failure of terminology. Benny's description
here is what I think Alok has been talking about.  All of the
good studies that I have seen indicate that in order to
best maintain data integrity it is important the checksum
be created as close to the creation of the data (the client
write in the case of NFS) and verified as close to the usage
(the client read). Ideally it would be the application
before it makes the syscall, but in the case of Solaris we don't
have that API (yet).

In such a scheme it is not required that anything below client
actually validate the checksum. The server and underlying filesystem
can simply store and retrieve the checksum and data. Of course
failing to validate makes error recovery much harder, the more layers
of the stack that understand the checksums the more opportunities to
re-read or re-write the data, or acquire another copy off a mirror.

 From an architectural perspective it is preferable for the checksum
to be created/validated at or above the NFS layer and passed
through the stack to the storage (and back) unmodified. This
means that the data in the READ and WRITE operations need their
own checksums.  In OpenSolaris we just happen to have a filesystem
that is checksum aware, connecting the NFS client and server
together allows us to build a strong stack all the way to the
rotating rust without touching either the data or the checksum.
To put the checksum in a lower layer such as RPC, would require
the NFS client and server to recalculate checksums, introducing
the risk of bad code or bad hardware creating undetected
corruption (think how TCP packets get corrupted even though
the ethernet CRC is quite strong). Alternatives to educate the
RPC layer about the structure of an NFS operation could be done,
but that rototilling of the layers is distasteful.

Initially, all implementations will have weak stacks.  The
NFS server can validate the checksums, but without help
from the underlying filesystem it will simply throw them away
or generate them after it has the data in memory. But as
implementations get better, either the NFS server stores the
checksum in some auxillary place in the filesystem, or the
underlying filesystems become checksum aware, the NFS protocol
architecture remains invarient. I believe the greatest
risk of corruption today is the network, and while not
perfect and only piecemeal, the integrity checks in storage
(FCP or iSCSI) are much less risk than the weak TCP checksum.

> recomputed if the client wrote zeros on the part of the block not being 

The problem of trying to allow arbitrary sized transfers, including
different sizes for read and write, is very hard to solve. Given that
in the vast majority of cases the data is sent in aligned and
filesystem preferred sizes it doesn't seem worth the effort.
The attributes maxread/maxwrite are reasonable first order
estimates at a preferred size, adding an explicit preferred
checksum blocksize could be useful.

	-David

---

From: Mike Eisler <[email-masked]>
Date: Tue, 14 Feb 2006 11:36:01 -0800 (PST)

> can simply store and retrieve the checksum and data. Of course

It's a nice concept but as soon as some client does a mis-aligned write
or read, it means the server has to calculate a checksum.

> rotating rust without touching either the data or the checksum.

So you are advocating we use the ZFS checksum across the board? :-)

Anyway, I thought the server's file system need to only store the checksum.
Is the server's filesystem part of the picture or not?

> corruption (think how TCP packets get corrupted even though

No it doesn't add risk per what Ric and I have pointed out.

> (FCP or iSCSI) are much less risk than the weak TCP checksum.

If that's the greatest risk then why aren't you advocating
checksuming the entire packet? (If a corrupted write is bad, I
can imagine a corrupted SETATTR being worse for some scenarios.).

E.g. you could advocate SCTP which as I understand it mandates CRC32C. 
Still leaves the IP header vulnerable (though maybe
SCTP commits a layering violating and covers the IP header too).

> different sizes for read and write, is very hard to solve. Given that

Well it's going to happen. So if the server doesn't have to be involved
in the checksumming what then?

---

From: Julian Satran <[email-masked]>
Date: Tue, 14 Feb 2006 21:52:23 +0200

I have to admit that I am completely puzzled. The disk checksum is not 
computed by the disk - it is computed by the writer of the data (the 
client that did the write) and stored on the disk. The reader gets it 
back. This type of checksum protects the whole path from writer to reader. 
The function question is an orthogonal one. With linear function you can 
compute it for parts of the data the compound the result with ease.

Julo

14/02/06 15:04

To

   The point that is apprently ignored is that you can't have end-2-end 
check 
   made out of two disparate items (the server check and the transport 
   check).

1) Compute Fd*2^Lt % CRC given just Fd % CRC from the file system
2) Compute (Mh*2^(Ld+Lt) + Mt) % CRC
3) Combine the two remainders to derive (Mh*2^(Ld+Lt) + Fd*2^Lt + Mt) % 
CRC

                 -Ric Werme

---

From: Julian Satran <[email-masked]>
Date: Tue, 14 Feb 2006 22:02:09 +0200

David - You are basically right. Benny just added that the write checksum 
can be verified by the server to avoid writing data possibly already 
corrupted in flight and thus make recovery easier. I don't see a 
contradiction here.

Julo

David Robinson <[email-masked]> 
Sent by: [email-masked]
14/02/06 21:09

To
[email-masked]
cc
Alok Aggarwal <[email-masked]>
Subject
Re: [nfsv4] Extensions to NFSv4 for Checksums

                 -David

---

From: David Robinson <[email-masked]>
Date: Tue, 14 Feb 2006 14:02:18 -0600

> or read, it means the server has to calculate a checksum.

Which just doesn't happen very often! So I am not worried about
a slight added risk for clients who want to do odd i/o's.

> So you are advocating we use the ZFS checksum across the board? :-)

Well there is no "ZFS checksum", it just supports multiple different
standard algorithms. Alok's proposal is designed to allow
different algorithms.

> Is the server's filesystem part of the picture or not?

The server filesystem only needs to store the checksum, but as I
said, if it also validates the checksum it can make error recovery
faster and more efficent. It is easier to re-read a disk block
from the server than to force the client to re-drive a request.
But that is just an optimization.

> No it doesn't add risk per what Ric and I have pointed out.

I am not much of a fan of intermixing the layers.

> can imagine a corrupted SETATTR being worse for some scenarios.).

I have nothing against an RPC level integrity check, and in
fact will suggest to anyone who asks to run krb5i. And I will
support a new mechanism that is integrity-only. But I simply
find requiring the NFS layer to do complex things to extract
out a checksum from RPC to be bad protocol design.

> SCTP commits a layering violating and covers the IP header too).

You must have heard my RPC/SCTP rants... why-o-why haven't we done
that obvious definition? More surprisingly is why the AD's let us
get away with NFSv4 without it....

> in the checksumming what then?

Doesn't have to be involved is different than won't be involved.
The expectation is that the server software will need to make
a decision if it will do a recalculation or simply ignore
the checksum. Of course, having a mechanism to tell the client
that a particular blocksize is not only preferred, but required,
would be useful.

	-David

---

From: Mike Eisler <[email-masked]>
Date: Tue, 14 Feb 2006 12:33:09 -0800 (PST)

> a slight added risk for clients who want to do odd i/o's.

Then over time, all files will have checksum holes, so I question the
utility.

> Well there is no "ZFS checksum", it just supports multiple different

Sure, the fact that juxtaposition of zfs and 
Alok's prime choice of checksum generate
lots of search engine hits is just a bug in google. :-)

> standard algorithms. Alok's proposal is designed to allow

So if client A wants to use algorithm X, and B wants to use algorithm Y,
what happens when A and B want to whack away on the same file?

> But that is just an optimization.

Sometimes I'm forced to admit the existence of other file access protocols
like NFSv3 and CIFS. File sharing among those protocols is highly likely.
For that reason alone (in addition the others) it
isn't realistic to suggest that a server can avoid being part of
checksumming picture.

> I am not much of a fan of intermixing the layers.

Not an RPC/RDMA fan either I reckon. :-)

Anyway, NICs have been calculating IP checksums for years now. You may
not be a fan of it, but that layering violation (the link layer doing
something for the transport layer) is what makes IP checksums
practical. Lack of such support is what makes iSCSI digest impractical.

> fact will suggest to anyone who asks to run krb5i. And I will

You suggest to people that they reduce their performance and raise their
CPU utilization to unacceptable levels? 

krb5i isn't practical. That's why CCM was invented. As it turns out, CCM
isn't practical (at least yet), because no one can build 10 giga bit
per second in-line IPsec solution for AES or 3DES, especially one
that doesn't require re-doing a data center's power system. IPsec
causes global climate change. :-)

That's why you aren't actually telling people to run krb5i.

My guess is that it would be practical for NICs to do CRC32C and
commit some layering violations. But the NIC designers don't seem
inclined to produce one. Maybe if there were more consumers they
would. 

> get away with NFSv4 without it....

Sure why do the ADs do anuthing before doing something about
IP, UDP, and TCP checksums? If CRC32C were in the IP and TCP layer
where it belongs, there'd be a rush to add CRC32C assist to NICs.

But it is a pointful as asking why a loving god allows evil (and no I
don't want answers to that question either).

Anyway, I think there needs to be consensus on what problem
we are trying to solve. What are the requirements? Everyone advocating
we do something in this space has different requirements.

---

From: Eric Werme USG <[email-masked]>
Date: Tue, 14 Feb 2006 16:02:31 -0500

I have to admit that I am completely puzzled.

Top-posting certainly doesn't help matters much.

   The disk checksum is not 
   computed by the disk - it is computed by the writer of the data (the 
   client that did the write) and stored on the disk.

Are you referring to my "The disk data comes with a precomputed checksum."?
I was trying to be generic.  A more specific statement would be that file
data comes from ZFS comes with a precomputed checksum that was recorded as
metadata when the file data was written.  I was thinking about the server
reading a file and was ignoring how the data and checksums were written.

What you are saying seems wrong to me.  You're saying that a client sends
write data along with a checksum and that the server records it on disk.
That implies the server isn't bothering to check the checksum, but is writing
what it's given.  That will lead to all sorts of fun when something finds
a bad checksum!  It also implies that the client writes data in the preferred
size which seems rather implausible/inflexible.

By checksumming the whole RPC message, then RPC decoding can check it
and NFS can soon resum the file data on its own terms.  If that window
is too large, I suppose RPC or NFS could later redo the message
checksum just to make sure nothing was corrupted while the file system
was checksumming the file data, picking up offsets, file handles, etc.
Overkill, I hope, but if you can't be too careful....

   The reader gets it 

My apologies for any awkward formalism in my language.  I'm not a computer
scientist.  What I tryed to convey fits in with Mike's equation, except
I was using three fragments to Mike's two.  Easy axiomatic extension.

   Julo

   
   To

   

   

   

   
   
                    -Ric Werme
-- 
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on Usenet and in e-mail?

---

From: David Robinson <[email-masked]>
Date: Tue, 14 Feb 2006 15:11:28 -0600

> Then over time, all files will have checksum holes, so I question the

If a vendor writes a server that is plays stupid, and only passes
the checksums, it will have to discard or generate no checksums.
Strictly conforming, but like with many other things in NFSv4, not
a very commercially viable product. Think of a server that only
implements mandatory attributes, functional but uninteresting.

The utility is that any sane vendor will do the extra bit of
work to recompute the checksums, accepting the added exposure.
Better vendors will make sure that their memory has double ECC
or other assurances and be very careful with this code.

> lots of search engine hits is just a bug in google. :-)

Not speaking for Alok, but if the WG did not declare his preferred
algorithm as mandatory to implement, I suspect he would still
implement it in Solaris and make it preferred. :-)

> what happens when A and B want to whack away on the same file?

Alok will have to jump in here on the details. But I thought I heard
him say that a file should have one true checksum (say X). But if the
server also supports and offers Y, it will have to do a transformation
and the associated risk. That said, a server shouldn't offer multiple
checksums unless it is confident in its code that the added risk is
minimal. The alternative is to just allow only X and let B not use
a checksum, which would clearly be even riskier.

> isn't realistic to suggest that a server can avoid being part of

Right.  There is theory and there is reality. A quality implementation
will have the server validating checksums. But a cheap NFSv4 only
appliance should not be required to, it just needs to store them.

> Not an RPC/RDMA fan either I reckon. :-)

So how much money has Netapp made off RPC/RDMA and/or DAFS?

> practical. Lack of such support is what makes iSCSI digest impractical.

Checksumming is just not that expensive.  Modern high clock rate,
multi-core, multi-thread cpus are here today and getting cheaper.
I disagree that iSCSI digest is impractical, it is like saying 10G
Ethernet is impractical. It may not be as cheap as we like today,
but the technology curve clearly shows it will be soon.  Internally,
our experience with ZFS checksums shows that the added overhead is
not significant, and getting less so. CPUs are just too fast to
matter anymore.

	-David

---

From: Mike Eisler <[email-masked]>
Date: Tue, 14 Feb 2006 13:25:34 -0800 (PST)

> size which seems rather implausible/inflexible.

Yes, I quite agree. The intent appears to make detection of corrupted
ranges (blocks) a client problem. That's not good enough.

> is saying that this proposal should not be included in 4.1 given the

Unless you want to alter the schedule for v4.1, it should not appear. What
is there needs to be better baked. Requirements need to be put forth and
agreed upon. Ric's insight alone justifies deferring this.

---

From: Alok Aggarwal <[email-masked]>
Date: Tue, 14 Feb 2006 14:37:56 -0700 (MST)

> a checksum, which would clearly be even riskier.

If the server is collaborating with the underlying file system,
then the files in that file system will be checksummed using
the native checksum algorithm (say X) - thus mandating the server 
advertise X as the supported algorithm for that particular fs.
Now if the server's capable enough to do transform checksum 
algorithm X into Y, it can advertise Y as well.

Once the client determines what the server supports for the given
file system, it can decide if it can find a union of it's own
checksum capabilities and the servers. And, if it can't agree on
a common checksum algorithm, it requests data as before - without
any checksum.

Alok

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 14 Feb 2006 15:41:13 -0600

> Top-posting certainly doesn't help matters much.

Not using normal quoting rules (i.e., prepend quoted text with '> ')
doesn't either :(

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 14 Feb 2006 15:44:40 -0600

>      messages, X and Y, and get the same result as C(X, Y). 

I think this is what in the world of rsync is termed a "rolling"
checksum.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 14 Feb 2006 13:45:42 -0800 (PST)

> implements mandatory attributes, functional but uninteresting.

NFSv4.x is getting more complex. It's one thing to give a server
or client a pass on block checksums, it is another to let them
do stupid things.

> implement it in Solaris and make it preferred. :-)

So much for meaningful interoperability.

> a checksum, which would clearly be even riskier.

Way too complex.

> So how much money has Netapp made off RPC/RDMA and/or DAFS?

You'll have dig into NetApp's annual report. But it is interesting
that you mention DAFS and per client/server pair checksums in the
same email message.

> Checksumming is just not that expensive.  Modern high clock rate,

Show me your iSCSI numbers with and without digest.

Show me your TCP/IP numbers with NIC-assist checksum off, and software 
checksum on or off.

If even 16 bit IP checksums were cheap, NICs wouldn't do them.

> not significant, and getting less so. CPUs are just too fast to

The file system's checksum cost 
shouldn't be significant if is it calculating
the checksum before writing to disk, and 
verifying it after read from disk. Disk wait eats
most of the latency. So I wouldn't expect checksums done
at those times to impact streaming throughput.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 14 Feb 2006 17:23:13 -0500

> size which seems rather implausible/inflexible.

The other point to be made is  that NFS does not have a notion of a
block. If the client wants to write two bytes across a disk block
boundary, the protocol is perfectly happy to allow it to do so.

Furthermore, there is the point that Mike made about interoperability
with other protocols (NFSv2, NFSv3, CIFS,...).

In such an environment, relying on client-computed checksums is
impractical at best.

Cheers,
  Trond

---

From: Julian Satran <[email-masked]>
Date: Wed, 15 Feb 2006 11:01:55 +0200

Mike,

You have good technical points but you don't seem to claim that they have 
no solution.
And some of them have already while for others we might have some fun 
trying to find solutions.
CRC32C in software is neither expensive nor unpractical - and can be done 
"in the shadow of the data copy operation" (see the paper on stripe CRC at 
the recent FAST conference).

Also most clients have a regular block cache. I think it would be 
acceptable to have the checksums only for files on which  all accesses are 
aligned to some server specified block size and have them switched off as 
soon as a client has an unaligned write (are appends an issue?).

The main thing to keep in mind is that block storage (SCSI) is going to 
have an end-2-end check and it seem to me unacceptable not to have the 
same level of protection for file storage.

Julo

14/02/06 22:33
Please respond to
[email-masked]

To
David Robinson <[email-masked]>, [email-masked]
cc
Alok Aggarwal <[email-masked]>
Subject
Re: [nfsv4] Extensions to NFSv4 for Checksums

---

From: Julian Satran <[email-masked]>
Date: Wed, 15 Feb 2006 13:12:07 +0200

comments in text

> size which seems rather implausible/inflexible.

As Benny has already stated the server may check the checksum and reject 
the write
Whether you make this behavior mandatory or not is a matter of discussion.
If you replace the checksum with an "authenticity proving digest (that 
includes a secret key)" then the server can't do anything but you have 
gained an authenticity check. I assume that this can be a legitimate 
alternative behavior.
 
> Overkill, I hope, but if you can't be too careful....

Redoing the checksum somewhere other than the client is defeating the 
purpose.
Remember the client does not trust the correctnes of anything in the 
chain.

---

From: Julian Satran <[email-masked]>
Date: Wed, 15 Feb 2006 14:39:58 +0200

> > a checksum, which would clearly be even riskier.

A selection "is way too complex" ?

> Show me your iSCSI numbers with and without digest.

The numbers are in the paper I quoted and others - 3.7Gbs on iSCSI with a 
1.7 Ghz pentium (I might be wrong)

> If even 16 bit IP checksums were cheap, NICs wouldn't do them.

NIC do it because it is relatively easy and helps them sell to "believers" 
(it sales lingo it is called "differentiation").

> at those times to impact streaming throughput.

I would not sate it this way since I/O is asynchronous. But checksum 
computation
while copying from buffer-to-buffer is free (most of the time is consumed 
by CPU cache misses so if you do sequence of "moves" or a sequence of 
"mov,op,op.." it'll eat up the same number of cycles (at least on a single 
threaded machine).

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 15 Feb 2006 10:46:42 -0600

> Remember the client does not trust the correctnes of anything in the 

Well, no, the client has to trust the server for some things.

E.g., the client has to trus the server when it comes to metat-data as
Alok's proposal doesn't, and can't address client checksumming of
filesystem meta-data.  That's because while file data is very simple
(octet strings), filesystem metat-data varies greatly, and there's no
way to canonically organize all the meta-data using NFSv4 types as the
bases either.

And the client has to trust that the server actually, you know, *writes*
the data somewhere it can be read from (as opposed to the bit bucket).

So, if the server has to redo checksums that's fine, as long as the
server, in so doing, can use the client's checksums to detect memory
errors on the server-side.

Nico
--

---

From: Julian Satran <[email-masked]>
Date: Wed, 15 Feb 2006 18:52:12 +0200

> errors on the server-side.

I agree

---

From: Mike Eisler <[email-masked]>
Date: Wed, 15 Feb 2006 09:01:48 -0800 (PST)

> A selection "is way too complex" ?

Yes. Too many choices makes it harder to test.

> 1.7 Ghz pentium (I might be wrong)

I looked at the paper. It talks about a user-level sandbox iSCSI implementation.
It mentions that without their optimizations to CRC32C, the checksum calcultation
is most of the processor cost. It also mentions there is a copy. I didn't
see any discussion about how their algorithm helps in zero copy implementations.
I didn't see any numbers that compared performance with CRC32C turn on and off.
I didn't see any discussion of the target's storage subsystem. 

> (it sales lingo it is called "differentiation").

Well I'm a believer. Apparently so are many other "rubes".
As for differentiation, most of them are doing it. But if they are
useless, show me the numbers -- on a non-sandbox.

> I would not sate it this way since I/O is asynchronous. But checksum 

The read and write system call in POSIX is synchronous. There are
clients that use async I/O, but no hints in the NFSv4 protocol to indicate
that. NFS servers and the file system and storage system they use may have
asynchrnous interfaces between the layers, but knowing that the application
on the client is likely in a synchronous wait, I/O latencies matter.

> "mov,op,op.." it'll eat up the same number of cycles (at least on a single 

Except on zero copy NFS and iSCSI servers.

> You have good technical points but you don't seem to claim that they have 

I thought I was clear on that. I'm claiming the solution is not fully cooked,
probably because the requirements aren't listed. It is difficult to evaluate
a solution when the requirements are either not listed or the advocates for the
solution can't agree on the requirements.

> "in the shadow of the data copy operation" (see the paper on stripe CRC at 

Can't have a shadow if there is no copy.

> soon as a client has an unaligned write (are appends an issue?).

Not for the proposals that make responsibility for
maintaining the block checksms only on the client.

> same level of protection for file storage.

Well NFS doesn't have it today and it seems to be accepted.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 15 Feb 2006 20:50:40 +0200

>the data somewhere it can be read from (as opposed to the bit bucket).
A rather infamous data corruption scenario happens when data is written 
correctly to disk, just not at the right location (due to e.g. 
corruption of the LBA on Parallel ATA wire which does not protect 
register transfers over the wire) .  Simple block checksums on storage 
will not detect that if the checksum is written atomically with the data 
at the wrong place.

Not referring specifically to the example above, does it seem realistic 
to provide a mechanism that will allow writing and reading data out of a 
subfile atomically with the main file's data.
With such a mechanism, a client could write any checksum it likes to 
stable storage atomically with the data and verify it when the data and 
subfile data are read back.  The checksum in this case could even be 
application specific if the client OS allows that and can take into 
account file metadata like a file identifier, data generation number, 
offset, etc. to satisfy the most paranoid apps.

Benny

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 15 Feb 2006 13:03:59 -0600

> will not detect that if the checksum is written atomically with the data 

That's why ZFS doesn't store data and checksums logically together;
instead it stores checksums and block *references* together (block
pointers include a block number and the checksum of that block's data).

This allows corruption like you describe to be detected.  Of course,
without redundancy (mirroring, RAID-5 (RAID-Z!)) you can't recover from
such errors.

(Some blocks in ZFS are self-checksumming, but that's another story and
not relevant here.)

> subfile atomically with the main file's data.

"Realistic"?  Sure -- if it can be implemented and if there's demand for
it, then it's realistic.  But your premise is incorrect.  Just because
the data and checksums are sent together doesn't mean that the scenario
you describe above is possible.

> offset, etc. to satisfy the most paranoid apps.

Yes, applications can do this today.

Since Alok's proposal can only provide end-to-end integrity protection
for data, but not meta-data, it's a fair question to ask what apps will
need this and why not let them do it directly, but I can't address it
from where I stand.

Nico
--

---

From: Benny Halevy <[email-masked]>
Date: Wed, 15 Feb 2006 21:07:42 +0200

Benny Halevy wrote:

 > Not referring specifically to the example above, does it seem 
realistic to provide a mechanism
 > that will allow writing and reading data out of a subfile atomically 
with the main file's data.
 > With such a mechanism, a client could write any checksum it likes to 
stable storage
 > atomically with the data and verify it when the data and subfile data 
are read back.  The
 > checksum in this case could even be application specific if the 
client OS allows that and can
 > take into account file metadata like a file identifier, data 
generation number, offset, etc. to
 > satisfy the most paranoid apps.

I just wanted to add that this could be implemented, maybe rather 
inefficiently, using byte range locks and a named attribute to store 
checksums.

By the way, many moons ago there was a discussion that didn't end 
anywhere about introducing a "current stateid" concept (e.g with a 
special stateid value that will refer to the last stateid returned in 
the same compound RPC).  With such a mechanism one could modify or read 
data atomically with a namedattr in the same compound.  Is there still 
interest in trying to add this to the protocol?

Benny

>>> Redoing the checksum somewhere other than the client is defeating 
>>> the purpose.
> A rather infamous data corruption scenario happens when data is 
> written correctly to disk, just not at the right location (due to e.g. 
> will not detect that if the checksum is written atomically with the 
> data at the wrong place.
>
> Not referring specifically to the example above, does it seem 
> realistic to provide a mechanism that will allow writing and reading 
> data out of a subfile atomically with the main file's data.
> stable storage atomically with the data and verify it when the data 
> and subfile data are read back.  The checksum in this case could even 
> be application specific if the client OS allows that and can take into

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 15 Feb 2006 14:34:28 -0600

> interest in trying to add this to the protocol?

Yes, I have a AI to raise this feature item again on the WG and 
drive consensus.  You have done the first part. :-)

Spencer

---

From: David Robinson <[email-masked]>
Date: Wed, 15 Feb 2006 14:44:18 -0600

On Feb 15, 2006, at 1:07 PM, Benny Halevy wrote:
> special stateid value that will refer to the last stateid returned  
> in the same compound RPC).  With such a mechanism one could modify  
> or read data atomically with a namedattr in the same compound.  Is  
> there still interest in trying to add this to the protocol?

Spencer has been beating me up on summarizing where we
were on that proposal.

Key question:  Do we want to go forward with this? In 4.1?

My I-D has expired, but I can resubmit it if need be.

The previous questions could be summarized as:

1) What about stateid hierarchy and ordering?

    In the only case where two stateid's can be returned (OPEN)
    the next usage with a special stateid will always be unambiguous
    which stateid to use. Ops requiring a delegation stateid should
    use it, all others use the open stateid. Brought up the interesting
    use case where the client could follow an OPEN with a
    DELEGRETURN in the same COMPOUND to efficiently always reject
    delegations.

2) The wording in section 2 that operations that do not return
    a stateid MUST zero the current stateid is overly restrictive.

    Yes, that is just bad language.  The intent was that any operation
    that changes the current filehandle without returning a stateid
    (e.g. PUTFH) MUST zero the current stateid. Other ops like
    GETATTR should not affect it.

Other than those clarifications, and some minor wording issues,
there were no serious objections.

	-David

---

From: Julian Satran <[email-masked]>
Date: Thu, 16 Feb 2006 10:25:55 +0200

Mike,

If you have a 0 copy solution (and I don't know about any commercially 
available one for NFS) you may wont to add your 6000 gates for the CRC to 
it (that is a parallel CRC32C for 10Gbs).

As for the rest of your comments - I wish I could be so confident about 
NFS's acceptance when:

there is strong evidence of undetected errors in the network
there is evidence of  disk errors causing data corruption
block storage has it

Enterprise class solution will be soon requiring something more than 
handwaving.
And it appears that at least the zFS architect saw that coming.

Julo

15/02/06 19:01
Please respond to
[email-masked]

To

> 1.7 Ghz pentium (I might be wrong)

I looked at the paper. It talks about a user-level sandbox iSCSI 
implementation.
It mentions that without their optimizations to CRC32C, the checksum 
calcultation
is most of the processor cost. It also mentions there is a copy. I didn't
see any discussion about how their algorithm helps in zero copy 
implementations.
I didn't see any numbers that compared performance with CRC32C turn on and 
off.
I didn't see any discussion of the target's storage subsystem. 

asynchrnous interfaces between the layers, but knowing that the 
application

> You have good technical points but you don't seem to claim that they 
have 

I thought I was clear on that. I'm claiming the solution is not fully 
cooked,
probably because the requirements aren't listed. It is difficult to 
evaluate
a solution when the requirements are either not listed or the advocates 
for the
solution can't agree on the requirements.

---

From: Julian Satran <[email-masked]>
Date: Thu, 16 Feb 2006 12:36:03 +0200

Thinking more about the metadata - I think that there are at least to 
categories - location and other internal metadata and user-defined 
metadata.
For the first - as those are not revealed to the client and/or not created 
by the client  they probably have to be taken care by the server at his 
own discretion.
The second category including user created metadata (extended attributes 
etc.) is probably another form of data and may benefit from protection the 
same way data does.

And again - if selection of the checksum is so onerous (to testing) - if 
you are really doing it end-2-end the exact checksum formula and its 
ingredients don't have to be negotiated - only the checksum size. 
Obviously then you loose the ability to eliminate corrupted data before 
they are written.

Julo

15/02/06 21:03

To
Benny Halevy <[email-masked]>
cc
Julian Satran/Haifa/IBM@IBMIL, [email-masked], Eric Werme USG 
<[email-masked]>, Alok Aggarwal <[email-masked]>, [email-masked]
Subject
Re: [nfsv4] Extensions to NFSv4 for Checksums

Nico
--