Subject: [nfsv4] Updated: checksum extensions I-D

From: Alok Aggarwal <[email-masked]>
Date: Mon, 15 May 2006 16:14:42 -0600 (MDT)

I've submitted an updated I-D for extensions to NFSv4 for Checksums
that can be found here -

http://www.ietf.org/internet-drafts/draft-aggarwal-nfsv4-cksum-01.txt

As per the discussions we had on the initial version of the draft, I've
updated the I-D to include a brief "Requirements" section for the work
and have also made corrections based on feedback recieved.

Discussion and suggestions for improvement are requested.

Thanks,
Alok

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Brent Callaghan <[email-masked]>
Date: Mon, 15 May 2006 17:27:10 -0700

The new requirements section adds little to justify the addition of
further complexity to what is already a very complex protocol.

By far, the most common challenge to data integrity is the
network - and krb5i already provides good protection for file
data, as well as the entire NFS call and reply.  What's more, it's
already available.  And ZFS can provide good protection
into and out of the server storage system.

While checksum extensions might provide some incremental
protection, the ultimate in end-to-end data protection is already
in use by applications themselves, e.g. archive checksums
and database record checksums.

Some discussion in the requirements that quantifies the extra level
of data protection provided by these checksum extensions would
make it easier to decide whether they make the protocol better,
or just more complex.

For instance, by how much do checksum extensions increase the
protection provided by krb5i + ZFS ?

	Brent

---

From: Julian Satran <[email-masked]>
Date: Mon, 15 May 2006 21:40:29 -0400

Brent,

Are you purposefully ignoring all previous discussion on the subject?
I think that some arguments for the client-2-client integrity where made 
and you do not even acknowledge them being made.
I find this "erasing of the past discussion" distasteful.

kerb5+zFS do not equal zFS storing clent generated checksums (that might 
include also keyed criptographic digests) and are
not relying on server (nor can they be affected by it). 

Julo

Brent Callaghan <[email-masked]> 
15/05/06 20:27

To
[email-masked]
cc

Subject
Re: [nfsv4] Updated: checksum extensions I-D

                 Brent

---

From: Brent Callaghan <[email-masked]>
Date: Mon, 15 May 2006 19:11:21 -0700

On May 15, 2006, at 6:40 PM, Julian Satran wrote:

> I think that some arguments for the client-2-client integrity where  
> made and you do not even acknowledge them being made.

I'm erasing nothing - my comment is specific to the requirements the
proposal addresses.

>
> kerb5+zFS do not equal zFS storing clent generated checksums (that  
> might include also keyed criptographic digests) and are

Sorry if I didn't make it clear.  If krb5i is used, then data on the  
network
is protected.  If ZFS is used, then data on the storage network and  
media
are protected.  The checksum extension would provide additional  
protection
only while the data are transiting server memory.  So my question was:
how much extra protection is that ?

	Brent

---

From: Julian Satran <[email-masked]>
Date: Mon, 15 May 2006 22:42:05 -0400

Sorry I did not make mysef clear _ the previous discussion clarified the 
end-to-end nature required for a client checksum - that does not rely on 
the server

Julo

---

From: [email-masked]
Date: Tue, 16 May 2006 08:22:06 -0400

I agree with Brent's question: Is the resulting extra protection worth
the effort ?
- mw

	Sorry I did not make mysef clear _ the previous discussion
clarified the end-to-end nature required for a client checksum - that
does not rely on the server 
	
	Julo

---

From: Julian Satran <[email-masked]>
Date: Tue, 16 May 2006 09:17:47 -0400

For many that see an end-2-end (client-2-client) protection as the only 
one good  in a service world (in which the NFS server is not necessarily 
trusted) it is definitely the only mechanism they can rely on (I would not 
qualify it as "extra protection").

julo

[email-masked] 
16/05/06 08:22

To
Julian Satran/Haifa/IBM@IBMIL, <[email-masked]>
cc
<[email-masked]>
Subject
RE: [nfsv4] Updated: checksum extensions I-D

I agree with Brent's question: Is the resulting extra protection worth the 
effort ?
- mw

Julo

---

From: Mike Eisler <[email-masked]>
Date: Tue, 16 May 2006 07:13:06 -0700 (PDT)

> Are you purposefully ignoring all previous discussion on the subject?

We could ask you and Alok the same question, but that would be impolite. 
Still, many holes and issues were pointed out, and they've yet to be addressed.

The requirements section of this i-d says:

3.  Requirements

   One of the key requirements for NFSv4 checksums is to protect the
   NFSv4 payload against network corruption occuring due to deficiencies
   in the underlying transports.

Brent has pointed out recently and in the past, 
respectively, that krb5i specifically,
and RPCSEC_GSS in general meets this requirement.

And as has been pointed out several times, 
the i-d doesn't even address the above
requirement. Indeed the RFC, before the 
requirements are listed, says:

   Checksums for NFSv4 might be implemented for the entire NFS payload
   or a part of the payload.  This document proposes extensions to NFSv4
   that might provide for a way to implement checksums for the READ/
   WRITE data portion of NFSv4 payload.

Back to the requirements ...

   As part of providing data integrity, it would be ideal if such a
   scheme would allow for extending the protection domain from as close
   as possible to where the data is generated, i.e. the application on
   the client; to as close as possible to where its stored, i.e. the
   storage on the server.  Since end-to-end integrity is undeniably a
   hard problem to solve, it would be a worthwhile to have the initial
   specification allow for extensions such that end-to-end integrity can
   be accomplished as part of future work.

And the i-d doesn't support that either. 

   Finally, any data integrity scheme should allow for providing the
   flexibility to choose from a set of checksum algorithms.

What about the requirement that this stuff perform? (I noted the 
existence of zero copy NFS stacks.) What about the issue of client generating
writes or issuing reads that are smaller than the server's local file system
checksum unit? Just to mention a couple issues that have been brought
up before that aren't addressed.

> I find this "erasing of the past discussion" distasteful.

Perhaps if the requirements captured and explained that discussion
it would be easier to avoid erasure, inadvertent or otherwise. 
Besides which IETF's mail archiving system is useless, and expecting
skeptics to wade through the archives is a non starter. The onus is
on the champions of this idea, not on those who are skeptical of it.

> not relying on server (nor can they be affected by it). 

The i-d says that the server generates the checksum on READ.

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 16 May 2006 09:38:53 -0500

> qualify it as "extra protection").

It's protection against otherwise potentially undetectable
memory/CPU/bus errors on the client, from the point the checksum is
taken (but *no earlier*), and on the server/storage.

This protection applies only to data, not metadata, and does not
entirely close the window of vulnerability to such errors, just mostly.

I can see how someone might like to argue that these sorts of errors are
vanishingly rare, and I can see why they might ask for more data on
this.

It's a fair question, IMO.  The I-D should include some informative
references to such data/studies.

Nico
--

---

From: Julian Satran <[email-masked]>
Date: Tue, 16 May 2006 10:51:47 -0400

Mike,

That is better discussion. The ID fails short (now) on many of the things 
I would like to see (and I would come soon with a complete review).
Nonetheless I was upset by the outright rejection of the idea.

julo

Mike Eisler <[email-masked]> 
16/05/06 10:13
Please respond to
[email-masked]

To
[email-masked]
cc

Subject

We could ask you and Alok the same question, but that would be impolite. 
Still, many holes and issues were pointed out, and they've yet to be 
addressed.

3.  Requirements

What about the requirement that this stuff perform? (I noted the 
existence of zero copy NFS stacks.) What about the issue of client 
generating
writes or issuing reads that are smaller than the server's local file 
system
checksum unit? Just to mention a couple issues that have been brought
up before that aren't addressed.

---

From: Alok Aggarwal <[email-masked]>
Date: Tue, 16 May 2006 09:42:29 -0600 (MDT)

> and RPCSEC_GSS in general meets this requirement.

And the "Introduction" section of the I-D accepts that
right? -

    The standard NFS protocol (without Kerberos) has historically relied
    on the underlying transport for data integrity and it itself does not
    make an attempt to ensure data integrity.  Kerberised NFS provides
    data integrity via the use of krb5i.

Really, the use case we're considering is protection in
the absense of RPCSEC_GSS, and absense of RPCSEC_GSS because
security mechanisms like Kerberos are hard to configure.

>   WRITE data portion of NFSv4 payload.

I'm not denying the need to protect the entire payload,
this is *just* a first step.

You've previously pointed out how this might be accomplished
with an RPC layer mechanism that uses RDMA-like chunking -
a proposal to outline such a mechanism would be good.

> And the i-d doesn't support that either.

It accepts that true end-to-end integrity is a hard problem 
to solve; thus, if we can even come close to making the situation 
better than it is today, i.e. no protection without RPCSEC_GSS/krb5i,
it would be good. Are you saying you'd rather wait for a 
*perfect* proposal?

> existence of zero copy NFS stacks.)

The choice of checksum algorithm will be one of the key components
that determines how this functionality performs, correct? So, if
the proposal can be flexible with the choice of algorithms, we
can address the performance requirement.

> What about the issue of client generating

The client generated checksum won't travel all the way to the
server's local file system in such a case. That is, the server
will have to bear the overhead of recomputing the checksum and
will have to pay a performance penalty.

But, given that mis-aligned writes don't happen that much we
won't have to pay the performance penalty often.

I should make mention of this in the I-D.

Alok

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 16 May 2006 11:09:46 -0500

> security mechanisms like Kerberos are hard to configure.

Alok, with all due respect, I don't buy this argument.  You could just
as easily introduce a new GSS-API mechanism that just doesn't provide
real authentication (i.e., it would only deal in GSS_C_NT_ANONYMOUS
principals).

That would be result in much better coverage on the wire than the
checksums proposal.

It wouldn't protect against memory/CPU errors.  I thought that
protection against memory/CPU errors (between the point where the
checksums are computed/verified all the way to storage) was the point of
this work.

> this is *just* a first step.

You're digging deeper in the wrong direction.  On-the wire protection
should not be addressed at this layer.

[...]
> it would be good. Are you saying you'd rather wait for a 

I don't know about Mike, but as far as I'm concerned this is not the way
to go about getting protection against errors on the wire w/o key
management.

You'd be better off making the argument about memory/CPU errors.  But
again, I think data on their frequency would be nice to have.

> won't have to pay the performance penalty often.

An attribute that the client could use to determine the natural block
size of the filesystem or file might be useful.  In the ZFS case not
having such an attribute simply means that I/O to small files (smaller
than the fs record size) will be slower than I/O to large files, but I
suspect that's OK.  Still, one might like to mention this.  Also, is
there any reason to think that the ZFS max record size won't get
significantly larger?  If it did then the trade-off might get painful.

> I should make mention of this in the I-D.

Yes.

Nico
--

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Tue, 16 May 2006 12:27:59 -0400

> real authentication (i.e., it would only deal in GSS_C_NT_ANONYMOUS

oh, you mean anonymous SPKM-3 :)

-->Andy

---

From: Mike Eisler <[email-masked]>
Date: Tue, 16 May 2006 09:31:21 -0700 (PDT)

> And the "Introduction" section of the I-D accepts that

No.

> security mechanisms like Kerberos are hard to configure.

Brent suggested at least once that one could create a trivial "security"
mechanism under GSS-API that does checksums on the entire payload,
for all operations (and all NFS versions for that matter).

> this is *just* a first step.

I don't see a compelling argument for a partial solution that will be
thrown away when the complete solution arrives.

> a proposal to outline such a mechanism would be good.

So where's the spec? Brent is basically saying, where's
beef (or perhaps, given his origins, "where's the mutton"? :-)

> it would be good. Are you saying you'd rather wait for a 

No, the complete proposal.

> that determines how this functionality performs, correct? So, if

All checksum algorithms introduce a potentially unacceptable 
performance impact on zero copy implementations. Many introduce
an unacceptable performance impact on single copy implementations.
I.e., customers will look at the cost, and not bother with it. Hence,
a waste of the WG's time.

> will have to pay a performance penalty.

Then put it in the i-d.

> won't have to pay the performance penalty often.

Seems wasteful for the apps that always do short or mis-aligned writes.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 16 May 2006 12:43:35 -0400

> a waste of the WG's time.

I thought part of the argument for this sort of end-to-end integrity was
that at least in some (hopefully common) cases, with sufficiently large
aligned writes, etc., the server would be able to just read/write the
client-calculated checksum from/to disk without doing any checksumming
itself, hence avoiding having to run all the data through its CPU in a
situation where other solutions (rpcsec_gss with integrity) would
require doing so.  Or is that not possible?

Of course this would be more easily done in the application.  So I guess
the draft should also state a requirement that the integrity protection
not require the cooperation of the application.

--b.

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 16 May 2006 11:49:49 -0500

> oh, you mean anonymous SPKM-3 :)

NO.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 16 May 2006 10:27:12 -0700 (PDT)

> require doing so.  Or is that not possible?

Ric Werme pointed out the pitfall. Specific problems include:

- what if a non-NFSv4 client goes to read the data? It has no inkling that
  bad data is being read back.

- The client that wrote data doesn't know the data arrived in a corrupted
  form. He thus has no opportunity to re-transmit or report the problem until
  much later when the data is read and found to be corrupted. But the client
  likely won't have the data available to re-transmit.

We seem to be reconstructing all the issues with this proposal. 
This isn't good process.

> not require the cooperation of the application.

Actually, applications SHOULD be originating the checksums for data
they create. That's the end to end argument. It requires APIs. Database
software already does this, and some storage systems already cooperate
with database software.

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Tue, 16 May 2006 13:47:36 -0400

> > oh, you mean anonymous SPKM-3 :)

what do you mean?

-->Andy

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 16 May 2006 12:52:29 -0500

> > > oh, you mean anonymous SPKM-3 :)

That that is not what I meant.

Nico
--

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 16 May 2006 15:24:55 -0400

> We seem to be reconstructing all the issues with this proposal. 

Yeah, OK, sorry for not keeping up.

> they create. That's the end to end argument.

I agree.  My point was just that if they're going to sell the proposal
then they need to make it explicit their requirement that applications
not be modified, so the value of that requirement can be judged....

--b.

---

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 16 May 2006 16:10:18 -0400

>Seems wasteful for the apps that always do short or mis-aligned writes.

Which is much more common than Alok indicates. For example,
databases doing O_DIRECT are, in effect, performing almost all
their block reads and writes "unaligned" (their table pagesize is
different from the server's blocksize). Of course, most databases
may be doing their own block checksums already.

I think much of this discussion comes down to APIs. The checksum
is most useful when it is exposed as a property of the service.
Attempts to hide the checksum beneath existing APIs leads to
inefficiencies and/or compromises in the service.

Depending on an API is a very treacherous way to introduce a
new feature (says he, having been there before). That said,
I think the end2end idea has merit.

Tom.