Subject: [nfsv4] The atomic bool of change_info4

From: "Zack Kirsch" <[email-masked]>
Date: Wed, 5 Oct 2011 12:31:16 -0700

Hey guys,

I've got a quick question about the atomic bool of the change_info4
structure.

In NFSv3, the server has the option of not returning a FH or attributes.
This is useful for cases where a non-idempotent operation like CREATE
successfully creates the file, but for some corner case reason, we can't
access the FH/attributes after the create. Rather than fail the CREATE
(which would clearly be wrong), we'd rather return success without the
optional information.

In NFSv4 (4.0 and 4.1), I'm wondering if the server has the same choice
about the 'after' changeid4. If we successfully CREATE an object but
cannot access its 'after' changeid4, is there a way we can tell the
client that?  I thought the atomic bool gave us that ability, but the
RFC (and Linux client code) seems a bit ambiguous. For example, in RFC
5661 section 10.8.1:

>   The post-operation change value needs to be saved as the basis for
future change_info4 comparisons.

Looking at the Linux client code, it seemed to use the atomic bool to
know if it could trust the 'before' changeid4, but it always stores the
'after' changeid4 (and I'm not sure what it does with it afterwards.)
My fear is that it stores an invalid 'after' changeid4, and then
proceeds to use it later to compare against a valid changeid4.

If that's true, it seems like returning something like (-1) may be the
right solution when we don't know the changeid4 to return?

Thoughts?

Thanks,
Zack Kirsch

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 05 Oct 2011 17:42:36 -0400

> RFC (and Linux client code) seems a bit ambiguous. For example, in RFC

No. I can see nothing in either RFC3530 or RFC5661 that states you can
provide invalid change attributes. All it says is that they can be
provided non-atomically w.r.t. the directory changes.

IOW: If the bool flag is set to false, then your server is not
guaranteeing that some other operation didn't change the directory after
the 'before' value was measured and before the 'after' value was
measured.

> proceeds to use it later to compare against a valid changeid4.

I see nothing in the text that states the server is allowed to lie. Only
that it can offer different levels of guarantee. If you see something
different, then please quote the exact text.

> right solution when we don't know the changeid4 to return?

Definitely not. That would confuse existing clients and cause them to
cache invalid information.

Cheers
  Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: <[email-masked]>
Date: Wed, 5 Oct 2011 18:23:06 -0400

Well, you can't return invalid change attributes, but I think the
requirements for a valid change attribute may make validity easier
to achieve than you might expect

RFC5661 says (section 5.8.1.4) a changeid4 is:

>    A value created by the server that the client can use to determine if
>    file data, directory contents, or attributes of the object have been
>    modified.  The server may return the object's time_metadata attribute
>    for this attribute's value, but only if the file system object cannot
>    be updated more frequently than the resolution of time_metadata.

I haven't been able to find much about exactly *how* the client is allowed to 
use the value to make this determination.  Two options are:

1) it may assume this is a numerically ascending value.

2) it only may compare for equality.

For (1) to be right, wraparound would have to be prohibited but I can't see that 
that is done explicitly.

If (2) is right, then returning atomic=false plus any after value that will not
be returned by GETATTR seems like it would meet the need and so avoid causing
the client to do something wrong.  It might not be the most forthright of server
responses but it wouldn't be a lie since it is only saying that the client
is not entitled to assume that the directory state after the CREATE matches any 
it has previously seen or will see in the future.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Trond Myklebust
Sent: Wednesday, October 05, 2011 5:43 PM
To: Kirsch, Zack (Isilon)
Cc: [email-masked]
Subject: Re: [nfsv4] The atomic bool of change_info4

Cheers
  Trond

NetApp

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 5 Oct 2011 18:41:46 -0400

> it has previously seen or will see in the future.

If the corner case in question were the directory getting destroyed
between the time you did the CREATE and the time you queried the
attributes, then in fact you probably could just reserve a single value
(like -1) for that case, and return that, confident that you wouldn't be
returning many more change attributes for that directory, or in the
unlikely event that you did, that they wouldn't be of much use....

I guess a harder case would be if your server somehow had its
permissions to look at the directory revoked at that moment.

At some point, what can you do?

--b.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 5 Oct 2011 15:45:14 -0700

> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, October 05, 2011 6:23 PM
> To: Myklebust, Trond; [email-masked]
> Cc: [email-masked]
> Subject: RE: [nfsv4] The atomic bool of change_info4
> 
requirements
> for a valid change attribute may make validity easier to achieve than
you
> might expect

No, the client may not assume this. There is nothing in either RFC that
says that the client may make any assumptions whatsoever about the
change attribute other than that it changes values when the data and/or
metadata change. 

In fact, this is why I'm proposing a new attribute in v4.2 that informs
the client on how the change attribute is expected to change over time.

> 2) it only may compare for equality.

Right now, this is the only option allowed by RFC3530 and RFC5661.
 
> For (1) to be right, wraparound would have to be prohibited but I
can't see
> that that is done explicitly.
> 
> If (2) is right, then returning atomic=false plus any after value that
will not be
> returned by GETATTR seems like it would meet the need and so avoid
> causing the client to do something wrong.  It might not be the most
forthright
> of server responses but it wouldn't be a lie since it is only saying
that the
> client is not entitled to assume that the directory state after the
CREATE
> matches any it has previously seen or will see in the future.

Right, but it will cause the client to assume that the directory has
changed _again_ when it issues a GETATTR and retrieves a completely
different value. Worse, if the server returns a fixed value such as
'-1', then the client may be doubly confused when it issues 2 CREATEs
and they both return the same change attribute...

> > successfully creates the file, but for some corner case reason, we
> > can't access the FH/attributes after the create. Rather than fail
the
> > CREATE (which would clearly be wrong), we'd rather return success
> > without the optional information.
> >
> > In NFSv4 (4.0 and 4.1), I'm wondering if the server has the same
> > choice about the 'after' changeid4. If we successfully CREATE an
> > object but cannot access its 'after' changeid4, is there a way we
can
> > tell the client that?  I thought the atomic bool gave us that
ability,
> > but the RFC (and Linux client code) seems a bit ambiguous. For
> > example, in RFC
> > 5661 section 10.8.1:
> 
provided non-
> atomically w.r.t. the directory changes.
> 
guaranteeing that
> some other operation didn't change the directory after the 'before'
value
> was measured and before the 'after' value was measured.
> 
> > know if it could trust the 'before' changeid4, but it always stores
> > the 'after' changeid4 (and I'm not sure what it does with it
> > afterwards.) My fear is that it stores an invalid 'after' changeid4,
> > and then proceeds to use it later to compare against a valid
changeid4.
> 
> I see nothing in the text that states the server is allowed to lie.
Only that it
> can offer different levels of guarantee. If you see something
different, then
> please quote the exact text.
> 
cache
> invalid information.

---

From: Rick Macklem <[email-masked]>
Date: Wed, 5 Oct 2011 19:22:45 -0400 (EDT)

David Noveck wrote:
[good stuff snipped for brevity]

> will not
> Trond Myklebust wrote:
I believe that what David is saying is correct in that, so long as -1 is
guaranteed to never be returned as a changeid4 value at any other time,
it would work.

Why? Well, it would reflect to the client that there has been a change when
in fact, there might not have been. This should result in a client not
caching when it could have, but not caching when it shouldn't have. Right?

Put another way...
- The changeid must change when the data or metadata changes.
- Ideally, the inverse would also be try. ie. The changeid wouldn't
  change when the data or metadata hasn't changed. However, an extraneous
  change of changeid when the data or metadata hasn't actually changed
  should only result in less efficient client caching.
--> The key is that this value for changeid (lets say it is -1) can only
    be returned once for the FH and never again, because if returned a
    second time, that would say the data and metadata hasn't changed when
    the server doesn't actually know the correct answer.

    One tricky case may be server crash recovery. I seem to recall that
    the Solaris10 client gets perturbed after a server crash/reboot, if
    it sees the changeid change from before the crash. (This probably only
    affects regular files, so it might not matter for directories?)
    I recall this from testing in the early days before FreeBSD stored the
    changeid on disk. (I don't remember the exact nature of the issue,
    just that the client didn't recover opens/locks well when the changeid
    changed after a server crash/reboot.)

I'll let the RFC lawyers decide whether or not this is actually permitted
by the RFCs or not;-) However, I think the above works in practice, assuming
you can ensure the (-1) is only returned once for an fh.

At least this is my take on the above, rick

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 5 Oct 2011 16:25:20 -0700

> From: Rick Macklem [mailto:[email-masked]]
> Sent: Wednesday, October 05, 2011 7:23 PM
> To: david noveck
> Cc: [email-masked]; Myklebust, Trond; zack kirsch
> > will not be returned by GETATTR seems like it would meet the need and
> > so avoid causing the client to do something wrong. It might not be the
> > most forthright of server responses but it wouldn't be a lie since it
> > is only saying that the client is not entitled to assume that the
> > directory state after the CREATE matches any it has previously seen or
> > will see in the future.
> >
> > Trond Myklebust wrote:
> > > If that's true, it seems like returning something like (-1) may be
> > > the right solution when we don't know the changeid4 to return?
> >
> guaranteed to never be returned as a changeid4 value at any other time, it
> would work.

Nope. It would require changes to _all_ existing clients and as such, is not an option.

Trond

---

From: Rick Macklem <[email-masked]>
Date: Wed, 5 Oct 2011 19:39:01 -0400 (EDT)

> Nope. It would require changes to _all_ existing clients and as such,
Ok, suppose there is a server that maintains changeid4 correctly and 2
clients (A and B) that create/modify "foo" in the following order:

- Client A creates "foo"
- server replies -1 for changeid4 (-1 just refers to all bits set for the unsigned)
  (since there is no defined initial value for a changeid, all bits set is fine)
* - Client B modifies "foo"
    (server changes changeid to 123)
- Client A does a getattr for changeid and gets 123
--> This tells the client that "foo" has changed since creation, so it should
    invalidate cached information for "foo".

The only difference from the above w.r.t. what I proposed is that the "*" step
hasn't actually occurred. All that happens is that it appears another client
changed "foo" when it didn't actually do so. And all this should do is make
Client A invalidate cached information for "foo" when it didn't really need
to do so. Less efficient? "yes". Requires existing clients to be modified?
"not that I can see".

rick

---

From: "Zack Kirsch" <[email-masked]>
Date: Wed, 5 Oct 2011 16:51:12 -0700

<snipped>

> Right, but it will cause the client to assume that the directory has
changed
> _again_ when it issues a GETATTR and retrieves a completely different
> value. Worse, if the server returns a fixed value such as '-1', then
the client
> may be doubly confused when it issues 2 CREATEs and they both return
the
> same change attribute...

I don't understand. If we return -1 for an after value after a CREATE
and then you issue a GETATTR and get a change value of 10, why would
that invalidate the metadata that I'm returning to you?

Zack

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 5 Oct 2011 16:52:46 -0700

> Sent: Wednesday, October 05, 2011 7:39 PM
> To: Myklebust, Trond
> Cc: zack kirsch; [email-masked]; david noveck
> > > guaranteed to never be returned as a changeid4 value at any other
> > > time, it would work.
> >
> Ok, suppose there is a server that maintains changeid4 correctly and 2 clients
> (A and B) that create/modify "foo" in the following order:
> 
> - Client A creates "foo"
> --> This tells the client that "foo" has changed since creation, so it
> --> should
> The only difference from the above w.r.t. what I proposed is that the "*"
> step hasn't actually occurred. All that happens is that it appears another client
> Client A invalidate cached information for "foo" when it didn't really need to
> do so. Less efficient? "yes". Requires existing clients to be modified?
> "not that I can see".

The changeinfo structure refers to the change attribute of the enclosing _directory_, not the object that is being created. The latter doesn't even have a 'before' change attribute...

If someone were to return '-1', on all CREATE changeids and then a different value for GETATTR, then my client, as it stands today, would end up having to assume that all cached dentries need to be revalidated (i.e. looked up again). Performance would suffer and, as I said earlier, there is the interesting case of CREATE followed by another CREATE (both returning change attribute == '-1')...

Trond

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 5 Oct 2011 16:55:06 -0700

> From: Zack Kirsch [mailto:[email-masked]]
> Sent: Wednesday, October 05, 2011 7:51 PM
> To: Myklebust, Trond
> Cc: [email-masked]
and then
> you issue a GETATTR and get a change value of 10, why would that
invalidate
> the metadata that I'm returning to you?

Think DNLC. You' ve just told my client that someone somewhere
mysteriously changed the contents of the directory. Suddenly, I can't
trust my cached lookups (or the readdir cache).

Trond

---

From: Rick Macklem <[email-masked]>
Date: Wed, 5 Oct 2011 20:07:55 -0400 (EDT)

Trond Myklebust wrote:
> > > > I believe that what David is saying is correct in that, so long
> > > > as
> > > > -1 is
> > > Nope. It would require changes to _all_ existing clients and as
> > > such,
> > > is not an option.
> > >
> The changeinfo structure refers to the change attribute of the
> enclosing _directory_, not the object that is being created. The
> latter doesn't even have a 'before' change attribute...
> 
Ok, yes, I realize that his case was for the directory. The above was
just easier to write without specifying that "foo" was a directory and
that the "*" occurred while a Create was in progress by Client A. (Note
that David said "atomic==false", so Client B could modify the directory
between the "before" and "after" attributes are acquired.)

> If someone were to return '-1', on all CREATE changeids and then a
> different value for GETATTR, then my client, as it stands today, would
> end up having to assume that all cached dentries need to be
> revalidated (i.e. looked up again). 
Yep, agreed. It is not what I would consider "good server behaviour" and
it will result in a performance hit. (I have assumed that Zack's situation
is a rare case which they can't fix easily in their server. Fixing the server
so that it always returns a valid changeid4 would be much preferable. Or, if
they can detect this situation before doing the Create, returning NFS4ERR_DELAY
for the Create instead of doing it would also be preferable, I suspect?

>Performance would suffer and, as I
> said earlier, there is the interesting case of CREATE followed by
> another CREATE (both returning change attribute == '-1')...
> 
I mentioned that the server can only return this reserved value (-1) once.
I agree that returning it more than once for a given FH would be incorrect,
from what I can see.

I think I should simply retract "my plan" and go with "if the server returns
NFS4_OK for Create, it had better be able to reply with legitimate before/after
changeid4 for the directory" because I agree that the client will probably
get hammered by the illegitimate after changeid.

rick

---

From: <[email-masked]>
Date: Thu, 6 Oct 2011 11:24:41 -0400

Let's not talk about this in term of confusion, whether single, double, or triple.  What we 
need to focus on is any case in which the client is caused to do something incorrect
by a changeid4 that it gets from the server.

Within that framework, the case that Trond has cited is concerning:

    Suppose I create X in directory D and get a changeid4
    of 0xffffffffffffffff.  If I subsequently do a create
    and get the same changeid4, the client would be able to
    conclude (perhaps incorrectly) that Y still existed in
    D.

    So a strategy to deal with this cannot just consider changeid4's
    returned by GETATTR.

What you could do depends on how often this happens.  Zack spoke of an 
"edge condition" but that could cover a lot of ground.  Are we talking about
a once-per-century edge condition (something like once-per-10**12 requests so
we'll call that pico-edge) or are we talking about something that is nano-edge or 
even more frequent?  That doesn't change the spec requirements but it does change 
what you might have to do to assure distinct otherwise unused values.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Myklebust, Trond
Sent: Wednesday, October 05, 2011 6:45 PM
To: Noveck, David; Kirsch, Zack (Isilon)
Cc: [email-masked]
Subject: Re: [nfsv4] The atomic bool of change_info4

requirements
you

 
can't see
will not be
forthright
that the
CREATE

the
can
ability,
provided non-
guaranteeing that
value
changeid4.
Only that it
different, then
the
cache

---

From: <[email-masked]>
Date: Thu, 6 Oct 2011 11:40:57 -0400

> Nope. 

Nope what?  You think Rick is wrong in his interpretation of what I'm saying?

Actually he is, since I only considered changeid4's returned by GETATTR.

So for now let's work from the formulation that Rick imputed to me, and
ignore the hand-wavy parts of what I actually said.

> It would require changes to _all_ existing clients and as such, is not an option. 

I would argue that, given Rick's formulation ("guaranteed to never be returned as 
a changeid4 value at any other time"), no correct client would require change.

To clarify, that would only be if "at any other time" meant just that and not "in any 
other situation".  Note that guarantee in question is one for the server to provide.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Myklebust, Trond
Sent: Wednesday, October 05, 2011 7:25 PM
To: Rick Macklem; Noveck, David
Cc: Kirsch, Zack (Isilon); [email-masked]
Subject: Re: [nfsv4] The atomic bool of change_info4

Trond

---

From: "Zack Kirsch" <[email-masked]>
Date: Thu, 6 Oct 2011 14:05:56 -0700

Let's say it's a nano-edge condition. :)  I'm okay with causing clients
to flush cache when this rarely happens because it would just be a
slight perf hit, but I'm not okay with Trond's point of (-1) matching
(-1) and _not_ flushing cache.

As an implementer, I'm also not willing to guarantee that I never return
the same value twice. As a funny solution, I could randomly generate a
64-bit number and have great odds of not hitting the correct current
changeid. :)

Looks like I'll have to fix this the real way (by always guaranteeing
the changeids I send back are correct). Thanks for the help guys.

Zack 

> Sent: Thursday, October 06, 2011 8:25 AM
> To: [email-masked]; Zack Kirsch
> Cc: [email-masked]
> Let's not talk about this in term of confusion, whether single,
double, or
> triple.  What we need to focus on is any case in which the client is
caused to
> do something incorrect by a changeid4 that it gets from the server.
> 
> a once-per-century edge condition (something like once-per-10**12
> requests so we'll call that pico-edge) or are we talking about
something that
> is nano-edge or even more frequent?  That doesn't change the spec
> requirements but it does change what you might have to do to assure
> distinct otherwise unused values.
> 
> No, the client may not assume this. There is nothing in either RFC
that says
> that the client may make any assumptions whatsoever about the change
> attribute other than that it changes values when the data and/or
metadata
> change.
> 
> In fact, this is why I'm proposing a new attribute in v4.2 that
informs the
> client on how the change attribute is expected to change over time.
> 
changed
the client
the
> > From: [email-masked] [mailto:[email-masked]] On
Behalf
> > Of Trond Myklebust

---

From: Malahal Naineni <[email-masked]>
Date: Thu, 6 Oct 2011 14:44:48 -0700

> 64-bit number and have great odds of not hitting the correct current

Reserving few funny changeids and recycling them may be better
than a random number here.

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 6 Oct 2011 19:14:02 -0700

A few funny changeids is not sufficient, because once recycled, they have
a high probability of messing up the client's cache.

On the theory that we picked a 64 bit width for the change attribute
because that was long enough to not worry about wrap around, then a 63 bit
width is probably as safe. Thus as safe algorithm would be to split his
changeid space into two 63 bit values:

- good changeids, where highest order bit is always zero
- funny changeids, where highest order bit is always one

Have the funny changeid be composed of a funny counter and the logical OR
of 0x8000000000000000 . Each time Zack;s server cannot get change
attribute for changeid, bump the funny counter.

The latter are to deal with the case when he cannot easily obtain the
parent directories.

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: <[email-masked]>
Date: Mon, 10 Oct 2011 13:52:34 -0400

That's certainly adequate.  9 quintillion funny change id's would even work for something that was not an edge condition as far as correctness goes, although client caching would be vitiated.  

I think we have to be ready for the newer persistent memory technologies (like MRAM) that may be happening during the lifetime of NFSv4.   So things like a billion CREATEs per second would be possible (hopeful that there would be some REMOVEs too).  At one billion funny changeid's per second, Mike's scheme would last about 3 centuries until it ran out.

Based on Zack's designation of this as a nano-edge condition, I was thinking of simply reserving everything that had all F's in the first 32 bits.  That gives you something that would last over a hundred years at one funny changeid per second.

If people are thinking of smaller numbers of funny changeid's than that, there are still numbers like 65K that are bigger than "a few".  The spec needs to be clearer about the scope of these.

It is clear that changeid's for different fh's are not going to be compared, but in a clustered arrangement it is hard for  a front-end processing unit to maintain information on a per-fh basis.  Per-client pools are more interesting.  Do we need some sort of statement that the client should only use the change attribute internally and not make it accessible to the application who might assume that it is going to always be ascending (or to other clients)? 

I think it may need to be said that these don't go away on client reboot since clients may save them to a persistent cache.  For RFC3530bis, we have the client-id string model nightmare to make us reluctant to touch this but should we fix RFC5661 at some point to make it clear that a v4.1 server may use different funny changeid pools to clients with different nfs_client_id4 id strings?

Also, it should be said that these will be preserved on migration so that might be an issue for migration-issues-02.