Subject: Re: [nfsv4] Device stateids

From: Robert Gordon <[email-masked]>
Date: Wed, 1 Aug 2007 03:45:50 -0500

On Jul 31, 2007, at 12:38 PM, Garth Goodson wrote:
>
> Robert Gordon wrote:
>> On Jul 31, 2007, at 11:50 AM, Garth Goodson wrote:
>>>
>>> Robert Gordon wrote:
>>>> On Jul 30, 2007, at 12:16 PM, Garth Goodson wrote:
>>>>> So my question is, is it possible to recall all device IDs for  
>>>>> a given FSID in a single call, and return in a single call  
>>>>> (like it is in the current text)?  If not, is everyone ok with  
>>>>> this.
>>>> it is, and i think that the current proposal is good.
>>>
>>> I guess there is still some confusion if you are claiming that it  
>>> is still possible to recall all device IDs and return them all in  
>>> a single call (how are all their stateids specified?).  It seems  
>>> a separate op is needed for each stateid.
>>>
>>> A few messages back Benny stated it was not possible and I tend  
>>> to agree.
>>>
>>> -Garth
>> Perhaps i misunderstood this:
>>> SEQUENCE will have a status flag added to indicate that
>>> the device mappings have been partially deleted, fully
>>> deleted (e.g. recall or lease expiry), changed, or added
>>> to. The partially deleted flag stays on until the client
>>> responds to any and all _DELETE notifications, or it does
>>> a GETDEVICELIST. The fully deleted flag stays on until
>>> DELEGRETURN returns the device stateid. The changed and
>>> added flags stay on until a GETDEVICELIST or GETDEVICEINFO
>>> gets the affected mappings.
>> I took it to mean:
>> if in SEQUENCE the server has indicated that the deviceid
>> mappings have been fully deleted ; then the client should
>> dump its current device list and issue GETDEVICELIST.
>> Robert.
> The thing that has me hung up is the sentence:
>
> 'The fully deleted flag stays on until DELEGRETURN returns the  
> device stateid.'
>
> Which seems to imply that each stateid needs to be returned.
>
> Spencer/Mike clarification?

I reviewed the WG maillist;

Mike was proposing a single device state id (devstateid) that
represented the mappings held at the client.

So the server wants to 'redo' the entire mapping, it will issue a
CB_RECALL for the devstateid, the client does DELEGRETURN with
the devstateid, then the client will do GETDEVLIST to fetch the
new mappings and new devstateid.

I think that there is still an issue as to the scope of the
device id. I'm viewing the scope to be global to that particular
client, where as some seem to want to tie the scope to fsid.

I think that a server has the ability to change or delete the
device ids on a per FSID basis via CB_NOTIFY.

Robert.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Robert Gordon <[email-masked]>
Date: Wed, 1 Aug 2007 04:29:22 -0500

Okay, well I'm willing to take a stab at tweaking
the proposal.

----

The NFS client maintains a mapping of device ids contained
in a layout, to the corresponding data server address.
The state of the mapping is represented by a single
recallable device stateid (devstateid). There is at most
one devstateid per { client ID, layout type } pair.

GETDEVICEINFO and GETDEVICELIST each return the devstateid.

The mappings stay in force until the devstateid is
recalled (via CB_RECALL) or lease expiration.

All layouts using the device IDs remain in force, but the
server MUST fence the client from accessing the affected
data servers until the client has obtained the re-mapped
device IDs via GETDEVICEINFO or GETDEVICELIST.

If the server supports CB_NOTIFY and the Client has
requested device id notifications via GETDEVICEINFO
or GETDEVICELIST; The server can inform the client
of an add, modify, or delete mapping.

The CB_NOTIFY notifications are:

NOTIFY4_DELETE_DEVICE_ID
	Deletes all mappings using the specified device ID.
	The server MUST stop using the device ID. Any
	layouts remain in force, but aren't useful until
	new mappings are available. The server fences the
	client from the affected data server until all
	layouts with the device ID are returned, or the
	client obtains new mappings. The server provides
	a hint as to whether new mappings will arrive and
	if so how long.

NOTIFY4_ADD_DEVICE_ID
	Adds a device ID to address mapping. This is
	for a device ID the that was not returned in a
	previous series GETDEVICELIST calls (where the
	series ended in gdlr_eof == TRUE).  The client
	calls GETDEVICEINFO or GETDEVICELIST to determine
	the mappings for the new device ID.

NOTIFY4_CHANGE_DEVICE_ID
	The client call GETDEVICEINFO or GETDEVICELIST
	to get updated device addresses for the device
	ID. Layouts remain in force. The client is fenced
	from any devices that were in the old address list,
	but not the new address list.

In addition to informing the client of mapping changes via
CB_NOTIFY and CB_RECALL, SEQUENCE carries a status flag to
indicate that device mappings have changed.

The flags are :-

SEQ4_STATUS_DEVID_DELETED
	Mapping has been partially deleted. If the client is
	using CB_NOTIFY it may issue a GETDEVICEINFO for each
	notified deletion or GETDEVICELIST to refresh the entire
	map. This flag stays on until the client responds to
	any and all _DELETE notifications, or it has issued a
	GETDEVICELIST.

SEQ4_STATUS_DEVID_DELETED_ALL
	This flag stays on until the client issues a DELEGRETURN
	using the devstateid.

SEQ4_STATUS_DEVID_CHANGED
SEQ4_STATUS_DEVID_ADDED
	These flags stay on until a GETDEVICELIST or GETDEVICEINFO
	(if using CB_NOTIFY) gets the affected mappings.

--

Robert.

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 1 Aug 2007 09:37:25 -0400

> The mappings stay in force until the devstateid is recalled 
> (via CB_RECALL) or lease expiration. 

For delegations, for example, recall is only a notification, and you
have the delegation until you return it (or it is revoked).  Is there a
way to return devstateid's?  If not, and this happens purely through
recall, when?  Presmably at the point the server receives the response
to the recall.

The state management chapter defines a general framework for freeing of
stateid's.  In addition to loss due to lease expiration, there is client
reboot, ADMIN_REVOKED which I would assume also applies here, and the
non-return of recallable state (the error now is DELEG_REVOKED but maybe
the name should be changed).

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Wednesday, August 01, 2007 5:29 AM
To: NFSv4
Subject: Re: [nfsv4] Device stateids

Okay, well I'm willing to take a stab at tweaking the proposal.

----

The NFS client maintains a mapping of device ids contained in a layout,
to the corresponding data server address.
The state of the mapping is represented by a single recallable device
stateid (devstateid). There is at most one devstateid per { client ID,
layout type } pair.

The mappings stay in force until the devstateid is recalled (via
CB_RECALL) or lease expiration.

All layouts using the device IDs remain in force, but the server MUST
fence the client from accessing the affected data servers until the
client has obtained the re-mapped device IDs via GETDEVICEINFO or
GETDEVICELIST.

If the server supports CB_NOTIFY and the Client has requested device id
notifications via GETDEVICEINFO or GETDEVICELIST; The server can inform
the client of an add, modify, or delete mapping.

NOTIFY4_DELETE_DEVICE_ID

NOTIFY4_ADD_DEVICE_ID

NOTIFY4_CHANGE_DEVICE_ID

In addition to informing the client of mapping changes via CB_NOTIFY and
CB_RECALL, SEQUENCE carries a status flag to indicate that device
mappings have changed.

The flags are :-

--

Robert.

---

From: Robert Gordon <[email-masked]>
Date: Thu, 2 Aug 2007 19:21:03 -0500

On Aug 1, 2007, at 8:37 AM, Noveck, Dave wrote:

> have the delegation until you return it (or it is revoked).

We could treat the device_mapping information as a file in that
the client holds a read delegation on that file, thus a delegation
stateid. It is just that the proposal called this special delegation
stateid the "device stateid".

> Is there a way to return devstateid's?

Since we treat the devstateid as a delegation stateid, then as the
proposal said we can use DELEGRETURN.

> If not, and this happens purely through recall, when?
> Presumably at the point the server receives the response
> to the recall.

so, going even further with our well understood delegation stateid
model. In draft 12 for both GETDEVICEINFO and GETDEVICELIST we say

'The current filehandle (cfh) is used to identify the file system;
device IDs are unique per file system (FSID) and are qualified by
the layout type.'

So I'll update the proposal such that the delegation stateid for
device mapping is not for the entire client but rather the file
system that the cfh refers. This way Benny and Marc (and others :) )
may CB_RECALL the device_mapping delegation stateid per FSID.

Does this sound reasonable ?

Robert.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 08 Aug 2007 18:58:41 +0300

Sorry for taking so long to comment on that.
Anyway, my comments are inline below.

>> have the delegation until you return it (or it is revoked).

right

> proposal said we can use DELEGRETURN.

Although this is technically possible I worry about using filehandles
to refer to something that is not a filesystem object as it may have a
profound effect on the protocol as we might need to specify for every use
case of a filehandle what to do about these "special" filehandles, what
error to return, etc.  Also, overloading the filehandle is very weird
for the server when decoding it as, for example, in the linux server
implementation decode_fh returns a struct dentry*, what will it return
for this case, should it not be called at all?  Can the nfsd implementation
of DELEGRETURN know that the delegation is for devices and not for a file
or a directory?  This leads me again to the conclusion that having
a "device set" identifier is helpful for managing the 

> Does this sound reasonable ?

I don't mind having the device address space global rather than
per filesystem. Recalling devices is rare to be worth the overhead
for simplicity.

> Robert.

With my above comments in mind I'll try to sketch a tweaked proposal
that does not rely on the filehandle abstraction.

in a layout, to the corresponding storage device address.

GETDEVICEINFO and GETDEVICELIST each take the layout type and
an optional devstateid and return the devstateid.  In case
the client holds a valid devstateid it MUST use it for subsequent
GETDEVICEINFO and GETDEVICELIST that update the device ID
mappings.  The devstateid argument is invalid whenever the
client has no valid devstateid (e.g. after reboot or after
the devstateid was previously returned via DELEGRETURN).
GETDEVICEINFO or GETDEVICELIST that operate on a device stateid
that was already recalled via CB_RECALL MUST be rejected by the
server with NFS4ERR_BADSTATEID.

GETDEVICEINFO and GETDEVICELIST also take a boolean flag
optionally asking for device notifications and return a
corresponding flag confirming the notification.

Device stateids follow the sequencing and use rules of all other
stateid types.  The mappings are invalidated when the devstateid is
returned via CB_DELEGRETURN or explicitly deleted via CB_NOTIFY.

All layouts using the device IDs remain in force, but the
server MUST fence the client from accessing the affected
devices until the client has obtained the re-mapped device
IDs via GETDEVICEINFO or GETDEVICELIST.

CB_RECALL takes a devstateid to notify the client it needs
to return all device IDs identified by the devstateid.  The
filehandle argument has no use in this case.  It does not change
the device stateid.

DELEGRETURN takes a devstateid to return all device IDs associated
with the devstateid. The current filehandle has no use in this case.

If the server supports CB_NOTIFY for devices and the Client has
requested device id notifications via GETDEVICEINFO
or GETDEVICELIST; The server can inform the client
of a delete, add, or change mapping.

CB_NOTIFY takes a device stateid qualifying a device ID
for the following device notifications:

NOTIFY4_DELETE_DEVICE_ID

NOTIFY4_ADD_DEVICE_ID

NOTIFY4_CHANGE_DEVICE_ID

CB_NOTIFY does not change the device stateid.

In addition to informing the client of mapping changes via
CB_NOTIFY and CB_RECALL, SEQUENCE carries status flags to
indicate that device mappings have changed.

The flags are :-

---

From: Robert Gordon <[email-masked]>
Date: Thu, 9 Aug 2007 01:33:47 -0500

In general i think this is good.

I've started to think that the SEQUENCE status flags could just be
two bits.

SEQ_STATUS_DEVID_CHANGED
	This flag stay on until a GETDEVICELIST or GETDEVICEINFO
	(if using CB_NOTIFY) gets the affected mappings.

Robert.

---

From: Benny Halevy <[email-masked]>
Date: Thu, 09 Aug 2007 10:22:50 +0300

> 	(if using CB_NOTIFY) gets the affected mappings.

The DEVID_DELETED cases should be covered by CHANGED as the client will
have to revalidate all the device mappings anyway to see what actually changed
and should either not see the deleted deviceid returned by GETDEVICELIST
or get an error from GETDEVICEINFO for that deviceid.

The DEVID_ADDED case is covered if the client is doing GETDEVICELIST
and the server supports it.  If the server does not support GETDEVICELIST
is must not set SEQ_STATUS_DEVID_CHANGED for added devices as the client
has no way to get the new device ID.  These will be discovered only
when the client will get new layouts referring to the new devices
(which is perfectly fine)

---

From: [email-masked]
Date: Sat, 11 Aug 2007 12:50:44 -0400

> device IDs via GETDEVICEINFO or GETDEVICELIST.

Has the right intent, but is too conservative.  Likely
implementations for the block layout will make the client
responsible for some of the fencing (block layout has to
trust the client in situations where the other layouts
don't have to).

This text should say something like:

All layouts using the device IDs remain in force, but the
client MUST NOT access the affected data servers until the
client has obtained the re-mapped device IDs via
GETDEVICEINFO or GETDEVICELIST.  The client MAY be prevented
from accessing the affected data servers until this is done,
but this prevention (fencing) is OPTIONAL.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Robert Gordon <[email-masked]>
Date: Mon, 13 Aug 2007 14:23:41 -0500

> but this prevention (fencing) is OPTIONAL.

Yes, i prefer that..

Robert.

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 20 Aug 2007 11:25:14 -0400

I have a great deal of concern about making this change.

The problem that requires data server fencing (at least in my view) is
that you can have situations in which the client can not be presumed to
be aware of the necessity for fencing.  It is not just that you are
worried about a rogue client which doesn't respect the requirements.  I
agree that if that were the case, then the argument that for blocks-type
layout, you are already subject to this issue would be convincing.  

But what we are talking about here happens in response to lease
expiration and to recall (which I interpret to mean when the server
*sends* a recall).  If you are going to make this the responsibility of
the client, how do you deal with cases where you cannot be assured that
the client has knowledge of the necessity for such fencing?  The fact
that lease expiration is one of the situations makes clear that we are
talking about situations in which reliable communication cannot be taken
for granted.

I think that we need a very clear story about how we can reliably (in
all cases) rely on the client to do this fencing before we contemplate
such a change.   

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Saturday, August 11, 2007 12:51 PM
To: [email-masked]; [email-masked]
Subject: [nfsv4] Device stateids and fencing

> client has obtained the re-mapped device IDs via GETDEVICEINFO or 

Has the right intent, but is too conservative.  Likely implementations
for the block layout will make the client responsible for some of the
fencing (block layout has to trust the client in situations where the
other layouts don't have to).

All layouts using the device IDs remain in force, but the client MUST
NOT access the affected data servers until the client has obtained the
re-mapped device IDs via GETDEVICEINFO or GETDEVICELIST.  The client MAY
be prevented from accessing the affected data servers until this is
done, but this prevention (fencing) is OPTIONAL.

Thanks,
--David

> The NFS client maintains a mapping of device ids contained in a 
> layout, to the corresponding data server address.

> If the server supports CB_NOTIFY and the Client has requested device 
> id notifications via GETDEVICEINFO or GETDEVICELIST; The server can 
> inform the client of an add, modify, or delete mapping.
> 
> In addition to informing the client of mapping changes via CB_NOTIFY 
> and CB_RECALL, SEQUENCE carries a status flag to indicate that device 
> mappings have changed.
> 
> The flags are :-
>

---

From: [email-masked]
Date: Tue, 21 Aug 2007 20:01:11 -0400

Dave,
 
> talking about situations in which reliable communication cannot be
> taken for granted.

At least for lease expiration, I expect the block clients to track
the lease behavior and do the right thing.

> all cases) rely on the client to do this fencing before we contemplate

For recall and the like, the server waits out the lease expiration,
and carries on, relying on the client to observe the lease expiration
restrictions.

Thanks,
--David

> server MUST 
> recallable device 
> server MUST 
> requested device 
> CB_NOTIFY

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 22 Aug 2007 09:26:35 -0400

> the lease behavior and do the right thing.

Maybe that's what you expect but for this to work, the protocol has
to require that this implementation be done, and describe, at least
in general terms, how it is to be done.  Given transmission delays,
for the client to determine when it is possible that the server
may see a lease expiration, is not trivial.  

With your "MUST NOT", all clients would be forced implement this and
I don't think that is reasonable if only blocks needs it.  I guess 
I would be OK, if we make this layout-type specific, and either 
describe this in the blocks document (if only blocks needs this),
or describe it in the base document and then say that it is only
required if you are using a particular layout type that requires
it (and the statement about whether a particular layout type 
requires this would be stated in the document for the layout type).

> and carries on, relying on the client to observe the lease expiration 

But there isn't necessarily going to be a lease expiration, that a
client could see, unless the server is going to do something drastic,
like not respond to any COMPOUNDs while a recall is outstanding.
If you respond to requests, the client has to assume that the lease
is being renewed, as it would be exected to be.

Suppose the server does a recall, and the request gets stuck in a 
buggy router, for a minute.  Meanwhile the client is doing stuff,
and getting his lease renewed.  He doesn't even know about the
recall.  Maybe he can find out about by a status bit in SEQUENCE,
but there is a fair amount of text to describe how this would
work and review it to make sure it really would work.  Until that
is done, and we know that this workable, a think we have to assume
that the data servers have the responsibility to do the fencing.    

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, August 21, 2007 8:01 PM
To: Noveck, Dave; [email-masked]
Subject: RE: [nfsv4] Device stateids and fencing

Dave,

> talking about situations in which reliable communication cannot be 

At least for lease expiration, I expect the block clients to track the
lease behavior and do the right thing.

> such a change.

For recall and the like, the server waits out the lease expiration, and
carries on, relying on the client to observe the lease expiration
restrictions.

Thanks,
--David

> re-mapped device IDs via GETDEVICEINFO or GETDEVICELIST.  The client 
> MAY be prevented from accessing the affected data servers until this 
> is done, but this prevention (fencing) is OPTIONAL.
> 
> Thanks,
> --David

---

From: [email-masked]
Date: Thu, 23 Aug 2007 11:08:50 -0400

Dave,

> may see a lease expiration, is not trivial.

I may be operating under a false assumption.  I had assumed that
clients would track lease expirations and know when the lease
expired locally.  The server then tracks that against its own clock
and adds a generous allowance for clock skew (this is an error case,
it just has to work, it doesn't have to be fast).  If clients aren't
already doing this, then you definitely have a point about the

> requires this would be stated in the document for the layout type).

My concerns are definitely block-layout-specific.  If the base
document could describe a) fencing and b) client cessation of
layout access after some sort of lack-of-communication timeout
(e.g., lease expiration) as options, then the block layout could
use the latter without imposing it on everyone.  There is quite
a bit of SAN filesystem "running code" out there that does not
do involuntary fencing of clients, and I'd like to avoid
invalidating that implementation approach.

> is being renewed, as it would be expected to be.

Ok, but the important case to make sure we get right first is loss
of communication.  If the client is uncommunicative or actually gone,
there has to be a deterministic way to recover its resources.

> that the data servers have the responsibility to do the fencing.    

Isn't this already a problem in the absence of pNFS?  What if the
request that gets stuck in a buggy router is a directory delegation
recall?

Thanks,
--David

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 23 Aug 2007 15:48:36 -0400

> expired locally. 

I don't know what it really means for the lease to expire locally.
Let's say I have this process that wakes up every lease-time minus
five seconds and a sends message if one hasn't been sent in the 
interim.  So assume that this process never, when the delay 
terminates, finds that it has been awoken more than five seconds
too late then, is it right to say that the lease never expires
"locally"?  If so, so what?  The point is that it matters when the
server receives the message, and you don't know that.  You only
know it happens some time between when you send it and when you 
get the reply.   

> The server then tracks that against its own clock
> it just has to work, it doesn't have to be fast).  

You add the generous allowance (five seconds in this case) to make
it unlikely that the lease will expire.  You can be as generous as 
you want, to reduce that probability, but you can't reduce it to
zero.

> If clients aren't

The issue is also (beside implementation work), specification work.
The spec does not describe how to correctly make this determination.
Given that you have N asynchronous parallel messages and you know
the send time and the reply time of each, how do you safely (false
positives are OK but false negatives are not) determine whether the 
server may have a gap of L seconds between receiving successive 
requests, with the only constraint being that each request is 
received after it is sent (no FTL-moving servers :-), and before 
the reply is received?

I'm not saying this can't be done, but so far nobody has clearly
described how to do it.  RFC 3530 had a vague hand-wavy statement
which basically told you to do it and use the determination to 
determine whether your lease had expired.  I don't know if anybody
actually tried to do this or what validation was done.  When we did
sessions, we scratched this and made this the responsibility of the
server to determine and created status bits for the server to tell 
the client as soon as it could of the current state of affairs.

> invalidating that implementation approach.

But that requires that you describe how to safely have the client
make that lease determination.

I have discussed this issue with a few other people who would not
be OK with this client-centered approach, even if it were limited
to the block-layout case only, so we may have difficulty getting to
consensus.  I'll let those people speak for themselves.

> there has to be a deterministic way to recover its resources.

I think the problem with phrasing things that way is that it implies
that other cases (transient loss of communication or delays) are not
important, and I think anything that can cause data corruption is very
important indeed.

The basic motivation of leasing is to find the client that is
uncommunicative or actually gone.  However, the server can only make
a determination that the client is likely to be done.  In the v4
model, his determination, whether correct or not as seen by an
omniscient observer, is definitive.  Once the server makes that
determination, he will release resources which may then be assigned
to others.  This is so whether the client really is uncommunicative 
or whether the client thinks he is uncommunicative.  For resources
on the server (MDS) itself, he can make sure that the client does 
not have access to freed resources.  For stuff on the DS's, that
won't work.  That leads to the requirement for fencing, and if you
want to change that and accept the clients' determination of whether 
the server may have freed resources, you have to be really sure that 
client can safely determine that the server could not possibly have 
judged him uncommunicative (as well as being astonishingly trusting 
in client implementers).

> request that gets stuck in a buggy router is a directory delegation

It is a problem and it is dealt with although you have pointed out an
issue that we have to clarify the spec on.

Let's take the case of an ordinary delegation first.  The next request
you issue after the revocation of the delegation, will have a status bit
telling you something is wrong.  Also any use of the delegation stateid
will get you a NFS4ERR_DELEG_REVOKED error.

Now with the directory delegation we will also get the status but we
have avoided the requirement to change CREATE, REMOVE, RENAME, etc to
take a stateid by using the session and saying that any reference to the
directory that that client has a delegation implicitly refers to that
delegation.  But that implies (and I don't think the spec says), that
if the delegation is revoked, we return an NFS4ERR_DELEG_REVOKED error,
even though the op does not have an explicit stateid parameter.
Similarly for NFS4ERR_EXPIRED and NFS4ERR_ADMIN_REVOKED.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Thursday, August 23, 2007 11:09 AM
To: Noveck, Dave; [email-masked]
Subject: RE: [nfsv4] Device stateids and fencing

Dave,

Thanks,
--David

---

From: "Everhart, Craig" <[email-masked]>
Date: Thu, 23 Aug 2007 17:17:37 -0400

Even if the client is conservative, as David alludes, about the client's
understanding of when the server thinks its lease will expire, what
happens if the client issues an I/O to a storage device just before its
lease (conservatively) expires?  That I/O request could get delayed for
quite a long time before reaching the storage device.  This seems to be
the basic weakness in client-based fencing.  I don't think you can come
up with a hard bound for that "quite a long time" value.

		Craig

---

From: [email-masked]
Date: Fri, 24 Aug 2007 15:44:17 -0400

Let me start with Craig's message, then I'll get to Dave's ...

> up with a hard bound for that "quite a long time" value.

The server needs to add in an allowance beyond when the lease should
have expired at the client.  As for the maximum time that an I/O can
be outstanding, there are practical limits.  In the blocks world, the
usual answer is 30 seconds, as that's the typical timeout used in SCSI
multi-path drivers for determining when to redrive an I/O for which
no response has been seen.  This is supported by engineering of the
infrastructure and devices - for example, Fibre Channel guarantees
to deliver or discard a frame in at most 10 seconds - while typical
delivery times are several orders of magnitude better, that is the
absolute maximum and is usually enforced by configuration limits.

Thanks,
--David

---

From: "Everhart, Craig" <[email-masked]>
Date: Fri, 24 Aug 2007 17:32:17 -0400

David,

Clearly, we're talking about the error-path scenario, not the typical
one, and just how bad it can get.  It's interesting to know that about
Fibre Channel.  I wonder if there are similar guarantees from other
schemes.  For example, I once heard a story (not from my current
employer) about debugging a problem in which a storage system had
delayed a write operation by tens of times more than what you quote as a
"usual answer"; presumably an awful lot of functionality had piled into
an enormous corner case.

I suppose there are diminishing returns in pNFS blocks clients making
larger and larger allowances, and the *possible* delays are inherent in
the blocks model.  Still, it would be attractive to have an end-to-end
case, particularly because we're mostly talking about some possibly
nasty corruption imposed by a stray write.  Individual disks can suffer
those with low, but noticeable, probability; but storage systems like to
defend agains those things.  In this case, there really aren't any
options for doing so.  It may be the state of the practice, for better
or worse.

		Craig

> Sent: Friday, August 24, 2007 3:44 PM
> To: Everhart, Craig; [email-masked]
> The server needs to add in an allowance beyond when the lease 
> should have expired at the client.  As for the maximum time 
> that an I/O can be outstanding, there are practical limits.  
> In the blocks world, the usual answer is 30 seconds, as 
> that's the typical timeout used in SCSI multi-path drivers 
> for determining when to redrive an I/O for which no response 
> has been seen.  This is supported by engineering of the 
> infrastructure and devices - for example, Fibre Channel 
> guarantees to deliver or discard a frame in at most 10 
> seconds - while typical delivery times are several orders of 
> magnitude better, that is the absolute maximum and is usually 
> enforced by configuration limits.
> 
> Thanks,
> --David

---

From: [email-masked]
Date: Fri, 24 Aug 2007 18:58:01 -0400

Dave,

This discussion seems to have morphed into trying to solve the

> received after it is sent (no FTL-moving servers :-), and before 

The problem is not that the client needs to know if/when the 
server thinks the lease expired.  The problem is the server
needs to know how long it has to wait to be assured that an
uncommunicative client has stopped using its layouts, making
unilateral reclaim of layout resources safe.  That's a different
problem, and it can be solved in the following fashion:
- After some period of time <X> of inability to talk to the
	server, the client stops using layouts.  The client uses
	a heartbeat operation as needed (e.g., every <X>/2) to
	stay inside of this.  The client side rule is simple:
	if <X> has elapsed since sending of an operation for which
	a server reply was received, then layouts cannot be used
	(if <X> is large enough that possible communication delays
	 are noise with respect to it, it's sufficient to track
	 when replies arrive from the server).  <X> should be set
	reasonably large - 5 seconds strikes me as too short, but
	I don't know what we use in practice off the top of my head.
- So, if the server hasn't heard from the client in 2 * <X>
	plus some allowance for I/O flight time (Note: not the
	smallest possible time period, but sufficient), the server
	knows that the client has stopped using its layouts, and
	can unilaterally reclaim resources from the client safely.
- Next time the server hears from the client, the client has to
	receive some sort of error notification - as opposed to the
	"you have a problem" status bit, forcing the client to
	behave as if the server has rebooted is a robust solution
	as it forces checks of everything the server could have
	reclaimed.
Note that whether to force the client to behave as if the server
rebooted is the server's decision, so the server can wait as long
as it likes before forcing the client to do this - if a client
request pops out after 5 * <X> and the server hasn't reclaimed
any state, the server can continue as if nothing went wrong and
the client will do likewise.  Keep in mind that <X> needs to be
set sufficiently large so that client's almost never run into it.

This isn't just invented off the top of my head - EMC has
deployed "running code" that uses this approach.  It does have
a dependence on the maximum I/O latency discussed in the exchange
with Craig - the server has to be sure that there are no client
I/O's in flight when it unilaterally reclaims client layout
resources.  I'd hoped to be able to use leasing for this as
opposed to reinvent something:

> won't work.

The extension I outlined above that makes this work is that the
server tells the client something about how the server will make
its decision - it tells the client what <X> is, and waits much
longer than <X> before reclaiming resources.  Given the level
of trust already placed in the block layout client, trusting it
to respect <X> and stop using layouts when <X> expires is
within reason.

I'm prepared to spec this out as part of the block layout (we
know how to do this, because EMC has working code that does it),
but we need the flexibility in the main draft to allow this
technique.  I'm concerned that a mandatory requirement for client
fencing at the DS's will severely constrain usability of the
block layout in practice, plus there's "running code" that
strongly suggests that such a requirement is not necessary.

There are lots of cases in which block storage systems
have things other than pNFS to do, and those can easily result
in administrative barriers to the managerial operations
necessary to fence off a client.  In addition, the interfaces
for client fencing (LUN mapping and masking) are not simple
in a number of ways, and have non-standard elements (e.g.,
most arrays don't use the SCSI standard LUN mapping and
masking commands).

Beyond this it sounds like the status bits you describe
may need to be applied to pNFS layout operations - for the
block layout, if layout resources are ever reclaimed
unilaterally, it may be better to force the client to behave
as if the server has rebooted.

Thanks,
--David

> From: Noveck, Dave [mailto:[email-masked]] 
> Sent: Thursday, August 23, 2007 3:49 PM
> To: Black, David; [email-masked]

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 26 Aug 2007 21:58:16 -0400

> The problem is not that the client needs to know if/when the server
thinks 
> the lease expired.  

I think it is, and trying to create a second problem that is almost the
same but different just adds to the complexity of the protocol and the
solution.

> The problem is the server needs to know how long it has to wait to be 
> assured that an uncommunicative client has stopped using its layouts, 
> making unilateral reclaim of layout resources safe.  That's a
different 
> problem, 

No, it is the same problem.  Leases are the way v4 determines whether a
client is uncommunicative.  If you create a second concept of
uncommunicativeness that is basically the same except that it replaces
the lease time by some other value <X>, you just make things more
complicated, for no good reason. 

For the client to stop using its layouts, it must be able to determine
that it is uncommunicative, or more precisely it must be able to
determine that it is not assured that the server believes it to be
communicative.  One server judgment about client communicativeness is
governed by lease expiration.  When a lease expires, the server has
judged that the client is uncommunicative.  There is no point in adding
a second concept of uncommunicativeness and then having to deal with
clients that are <X>-uncommunicative and still have a valid lease, or
those that have a lease expired but are not <X>-uncommunicative.  That
makes my head hurt and I suspect that as people try to implement and
debug this stuff, there will many headaches ensuing. 

If the server needs to wait for an "uncommunicative client" to stop
using its layouts, it must be able to determine when that client will
believe that it is uncommunicative.  If we keep to one concept of
communicativeness, that means when the client will determine that the
server may have expired the lease. 

> and it can be solved in the following fashion: 

> The client side rule is simple:

I believe this can work and if <X> is the lease time, then you have the
client determining whether the server may have expired the client's
lease (with possible false positives but no false negatives).  For each
slot you store the send time of the requests.  When you get a response,
if that response successfully executed the SEQUENCE then you store the
send time as the currently updated renew time for the client associated
with that session, if the time is greater than the previous such time,
and not more than the lease time greater than such previous time.

This doesn't deal with the REVOKE cases, but I think you can deal with
those by having the server wait 2X (plus the maximum layout IO service
time) before assuming the layouts have been quasi-fenced.  This gives
the client time to the set the SEQUENCE status bit and for the client to
be sure of seeing it or being judged uncommunicate if there is chance he
hasn't seen it.

Note:  It occurs to me that CREATE_SESSION should probably be treated as
lease-renewing (in addition to SEQUENCE) but this would cause an issue
for the paragraph above since you don't get back the sequence status
bits in that case.

That works if <X> is the lease time, at least I don't see any big holes
right now.  I can't see why we would use anything else.  If you have
something else that there has to be a way to communicate it, etc (e.g.
an attribute).  The lease is a concept to determine client
uncommunicativeness.  Having two such concepts is one too many.

>	as it forces checks of everything the server could have

Maybe it is robust but it seems totally unnecessary.  We have means of
indicating that the server has rebooted.  Why do we need another one?
If the server wants to pretend he has rebooted, he can do that.

But I don't see any need for that.  We have status bits to indicate that
all or some state has been lost and a means to check whether state is
still valid (i.e. TEST_STATEID).  I think we should make the MDS part of
the protocol for all types of DS's as similar as we possibly can and not
create layout-specific mechanisms if we avoid it.

> request pops out after 5 * <X> and the server hasn't reclaimed any 
> state, the server can continue as if nothing went wrong and the client

> will do likewise.  Keep in mind that <X> needs to be set sufficiently 
> large so that client's almost never run into it.

If the client hasn't lost any state, then no status bits and are set and
the client knows has hasn't lost any state.  Since the client
uncommunicativeness determination cannot avoid false positives (you have
choice between avoiding false positives and false negatives and you have
to choose the latter), the client must not do anything irrevocable until
the server indicates that he has.  Up until then, the client only is
delaying things as a (quasi)-fencing participant. 

> The extension I outlined above that makes this work is that the server

> tells the client something about how the server will make its decision
- 
> it tells the client what <X> is, and waits much longer than <X> before

> reclaiming resources.  Given the level of trust already placed in the 
> block layout client, trusting it to respect <X> and stop using layouts

> when <X> expires is within reason.

I don't know why the client needs to know anything but the lease time.
Once it is possible that the lease may have expired or he gets an
indication that state has been revoked, he stops doing IO to the DS's
(for the block layout).  The server has to delay actually effectuating
the transfer of resources until it is sure that any IO's in the pipeline
are no longer active, but the client doesn't see this value.  If the
lease time is sixty seconds and the time to flush DS IO's, the client
stops doing IO's after the sixty-second value.  If he added the extra
ten seconds things wouldn't work.  It is the server who has to delay the
extra ten seconds and only as far as transferring the resources freed,
the stateid become invalid immediately as well as the appropriate
sequence status bit getting set immediately.

> Given the level of trust already placed in the block layout client, 
> trusting it to respect <X> and stop using layouts when <X> expires is 
> within reason.

Within approximately the same degree of reason.

> Beyond this it sounds like the status bits you describe may need to 
> be applied to pNFS layout operations - for the block layout, if layout

> resources are ever reclaimed unilaterally, it may be better to force 
> the client to behave as if the server has rebooted.

If they have stateids, they will be covered.  As far as state
revocation, It think it would be cleaner to set the appropriate sequence
status bits. 

It seem like this can work for an appropriate definition of "work".  My
concern would be about making changes to the protocol at a fairly late
date.  It was my understanding that this stuff was always intended to be
handled via fencing and it is kind of late in the day to add this new
requirement.

I'd like to hear what other people have to say about this.  Not too many
people have been part of the discussion.  Maybe they're all saying, "I
wish the two Dave's would just shut up", but if so we'll hear that
sooner or later.  I'd like to see if there is a group consensus about
this (either about us shutting up or the underlying issue of what to do
about this :-).  

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, August 24, 2007 6:58 PM
To: Noveck, Dave; [email-masked]
Subject: Client fencing

Dave,

This discussion seems to have morphed into trying to solve the wrong
problem:

> Given that you have N asynchronous parallel messages and you know the 
> send time and the reply time of each, how do you safely (false 
> requests, with the only constraint being that each request is received

> after it is sent (no FTL-moving servers :-), and before the reply is 
> received?

The problem is not that the client needs to know if/when the server
thinks the lease expired.  The problem is the server needs to know how
long it has to wait to be assured that an uncommunicative client has
stopped using its layouts, making unilateral reclaim of layout resources
safe.  That's a different problem, and it can be solved in the following
fashion:
- After some period of time <X> of inability to talk to the
	server, the client stops using layouts.  The client uses
	a heartbeat operation as needed (e.g., every <X>/2) to
	stay inside of this.  The client side rule is simple:
	if <X> has elapsed since sending of an operation for which
	a server reply was received, then layouts cannot be used
	(if <X> is large enough that possible communication delays
	 are noise with respect to it, it's sufficient to track
	 when replies arrive from the server).  <X> should be set
	reasonably large - 5 seconds strikes me as too short, but
	I don't know what we use in practice off the top of my head.
- So, if the server hasn't heard from the client in 2 * <X>
	plus some allowance for I/O flight time (Note: not the
	smallest possible time period, but sufficient), the server
	knows that the client has stopped using its layouts, and
	can unilaterally reclaim resources from the client safely.
- Next time the server hears from the client, the client has to
	receive some sort of error notification - as opposed to the
	"you have a problem" status bit, forcing the client to
	behave as if the server has rebooted is a robust solution
	as it forces checks of everything the server could have
	reclaimed.
Note that whether to force the client to behave as if the server
rebooted is the server's decision, so the server can wait as long as it
likes before forcing the client to do this - if a client request pops
out after 5 * <X> and the server hasn't reclaimed any state, the server
can continue as if nothing went wrong and the client will do likewise.
Keep in mind that <X> needs to be set sufficiently large so that
client's almost never run into it.

This isn't just invented off the top of my head - EMC has deployed
"running code" that uses this approach.  It does have a dependence on
the maximum I/O latency discussed in the exchange with Craig - the
server has to be sure that there are no client I/O's in flight when it
unilaterally reclaims client layout resources.  I'd hoped to be able to
use leasing for this as opposed to reinvent something:

> uncommunicative or actually gone.  However, the server can only make a

> determination that the client is likely to be done.  In the v4 model, 
> his determination, whether correct or not as seen by an omniscient 
> observer, is definitive.  Once the server makes that determination, he

> will release resources which may then be assigned to others.  This is 
> so whether the client really is uncommunicative or whether the client 
> thinks he is uncommunicative.  For resources on the server (MDS) 
> itself, he can make sure that the client does not have access to freed

> resources.  For stuff on the DS's, that won't work.

The extension I outlined above that makes this work is that the server
tells the client something about how the server will make its decision -
it tells the client what <X> is, and waits much longer than <X> before
reclaiming resources.  Given the level of trust already placed in the
block layout client, trusting it to respect <X> and stop using layouts
when <X> expires is within reason.

I'm prepared to spec this out as part of the block layout (we know how
to do this, because EMC has working code that does it), but we need the
flexibility in the main draft to allow this technique.  I'm concerned
that a mandatory requirement for client fencing at the DS's will
severely constrain usability of the block layout in practice, plus
there's "running code" that strongly suggests that such a requirement is
not necessary.

There are lots of cases in which block storage systems have things other
than pNFS to do, and those can easily result in administrative barriers
to the managerial operations necessary to fence off a client.  In
addition, the interfaces for client fencing (LUN mapping and masking)
are not simple in a number of ways, and have non-standard elements
(e.g., most arrays don't use the SCSI standard LUN mapping and masking
commands).

Beyond this it sounds like the status bits you describe may need to be
applied to pNFS layout operations - for the block layout, if layout
resources are ever reclaimed unilaterally, it may be better to force the
client to behave as if the server has rebooted.

Thanks,
--David

> terminates, finds that it has been awoken more than five seconds too 
> late then, is it right to say that the lease never expires "locally"?

> If so, so what?  The point is that it matters when the server receives

> the message, and you don't know that.  You only know it happens some 
> time between when you send it and when you
> get the reply.   
> 
> > The server then tracks that against its own clock and adds a 
> > generous allowance for clock skew (this is an error case, it just 
> > has to work, it doesn't have to be fast).
> 
> You add the generous allowance (five seconds in this case) to make it 
> unlikely that the lease will expire.  You can be as generous as you 
> want, to reduce that probability, but you can't reduce it to zero.
> 
> > If clients aren't
> Given that you have N asynchronous parallel messages and you know the 
> send time and the reply time of each, how do you safely (false 
> requests, with the only constraint being that each request is received

> after it is sent (no FTL-moving servers :-), and before the reply is 
> received?
> 
> server to determine and created status bits for the server to tell the

> client as soon as it could of the current state of affairs.
> 
> > document could describe a) fencing and b) client cessation of layout

> > access after some sort of lack-of-communication timeout (e.g., lease

> > expiration) as options, then the block layout could use the latter 
> > without imposing it on everyone.  There is quite a bit of SAN 
> > filesystem "running code" out there that does not do involuntary 
> > fencing of clients, and I'd like to avoid invalidating that 
> > implementation approach.
> 
> But that requires that you describe how to safely have the client make

> that lease determination.
> 
> I have discussed this issue with a few other people who would not be 
> OK with this client-centered approach, even if it were limited to the 
> block-layout case only, so we may have difficulty getting to 

> uncommunicative or actually gone.  However, the server can only make a

> determination that the client is likely to be done.  In the v4 model, 
> his determination, whether correct or not as seen by an omniscient 
> observer, is definitive.  Once the server makes that determination, he

> will release resources which may then be assigned to others.  This is 
> so whether the client really is uncommunicative or whether the client 
> thinks he is uncommunicative.  For resources on the server (MDS) 
> itself, he can make sure that the client does not have access to freed

> resources.  For stuff on the DS's, that won't work.  That leads to the

> requirement for fencing, and if you want to change that and accept the

> clients' determination of whether the server may have freed resources,

> you have to be really sure that client can safely determine that the 
> server could not possibly have judged him uncommunicative (as well as 
> being astonishingly trusting in client implementers).
> 
> > > but there is a fair amount of text to describe how this would work

> > > and review it to make sure it really would work.  Until that is 
> > > done, and we know that this workable, a think we have to assume 

> you issue after the revocation of the delegation, will have a status 
> bit telling you something is wrong.  Also any use of the delegation 
> stateid will get you a NFS4ERR_DELEG_REVOKED error.
> 
> take a stateid by using the session and saying that any reference to 
> the directory that that client has a delegation implicitly refers to 
> that delegation.  But that implies (and I don't think the spec says), 
> that if the delegation is revoked, we return an NFS4ERR_DELEG_REVOKED 
> error, even though the op does not have an explicit stateid parameter.
> > for the client to determine when it is possible that the server may 
> > see a lease expiration, is not trivial.
> 
> clients would track lease expirations and know when the lease expired 
> locally.  The server then tracks that against its own clock and adds a

> generous allowance for clock skew (this is an error case, it just has 
> to work, it doesn't have to be fast).  If clients aren't already doing

> this, then you definitely have a point about the additional 
> implementation work:
> 

> > I don't think that is reasonable if only blocks needs it.  I guess I

> > would be OK, if we make this layout-type specific, and either 
> > describe this in the blocks document (if only blocks needs this), or

> > describe it in the base document and then say that it is only 
> > required if you are using a particular layout type that requires it 
> > (and the statement about whether a particular layout type requires 
> > this would be stated in the document for the layout type).
> 
> document could describe a) fencing and b) client cessation of layout 
> access after some sort of lack-of-communication timeout (e.g., lease 
> expiration) as options, then the block layout could use the latter 
> without imposing it on everyone.  There is quite a bit of SAN 
> filesystem "running code" out there that does not do involuntary 
> fencing of clients, and I'd like to avoid invalidating that 
> implementation approach.
> 
> Ok, but the important case to make sure we get right first is loss of 
> communication.  If the client is uncommunicative or actually gone, 
> > but there is a fair amount of text to describe how this would work 
> > and review it to make sure it really would work.  Until that is 
> > done, and we know that this workable, a think we have to assume

---

From: Robert Gordon <[email-masked]>
Date: Mon, 27 Aug 2007 01:21:54 -0500

Let's re-wind back to this text...

> but this prevention (fencing) is OPTIONAL.

I'm 'recalling' my previous "Yes, i prefer that.." comment.

I'm proposing that we state this :-

All layouts using the device IDs remain in force, but the
server MAY fence the client from accessing the affected
data servers until the client has obtained the re-mapped
device IDs via GETDEVICEINFO or GETDEVICELIST.
	
--  

The client already has a mechanism in place to establish if the state
it holds is valid; That is via the lease renewal, either explicitly
sending just a SEQUENCE or implicitly via other compounds with a
SEQUENCE op. If the client has not successfully renewed it's lease
for one lease period then it MUST assume the state it holds is invalid.

If an MDS expires the clients state, then it MUST expire the state at
any associated data-servers.

If an MDS wants to be hard nosed about a DeviceID change, and does not
want a client to perform IO against the data-server then it has the  
option
to recall the effected layouts.

Robert;

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 27 Aug 2007 18:33:16 -0400

> know it happens some time between when you send it and when you 

Even if it could reliably track the lease expirations, the client has no
way to ensure that a write that is sent at time a, will complete before
time b. When should the client stop sending read or write requests in
order to ensure that it respects the lease?

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 27 Aug 2007 18:41:56 -0400

According to Dave Black there is such a way for writes to block devices
(which is what is at issue, here).  I'm unable to verify or contradict
that but I have been working under the assumption that it is the case. 

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Monday, August 27, 2007 6:33 PM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]
Subject: RE: [nfsv4] Device stateids and fencing

Trond

---

From: "Everhart, Craig" <[email-masked]>
Date: Tue, 28 Aug 2007 10:17:32 -0400

I think David Black mentioned working upper bounds in practice, but I
wanted to push on that some to see if there *are* hard limits.  I am
concerned that there are not. 

> From: Noveck, Dave 
> Sent: Monday, August 27, 2007 6:42 PM
> To: Trond Myklebust
> According to Dave Black there is such a way for writes to 
> block devices (which is what is at issue, here).  I'm unable 
> to verify or contradict that but I have been working under 
> the assumption that it is the case. 
> 
> Even if it could reliably track the lease expirations, the 
> client has no way to ensure that a write that is sent at time 
> a, will complete before time b. When should the client stop 
> sending read or write requests in order to ensure that it 
> respects the lease?
> 
> Trond
>

---

From: Benny Halevy <[email-masked]>
Date: Thu, 30 Aug 2007 11:52:18 +0300

> this (either about us shutting up or the underlying issue of what to do

Dave N. asked for more opinions so FWIW, here's mine. :)

Given the limitations of the block layout-type the server SHOULD fence off
clients from storage when it revokes the client's layout or deviceid state.
Specific layout types can and should require that as a MUST if possible.

The client SHOULD keep track of the lease expiration time and refrain from
directly accessing any storage device if the lease has not been successfully 
renewed to provide enough time for the I/O operation to be executed, given
the typical timing characteristics of the client and network implementation.
Or has there been an indication for state revocation in the SEQUENCE
results flag. [and there are more details on that as Dave N. mentioned. This
should probably be described in more details in the spec.]

I'm not sure whether specific layout types can require that as a MUST, even
if the client can determine guaranteed deterministic bounds on the I/O
execution time as unless it has a strict real time operating system,
I doubt it can deterministically bound the queuing time of requests
until they are dispatched on the storage network.

Therefore I agree with Dave N. that the server must be the authoritative entity
with regards to layout and device state revocation.

I also agree that the lease time and the SEQUENCE result flags should be used as
the basis for determining that the client realized that its state may have expired
or revoked. [And the text describing the SEQUENCE status flags should be extended
to cover layouts and deviceids].

I still feel uncomfortable with letting the server reassign client resources
without fencing off clients but I believe Dave B. that we're not making
the current implementation any worse.  That said, I think that the server
behavior in the non-fencing case should be described in general details in the
nfsv4.1 spec rather than the layout-type specific document. And that the block
layout should only add block layout specifics.

Benny

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 31 Aug 2007 11:44:05 -0400

> I still feel uncomfortable with letting the server reassign client 
> resources without fencing off clients 

As I understand it, your discomfort is due to the fact that you have no assurance that this approach can be made to work reliably.  And that means not that you have individual working implementations, but that you believe that you can define a protocol, which when correctly implemented, will prevent the corruption you are worried about from ever happening.

On the other hand, we know the fencing approach would work and that is in line with all of the other pnfs variants. 

> but I believe Dave B. that we're not making the current implementation any worse. 

I do also but I don't think that is the question.  Not requiring fencing seems that it would make the protocol being specified worse (i.e. less reliable), compared to one that does require fencing.  If there is a consensus that that gap can be closed so that we believe this change would not create a hole, then that would be fine.

I'm not naïve.  I know that despite our best efforts including formal review, we may define things that have holes in them.  We are doing our best to avoid them, mainly because dealing with holes found after the fact is a real drag.  But we should not close our eyes to a hole that exists in a standards track protocol or knowingly create one.

Here's where I think we should be.  In this I'm going to use SHOULD and MUST but since these statements are about what the layout-specific protocols should or must do rather than implementations directly, I'm not sure what verbiage is appropriate, but I hope I'm being clear in any case:

The protocol MUST ensure that when resources are transferred to another client, they are not used by the client originally owning them and this must be ensured against any possible combination of partitions and delays among all of the participants to the protocol (DS, MDS, client).

The protocol SHOULD do this by providing fencing, that is, a method where the MDS's decision is made known to the DS's so that any attempt at doing an inappropriate access will be prevented.  Until and unless there is confirmation that the DS's have been notified to reject such use by the original client, transfer of resources to another client MUST NOT occur.

A specific protocol MAY instead use the client to enforce this isolation, for example by using a time-based approach.  However, if it does so, it MUST ensure that no access by the client to the MDS be done or be continuing, when the server transfers these resources, and it must be clear how this guarantee is ensured.  In particular, where the protocol depends on timing bounds, the basis for these bounds must be clear and their reliability convincing. 

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Thursday, August 30, 2007 4:52 AM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] RE: Client fencing

> I'd like to hear what other people have to say about this.  Not too 
> many people have been part of the discussion.  Maybe they're all 
> saying, "I wish the two Dave's would just shut up", but if so we'll 
> hear that sooner or later.  I'd like to see if there is a group 
> consensus about this (either about us shutting up or the underlying 
> issue of what to do about this :-).

Given the limitations of the block layout-type the server SHOULD fence off clients from storage when it revokes the client's layout or deviceid state.
Specific layout types can and should require that as a MUST if possible.

The client SHOULD keep track of the lease expiration time and refrain from directly accessing any storage device if the lease has not been successfully renewed to provide enough time for the I/O operation to be executed, given the typical timing characteristics of the client and network implementation.
Or has there been an indication for state revocation in the SEQUENCE results flag. [and there are more details on that as Dave N. mentioned. This should probably be described in more details in the spec.]

I'm not sure whether specific layout types can require that as a MUST, even if the client can determine guaranteed deterministic bounds on the I/O execution time as unless it has a strict real time operating system, I doubt it can deterministically bound the queuing time of requests until they are dispatched on the storage network.

Therefore I agree with Dave N. that the server must be the authoritative entity with regards to layout and device state revocation.

I also agree that the lease time and the SEQUENCE result flags should be used as the basis for determining that the client realized that its state may have expired or revoked. [And the text describing the SEQUENCE status flags should be extended to cover layouts and deviceids].

I still feel uncomfortable with letting the server reassign client resources without fencing off clients but I believe Dave B. that we're not making the current implementation any worse.  That said, I think that the server behavior in the non-fencing case should be described in general details in the
nfsv4.1 spec rather than the layout-type specific document. And that the block layout should only add block layout specifics.

Benny

Noveck, Dave wrote:
> I think it is, and trying to create a second problem that is almost 
> the same but different just adds to the complexity of the protocol and 
> the solution.
> 
> No, it is the same problem.  Leases are the way v4 determines whether 
> a client is uncommunicative.  If you create a second concept of 
> judged that the client is uncommunicative.  There is no point in 
> adding a second concept of uncommunicativeness and then having to deal 
> with clients that are <X>-uncommunicative and still have a valid 
> lease, or those that have a lease expired but are not 
> <X>-uncommunicative.  That makes my head hurt and I suspect that as 
> people try to implement and debug this stuff, there will many headaches ensuing.
> 
>> if <X> has elapsed since sending of an operation for which a server 
>> reply was received, then layouts cannot be used
> 
> I believe this can work and if <X> is the lease time, then you have 
> the client determining whether the server may have expired the 
> client's lease (with possible false positives but no false negatives).  
> For each slot you store the send time of the requests.  When you get a 
> response, if that response successfully executed the SEQUENCE then you 
> store the send time as the currently updated renew time for the client 
> associated with that session, if the time is greater than the previous 
> such time, and not more than the lease time greater than such previous time.
> 
> the client time to the set the SEQUENCE status bit and for the client 
> to be sure of seeing it or being judged uncommunicate if there is 
> chance he hasn't seen it.
> 
> Note:  It occurs to me that CREATE_SESSION should probably be treated 
> as lease-renewing (in addition to SEQUENCE) but this would cause an 
> issue for the paragraph above since you don't get back the sequence 
> status bits in that case.
> 
> That works if <X> is the lease time, at least I don't see any big 
> holes right now.  I can't see why we would use anything else.  If you 
> have something else that there has to be a way to communicate it, etc (e.g.
> But I don't see any need for that.  We have status bits to indicate 
> that all or some state has been lost and a means to check whether 
> state is still valid (i.e. TEST_STATEID).  I think we should make the 
> MDS part of the protocol for all types of DS's as similar as we 
> possibly can and not create layout-specific mechanisms if we avoid it.
> 
>> rebooted is the server's decision, so the server can wait as long as 
>> it likes before forcing the client to do this - if a client request 
>> pops out after 5 * <X> and the server hasn't reclaimed any state, the 
>> server can continue as if nothing went wrong and the client
> 
> If the client hasn't lost any state, then no status bits and are set 
> and the client knows has hasn't lost any state.  Since the client 
> uncommunicativeness determination cannot avoid false positives (you 
> have choice between avoiding false positives and false negatives and 
> you have to choose the latter), the client must not do anything 
> irrevocable until the server indicates that he has.  Up until then, 
> the client only is delaying things as a (quasi)-fencing participant.
> 
>> server
> the transfer of resources until it is sure that any IO's in the 
> pipeline are no longer active, but the client doesn't see this value.  
> If the lease time is sixty seconds and the time to flush DS IO's, the 
> client stops doing IO's after the sixty-second value.  If he added the 
> extra ten seconds things wouldn't work.  It is the server who has to 
> delay the extra ten seconds and only as far as transferring the 
> resources freed, the stateid become invalid immediately as well as the 
> appropriate sequence status bit getting set immediately.
> 
> revocation, It think it would be cleaner to set the appropriate 
> sequence status bits.
> 
> It seem like this can work for an appropriate definition of "work".  
> My concern would be about making changes to the protocol at a fairly 
> late date.  It was my understanding that this stuff was always 
> intended to be handled via fencing and it is kind of late in the day 
> to add this new requirement.
> 
> I'd like to hear what other people have to say about this.  Not too 
> many people have been part of the discussion.  Maybe they're all 
> saying, "I wish the two Dave's would just shut up", but if so we'll 
> hear that sooner or later.  I'd like to see if there is a group 
> consensus about this (either about us shutting up or the underlying 
> issue of what to do about this :-).
> 
> 
>> received
> stopped using its layouts, making unilateral reclaim of layout 
> resources safe.  That's a different problem, and it can be solved in 
> the following
> fashion:
> rebooted is the server's decision, so the server can wait as long as 
> it likes before forcing the client to do this - if a client request 
> pops out after 5 * <X> and the server hasn't reclaimed any state, the 
> server can continue as if nothing went wrong and the client will do likewise.
> unilaterally reclaims client layout resources.  I'd hoped to be able 
> to use leasing for this as opposed to reinvent something:
> 
>> a
> - it tells the client what <X> is, and waits much longer than <X> 
> before reclaiming resources.  Given the level of trust already placed 
> in the block layout client, trusting it to respect <X> and stop using 
> layouts when <X> expires is within reason.
> 
> to do this, because EMC has working code that does it), but we need 
> the flexibility in the main draft to allow this technique.  I'm 
> concerned that a mandatory requirement for client fencing at the DS's 
> will severely constrain usability of the block layout in practice, 
> plus there's "running code" that strongly suggests that such a 
> requirement is not necessary.
> 
> There are lots of cases in which block storage systems have things 
> other than pNFS to do, and those can easily result in administrative 
> barriers to the managerial operations necessary to fence off a client.  
> In addition, the interfaces for client fencing (LUN mapping and 
> masking) are not simple in a number of ways, and have non-standard 
> elements (e.g., most arrays don't use the SCSI standard LUN mapping 
> and masking commands).
> 
>> received
>> the
>> make
>> a