Subject: Re: [nfsv4] RE: Client fencing

From: Benny Halevy <[email-masked]>
Date: Sun, 02 Sep 2007 09:44:14 +0300

Noveck, Dave wrote:
>> I still feel uncomfortable with letting the server reassign client 
>> resources without fencing off clients 
> 
> As I understand it, your discomfort is due to the fact that you have no assurance that this approach can be made to work reliably.  And that means not that you have individual working implementations, but that you believe that you can define a protocol, which when correctly implemented, will prevent the corruption you are worried about from ever happening.
> 
> On the other hand, we know the fencing approach would work and that is in line with all of the other pnfs variants. 
> 
>> but I believe Dave B. that we're not making the current implementation any worse. 
> 
> I do also but I don't think that is the question.  Not requiring fencing seems that it would make the protocol being specified worse (i.e. less reliable), compared to one that does require fencing.  If there is a consensus that that gap can be closed so that we believe this change would not create a hole, then that would be fine.
> 
> I'm not naïve.  I know that despite our best efforts including formal review, we may define things that have holes in them.  We are doing our best to avoid them, mainly because dealing with holes found after the fact is a real drag.  But we should not close our eyes to a hole that exists in a standards track protocol or knowingly create one.
> 
> Here's where I think we should be.  In this I'm going to use SHOULD and MUST but since these statements are about what the layout-specific protocols should or must do rather than implementations directly, I'm not sure what verbiage is appropriate, but I hope I'm being clear in any case:
> 
> The protocol MUST ensure that when resources are transferred to another client, they are not used by the client originally owning them and this must be ensured against any possible combination of partitions and delays among all of the participants to the protocol (DS, MDS, client).
> 
> The protocol SHOULD do this by providing fencing, that is, a method where the MDS's decision is made known to the DS's so that any attempt at doing an inappropriate access will be prevented.  Until and unless there is confirmation that the DS's have been notified to reject such use by the original client, transfer of resources to another client MUST NOT occur.
> 
> A specific protocol MAY instead use the client to enforce this isolation, for example by using a time-based approach.  However, if it does so, it MUST ensure that no access by the client to the MDS be done or be continuing, when the server transfers these resources, and it must be clear how this guarantee is ensured.  In particular, where the protocol depends on timing bounds, the basis for these bounds must be clear and their reliability convincing. 

This approach sounds reasonable.

Benny

> 
> -----Original Message-----
> From: Benny Halevy [mailto:[email-masked]] 
> Sent: Thursday, August 30, 2007 4:52 AM
> To: Noveck, Dave
> Cc: [email-masked]; [email-masked]
> Subject: Re: [nfsv4] RE: Client fencing
> 
>> I'd like to hear what other people have to say about this.  Not too 
>> many people have been part of the discussion.  Maybe they're all 
>> saying, "I wish the two Dave's would just shut up", but if so we'll 
>> hear that sooner or later.  I'd like to see if there is a group 
>> consensus about this (either about us shutting up or the underlying 
>> issue of what to do about this :-).
> 
> Dave N. asked for more opinions so FWIW, here's mine. :)
> 
> Given the limitations of the block layout-type the server SHOULD fence off clients from storage when it revokes the client's layout or deviceid state.
> Specific layout types can and should require that as a MUST if possible.
> 
> The client SHOULD keep track of the lease expiration time and refrain from directly accessing any storage device if the lease has not been successfully renewed to provide enough time for the I/O operation to be executed, given the typical timing characteristics of the client and network implementation.
> Or has there been an indication for state revocation in the SEQUENCE results flag. [and there are more details on that as Dave N. mentioned. This should probably be described in more details in the spec.]
> 
> I'm not sure whether specific layout types can require that as a MUST, even if the client can determine guaranteed deterministic bounds on the I/O execution time as unless it has a strict real time operating system, I doubt it can deterministically bound the queuing time of requests until they are dispatched on the storage network.
> 
> Therefore I agree with Dave N. that the server must be the authoritative entity with regards to layout and device state revocation.
> 
> I also agree that the lease time and the SEQUENCE result flags should be used as the basis for determining that the client realized that its state may have expired or revoked. [And the text describing the SEQUENCE status flags should be extended to cover layouts and deviceids].
> 
> I still feel uncomfortable with letting the server reassign client resources without fencing off clients but I believe Dave B. that we're not making the current implementation any worse.  That said, I think that the server behavior in the non-fencing case should be described in general details in the
> nfsv4.1 spec rather than the layout-type specific document. And that the block layout should only add block layout specifics.
> 
> Benny
> 
> Noveck, Dave wrote:
>>> The problem is not that the client needs to know if/when the server
>> thinks
>>> the lease expired.  
>> I think it is, and trying to create a second problem that is almost 
>> the same but different just adds to the complexity of the protocol and 
>> the solution.
>>
>>> The problem is the server needs to know how long it has to wait to be 
>>> assured that an uncommunicative client has stopped using its layouts, 
>>> making unilateral reclaim of layout resources safe.  That's a
>> different
>>> problem,
>> No, it is the same problem.  Leases are the way v4 determines whether 
>> a client is uncommunicative.  If you create a second concept of 
>> uncommunicativeness that is basically the same except that it replaces 
>> the lease time by some other value <X>, you just make things more 
>> complicated, for no good reason.
>>
>> For the client to stop using its layouts, it must be able to determine 
>> that it is uncommunicative, or more precisely it must be able to 
>> determine that it is not assured that the server believes it to be 
>> communicative.  One server judgment about client communicativeness is 
>> governed by lease expiration.  When a lease expires, the server has 
>> judged that the client is uncommunicative.  There is no point in 
>> adding a second concept of uncommunicativeness and then having to deal 
>> with clients that are <X>-uncommunicative and still have a valid 
>> lease, or those that have a lease expired but are not 
>> <X>-uncommunicative.  That makes my head hurt and I suspect that as 
>> people try to implement and debug this stuff, there will many headaches ensuing.
>>
>> If the server needs to wait for an "uncommunicative client" to stop 
>> using its layouts, it must be able to determine when that client will 
>> believe that it is uncommunicative.  If we keep to one concept of 
>> communicativeness, that means when the client will determine that the 
>> server may have expired the lease.
>>
>>> and it can be solved in the following fashion: 
>>> The client side rule is simple:
>>> if <X> has elapsed since sending of an operation for which a server 
>>> reply was received, then layouts cannot be used
>> I believe this can work and if <X> is the lease time, then you have 
>> the client determining whether the server may have expired the 
>> client's lease (with possible false positives but no false negatives).  
>> For each slot you store the send time of the requests.  When you get a 
>> response, if that response successfully executed the SEQUENCE then you 
>> store the send time as the currently updated renew time for the client 
>> associated with that session, if the time is greater than the previous 
>> such time, and not more than the lease time greater than such previous time.
>>
>> This doesn't deal with the REVOKE cases, but I think you can deal with 
>> those by having the server wait 2X (plus the maximum layout IO service
>> time) before assuming the layouts have been quasi-fenced.  This gives 
>> the client time to the set the SEQUENCE status bit and for the client 
>> to be sure of seeing it or being judged uncommunicate if there is 
>> chance he hasn't seen it.
>>
>> Note:  It occurs to me that CREATE_SESSION should probably be treated 
>> as lease-renewing (in addition to SEQUENCE) but this would cause an 
>> issue for the paragraph above since you don't get back the sequence 
>> status bits in that case.
>>
>> That works if <X> is the lease time, at least I don't see any big 
>> holes right now.  I can't see why we would use anything else.  If you 
>> have something else that there has to be a way to communicate it, etc (e.g.
>> an attribute).  The lease is a concept to determine client 
>> uncommunicativeness.  Having two such concepts is one too many.
>>
>>> - Next time the server hears from the client, the client has to
>>> 	receive some sort of error notification - as opposed to the
>>> 	"you have a problem" status bit, forcing the client to
>>> 	behave as if the server has rebooted is a robust solution
>>> 	as it forces checks of everything the server could have
>>> 	reclaimed.
>> Maybe it is robust but it seems totally unnecessary.  We have means of 
>> indicating that the server has rebooted.  Why do we need another one?
>> If the server wants to pretend he has rebooted, he can do that.
>>
>> But I don't see any need for that.  We have status bits to indicate 
>> that all or some state has been lost and a means to check whether 
>> state is still valid (i.e. TEST_STATEID).  I think we should make the 
>> MDS part of the protocol for all types of DS's as similar as we 
>> possibly can and not create layout-specific mechanisms if we avoid it.
>>
>>> Note that whether to force the client to behave as if the server 
>>> rebooted is the server's decision, so the server can wait as long as 
>>> it likes before forcing the client to do this - if a client request 
>>> pops out after 5 * <X> and the server hasn't reclaimed any state, the 
>>> server can continue as if nothing went wrong and the client
>>> will do likewise.  Keep in mind that <X> needs to be set sufficiently 
>>> large so that client's almost never run into it.
>> If the client hasn't lost any state, then no status bits and are set 
>> and the client knows has hasn't lost any state.  Since the client 
>> uncommunicativeness determination cannot avoid false positives (you 
>> have choice between avoiding false positives and false negatives and 
>> you have to choose the latter), the client must not do anything 
>> irrevocable until the server indicates that he has.  Up until then, 
>> the client only is delaying things as a (quasi)-fencing participant.
>>
>>> The extension I outlined above that makes this work is that the 
>>> server
>>> tells the client something about how the server will make its 
>>> decision
>> -
>>> it tells the client what <X> is, and waits much longer than <X> 
>>> before
>>> reclaiming resources.  Given the level of trust already placed in the 
>>> block layout client, trusting it to respect <X> and stop using 
>>> layouts
>>> when <X> expires is within reason.
>> I don't know why the client needs to know anything but the lease time.
>> Once it is possible that the lease may have expired or he gets an 
>> indication that state has been revoked, he stops doing IO to the DS's 
>> (for the block layout).  The server has to delay actually effectuating 
>> the transfer of resources until it is sure that any IO's in the 
>> pipeline are no longer active, but the client doesn't see this value.  
>> If the lease time is sixty seconds and the time to flush DS IO's, the 
>> client stops doing IO's after the sixty-second value.  If he added the 
>> extra ten seconds things wouldn't work.  It is the server who has to 
>> delay the extra ten seconds and only as far as transferring the 
>> resources freed, the stateid become invalid immediately as well as the 
>> appropriate sequence status bit getting set immediately.
>>
>>> Given the level of trust already placed in the block layout client, 
>>> trusting it to respect <X> and stop using layouts when <X> expires is 
>>> within reason.
>> Within approximately the same degree of reason.
>>
>>> Beyond this it sounds like the status bits you describe may need to 
>>> be applied to pNFS layout operations - for the block layout, if 
>>> layout
>>> resources are ever reclaimed unilaterally, it may be better to force 
>>> the client to behave as if the server has rebooted.
>> If they have stateids, they will be covered.  As far as state 
>> revocation, It think it would be cleaner to set the appropriate 
>> sequence status bits.
>>
>> It seem like this can work for an appropriate definition of "work".  
>> My concern would be about making changes to the protocol at a fairly 
>> late date.  It was my understanding that this stuff was always 
>> intended to be handled via fencing and it is kind of late in the day 
>> to add this new requirement.
>>
>> I'd like to hear what other people have to say about this.  Not too 
>> many people have been part of the discussion.  Maybe they're all 
>> saying, "I wish the two Dave's would just shut up", but if so we'll 
>> hear that sooner or later.  I'd like to see if there is a group 
>> consensus about this (either about us shutting up or the underlying 
>> issue of what to do about this :-).
>>
>>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]]
>> Sent: Friday, August 24, 2007 6:58 PM
>> To: Noveck, Dave; [email-masked]
>> Subject: Client fencing
>>
>> Dave,
>>
>> This discussion seems to have morphed into trying to solve the wrong
>> problem:
>>
>>> Given that you have N asynchronous parallel messages and you know the 
>>> send time and the reply time of each, how do you safely (false 
>>> positives are OK but false negatives are not) determine whether the 
>>> server may have a gap of L seconds between receiving successive 
>>> requests, with the only constraint being that each request is 
>>> received
>>> after it is sent (no FTL-moving servers :-), and before the reply is 
>>> received?
>> The problem is not that the client needs to know if/when the server 
>> thinks the lease expired.  The problem is the server needs to know how 
>> long it has to wait to be assured that an uncommunicative client has 
>> stopped using its layouts, making unilateral reclaim of layout 
>> resources safe.  That's a different problem, and it can be solved in 
>> the following
>> fashion:
>> - After some period of time <X> of inability to talk to the
>> 	server, the client stops using layouts.  The client uses
>> 	a heartbeat operation as needed (e.g., every <X>/2) to
>> 	stay inside of this.  The client side rule is simple:
>> 	if <X> has elapsed since sending of an operation for which
>> 	a server reply was received, then layouts cannot be used
>> 	(if <X> is large enough that possible communication delays
>> 	 are noise with respect to it, it's sufficient to track
>> 	 when replies arrive from the server).  <X> should be set
>> 	reasonably large - 5 seconds strikes me as too short, but
>> 	I don't know what we use in practice off the top of my head.
>> - So, if the server hasn't heard from the client in 2 * <X>
>> 	plus some allowance for I/O flight time (Note: not the
>> 	smallest possible time period, but sufficient), the server
>> 	knows that the client has stopped using its layouts, and
>> 	can unilaterally reclaim resources from the client safely.
>> - Next time the server hears from the client, the client has to
>> 	receive some sort of error notification - as opposed to the
>> 	"you have a problem" status bit, forcing the client to
>> 	behave as if the server has rebooted is a robust solution
>> 	as it forces checks of everything the server could have
>> 	reclaimed.
>> Note that whether to force the client to behave as if the server 
>> rebooted is the server's decision, so the server can wait as long as 
>> it likes before forcing the client to do this - if a client request 
>> pops out after 5 * <X> and the server hasn't reclaimed any state, the 
>> server can continue as if nothing went wrong and the client will do likewise.
>> Keep in mind that <X> needs to be set sufficiently large so that 
>> client's almost never run into it.
>>
>> This isn't just invented off the top of my head - EMC has deployed 
>> "running code" that uses this approach.  It does have a dependence on 
>> the maximum I/O latency discussed in the exchange with Craig - the 
>> server has to be sure that there are no client I/O's in flight when it 
>> unilaterally reclaims client layout resources.  I'd hoped to be able 
>> to use leasing for this as opposed to reinvent something:
>>
>>> The basic motivation of leasing is to find the client that is 
>>> uncommunicative or actually gone.  However, the server can only make 
>>> a
>>> determination that the client is likely to be done.  In the v4 model, 
>>> his determination, whether correct or not as seen by an omniscient 
>>> observer, is definitive.  Once the server makes that determination, 
>>> he
>>> will release resources which may then be assigned to others.  This is 
>>> so whether the client really is uncommunicative or whether the client 
>>> thinks he is uncommunicative.  For resources on the server (MDS) 
>>> itself, he can make sure that the client does not have access to 
>>> freed
>>> resources.  For stuff on the DS's, that won't work.
>> The extension I outlined above that makes this work is that the server 
>> tells the client something about how the server will make its decision 
>> - it tells the client what <X> is, and waits much longer than <X> 
>> before reclaiming resources.  Given the level of trust already placed 
>> in the block layout client, trusting it to respect <X> and stop using 
>> layouts when <X> expires is within reason.
>>
>> I'm prepared to spec this out as part of the block layout (we know how 
>> to do this, because EMC has working code that does it), but we need 
>> the flexibility in the main draft to allow this technique.  I'm 
>> concerned that a mandatory requirement for client fencing at the DS's 
>> will severely constrain usability of the block layout in practice, 
>> plus there's "running code" that strongly suggests that such a 
>> requirement is not necessary.
>>
>> There are lots of cases in which block storage systems have things 
>> other than pNFS to do, and those can easily result in administrative 
>> barriers to the managerial operations necessary to fence off a client.  
>> In addition, the interfaces for client fencing (LUN mapping and 
>> masking) are not simple in a number of ways, and have non-standard 
>> elements (e.g., most arrays don't use the SCSI standard LUN mapping 
>> and masking commands).
>>
>> Beyond this it sounds like the status bits you describe may need to be 
>> applied to pNFS layout operations - for the block layout, if layout 
>> resources are ever reclaimed unilaterally, it may be better to force 
>> the client to behave as if the server has rebooted.
>>
>> Thanks,
>> --David
>>
>>
>>> -----Original Message-----
>>> From: Noveck, Dave [mailto:[email-masked]]
>>> Sent: Thursday, August 23, 2007 3:49 PM
>>> To: Black, David; [email-masked]
>>> Subject: RE: [nfsv4] Device stateids and fencing
>>>
>>>>>> At least for lease expiration, I expect the block
>>> clients to track
>>>>>> the lease behavior and do the right thing.
>>>>> Maybe that's what you expect but for this to work, the
>>> protocol has
>>>>> to require that this implementation be done, and
>>> describe, at least
>>>>> in general terms, how it is to be done.  Given
>>> transmission delays,
>>>>> for the client to determine when it is possible that the server may 
>>>>> see a lease expiration, is not trivial.
>>>> I may be operating under a false assumption.  I had assumed that 
>>>> clients would track lease expirations and know when the lease 
>>>> expired locally.
>>> I don't know what it really means for the lease to expire locally.
>>> Let's say I have this process that wakes up every lease-time minus 
>>> five seconds and a sends message if one hasn't been sent in the 
>>> interim.  So assume that this process never, when the delay 
>>> terminates, finds that it has been awoken more than five seconds too 
>>> late then, is it right to say that the lease never expires "locally"?
>>> If so, so what?  The point is that it matters when the server 
>>> receives
>>> the message, and you don't know that.  You only know it happens some 
>>> time between when you send it and when you
>>> get the reply.   
>>>
>>>> The server then tracks that against its own clock and adds a 
>>>> generous allowance for clock skew (this is an error case, it just 
>>>> has to work, it doesn't have to be fast).
>>> You add the generous allowance (five seconds in this case) to make it 
>>> unlikely that the lease will expire.  You can be as generous as you 
>>> want, to reduce that probability, but you can't reduce it to zero.
>>>
>>>> If clients aren't
>>>> already doing this, then you definitely have a point about the 
>>>> additional implementation work:
>>> The issue is also (beside implementation work), specification work.
>>> The spec does not describe how to correctly make this determination.
>>> Given that you have N asynchronous parallel messages and you know the 
>>> send time and the reply time of each, how do you safely (false 
>>> positives are OK but false negatives are not) determine whether the 
>>> server may have a gap of L seconds between receiving successive 
>>> requests, with the only constraint being that each request is 
>>> received
>>> after it is sent (no FTL-moving servers :-), and before the reply is 
>>> received?
>>>
>>> I'm not saying this can't be done, but so far nobody has clearly 
>>> described how to do it.  RFC 3530 had a vague hand-wavy statement 
>>> which basically told you to do it and use the determination to 
>>> determine whether your lease had expired.  I don't know if anybody 
>>> actually tried to do this or what validation was done.  When we did 
>>> sessions, we scratched this and made this the responsibility of the 
>>> server to determine and created status bits for the server to tell 
>>> the
>>> client as soon as it could of the current state of affairs.
>>>
>>>>> With your "MUST NOT", all clients would be forced
>>> implement this and
>>>>> I don't think that is reasonable if only blocks needs it. 
>>>  I guess
>>>>> I would be OK, if we make this layout-type specific, and either 
>>>>> describe this in the blocks document (if only blocks needs this), 
>>>>> or describe it in the base document and then say that it is only 
>>>>> required if you are using a particular layout type that requires it 
>>>>> (and the statement about whether a particular layout type requires 
>>>>> this would be stated in the document for the
>>> layout type).
>>>
>>>> My concerns are definitely block-layout-specific.  If the base 
>>>> document could describe a) fencing and b) client cessation of layout
>>>> access after some sort of lack-of-communication timeout (e.g., lease
>>>> expiration) as options, then the block layout could use the latter 
>>>> without imposing it on everyone.  There is quite a bit of SAN 
>>>> filesystem "running code" out there that does not do involuntary 
>>>> fencing of clients, and I'd like to avoid invalidating that 
>>>> implementation approach.
>>> But that requires that you describe how to safely have the client 
>>> make
>>> that lease determination.
>>>
>>> I have discussed this issue with a few other people who would not be 
>>> OK with this client-centered approach, even if it were limited to the 
>>> block-layout case only, so we may have difficulty getting to 
>>> consensus.  I'll let those people speak for themselves.
>>>
>>>>>> For recall and the like, the server waits out the lease
>>> expiration,
>>>>>> and carries on, relying on the client to observe the
>>> lease expiration
>>>>>> restrictions.
>>>>> But there isn't necessarily going to be a lease expiration, that a
>>>>> client could see, unless the server is going to do
>>> something drastic,
>>>>> like not respond to any COMPOUNDs while a recall is outstanding.
>>>>> If you respond to requests, the client has to assume that
>>> the lease
>>>>> is being renewed, as it would be expected to be.
>>>> Ok, but the important case to make sure we get right first is loss 
>>>> of communication.  If the client is uncommunicative or
>>> actually gone,
>>>> there has to be a deterministic way to recover its resources.
>>> I think the problem with phrasing things that way is that it implies 
>>> that other cases (transient loss of communication or delays) are not 
>>> important, and I think anything that can cause data corruption is 
>>> very
>>> important indeed.
>>>
>>> The basic motivation of leasing is to find the client that is 
>>> uncommunicative or actually gone.  However, the server can only make 
>>> a
>>> determination that the client is likely to be done.  In the v4 model, 
>>> his determination, whether correct or not as seen by an omniscient 
>>> observer, is definitive.  Once the server makes that determination, 
>>> he
>>> will release resources which may then be assigned to others.  This is 
>>> so whether the client really is uncommunicative or whether the client 
>>> thinks he is uncommunicative.  For resources on the server (MDS) 
>>> itself, he can make sure that the client does not have access to 
>>> freed
>>> resources.  For stuff on the DS's, that won't work.  That leads to 
>>> the
>>> requirement for fencing, and if you want to change that and accept 
>>> the
>>> clients' determination of whether the server may have freed 
>>> resources,
>>> you have to be really sure that client can safely determine that the 
>>> server could not possibly have judged him uncommunicative (as well as 
>>> being astonishingly trusting in client implementers).
>>>
>>>>> Suppose the server does a recall, and the request gets stuck in a 
>>>>> buggy router, for a minute.  Meanwhile the client is doing stuff, 
>>>>> and getting his lease renewed.  He doesn't even know about the 
>>>>> recall.  Maybe he can find out about by a status bit in SEQUENCE, 
>>>>> but there is a fair amount of text to describe how this would work
>>>>> and review it to make sure it really would work.  Until that is 
>>>>> done, and we know that this workable, a think we have to assume 
>>>>> that the data servers have the responsibility to do the
>>> fencing.    
>>>> Isn't this already a problem in the absence of pNFS?  What if the 
>>>> request that gets stuck in a buggy router is a directory delegation 
>>>> recall?
>>> It is a problem and it is dealt with although you have pointed out an 
>>> issue that we have to clarify the spec on.
>>>
>>> Let's take the case of an ordinary delegation first.  The next 
>>> request
>>> you issue after the revocation of the delegation, will have a status 
>>> bit telling you something is wrong.  Also any use of the delegation 
>>> stateid will get you a NFS4ERR_DELEG_REVOKED error.
>>>
>>> Now with the directory delegation we will also get the status but we 
>>> have avoided the requirement to change CREATE, REMOVE, RENAME, etc to 
>>> take a stateid by using the session and saying that any reference to 
>>> the directory that that client has a delegation implicitly refers to 
>>> that delegation.  But that implies (and I don't think the spec says), 
>>> that if the delegation is revoked, we return an NFS4ERR_DELEG_REVOKED 
>>> error, even though the op does not have an explicit stateid parameter.
>>> Similarly for NFS4ERR_EXPIRED and NFS4ERR_ADMIN_REVOKED.
>>>
>>>
>>> -----Original Message-----
>>> From: [email-masked] [mailto:[email-masked]]
>>> Sent: Thursday, August 23, 2007 11:09 AM
>>> To: Noveck, Dave; [email-masked]
>>> Subject: RE: [nfsv4] Device stateids and fencing
>>>
>>>
>>> Dave,
>>>
>>>>> At least for lease expiration, I expect the block clients
>>> to track
>>>>> the lease behavior and do the right thing.
>>>> Maybe that's what you expect but for this to work, the protocol has 
>>>> to require that this implementation be done, and describe, at least 
>>>> in general terms, how it is to be done.  Given transmission delays, 
>>>> for the client to determine when it is possible that the server may 
>>>> see a lease expiration, is not trivial.
>>> I may be operating under a false assumption.  I had assumed that 
>>> clients would track lease expirations and know when the lease expired 
>>> locally.  The server then tracks that against its own clock and adds 
>>> a
>>> generous allowance for clock skew (this is an error case, it just has 
>>> to work, it doesn't have to be fast).  If clients aren't already 
>>> doing
>>> this, then you definitely have a point about the additional 
>>> implementation work:
>>>
>>>> With your "MUST NOT", all clients would be forced implement this and
>>>> I don't think that is reasonable if only blocks needs it.  I guess I
>>>> would be OK, if we make this layout-type specific, and either 
>>>> describe this in the blocks document (if only blocks needs this), or
>>>> describe it in the base document and then say that it is only 
>>>> required if you are using a particular layout type that requires it 
>>>> (and the statement about whether a particular layout type requires 
>>>> this would be stated in the document for the layout type).
>>> My concerns are definitely block-layout-specific.  If the base 
>>> document could describe a) fencing and b) client cessation of layout 
>>> access after some sort of lack-of-communication timeout (e.g., lease
>>> expiration) as options, then the block layout could use the latter 
>>> without imposing it on everyone.  There is quite a bit of SAN 
>>> filesystem "running code" out there that does not do involuntary 
>>> fencing of clients, and I'd like to avoid invalidating that 
>>> implementation approach.
>>>
>>>>> For recall and the like, the server waits out the lease
>>> expiration,
>>>>> and carries on, relying on the client to observe the
>>> lease expiration
>>>>> restrictions.
>>>> But there isn't necessarily going to be a lease expiration, that a 
>>>> client could see, unless the server is going to do
>>> something drastic,
>>>> like not respond to any COMPOUNDs while a recall is outstanding.
>>>> If you respond to requests, the client has to assume that the lease 
>>>> is being renewed, as it would be expected to be.
>>> Ok, but the important case to make sure we get right first is loss of 
>>> communication.  If the client is uncommunicative or actually gone, 
>>> there has to be a deterministic way to recover its resources.
>>>
>>>> Suppose the server does a recall, and the request gets stuck in a 
>>>> buggy router, for a minute.  Meanwhile the client is doing stuff, 
>>>> and getting his lease renewed.  He doesn't even know about the 
>>>> recall.  Maybe he can find out about by a status bit in SEQUENCE, 
>>>> but there is a fair amount of text to describe how this would work 
>>>> and review it to make sure it really would work.  Until that is 
>>>> done, and we know that this workable, a think we have to assume
>>>> that the data servers have the responsibility to do the fencing.    
>>> Isn't this already a problem in the absence of pNFS?  What if the 
>>> request that gets stuck in a buggy router is a directory delegation 
>>> recall?
>>>
>>> Thanks,
>>> --David
>>> ----------------------------------------------------
>>> David L. Black, Senior Technologist
>>> EMC Corporation, 176 South St., Hopkinton, MA  01748
>>> +1 (508) 293-7953             FAX: +1 (508) 293-7786
>>> [email-masked]        Mobile: +1 (978) 394-7754
>>> ----------------------------------------------------
>>>
>>>
>> _______________________________________________
>> nfsv4 mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: [email-masked]
Date: Tue, 11 Sep 2007 01:11:48 -0400

Getting back to this after having been diverted by other things going on ... 

> > As I understand it, your discomfort is due to the fact that 
> > you have no assurance that this approach can be made to work 
> > reliably.  And that means not that you have individual 
> > working implementations, but that you believe that you can 
> > define a protocol, which when correctly implemented, will 
> > prevent the corruption you are worried about from ever happening.
> > 
> > On the other hand, we know the fencing approach would work 
> > and that is in line with all of the other pnfs variants. 
> > 
> > I do also but I don't think that is the question.  Not 
> > requiring fencing seems that it would make the protocol being 
> > specified worse (i.e. less reliable), compared to one that 
> > does require fencing.  If there is a consensus that that gap 
> > can be closed so that we believe this change would not create 
> > a hole, then that would be fine.
> > 
> > I'm not naïve.  I know that despite our best efforts 
> > including formal review, we may define things that have holes 
> > in them.  We are doing our best to avoid them, mainly because 
> > dealing with holes found after the fact is a real drag.  But 
> > we should not close our eyes to a hole that exists in a 
> > standards track protocol or knowingly create one.
> > 
> > Here's where I think we should be.  In this I'm going to 
> > use SHOULD and MUST but since these statements are about what 
> > the layout-specific protocols should or must do rather than 
> > implementations directly, I'm not sure what verbiage is 
> > appropriate, but I hope I'm being clear in any case:
> > 
> > The protocol MUST ensure that when resources are 
> > transferred to another client, they are not used by the 
> >client originally owning them and this must be ensured 
> > against any possible combination of partitions and delays 
> > among all of the participants to the protocol (DS, MDS, client).

I agree.

> > The protocol SHOULD do this by providing fencing, that is, 
> > a method where the MDS's decision is made known to the DS's 
> > so that any attempt at doing an inappropriate access will be 
> > prevented.  Until and unless there is confirmation that the 
> > DS's have been notified to reject such use by the original 
> > client, transfer of resources to another client MUST NOT occur.

I won't argue with this, as fencing is certainly the most
straightforward way to do this, when its reasonably feasible.
There are block layout cases in which fencing is feasible, and
there are cases where it is rather difficult.

> > A specific protocol MAY instead use the client to enforce 
> > this isolation, for example by using a time-based approach.  
> > However, if it does so, it MUST ensure that no access by the 
> > client to the MDS be done or be continuing, when the server 
> > transfers these resources, and it must be clear how this 
> > guarantee is ensured.  In particular, where the protocol 
> > depends on timing bounds, the basis for these bounds must be 
> > clear and their reliability convincing. 
> 

I think the block layout can live with this ...

The block layout will make client tracking of possible lease
expiration, and the client ceasing to use the layout when a lease
may have expired a MUST.  We will also craft some text on upper bounds
on I/Os.  In Fibre Channel environments, 30 sec is a typical upper
bound, as multipath drivers will retry an I/O after 30sec and HA
cluster software will declare a failure after 60sec (and FC guarantees
that a frame will be delivered or dropped within 10 sec).  Having
I/O's linger after a retry, or especially after a failure is declared
can cause all sorts of trouble, so systems are engineered to ensure
that this does not happen.  iSCSI may be somewhat different in that
the corresponding loss of communication issues require timing out
TCP and tearing down the TCP connection, bringing TCP timeouts into
play.

Making client fencing a MUST for the block layout runs the risk of
being fatal to a significant number of possible implementations.

Thanks,
--David