Subject: [nfsv4] WRITE_PLUS offloading

From: Anna Schumaker <[email-masked]>
Date: Tue, 5 Nov 2013 15:54:30 -0500

I have yet another WRITE_PLUS implementation question! :)

It's been pointed out on the linux-nfs mailing list that hole punches and extent zeroing is implemented through metadata operations, and therefore should run very fast.  The client is required to support CB_OFFLOAD for this operation, which creates a decent amount of extra work for tracking the offload stateid and waiting for the callback to finish.  Is there any way for the client to signal that it doesn't want to go async for hole punching?  Maybe through a new flag or through sync / async arms of the discriminated union?

Anna

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 6 Nov 2013 01:15:14 +0000

On Nov 5, 2013, at 12:54 PM, Anna Schumaker <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

Given we had such a process:

What would the client do if it wanted to hole punch synchronously and the
server was going to take 30 minutes?

Is the client able to wait?

How does the client tell the difference between thinking the server is not
responding and the server is actually taking that amount of time to get the
action done?

---

From: Anna Schumaker <[email-masked]>
Date: Wed, 6 Nov 2013 11:43:14 -0500

> server was going to take 30 minutes?

If the server can detect that the punch is going to take a long time then it could return an error to the client and disallow the punch.

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 6 Nov 2013 17:10:07 +0000

On Nov 6, 2013, at 8:43 AM, Anna Schumaker <[email-masked]<mailto:[email-masked]>> wrote:

On Nov 5, 2013, at 12:54 PM, Anna Schumaker <[email-masked]<mailto:[email-masked]>> wrote:

Anna
_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

What defines "long"? The client? Will it send the time it is willing to wait?

IMHO this all adds complexity.

There are scenarios where the client may have to wait because of the action the server is taking. Given that a hole
punch that requires deallocation of blocks or writing zeros to blocks can be much larger than a normal write,
we acknowledge the asynchronous nature of the hole punch.

---

From: Marc Eshel <[email-masked]>
Date: Wed, 6 Nov 2013 09:59:47 -0800

I think that there should be a synchronize option, the server will 
probably not use callback because it doesn't really know how long the 
operation will take or because call back is not possible. So the client 
should also have the option of not having to support async WRITE_PLUS.
Marc.

From:   "Haynes, Tom" <[email-masked]>
To:     "Schumaker, Bryan" <[email-masked]>, 
Cc:     "Haynes, Tom" <[email-masked]>, "[email-masked]" 
<[email-masked]>
Date:   11/06/2013 09:10 AM
Subject:        Re: [nfsv4] WRITE_PLUS offloading
Sent by:        [email-masked]

On Nov 6, 2013, at 8:43 AM, Anna Schumaker <[email-masked]> wrote:

It's been pointed out on the linux-nfs mailing list that hole punches and 
extent zeroing is implemented through metadata operations, and therefore 
should run very fast.  The client is required to support CB_OFFLOAD for 
this operation, which creates a decent amount of extra work for tracking 
the offload stateid and waiting for the callback to finish.  Is there any 
way for the client to signal that it doesn't want to go async for hole 
punching?  Maybe through a new flag or through sync / async arms of the 
discriminated union?

Anna

If the server can detect that the punch is going to take a long time then 
it could return an error to the client and disallow the punch.

What defines "long"? The client? Will it send the time it is willing to 
wait?

There are scenarios where the client may have to wait because of the 
action the server is taking. Given that a hole
punch that requires deallocation of blocks or writing zeros to blocks can 
be much larger than a normal write,
we acknowledge the asynchronous nature of the hole punch.

How does the client tell the difference between thinking the server is not
responding and the server is actually taking that amount of time to get 
the
action done?

---

From: Anna Schumaker <[email-masked]>
Date: Tue, 19 Nov 2013 14:59:27 -0500

Shouldn't we be able to use the wp_stable (stable_how4) field to mimic a synchronous option?  If the client requests FILE_SYNC, then the operation is synchronous.  If the client requests UNSTABLE then that could either mean the client supports the callback or that it will send a commit later to make sure the operation has finished.

Am I misunderstanding something that would keep this idea from working?

Anna

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 19 Nov 2013 21:03:36 +0100

> Am I misunderstanding something that would keep this idea from working?

I've thought about this a bit, and I suspect one easy way out is to formalize
the option for the client to simply not implement CB_OFFLOAD (or not even
have a backchannel available) and assign a specific error to it.  In that
case a server that knows it wants to run a long-running operation can
tell the client "I need an offload" and a client not wanting a long-operation
can simplify return EOPNOTSUPP to the application for say the Linux
fallocate system call or alternatively only set up the whole callback
infrastructure for those server that have bad enough performance to need it.

---

From: Marc Eshel <[email-masked]>
Date: Wed, 20 Nov 2013 09:00:02 -0800

I think this is a good idea.
Marc.

From:   Anna Schumaker <[email-masked]>
To:     Marc Eshel/Almaden/IBM@IBMUS, "Haynes, Tom" 
<[email-masked]>, 
Cc:     "Schumaker, Bryan" <[email-masked]>, "[email-masked]" 
<[email-masked]>, <[email-masked]>
Date:   11/19/2013 11:59 AM
Subject:        Re: [nfsv4] WRITE_PLUS offloading

Shouldn't we be able to use the wp_stable (stable_how4) field to mimic a 
synchronous option?  If the client requests FILE_SYNC, then the operation 
is synchronous.  If the client requests UNSTABLE then that could either 
mean the client supports the callback or that it will send a commit later 
to make sure the operation has finished.

Anna

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 20 Nov 2013 19:21:33 +0100

> I think this is a good idea.

Overloading the existing flags seems fairly confusing.

But with a new OFFLOAD4 value for the stable_how4 the client could
explicitly say that it's fine with a long-running offloaded operation.