Subject: [nfsv4] seeking clarifications in server-side copy CB_OFFLOAD feature

From: Olga Kornievskaia <[email-masked]>
Date: Thu, 2 Mar 2017 16:07:42 -0500

Hi folks,

I'm seeking some clarification for doing the server-side copy feature.
EINVAL in read beyond the end of the file has been already raised in
another thread so I'll leave that aside. So here are a few more :
-- possible inconsistent behavior in presence of an error from doing
an async copy vs sync. CB_OFFLOAD can return an error and a partial
copy result but there is no mechanism to do so in synchronous COPY. If
CB_OFFLOAD returned an error and a partial copy, is that really an
error or should it be a short copy? I can see a case where an error
should not be ignore -- if error is ENOSPC. Should the client if
received an error in CB_OFFLOAD propagate the error to the
application? Should it be server's responsibility to then if some
error occurred but a partial copy succeeded to then not return the
error to the client (that leads to the 2nd question).

-- related to errors returned in CB_OFFLOAD, there is no guidance with
regards to the reboot recovery. If the destination server unable to
finish the copy due to the source server rebooting, should it reply
back with a successful short copy or should it return some error (and
short copy of possibly 0bytes).

To address these two questions I find some wording in the spec that
confuse me (section 15.2.3):

"If a failure does occur for a synchronous copy, wr_count will be set
to the number of bytes copied to the destination file before the error
occurred." ... it also talks about consecutive bits there but then it
proceeds to also say that "If the failure occurred for an asynchronous
copy, .... It will be able to determine the bytes copied from the
coa_bytes_copied in the CB_OFFLOAD argument."

Since it talks about "wr_count", then it must means that COPY's
nfsstat4 status is not an error. Does this then mean that if copy
"failed" then the error must be ignored on the server and a partial
copy results propagated to the client. But for the async copy,
"coa_bytes_copied" means that CB_OFFLOAD also set nfsstat4 to some
error. So error is not ignored in the async case?

-- the spec lists the kind of errors CB_OFFLOAD reply can return but
there is no wording for what kind of errors be included in
CB_OFFLOAD's call. For instance, I can see that a source server reboot
can translate to ERR_BAD_HANDLE or it could be BAD_SESSION or
STALE_CLIENTID or BAD_STATEID. It is not clear what errors to check
for and recover by restarting the copy and what errors shouldn't
happen. Something like BAD_SESSION implies that server-to-server
protocol was NFS4.x x>=1. Can the client receive an NFSv3 error in
CB_OFFLOAD?

While I realize that server-to-server copy details is not a part of
the RFC7862, but what the client receives in CB_OFFLOAD and how to
interpret that is unclear.

Thank you.

---

From: [email-masked] (J. Bruce Fields)
Date: Fri, 3 Mar 2017 13:48:52 -0500

On Thu, Mar 02, 2017 at 04:07:42PM -0500, Olga Kornievskaia wrote:
> error or should it be a short copy?

On the client side, in the Linux case, the client is implementing
copy_file_range(), which doesn't have the option of returning both an
error and a number of bytes copied.  Returning an error to the
application when the destination was actually modified is unacceptable.
Returning success and a short copy is OK, I think--if the error's
important then the application will get it when it tries to copy the
rest.

> I can see a case where an error should not be ignore -- if error is
> ENOSPC. Should the client if received an error in CB_OFFLOAD propagate
> the error to the application?

I think it's OK to ignore the ENOSPC.  The application will try to write
the rest, the client will turn that into a new COPY request, this time
the response will probably be 0 bytes copied and ENOSPC, so this time
the client can return the error to the client.

Alternatively I guess the client could cache that ENOSPC result and
return it on the next copy without talking to the server?  But then
there's a risk that the client could miss the fact that space was freed
up since the previous COPY.

> Should it be server's responsibility to then if some

I'd rather servers didn't use this odd quirk of the protocol, but maybe
it's too late to stop them.  And it seems easy enough just to ignore the
error on the client side in the case bytes were copied.

> short copy of possibly 0bytes).

A short copy sounds reasonable to me.

--b.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: David Noveck <[email-masked]>
Date: Fri, 3 Mar 2017 15:20:54 -0500

> another thread so I'll leave that aside.

Good.  I'm tired of talking about that.

> So here are a few more :

uh-oh.

> copy result but there is no mechanism to do so in synchronous COPY.

Definitely inconsistent behavior.

The only questions are how/why it got that way and what to do about.

> If

It's really both but, in many cases, the client will have to make a choice
to treat it as one or the other.

> I can see a case where an error
> should not be ignore -- if error is ENOSPC.

Agree.

> Should the client if
> application?

In general it should.  The only exception should be cases where the
client can do recovery to complete the requested copy.

> error to the client (that leads to the 2nd question).

I don't think so, in the CB_OFFLOAD case.  Since the server
can indicate both the error and that the partial copy was done,
there doesn't seem any  to be any point in suppressing that information.

> regards to the reboot recovery.

There should be some :-(

> If the destination server unable to

I think it should return some error.  In the case of enormous copies, the
client
could want to continue the operation, although I'm not sure what it could do
if the sequential flag is not set).  Given that the open is to be done by
the client
he was to have some indication that he can use to decide to get the lock
back within
the grace period.  If this is the only stateid for the server, an he does
not have a clear indicatin of the reason for the failure, the grace period
might expire before he decides to try to get his open back

> confuse me (section 15.2.3):

Me too.  I sometimes feel that, if you are not confused by this, it
is a sign that you haven't read it carefully enough.

> occurred."

This sentence needs to be clarified, given that wr_count is only returned
in the success case.  I'll discuss possible replacements below.

> it also talks about consecutive bits there but then it
> proceeds to also say that
> "If the failure occurred for an asynchronous

To me, this s the most troubling part of this whole thing.  At some
point, someone was aware that CB_OFFLOAD had the ability to inform
the client of a partial successful copy together with  later failure while
the
result of CB_COPY did not.  However, nobody bothered to either:

   - Explain why this discrepancy was valid.
   - Sought to correct/eliminate the discrepancy.

> Since it talks about "wr_count", then it must mean that COPY's
> nfsstat4 status is not an error.

That's the implication.

> Does this then mean that if copy
> copy results propagated to the client.

Yes, as you recognize in putting the word *failed* in quotation marks,
perhaps there
was no true failure here.

If the curious lead sentence were rewritten to say:

If, a copy proceeds for a while and sitution develops that makes it
impossible to
continue with the copy, wr_count will be set to the number of bytes copied
to
the destination file before it became impossible to continue the operation

further. In this case the operation is treated  as a success, even though
not all bytes

requested were copied.

Would that make things clearer?  if so, is it important enough to file an
errata?

> But for the async copy,

Yes.  CB_OFFLOAD has a design that allows a partially successful
copy together with a later error to be reported to the client.

Unfortunately, the synchronous case does not

> CB_OFFLOAD's call.

True but given the fact that this is asychronous way of indicated completion
operation, I believe that the errors returned by COPY are the plce to start.

> For instance, I can see that a source server reboot
> happen.

I think that is primarily riven my implementer preference and the semantics
of the client-side API"s

>Something like BAD_SESSION implies that server-to-server
> protocol was NFS4.x x>=1.

I suppose so.

> Can the client receive an NFSv3 error in CB_OFFLOAD?

GIven that this is an nfsstat4, I think it can't.

> interpret that is unclear.

I think there are some lapses in clarity but some of the issues are not
questions about how to interpret the results of CB_OFFLOAD but deal
with what to do about the results.

---

From: Olga Kornievskaia <[email-masked]>
Date: Fri, 3 Mar 2017 17:28:17 -0500

Thanks for the comments Bruce!

> application when the destination was actually modified is unacceptable.

Are you saying it is not acceptable to fail copy_file_range() and do a
partial copy or am I not reading it correctly? Why is this not
acceptable? Currently if I do a "cp /mnt/foo /mnt2/bar" and say there
is not enough space "cp" will display ENOSPC error but it will copy as
much as it could in /mnt/bar until it failed.

> the client can return the error to the client.

Ok I agree about ENOSPC.

But, in general, 0 + some ERROR might not necessarily mean an error.
When the server rebooted and copied 0 bytes and maybe returned an
error (?), it should not be considered an error on the (NFS) client.

I'm actually hoping that server will return -1 and ENOSPC. But the
spec doesn't dictate that failure should return -1.

> up since the previous COPY.

ENOSPC is typically treated as a "permanent" error. By that I mean
even now in v4 WRITE can fail with ENOSPC, server could a sec later
freed space and if client send it again it would succeed but we fail
the operation.

I would have like the COPY to fail on 1st occurrence of ENOSPC
(instead of ignoring it) but I guess I'm fine to send it again and
then finally replying to application with an error.

> error on the client side in the case bytes were copied.

I can argue that it's easier for the server to ignore the error due to
the reboot. There is no error to represent server reboot. There is no
wording in the spec to state that if CB_OFFLOAD.error = X it means
source server reboot. So if client got CB_OFFLOAD.error = X it is
really guessing if it means server rebooted or something else.

I guess I'm afraid that by ignoring all errors I'm not thinking of
some example where it was the wrong thing to do. Currently this is how
I've coded up the client to ignore the errors (all except ENOSPC which
I guess I can ignore as well let client got another ENOSPC) but I have
a nagging feeling I'm missing something.

> A short copy sounds reasonable to me.

short copy with an error, right?

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 3 Mar 2017 21:07:26 -0500

> much as it could in /mnt/bar until it failed.

Maybe cp doesn't use the information that the write() system call gives
it, but I think the system call interface should still try to provide
this information.

So, I'd still be in favor of turning nonzero copy + error reply from
server into a succesful copy_file_range() call in every case.

The one exception might be errors that the client can handle itself (say
by retrying)?  I don't know what those might be.

> error (?), it should not be considered an error on the (NFS) client.

I'm trying to think about the server reboot cases, but I'm confused....

In the case of just a single server, or in the server-to-server case
when the target server reboots, I think the client just retries any
COPYs it had in progress, the same as it would if it had a write that
was in progress.  I don't there's anything else it can do.  The server
doesn't have any way to return an error to the COPY after it reboots.
Our server won't even remember it was in the middle of a copy after it
comes back up.

If the source server reboots.... Yes, I guess it could just return a
short copy result, but as you say if it didn't manage to copy any bytes,
that might be confusing, as it might look like it hit end of file.

Maybe it should return BAD_STATEID to indicate that the stateid that the
server got in ca_src_stateid isn't accepted by the source server any
more?  Or maybe PARTNER_NO_AUTH?

> spec doesn't dictate that failure should return -1.

I'm not sure what you mean by returning -1.  You mean returning with
wr_count == -1?

> short copy with an error, right?

I was thinking without an error, but at least in the case where there
were 0 bytes I think you're right there needs to be an error.

--b.

---

From: Olga Kornievskaia <[email-masked]>
Date: Mon, 6 Mar 2017 11:54:44 -0500

> Our server won't even remember it was in the middle of a copy after it

Correct client will reboot the copy from scratch.

> more?  Or maybe PARTNER_NO_AUTH?

Due to the fact that VFS copy_file_range() can't return anything of
that sort, it returns EBADF. NFSD will return partial copy and
BAD_FILEHANDLE.

> I'm not sure what you mean by returning -1.  You mean returning with

As far as I can tell, "wr_count" could never be -1. wr_count is only
present on a successful copy (meaning CB_OFFLOAD error is NFS4_OK or
for synchronous case COPY's status is NFS4_OK). If error is set, then
coa_bytes_copied will be set to something. I'm suggesting it should be
-1.

> were 0 bytes I think you're right there needs to be an error.

I'm confused :( The way I read things you say server shouldn't hide
errors but here you are saying it should return a short copy without
an error. I guess I'll wait for you to comment on the actual code to
let me know what you think should happen there.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 6 Mar 2017 17:06:59 -0500

> let me know what you think should happen there.

I'd reserve the case of a succesful copy with 0 bytes copied for
end-of-file.  So if we're forced to return without copying anything, and
it's not because of end of file, we should return an error.  If it's a
problem like source-server reboot that would be cured by the client
retrying, the it should be an error that prompts the client to do that.

In the case of a short copy of some number of bytes more than 0, I don't
understand why the protocol allows an error return, but I guess I can't
think of a reason to care whether we return one or not.

--b.