Subject: [nfsv4] issue 164 : Can a replier send a reply on a connection other than the one the requester sent the request on?

From: Mike Eisler <[email-masked]>
Date: Wed, 7 Nov 2007 11:47:30 -0800 (PST)

I think the answer should be no. I propose adding the following
text to the specification:

  <t>
   When sending a reply, the replier MUST send the reply to
   the same full network address (e.g. if using an IP-based
   transport, the source port of the requester is part of
   the full network address) that the requester issued the
   request from. If using a connection-oriented transport,
   replies MUST be send on the same connection the request
   was sent from. If a connection is dropped after the
   replier receives the request but before the replier
   sends the reply, and if a connection is established
   with the same source and destination full network
   address as the dropped connection, then the replier
   MUST NOT send the reply until the client retries the
   request. The reason for this prohibition is that the
   client MAY retry a request over a different connection
   that is associated with the session.

  </t>

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 07 Nov 2007 13:54:04 -0600

Agreed.

---

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 07 Nov 2007 15:44:33 -0500

>I think the answer should be no. I propose adding the following

Definitely no.

>   replies MUST be send on the same connection the request

"sent"

>   was sent from. If a connection is dropped after the

"received"

>   that is associated with the session.

"than" (not "that"), but see comment:

There's more in this than is covered in the requirement of
those last two sentences. The issue is that the new test
 _requires_ the server to detect this, which is a problem.
Yes, an unsolicited reply on the new connection breaks
RPC semantics, blows the newly reallocated slot table,
and may break the connection if it's over RDMA. But, it's
a big reach to forbid it from happening altogether since it
implies a merging of the RPC and transport layers.

I think the language here needs to be SHOULD NOT, and
it needs to be clear that the client SHOULD drop any such
unsolicited reply. We could go so far as to recommend that
the client disconnect and back off, if it should happen.

The server clearly must never buffer an actual reply for
sending when the client reconnects. Servers only speak
when directly spoken to. Besides this is what the reply
cache is all about.

Tom.

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 07 Nov 2007 14:53:52 -0600

> I think the language here needs to be SHOULD NOT, and

I agree with Mike's proposal in that it should be a "MUST NOT".

Spencer

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 07 Nov 2007 16:22:59 -0500

> the client disconnect and back off, if it should happen.

Why would backing off help? What scenario are we looking at where
unsolicited replies might occur despite the MUST that Mike is proposing?

Cheers
  Trond

---

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 07 Nov 2007 16:31:18 -0500

>unsolicited replies might occur despite the MUST that Mike is proposing?

It's only advisable in the SHOULD case, and then only if unsolicited
replies are received.

If server implementors are happy that this MUST can be implemented,
then I'm fine with it. A similar requirement was in a version of my original
sessions proposal, but it was removed after implementation experience.
Most existing servers do not check the state of the socket before
sending their reply, and they certainly don't monitor the connection
while processing.

Tom.

---

From: Peter Staubach <[email-masked]>
Date: Wed, 07 Nov 2007 16:36:27 -0500

> sending their reply, and they certainly don't monitor the connection

I am curious what, more precisely, the implementation experience was.
The server should just send off the response and ignore any errors
that come back from the transport that indicated that it couldn't
send the response for any reason other than things like local memory
shortage.  Isn't the transport generally responsible for dropping
any undeliverable data due to broken connections and such?

    Thanx...

       ps

---

From: "Talpey, Thomas" <[email-masked]>
Date: Wed, 07 Nov 2007 19:11:55 -0500

>any undeliverable data due to broken connections and such?

Sure.

The problem isn't actually sending on a broken connection. That's
the good case - the reply isn't going anywhere and there is no issue
of misdirecting it. The issue is when there *is* a connection, but it's
the wrong one.

When implementing to sessions, the server implementations so far
have all referenced the session from the in-progress request. This is
very convenient, as the session contains all the server context needed
to execute the request, including the reply cache, the RPC endpoint,
and the clientid. The thing about the session is that it's an NFS layer
object, which references an RPC object, which in turn contains
the TCP connection.

If that TCP connection changes, there's no existing API in most systems
to signal that to NFS - all the upper layer can do is send and hopefully
get an error. To fix this, either the local RPC has to change, or some
kind of exposure of the socket, and a reference, needs to be taken.

There were other issues in the early sessions spec related to this.
You may recall it had channel objects, and a binding operation for
connections. These ops basically "registered" the TCP socket with
NFS. But this was hard for both implementations, and even harder
for them to propagate state. So it was modified to what you see
today.

Changing the server requirement to a MUST is a good goal, and frankly
I agree with it, in principle. But it will have implications. As a SHOULD,
it is a strong recommendation, but implementations can still have
some freedom to evolve to it.

BTW, either way this is going to be a devilishly hard condition to test
for. Maybe even harder than testing the reply cache.

Tom.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 9 Nov 2007 18:59:53 -0800 (PST)

--- "Talpey, Thomas" <[email-masked]> wrote:

> to signal that to NFS - all the upper layer can do is send and

There's probably no existing API for implementating sessions either.

By specifying this issue as a MUST NOT, that guides the design
of the API.

Most existing NFS/TCP servers have a link to the connection end point
attached to the request so I don't see why this is ahrd.

> hopefully