Subject: [nfsv4] https://datatracker.ietf.org/doc/draft-myklebust-nfsv4-pnfs-backend/

From: Thomas Haynes <[email-masked]>
Date: Wed, 17 Nov 2010 16:04:12 -0800

Trond,

Some questions I had after reading this:

1) What keeps another client from spoofing a file handle to get at a cached file?

Note that there is no export control on cached files. The caching server
has to assume that the MDS did the access checks. With AUTH_UNIX,
this means that any client can get access with a spoofed file handle.

Another implementation issue here is that the caching server, if it does
per op access checks, will now need to know to skip such checks on these
files.

2) What keeps a file handle clash from occurring?

I.e., the cached file has a file handle which maps to a valid file on the caching
server.

3) This also goes with the following from the draft:

>    When the data server receives a READ request from a client with a
> 
> 
> 
> 
> Myklebust                Expires January 7, 2010                [Page 5]
>  
> Internet-Draft      pNFS back end protocol extensions          July 2009
> 
> 
> 
>    stateid or a data server filehandle that it does not recognise, it
>    attempts to validate that request using the PROXY_OPEN call.  This
>    operation will convert the data server filehandle as provided by the
>    layout into a real filehandle, that the data server can use to access
>    the file on the metadata server.  In order to make it easy for the
>    data server to identify the file, the real filehandle SHOULD match
>    the filehandle that was returned to the client when it received the
>    read delegation.

Which I take to imply that stateids must be unique on the MDS and
these caching data servers. I.e., if a client presents a stateid that is
already being used on the caching data server and it presents a valid
filehandle from that caching data server, then the caching data server
has to assume that the client is trying to access a non-cached file.

4) I'm also reading this as saying that the data server filehandle SHOULD
match the real filehandle presented in the layout. Indeed if it does not,
then the data server will need to pass any filehandle it gets back to the
MDS to see if it is a file handle it is caching.

5) 

> 
>    The PROXY_OPEN call also checks the access rights that were granted
>    by the layout and the READ stateid for validity.  If the pNFS client
>    in question does not hold a layout for this file, the PROXY_OPEN
>    request from the data server will return NFS4ERR_PNFS_NO_LAYOUT.  In
>    this case, the data server should not attempt to service the READ
>    request, but should pass the error on to the pNFS client.
> 

Is NFS4ERR_PNFS_NO_LAYOUT the only error that can be set here?

a) File does not exist (NFS4ERR_STALE)
b) Client does not have permission to the underlying export (NFS4ERR_ACCESS)
c) ...
d) stateid is no longer valid, etc 

Ideally, the client should have gone through all of these steps in order to get
the filehandle from the MDS, but nothing stops the client from holding onto the
filehandle and the MDS deleting it, etc.

Thanks,
Tom

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 17 Nov 2010 20:59:50 -0500

> per op access checks, will now need to know to skip such checks on these

The client acting as the DS is supposed to check with the MDS that the
file handle is valid and that the client 2 is authorised to access it.
That's precisely what the PROXY_OPEN is for: to allow the MDS to check
that client 2 holds a layout.

> I.e., the cached file has a file handle which maps to a valid file on the caching

It is up to the MDS to generate data server filehandles that do not
clash.

> has to assume that the client is trying to access a non-cached file.

What do you mean by 'valid filehandle'? Data server filehandles are
_not_ the same as ordinary filehandles; they are filehandles that the
MDS has handed out as part of a layout and that need to be translated
via PROXY_OPEN. It is expected that they will be invalidated (and
presumably forgotten by the MDS and DS) when the layout is revoked.

> MDS to see if it is a file handle it is caching.

It needs to do this once in order to translate the filehandle. Once that
is done, it can rely on the server to call it back when the layout is
revoked (and the filehandle is invalidated).

> Is NFS4ERR_PNFS_NO_LAYOUT the only error that can be set here?

No. See the next paragraph.

> filehandle and the MDS deleting it, etc.

The DS holds a delegation for the file. The MDS can't delete it or
change access permissions; that would revoke the delegation, and hence
all layouts.

Cheers
  Trond

---

From: Tigran Mkrtchyan <[email-masked]>
Date: Thu, 18 Nov 2010 08:18:54 +0100

> that client 2 holds a layout.
For multihomed client 2 it's possible that MDS and  the client acting as
a DS  will see different IP. I think this may produce 'false negatives'
and some requests will be denied.

BTW, PROXY_OPEN desctiprions says:

   The PROXY_OPEN function authenticates the READ request by the pNFS
   client.  If the data filehandle is valid, and the user identified by
   the popa_user_id is authorised to access the file, then the metadata
   server returns the true filehandle (as returned by LOOKUP and/or

   OPEN) of the file.

but XDR says:

     union PROXY_OPEN4res switch (nfsstat4 status) {
           case NFS4_OK:
                   /* CURRENTFH: true filehandle */
                   stateid4        popr_proxy_stateid;
           default:
                   void;
     };

In other words, description promises filehandle, but XDR returns
stateid. Did I get it wrong?

Tigran.
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 18 Nov 2010 10:19:58 -0500

> and some requests will be denied.

I don't see how: the DS doesn't exchange client2's IP address with the
MDS.

We could perhaps pass the clientid that client2 sent as part of the
EXCHANGE_ID, but I'm not sure that would significantly enhance the
security of the system.

> stateid. Did I get it wrong?

The client can/should retrieve the filehandle using a GETFH op.

Cheers
  Trond

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 18 Nov 2010 10:26:48 -0800

> that client 2 holds a layout.

So I can read that in your text now. The first time through, that was not
what I got. In retrospect, I kept confusing whether the caching client or
client 2 was the one with the layout.

> It is up to the MDS to generate data server filehandles that do not

The MDS can not know what filehandles the caching client has in use.

> presumably forgotten by the MDS and DS) when the layout is revoked.

Again, there is nothing to prevent the MDS from handing out a filehandle
which happens to be one in use by the caching client. If client 2 already has
that filehandle from an earlier action, then it may be the case that it presents
it back to caching client.

> revoked (and the filehandle is invalidated).

Yes, once it has a mapping of valid data server filehandles, it can cache
them.

> No. See the next paragraph.

I'd swear you added that after I copied the text from the draft. :->

I probably became laser focused.

> change access permissions; that would revoke the delegation, and hence

I'm thinking a couple of pictures would help in the final text - I can see all of this
now, but a concrete example could have helped.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 18 Nov 2010 13:53:35 -0500

> The MDS can not know what filehandles the caching client has in use.

When it sent the REGISTER_DS request, the DS client included a unique
rea_mds_identifier which the server is supposed to put as the first 8
bytes of the data server filehandle. This ensures that the DS client can
trace the data server filehandle as belonging to a particular MDS. The
MDS is then at liberty to define the rest of the file handle as it
pleases.

> it back to caching client.

The DS client knows the difference between an ordinary filehandle and a
DS filehandle. The latter is _only_ used as part of a pNFS READ or WRITE
request by client 2 and will have been appropriately generated by the
MDS (see above).

Nothing stops the MDS from including information about the layout
stateid in the DS filehandle.

> >> match the real filehandle presented in the layout.

BTW: this is false. The DS filehandle MUST NOT match the real
filehandle. See above.

>  Indeed if it does not,

That's the whole point of distinguishing DS filehandles and ordinary
filehandles. The former represents the layout.

> now, but a concrete example could have helped.

OK. I'll think about how to add this info.

I apparently also need to add in a paragraph about DS filehandles vs.
real filehandles.

Cheers
  Trond