Subject: [nfsv4] [FedFS] FSL caching

From: James Lentini <[email-masked]>
Date: Wed, 20 May 2009 17:21:35 -0400 (EDT)

At our FedFS meeting on 4/9, we discussed FSL caching. Based on that, 
I've put together the recommendations below which I would propose 
adding in a new section (3.3.2) of the 
draft-ietf-nfsv4-federated-fs-protocol I-D. I'll set aside some time 
to discuss this at tomorrow's FedFS meeting.

-james

---

3.3.2 Caching of Fileset Locations

 To resolve an FSN to a set of FSL records, the fileserver queries the
 appropriate NSDB for the FSL records. A fileserver MAY cache these
 FSL records for a limited period of time. The period of time, if any,
 during which FSL records are cached is implementation dependent.

 The combination of FSL caching and FSL migration presents a challenge. 
 For example, suppose there are three fileservers named 
 A, B, and C and fileserver A contains a junction to fileset X
 stored on fileserver B. Now suppose that fileset X is migrated from 
 fileserver B to fileserver C and the corresponding FSL information for 
 fileset X in the appropriate NSDB is updated. If fileserver A has a 
 cached FSL for fileset X, a user traversing the junction on fileserver A 
 will be referred to fileserver B even though fileset X has migrated to 
 fileserver C. If fileserver A was not caching FSL records, it would have 
 obtained the correct location of fileset X from the NSDB.

 Administrators are advised to be aware of FSL caching when performing 
 a migration. When migrating a fileset, administrators SHOULD create a 
 junction from the fileset's old location to the new location. This 
 junction will redirect an users who follow stale FSL information to 
 the correct location.

---

From: "Everhart, Craig" <[email-masked]>
Date: Thu, 21 May 2009 11:15:06 -0400

In the example James cites, NFS does as well with FedFS as without it.
NFSv4.1 talks about leaving a non-FedFS junction in the old location so
that the client can track it.  James's prose simply adapts that to the
FedFS case.

Given the FedFS charter, it's too bad that FedFS can do no better.  We
could pick an arbitrary hard limit for how long such a junction would
need to stay around, where the hard limit was set architecturally.
Alternatively, one agent (the NSDB?) could pick a duration and tell the
other players, and fileserver A could use that duration as a limit on
caching the location data for fileset X without revalidation, and the
junction could be removed after that limit had expired.

I like either of a couple of other ideas better:

(a) Leave the junction in B as a matter of course.  Handle both the
cases where B contains an instance of the parent of fileset X and where
it does not.
(a1) B has an instance of the parent fileset.  Since that instance
should have a copy of the same referral to fileset X, then anybody
asking about a given FID within the parent fileset X will naturally be
redirected to the new location for X.  This would ideally happen by
fileserver B consulting the NSDB, possibly invalidating and refreshing
its own NSDB cache upon noticing that its cached NSDB entry for X has an
FSL pointing to B that it, in actuality, does not possess.
(a2) B does not have an instance of the parent fileset.  In the
handshake that happens between client and fileserver B to follow the
referral to the root of fileset X, fileserver B could observe that the
fileset is not resident on itself.  Using the federation interfaces, and
possibly invalidating its own NSDB cache as in (a1), fileserver B could
obtain accurate information about where fileset X is and return it to
the client.
(a-both) Eventually, the NSDB cache on A would be revalidated and the
extra hop on fileserver B could be avoided.

(b) Feed back the staleness of A's cache from the client back to
fileserver A.  Define a new optional NFSv4.1+ operation by which the
client could notify a fileserver that the location information it gave
out for a given fileset (NFSv4 filesystem) was obsolete.

If option (a) can be made to work, it seems to be an elegant solution.
The requirement there is basically that a fileserver be able to
recognize an FSN in an incoming NFSv4 FID.  I don't know how well this
would work using actual federated FS's; it could be hard to make the
(a2) case work correctly without the full FSN information, including
which NSDB to use.

		Craig

> -----Original Message-----
> From: Lentini, James 
> Sent: Wednesday, May 20, 2009 5:22 PM
> To: [email-masked]
> Subject: [nfsv4] [FedFS] FSL caching
> 
> 
> At our FedFS meeting on 4/9, we discussed FSL caching. Based 
> on that, I've put together the recommendations below which I 
> would propose adding in a new section (3.3.2) of the 
> draft-ietf-nfsv4-federated-fs-protocol I-D. I'll set aside 
> some time to discuss this at tomorrow's FedFS meeting.
> 
> -james
> 
> ---
> 
>  To resolve an FSN to a set of FSL records, the fileserver 
> queries the  appropriate NSDB for the FSL records. A 
> fileserver MAY cache these  FSL records for a limited period 
> of time. The period of time, if any,  during which FSL 
> records are cached is implementation dependent.
> 
>  For example, suppose there are three fileservers named  A, 
> B, and C and fileserver A contains a junction to fileset X  
> stored on fileserver B. Now suppose that fileset X is 
> migrated from  fileserver B to fileserver C and the 
> corresponding FSL information for  fileset X in the 
> appropriate NSDB is updated. If fileserver A has a  cached 
> FSL for fileset X, a user traversing the junction on 
> fileserver A  will be referred to fileserver B even though 
> fileset X has migrated to  fileserver C. If fileserver A was 
> not caching FSL records, it would have  obtained the correct 
> location of fileset X from the NSDB.
> 
>  Administrators are advised to be aware of FSL caching when 
> performing  a migration. When migrating a fileset, 
> administrators SHOULD create a  junction from the fileset's 
> old location to the new location. This  junction will 
> redirect an users who follow stale FSL information to  the 
> correct location.
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Thu, 21 May 2009 13:28:07 -0400

How many servers will actually cache an FSL? If the name space is
hierarchical, the answer is the number of replicas. In any case, this
should not be a large number.

Why not have a <FSL, server1, server2, server3...serverN> record in
the NSDB that is optional, and that is maintained each time a junction
is created? This record stores 'which server is caching the FSL'.

In Jame's example, when the junction on fileserver A that refers to
fileset X is created this record is stored in the NSDB

<FSL(X), Server A>

Then when fileset X is migrated from fileserver B to fileserver C and
the corresponding FSL information for fileset X in the appropriate
NSDB is updated, the record is accessed using the FSL (x) as a key and
each server in the list, Server A in the exmple, is told to flush the
cache - or perhaps told of the new location.....

-->Andy

---

From: James Lentini <[email-masked]>
Date: Fri, 22 May 2009 16:47:52 -0400 (EDT)

See reply below:

> (a2) case work correctly without the full FSN information, including

These are interesting idea.

As you alluded above, the FSN is not known to the client or the server 
storing the FSL. I'm afraid (a) will not be feasible with our existing 
mechanisms.

Option (b) is an interesting idea, but it requires client software 
changes and cooperation which may be difficult to realize.

---

From: James Lentini <[email-masked]>
Date: Fri, 22 May 2009 16:47:58 -0400 (EDT)

This is an interesting idea. My only concern is that the list of all 
fileservers caching the FSL will likely degenerate into a list of all 
fileservers storing the FSL. As you say, the total number of 
fileserver may be relatively small. On the other hand, updating long 
list of fileservers every time any FSL in the federation changed would 
limit scalability.

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Tue, 26 May 2009 12:32:41 -0400

> list of fileservers every time any FSL in the federation changed would

Do you think there will be a large number of FSL migrations? I think a
migration event will be an unusual event. At least it will not be a
common event, and so will not limit scalability. I wonder if there are
some AFS stats on how often volumes were moved...

-->Andy

---

From: James Lentini <[email-masked]>
Date: Tue, 26 May 2009 16:49:12 -0400 (EDT)

> some AFS stats on how often volumes were moved...

I don't know how often an FSL will migrate. Like you, my intuition is 
that it will be an infrequent event. Some statistics to back that up 
would be useful.

---

From: James Lentini <[email-masked]>
Date: Wed, 27 May 2009 09:43:29 -0400 (EDT)

> that it will be an infrequent event. Some statistics to back that up 

A thought occurred to me today. If migrations are "cheap" they may 
happen much more often than we would guess. Given the namespace, 
administrative software could monitor and automatically migrate data 
back and forth between a high performance, high cost storage tier and 
a lower performance, lower cost tier depending on usage or other 
factors. In other words, I don't know if we can predict the frequency 
of migrations in the future from past data.

-james

---

From: Nick Williams <[email-masked]>
Date: Thu, 28 May 2009 14:11:57 +0100

I definitely support James' thinking. We're a large AFS installation and we
had so many volume migrations in the past (moving volumes around to make
space, ensure even load on fileservers, migrating for maintenance) that we
implemented some of the maintenance infrastructure suggested by James: the
whole balancing operation was automated into a nightly job and we have
migrations constantly "in-flight".

Cheers,
Nick.