Subject: RE: [nfsv4] Issue53 - Add data retention support

From: "Everhart, Craig" <[email-masked]>
Date: Tue, 11 Jul 2006 10:26:00 -0400

I had a couple of questions, but Alan's points about compliance--specifically, what kinds of requirements we're aiming to comply with--help.

(a) What happens to snapshots?  Can I delete a filesystem that has an actively-retained file in it?  How about if it's just a (temporary) snapshot of an active filesystem?
	- Probably the answer is that the vendor makes a decision, then the customer, or the vendor, or the customer's or the vendor's lawyers, make a call about what Alan describes as chain-of-custody: given the manufacturer's claims, what can we claim about the compliance model?  With what statutes does and doesn't it comply?

(b) Can you rename a directory that contains (some levels down) an actively-retained file?  Would you have had to set the Retain attribute on all parent directories in order to preserve a file name, if that's what you wanted?  Heck, with a naïve implementation of Retain, you could rename a Retained file directly: if you're just Retaining an inode.  Can I add a new hard link to a Retained file?  Reduce the link count from 2 to 1?  Probably I would be prohibited from reducing the link count from 1 to 0.

(c) What's special about Alan's guess about making a temp (un-Retained) copy of a Retained file doesn't seem special to me.  All naïve copies would likely be un-Retained.  What would be a very special operation indeed is to transfer the Retained property from one copy to another, so you'd require the ability to un-mark a file as Retained under this special operation.  The alternative is that all copies of a Retained file themselves become Retained, potentially producing a lot of litter.

		Craig

> -----Original Message-----
> From: Yoder, Alan 
> Sent: Monday, July 10, 2006 7:10 PM
> To: Brent Callaghan; [email-masked]
> Subject: RE: [nfsv4] Issue53 - Add data retention support
> 
> inline.... 
> 
> > -----Original Message-----
> > From: Brent Callaghan [mailto:[email-masked]]
> > Sent: Monday, July 10, 2006 12:50 PM
> > To: [email-masked]
> > Subject: Re: [nfsv4] Issue53 - Add data retention support
> > 
> > 
> > Interesting.  Some questions/comments:
> > 
> > o Would the retention attribute be advisory or mandatory ?
> >     For instance, could a client (or server app) that doesn't know
> >     of the attribute just blow the file away anyway ?
> 
>    There needs to be a way for servers not to support it (per 
> exported filesystem I would think), but in servers that do, 
> and on "compliance filesystems", the bit is there precisely 
> to prevent clients or other apps from blowing away files.  
> Possibly a new error code would be appropriate.
>    The question of whether the bit should follow the file 
> when it's copied is more interesting.  At present we are 
> thinking that if a file were copied from one compliance 
> volume to another, the bit would be preserved; if you want a 
> temporary copy, you have to copy the file to a non-compliance 
> volume, that doesn't know about the bit.  But this is 
> client-side behavior.  Can that be written into the spec?
> 
> > 
> > o If the file is in the form of a log, then you'd like to retain
> >      previous entries, yet be able to add new ones.  The
> >      retention attribute would preserve old entries, but prevent
> >      the addition of new ones.
> 
>    A nice-to-have.  Servers aren't usually in the business of 
> supporting append-only semantics, are they?  If v4 has that, 
> then this would be easy.  Otherwise, normal practice is to 
> write the log, and when you roll over to a new log file, set 
> the retention stuff on the old one and lock it down.
> 
> > 
> > o How would the attribute work with a collection of related
> >      files in a directory ?  Would it also apply to directories as
> >      well as files ?
> 
>    In the FCAS space there's no such thing as a directory, so 
> I don't think it makes that much sense to try and get this to 
> work with directories.
> 
> > 
> > o A problem with any new attribute: the attribute is easily lost
> >      if archived on a filesystem, archive or device that doesn't
> >      know about it, i.e. the retention attribute wouldn't 
> be retained.
> 
>    Yep.  Compliance is more involved than just a couple 
> attributes on a file.  There's a lot of auditing and 
> chain-of-custody stuff that has to be tracked to have a 
> complete story.  But if the chain of custody shows a file has 
> always been locked down on a server that understands these 
> bits, then you can show you've preserved the file unchanged 
> to the best of your ability.  In the FCAS space they use 
> digests as a further proof.  In the v4 space a named 
> attribute could be used to store a digest or any other info 
> that is deemed necessary to show compliance, in addition to 
> the proposed attribute and retention time.
> 
> > 
> > o  How is the attribute an improvement on existing data retention
> >      policies, e.g. archival backup, snapshots etc.
> 
>    See question 1.  This stuff is based on Netapp experience 
> with our SnapLock and LockVault products.  Basically, if you 
> want to keep the stuff in the live filesystem, you need some 
> way of locking it down and emulating true WORM storage, a la 
> question 2.
> 
> Cheers,
> 
> Alan
> 
> ===============================================================
> Alan G. Yoder                                    [email-masked]
> Technical Staff                           
> Network Appliance, Inc.                            408-822-6919
> ===============================================================
> 
> 
> > 
> > 	Brent
> > 
> > 
> > On Jul 10, 2006, at 11:57 AM, Mike Eisler wrote:
> > 
> > > Here is an outline of a proposal, which is based on conversations 
> > > with Alan Yoder and Jay Moorthi of Network Appliance, who work on 
> > > NetApp's data retention feature. Errors are mine. I'm 
> sure Alan will 
> > > correct me.
> > >
> > > My understanding is that Data Retention is a concept that 
> has been 
> > > driven by legislation and regulations passed by many countries in 
> > > recent years.
> > >
> > > The rationale is political and other jurisdictions have 
> decided that 
> > > some data (e.g. health records) must be retained for a sufficient 
> > > period of time in order for the stake holders
> > associated
> > > with data to not be harmed due to deletion (unintended or 
> otherwise) 
> > > of the data.
> > >
> > > I'm told it would be difficult to impossible to normalize 
> all these 
> > > regulations and support a full range of retention attributes.
> > > That seems more like the place for applications using 
> higher level 
> > > APIs than what NFSv4 is intended to support, such as the 
> work that 
> > > SNIA is doing ( http://www.snia-dmf.org/xam/faq.shtml ). 
> That said, 
> > > the "low hanging fruit" consists of two attributes:
> > >
> > > - Retained. This is a boolean attribute. By default it is FALSE.
> > >   Once it is TRUE, a file, its named attributes, and some other
> > >   attributes, including modification time, change, meta data time,
> > >   MUST NOT change, ever. The Retained attribute once set to TRUE,
> > >   MUST stay true.
> > >
> > > - Retention Time. This is a time (date really) to which
> > >   a file object (and its named attributes) are guaranteed to
> > >   be retained. The value is provisional until Retained is set
> > >   to TRUE, and thus has no meaning for compliance with
> > >   retention regulations until Retained is TRUE. Once Retained
> > >   is set to TRUE, the Retention Time MAY be changed, but it
> > >   MUST not be set to a new value less than its current
> > value. Once the
> > >   retention time is reached:
> > >
> > >   - the file MAY be deleted. However, individual named attributes
> > >     MUST NOT be deleted.
> > >
> > >   - the contents of the file, its named attributes, and some
> > >     other attributes MUST NOT change as long as the file exists.
> > >
> > > Normally only the owner of a file can set Retained or
> > Retention Time,
> > > though the use of ACLs may extend modification rights for these 
> > > attributes to other principals. Just having attribute 
> modification 
> > > rights to the file object does not guarantee the right to set 
> > > Retained or update Retention Time. NFSv4.1 server implementations 
> > > MAY support policies that allow the storage administrator 
> to limit 
> > > the capability of users can set retention attributes (including 
> > > whether a user can set a retention attribute and the range of 
> > > retention times).
> > >
> > > I look forward to discussion on this mailing list. I see 
> I am on the 
> > > agenda 
> (http://www3.ietf.org/proceedings/06jul/agenda/nfsv4.txt ) to 
> > > discuss this tommorrow, so I look forward to the discussion in 
> > > Montreal too.
> > >
> > >   -mre
> > >
> > >
> > > _______________________________________________
> > > nfsv4 mailing list
> > > [email-masked]
> > > https://www1.ietf.org/mailman/listinfo/nfsv4
> > 
> > 
> > _______________________________________________
> > nfsv4 mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/nfsv4
> > 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Tue, 11 Jul 2006 08:10:22 -0700 (PDT)

--- "Everhart, Craig" <[email-masked]> wrote:

> I had a couple of questions, but Alan's points about
> compliance--specifically, what kinds of requirements we're aiming to
> comply with--help.
> 
> (a) What happens to snapshots?  Can I delete a filesystem that has an

A snapshot that contains a file with Retain == TRUE was taken at time
t when there was an file in the active file system with Retain == TRUE.
Therefore the active file still exists in the same state it does
in the snapshot. Therefore the snapshot can be deleted.

> actively-retained file in it?  How about if it's just a (temporary)

No you can't delete a file system with an active file with Retain ==
TRUE.

> snapshot of an active filesystem?

Temporary or not the answer is the same.

> 	- Probably the answer is that the vendor makes a decision, then the
> customer, or the vendor, or the customer's or the vendor's lawyers,
> make a call about what Alan describes as chain-of-custody: given the
> manufacturer's claims, what can we claim about the compliance model? 

True. A vendor might decide for variois reasons that snapshots 
containing Retained files can't be deleted. 

> With what statutes does and doesn't it comply?

We can't make claims; statutes are more fluid than IETF standards.
This somewhat analagous to RFC1984. 

> (b) Can you rename a directory that contains (some levels down) an
> actively-retained file?  Would you have had to set the Retain

I think that is implementation specific, and implementations
will perhaps adapt to changes in regulations or interpretations.

> attribute on all parent directories in order to preserve a file name,
> if that's what you wanted?  Heck, with a naï¿½ve implementation of
> Retain, you could rename a Retained file directly: if you're just

I don't think so, because I believe this would alter the 
metadata change time, and the change attribute, and those 
are locked down.

> Retaining an inode.  Can I add a new hard link to a Retained file? 

No.

> Reduce the link count from 2 to 1?  Probably I would be prohibited

No.

> from reducing the link count from 1 to 0.

That's equivalent to deleting the file in most file
systems, so yes you would.

> 
> (c) What's special about Alan's guess about making a temp
> (un-Retained) copy of a Retained file doesn't seem special to me. 
> All naï¿½ve copies would likely be un-Retained.  What would be a very

Yes.

> special operation indeed is to transfer the Retained property from
> one copy to another, so you'd require the ability to un-mark a file
> as Retained under this special operation.  The alternative is that

Retained is like the roach motel. Once you check in, you
don't check out. If the copy program copies Retained, too bad,
the file is around for a long time.

You might want a variant of "cp -p" that says copying all the
metadata and attributes (as well as the data) except for Retained.

> all copies of a Retained file themselves become Retained, potentially
> producing a lot of litter.
> 
> 		Craig