Subject: [nfsv4] Issue53 - Add data retention support

From: Mike Eisler <[email-masked]>
Date: Mon, 10 Jul 2006 11:57:21 -0700 (PDT)

Here is an outline of a proposal, which is based on conversations
with Alan Yoder and Jay Moorthi of Network Appliance, who work on
NetApp's data retention feature. Errors are mine. I'm sure
Alan will correct me.

My understanding is that Data Retention is a concept that
has been driven by legislation and regulations passed by
many countries in recent years. 

The rationale is political and other jurisdictions have 
decided that some data (e.g. health records) must be retained for
a sufficient period of time in order for the stake holders associated
with data to not be harmed due to deletion (unintended or otherwise)
of the data. 

I'm told it would be difficult to impossible to normalize all these
regulations and support a full range of retention attributes.
That seems more like the place for applications using
higher level APIs than what NFSv4 is intended to support,
such as the work that SNIA is doing (
http://www.snia-dmf.org/xam/faq.shtml ). That said, 
the "low hanging fruit" consists of two attributes:

- Retained. This is a boolean attribute. By default it is FALSE.
  Once it is TRUE, a file, its named attributes, and some other
  attributes, including modification time, change, meta data time,
  MUST NOT change, ever. The Retained attribute once set to TRUE,
  MUST stay true.

- Retention Time. This is a time (date really) to which 
  a file object (and its named attributes) are guaranteed to 
  be retained. The value is provisional until Retained is set 
  to TRUE, and thus has no meaning for compliance with 
  retention regulations until Retained is TRUE. Once Retained 
  is set to TRUE, the Retention Time MAY be changed, but it 
  MUST not be set to a new value less than its current value. Once the
  retention time is reached:

  - the file MAY be deleted. However, individual named attributes
    MUST NOT be deleted. 

  - the contents of the file, its named attributes, and some
    other attributes MUST NOT change as long as the file exists.

Normally only the owner of a file can set Retained or Retention Time,
though the use of ACLs may extend modification rights for these
attributes to other principals. Just having attribute modification
rights to the file object does not guarantee the right to set 
Retained or update Retention Time. NFSv4.1 server implementations 
MAY support policies that allow the storage administrator to limit
the capability of users can set retention attributes (including
whether a user can set a retention attribute and the range
of retention times).

I look forward to discussion on this mailing list. I see
I am on the agenda
(http://www3.ietf.org/proceedings/06jul/agenda/nfsv4.txt ) to 
discuss this tommorrow, so I look forward to the discussion
in Montreal too.

  -mre

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Brent Callaghan <[email-masked]>
Date: Mon, 10 Jul 2006 12:49:42 -0700

Interesting.  Some questions/comments:

o Would the retention attribute be advisory or mandatory ?
    For instance, could a client (or server app) that doesn't know
    of the attribute just blow the file away anyway ?

o If the file is in the form of a log, then you'd like to retain
     previous entries, yet be able to add new ones.  The
     retention attribute would preserve old entries, but prevent
     the addition of new ones.

o How would the attribute work with a collection of related
     files in a directory ?  Would it also apply to directories as
     well as files ?

o A problem with any new attribute: the attribute is easily lost
     if archived on a filesystem, archive or device that doesn't
     know about it, i.e. the retention attribute wouldn't be retained.

o  How is the attribute an improvement on existing data retention
     policies, e.g. archival backup, snapshots etc.

	Brent

---

From: Mike Eisler <[email-masked]>
Date: Mon, 10 Jul 2006 13:32:59 -0700 (PDT)

--- Brent Callaghan <[email-masked]> wrote:
Brent Callaghan <[email-masked]> wrote:

Brent,

Thanks for the fast feedback.

>     of the attribute just blow the file away anyway ?

Mandatory. A client that doesn't know about the attribute doesn't get
to skirt the legal framework these attributes are intended to support.

Think of this as simulating a blank CD-R, only much more flexible.

>      the addition of new ones.

Yes that is an interesting point. On the other hand, to implement
it requires correct implementation of append writes, which
we've punted again for good reasons. 

>      files in a directory ?  Would it also apply to directories as

As proposed, setting Retain on a directory prevents the directory from
having entries added or removed, but doesn't affect the contents of
each directory entry. That said, I can see arguments for setting
Retain on a group of files, whether it be identified by directory
or some other list identifier. However, these arguments
could apply to setting of other attributes or other state change.
So I would prefer to see a general mechanism for changing a set of
file objects, versus a single mechanism for these two attributes.

>      know about it, i.e. the retention attribute wouldn't be

Yes. But the data still persists on the primary until there is
a disaster. 

Presumably the regulatory requirements would (or will someday)
compel the customers of this feature to have
a disaster recovery plan in compliance with 
Retention regulations. E.g. use archival/backup systems 
that preserve the retention bit. It is beyond the scope of
the NFSv4.1 protocol to enforce that.

>      policies, e.g. archival backup, snapshots etc.

I can delete archived backups, snapshots, etc., but short of violence
rendered upon the storage subsystem, I can't delete Retained data.
Thus Retention is aligned with legal requirements, and snapshots, etc.
are not.

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 13 Jul 2006 01:29:14 -0500

On Mon, Mike Eisler wrote:

...

>   retention time is reached:
...

It seems logical that NFSv4.1 should attributes such as these given
that we have products in the market today that doing this by overloading
existing attributes.

Is there any interest in adding attributes to this proposal to
accomplish "administrative hold" and "remove at" features?

In the case of administrative hold, a retained file would be held
regardless of the retention time until the hold is removed.
In the case of remove at, the file is removed at the specified
time given that it is greater than the retention time.

Spencer

---

From: Mike Eisler <[email-masked]>
Date: Thu, 13 Jul 2006 00:11:51 -0700 (PDT)

> regardless of the retention time until the hold is removed.

Hold makes sense (are there products that support that today?). 

> time given that it is greater than the retention time.

I'm not in favor of a RemoveAt that says the file MUST or SHOULD be
automatically deleted by the server. Bugs, administrative errors, etc.
can lead to data being accidentally deleted in error, which 
defeats the point
of data retention. Also, mandated removal is sometimes associated
with regulations for electronic shredding, which is something
else we should avoid specifying.

However, a RemoveAt that just says: this is the date the file
should be removed, and leaving it up to mechanisms outside the
NFSv4.1 specification to do the removal/shred is reasonable. 

Some files won't have a RemovedAt time, so we will
need to represent it as a variable length array of up to 1 entry.

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 13 Jul 2006 02:31:58 -0500

> NFSv4.1 specification to do the removal/shred is reasonable. 

This would be useful and fine with me.

Spencer