Subject: Re: tar for NFS V4

From: Jim Rees <[email-masked]>
Date: Fri, 31 May 2002 10:07:53 -0400

POSIX defines a backward compatible, extensible tar format that can be used
for things like acls and ea.  I think the Solaris tar/pax takes advantage of
this, and linux star (http://acl.bestbits.at/star.html) uses it to encode
acls but not ea.

This is probably outside the scope of what the WG is doing.

---

From: Peter Staubach <[email-masked]>
Date: Fri, 31 May 2002 09:36:15 -0600 (MDT)

> 
> On Thu, May 30, 2002 at 03:21:10PM -0700, Eric Kadison wrote:
> > Has anyone looked into updating the tar program to support new NFS V4
> > features (e.g. the new attributes)?
> 
> "tar" wouldn't know NFS from a hole in the ground.
> 
> If there are APIs in the OSes on which "tar" runs that allow
> applications such as "tar" access to those features, the "tar" that
> comes with those OSes might be updated to use those APIs.

In this vein, the various archivers delivered with Solaris 9 have been
updated to understand the extended attributes model which was
introduced in Solaris 9.

As Guy said, these things shouldn't be file system specific or over
the wire protocol specific.

		ps

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 31 May 2002 10:42:13 -0600 (MDT)

> Just as the the NFSv4 community is going outside its
> traditional scope and getting interested in file system
> migration, I think the NDMP cmmunity have to take a step outside
> our traditional turf and consider seriously the task 
> of backing up date with richer metadata, such as NFSv4, 
> NTFS etc.

Correct me if I'm wrong here Harald, but doesn't NDMP support
these things if only because the server owing the files gets
to decide the backup data stream stored and retrived, and can
put richer metadata into that stream?

That said, I'd like to see a definition of a canonical data
format that many servers can understand, so that they can
exchange data more readily.  I like the AFS model - on any
architecture AFS supports, you can get a bytestream which
includes all data and metadata AFS can store, and you can
then put it out on tape as a backup, restore it, or use it
to do replication and migration.  With NDMP+security fixes,
a good canonical (and extensible) data format could solve
the replication and migration problem for NFSv4 and other
areas.  As a later goal, this could also be replacement for
tar/cpio/pax and friends.

> I could see this being a "joint venture" between NDMP and NFS,
> and possibly others. (WebDAV?). It seems as if there are 
> two deliverables, a data format specification, and an open
> source implementation for Linux.
> 
> First question: What is the general interest level in the NFSv4
> group and in the NDMP collaborative in solving this problem 
> together?

I'd be much in favor of it.  We have some problems logistically
until the Network Backup and Restore (NBR) workgroup is approved
by the IETF, but after that, I think much is possible.  Clearly,
we need this not to be too much of a distraction to the NBR WG.

Is anyone interested in a bar BOF in Yokohama for this stuff?

Rob T

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 31 May 2002 11:20:25 -0600 (MDT)

> Otherwise I very much agree with your thinking. This AFS model,
> is there a handy web page, and is there already a utility
> for it?

Sorry Harald, I was probably too terse with my shorthand.
AFS is the proprietary distributed filesystem from Transarc
Corp., now a wholly-owned subsidiary of IBM.  They have
made OpenAFS available; see http://www.openafs.org/ for more.
I don't know that much internals info is available anywhere,
and know about it because I led a team which ported it to
our OS at a former company.

Rob T

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 31 May 2002 10:48:15 -0700

> Furthermore, GNU tar has a special ability to archive sparse files (I don't
> know how it knows sparseness).  Does that method continue to work under NFS
> V4?

I don't think there's a way in the V4 protocol to distinguish a hole from a 
large run of zero's.  

I think this is information that should be transferred by a migration 
protocol.  In the general case, you can't do it reliably.  If I have a 
4K hole and you have an 8K block size, there's no way.  Nevertheless,
turning a gigabyte hole into a gigabyte of zero's on migration is 
liable to be considered rather anti-social.

-----Original Message-----
From: Eric Kadison [mailto:[email-masked]]
Sent: Friday, May 31, 2002 1:21 PM
To: [email-masked]'
Subject: RE: tar for NFS V4

I agree that a joint effort between the two communities is a very good
long-term idea.  My immediate interest was short term.  tar is currently
used as both a backup program and a standard archive format for
distribution.  It seems to me that customers would want a new version of tar
that:

1.  is aware of the new attributes available under NFS V4.  e.g. tar can
currently set the last modified time of a file during extraction.  Wouldn't
it be desirable to also be able to set the last backup time?  Aren't there
other new attributes, e.g. archive, hidden, in the same class.  

2.  is backward compatible with pre-NFS V4 file systems and existing tar
balls.

3.  is public domain, like GNU tar.

Furthermore, GNU tar has a special ability to archive sparse files (I don't
know how it knows sparseness).  Does that method continue to work under NFS
V4?

Eric

> -----Original Message-----
> From: Robert Thurlow [mailto:[email-masked]]
> Sent: Friday, May 31, 2002 9:42 AM
> To: Skardal, Harald
> Cc: 'Gordon Waidhofer'; Harris, Guy; Eric Kadison;
> [email-masked]; [email-masked]
> Subject: RE: tar for NFS V4
> 
>

---

From: "Skardal, Harald" <[email-masked]>
Date: Fri, 31 May 2002 12:06:29 -0700

Resent to this mailing list since it bounced earlier according
to Spencer.

    Harald

    -----Original Message-----
    From: Skardal, Harald 
    Sent: Friday, May 31, 2002 8:38 AM
    To: 'Gordon Waidhofer'; Harris, Guy
    Cc: Eric Kadison; [email-masked]; [email-masked]
    Subject: RE: tar for NFS V4

    All,
    
    Just as the the NFSv4 community is going outside its
    traditional scope and getting interested in file system
    migration, I think the NDMP cmmunity have to take a step outside
    our traditional turf and consider seriously the task 
    of backing up date with richer metadata, such as NFSv4, 
    NTFS etc. I can not say what the right approach is
    (tar+ vs new utility), but in order to push NDMP along
    we need to make sure that the right underlying tools exists
    such that use of NDMP can be expanded.
    
    There are numerous new technologies available now that might
    help, as compared to when tar was conceived. I would think 
    that XML could be used, either to construct one DTD for
    the superset of all possible file system attributes, or 
    a format where the DTD was included such that all metadata
    encoded in the "tar+" file bu the originating file system 
    could be decoded. 
    Translation into a different file system is a different issue, 
    in some cases attributes may not be translatable. To me, the 
    most important thing here is to be able to store the meaning 
    of the attributes in the source file system with the actual 
    encoded attributes and the data.

    
    Cheers,
    
               Harald 
    
        -----Original Message-----
        From: Gordon Waidhofer [mailto:[email-masked]]
        Sent: Thursday, May 30, 2002 8:33 PM
        To: Guy Harris
        Cc: Eric Kadison; [email-masked]; 
    [email-masked]
        Subject: Re: tar for NFS V4

        The question of enhancing the tar format for new attributes
        appeared on the NDMP (Network Data Movement Protocol,
        [email-masked])
        mail list some time ago. The scope of the question is 
    broader than
        NFSv4. For example, NTFS and MacOS file models have 
    many attributes
        that tar wouldn't know from a hole in the ground :)
        
        Perhaps something akin the Rock Ridge (CD-ROM file system)
        tagged attributes would do the trick. Perhaps leave tar
        alone and do something new. Perhaps.....
        
        Clearly the backup format question is outside the scope of 
        NFSv4 work.
        So too beyond the strict scope of NDMP. Yet participants in
        both these efforts will surely have interest in backup formats,
        file attributes, interoperability/exchangability, and so on.
        There is a worthwhile overlap. Hopefully the day will come
        when we can all take up this issue.
        
        But not today......
        
        Regards,
        	-gww
        
        > comes with those OSes might be updated to use those APIs.
        To remove yourself from the [email-masked] mailing list,
        send mail to <[email-masked]> with the following
        command in the body of your email message:
        unsubscribe xdl-ndmp-tech [your email address]

---

From: Eric Kadison <[email-masked]>
Date: Fri, 31 May 2002 12:40:59 -0700

gww wrote: 
> Harder still is to cook up an archive extension
> method that doesn't make unextended tar's cough
> up fur balls. GNU tar has a few tricks (long file
> names, sparse files, etc). But you still get fur
> balls when you extract with an old tar. And it
> looks like it won't be long before today's GNU
> is the unextended, fur ball coughing tar. And then
> there are present day extensions like Solaris 9.
> Such are future legacy at least. I suppose a
> certain amount of divergence before convergence
> is to be expected.

Here, Here!! Well Said!??  Backward compatibility is a big problem.  GNU tar
is a standard for distribution beyond any one OS.  NFS V4 and NDMP both need
more capability that, implemented by each of us in our own semi-proprietary
way, cause GNU tar to cough up.  Before long, someone needs to perform an
update to GNU tar or it will become obsolete.

It might be too late to create an extension mechanism that won't cause
classical GNU tar to fail.  But it would not be difficult to create a new
version of GNU tar that would recognize and ignore extensions it did not
support.  This would be a much simpler upgrade for the existing customer
base.

Eric