Subject: Re: [ftpext] Followup on FTPEXT2 BOF in Maastricht

From: Alfred Hönes <[email-masked]>
Date: Fri, 29 Oct 2010 19:46:30 +0200 (MESZ)

John,
thanks for your additional considerations.

You wrote, in response to Alexey's appeal:

>>> [Alexey]
>>> In particular, I would like this WG to revise the FTP URI spec,
>>> but I understand that there might be no interest in doing that.
>
> [John]
> I'm less concerned about interest than I am about the right mix
> of skills.  I note that draft-hoffman-ftp-uri-04 expired over
> five years ago and that there has been a lot of thinking about
> URIs since.  I wonder if that draft would even still be a good
> foundation (I just glanced at it but haven't looked carefully at
> it in a couple of years).  My recollection is that it carries
> forward the URI definition from RFC 1738 and therefore has no

That's what has Paul confirmed.  That draft essentially only was
an excerpt from the old RFCs and there was a lack of energy and
resources, and of interest from the community, in doing improvements.

> provisions for extensions (commands or functions not found in
> RFC 959) at all.  It also appears to have assumed that FTP was
> going to be static: not only is that unlikely to be true in a
> world in which we have an extension mechanism and are using it
> but overloading NLST (or other commands) on TYPE could turn out
> to be a really bad plan.
>
> My instinct is that it might be reasonable to put "new/updated
> FTP URI scheme" somewhere in the WG's queue, but that, while
> such an effort should consider the RFC 1738 /
> draft-hoffman-ftp-uri approach, it may well want to start over
> and design something appropriate for "modern" FTP, even if that
> requires a different scheme keyword.

My existing pre-draft already _is_ different from
draft-hoffman-ftp-uri-04, but it is still imcomplete and
not yet ready to be submitted.

I indeed plan to do that ASAP, most likely shortly after the IETF.

One of the points where I got stuck when this work started out also
was related to the perceived mismatch between the very conservative
approach of the old 'ftp' URI scheme and the capabilities of more
modern FTP servers that you describe.

In particular, one raw idea I had started off during working on the
text to describe the tedious step-by-step navigation procedures
prescribed in the old spec for a "consumer" of an 'ftp' URI
(see Section 3.2.2, "FTP url-path", of RFC 1738).
This procedure was based on the lack of an accepted model for the
(abstract or concrete) hierarchical naming system (supported by URIs
in general, but not expected by FTP servers at the time of RFC 959).

Since those days, the old FTPEXT WG has produced a specification
of the TVFS, published in RFC 3659.

My idea then was:  If an 'ftp' URI could carry an indication that the
server specified in the <authority> part if the URI supports the TVFS
model (it will announce TVFS capability if asked), and therefore that
URI follows TVFS semantics, an application interpreting such URIs
could save a lot of round trips and gain performance (on both sides
of the control connection and on the network) by directly changing
directory in a single step over multiple hierarchical levels.

The proper way I envision to encode this extra information is via a
URI parameter, say  ";tvfs".

This would maintain backwards compatibility (a client that doesn't
understand and support the parameter would ignore it -- following the
robustness principle -- and behave as inefficiently as it did before,
and the original creator/publisher of an 'ftp' URI is expected to
know the capabilities of her server(s) and therefore would only add
this parameter if the related expected "new" client behavior would
be supported by the server(s).

Similar extensions could be conceived for other server capabilities
(yet to be determined) for which it would make sense to make
prospective URI consumers aware of beforehands, without having to
negotiate with the server.

Questions:
+++++++++

A) John:  Does this idea match what you had in mind ?

B) to the list:

(1)  What do you think about this idea ?   In detail:

(2)  Is there actually a problem in practice, i.e. are there
     opportunities and needs, e.g. for optimization, that actually
     need to be addressed?

(3)  Does the general approach of encoding server capabilities
     in 'ftp' URIs make sense?

(4)  Would there be good chance this would be supported by
     (client) implementations and 'ftp' URI creators ?

(5)  Does the encoding in a URI parameter make sense to you?

(6)  Is the above idea worth being specified in the inital new
     ftp-uri draft ?

Feedback welcome!

Kind regards,
  Alfred Hönes.

-- 

+------------------------+--------------------------------------------+
| TR-Sys Alfred Hoenes   |  Alfred Hoenes   Dipl.-Math., Dipl.-Phys.  |
| Gerlinger Strasse 12   |  Phone: (+49)7156/9635-0, Fax: -18         |
| D-71254  Ditzingen     |  E-Mail:  [email-masked]                     |
+------------------------+--------------------------------------------+

Appendix -- for more context:
++++++++

The introduction of my current pre-draft currently says:

<snip>

1.  Introduction

   Uniform Resource Identifiers (URIs) were initially described for the
   IETF in RFC 1630 [RFC1630], and later on more precisely specified,
   first in RFC 1738 [RFC1738] and RFC 1808 [RFC1808], then by RFC 2396
   [RFC2396], which eventually was obsoleted by STD 66, RFC 3986
   [RFC3986].  The earlier documents also specified how to define
   schemes for URIs; the applicable rules and procedures for URI schemes
   are now prescribed by BCP 35, RFC 4395 [RFC4395].

   The first definition for many 'classical' URI schemes also appeared
   in RFC 1630 [RFC1630]; the first standard specifications for these
   were contained in RFC 1738 [RFC1738], and RFC 1808 [RFC1808] has
   elaborated on "relative URLs" (now more precisely called "relative
   URI references" in STD 66).  These documents were made obsolete
   without creating replacement documents for all parts of their URI
   scheme specific content.  Thereby, some basic URI schemes that are in
   widespread use have been orphaned, leaving the IANA Registry of URI
   schemes without a valid reference to a normative definition document.

   Most of these gaps already have been closed (by RFCs 4156, 4157,
   4248, 4266, and 5538).  This document recollects and updates the
   definition of the 'ftp' URI scheme from the older RFCs to allow that
   material to remain on the Standards Track and bring it in alignment
   with current IETF STD and BCP documents.

   Note that the 'file' and 'ftp' URIs are not the same, even when the
   target of the 'ftp' URI is the local host.  Specification of the
   'file' URI scheme is out of scope for this draft.

</snip>

And this is what I have in the central syntax and semantics part,
*without* any mention of the aforementioned extension:

<snip>

2.4.  FTP url-path

   The <url-path> of an FTP URI has the following form:

             <cwd1>/<cwd2>/.../<cwdN>/<name>[;type=<typecode>]

   <cwd1> through <cwdN> and <name> are (possibly encoded) strings and
   <typecode> is one of the characters "a", "i", or "d".  The <cwdx> and
   <name> parts may be empty.  The part ";type=<typecode>" may be
   omitted.  In fact, the whole url-path, including the leading "/", may
   be omitted.

   The <url-path> is interpreted as a series of FTP commands as follows:

   o  Each of the <cwd> elements is to be supplied, sequentially, as the
      argument to a CWD (change working directory) command.

   o  If the typecode is "d", perform a NLST (name list) command with
      <name> as the argument, and interpret the results as a file
      directory listing.

   o  Otherwise, perform a TYPE command with <typecode> as the argument,
      and then access the file whose name is <name> (for example, using
      the RETR command.)

   Within a <name> or <cwdx> component, the characters "/" and ";" are
   reserved and must be encoded.  The components are decoded prior to
   their use in the FTP protocol.  In particular, if the appropriate FTP
   sequence to access a particular file requires supplying a string
   containing a "/" as an argument to a CWD or RETR command, it is
   necessary to encode each "/".

   Historical note: Most FTP client implementations precede the <cwd1>
   with a "/" before sending the CWD command.  This is in conflict with
   RFC 1738, although the practice is quite widespread.  Thus, a client
   that is presented with the URI ftp://[email-masked]/abc/def might
   send the two commands "CWD /abc" and "RETR def" or it might send the
   two commands "CWD abc" and "RETR def".  Server implementers should be
   aware of these two different interpretations of the same URI.

   FTP URIs may also be used for other operations; for example, it is
   possible to update a file on a remote file server, or infer
   information about it from the directory listings or FTP extension
   commands.  The mechanism for doing so is not spelled out here.

2.5.  FTP Typecode is Optional

   The entire ;type=<typecode> part of a FTP URI is optional and is
   rarely used.  Historically, the typecode option was rarely
   implemented and, in practice, dereferencing is done by guessing.  If
   the typecode is omitted, the client program interpreting the URI must
   guess the appropriate mode to use.  In general, the data content type
   of a file can only be guessed from the name, such as from the suffix
   of the name; the appropriate type code to be used for the transfer of
   the file can then be deduced from the data content of the file.

</snip>

Additional question to the list:

  I have been told that in practice RFC 1738 behavior described above
  is not followed and RFC 1630 behavior is implemented:

     In current practice, a trailing "/" in the <url-path> (i.e. an
     empty <name> is regarded as equivalent to (and a substitute for)
     ";type=d".

  Is that correct ?

---

From: John C Klensin <[email-masked]>
Date: Fri, 29 Oct 2010 15:54:20 -0400

--On Friday, October 29, 2010 19:46 +0200 "Alfred H?nes"
<[email-masked]> wrote:

>... 

No, but mostly because I'm trying to remain agnostic.
Historically, the reason why the navigation facilities of RFC
959 are so primitive is because, at the time FTP was developed,
we had a wide variety of file system organizations and no
general model was possible.   With RFC 3659 and TVFS, the
community recognized that virtually all contemporary file
systems use a hierarchical model that, when abstracted from
issues of syntax, actual file storage forms, and encoding of
content types into file names, all turn out to be pretty much
the same.  If we expect that to remain the case, or decide that
FTP should make that case easy and any other cases hard, then
your idea, plus or minus details, makes a lot of sense.

However, consider the following strawman fantasy: say that
various "semantic connections among data" take off and that some
were to actually develop a file system and data model that is
semantic-relational rather than hierarchical or
multihierarchical.  At that point, TVFS becomes as irrelevant as
some of the historical data storage models are to the
hierarchical one.

>      opportunities and needs, e.g. for optimization, that
> actually need to be addressed?

Note that these issues might be an additional argument for
"streamlined" or combination commands such as those for which
there is at least one proposal in the hopper.  I'd rather see
aggressive, asynchronous-command pipelining (which is part of
what we had in mind in the early 70s when the two-stream model
was defined) but it may be too late for that.

>      in 'ftp' URIs make sense?

I rather doubt it, but that is a much more complicated question.

>...

    john

p.s. This list is not on the non-WG mailing lists page.  Unless
we anticipate a charter approval _really_ quickly, that should
probably be fixed.