Subject: Re: [ftpext] New Version of FTP HASH, RANG, LOCK, and HOST

From: Anthony Bryan <[email-masked]>
Date: Fri, 1 Feb 2013 23:26:05 -0500

right, we could leave the boundary REQUIRED but have an OPTIONAL size
that would be used (probably the majority of the time?)

btw, we are working on some of the issues that have been brought up w/
RANG & LOCK so we put new versions out.
we'll continue on this & should have more updates soon.

the RANG text has been re-organized to reflect that it is not transfer
specific, but just about selecting a range, that can be used with
transfers, HASH, etc, as it always has been.

fancy diffs:
http://tools.ietf.org/html/draft-bryan-ftp-range
http://tools.ietf.org/html/draft-bryan-ftp-lock

On Sun, Jan 27, 2013 at 6:52 PM, Bruce Blackshaw <[email-masked]> wrote:
> It's true that you don't always know the size at the start.
>
> Nonetheless, it could be be specified that iff the size of the raw data is
> known, then "size" could be used. In the many cases that this is possible,
> performance should be faster (although I can't quantify by how much).
>
> regards
>
> Bruce Blackshaw
>
>
> On 28/01/2013 4:32 AM, Alun Jones wrote:
>>>
>>> Of Ángel González
>>>
>>> Bruce Blackshaw wrote:
>>>>
>>>> Re LOCK, I think it would be useful to have an alternative to
>>>> "boundary" called "size", that could provide the size of the raw data
>>>> for binary transfers.  That might provide quite a performance
>>>> advantage compared to having to search for the boundary.
>>>>
>>>> regards
>>>
>>> Good point. I think it would be much more useful than the boundary
>>> method.
>>> The value of such size parameter should be the amount of data sent on the
>>> wire.
>>
>> This has an obvious problem - what if the size of the file changes during
>> the transfer?
>>
>> There are many uses of FTP (webcam being the one that comes immediately to
>> mind) where a "file" is retrieved, and is handled as if it is a
>> potentially-infinite stream. We have a webcam at work that encodes video
>> as
>> an animated GIF that never ends. Doesn't display in Internet Explorer,
>> because IE is paranoid, but it's a use-case that should be considered. The
>> boundary method works for this use, where the size method doesn't.
>>
>> Alun.
>> ~~~~
>>
>>
>
> _______________________________________________
> ftpext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ftpext

--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads

---

From: Hans Andersen <[email-masked]>
Date: Sat, 2 Feb 2013 15:59:35 +1000

I just wanted to back my colleague, Bruce, on the performance issue related
to detecting boundaries.  A few weeks ago I wrote code that detects MIME
boundaries in HTTP-POSTs while piping the data to a file.  The best I
managed to achieve was significantly slower than simple piping.  I
therefore think that optionally including the size is important for
performance.

---

From: Tim Kosse <[email-masked]>
Date: Sun, 03 Feb 2013 22:05:03 +0100

Hi,

here's my review of the LOCK draft:

1. From the draft: "As mentioned, the use of two ports with FTP became
problematic with the advent of firewalls and NATs. By using a single
port, like many other protocols, these problems are eliminated."

For plain FTP, transmitting arbitrary data over the control connection
will cause massive problems. There are plenty of firewalls and NAT
devices that close the FTP connection if there's strange data on the
control connection.

Worse, many NAT devices act as transparent FTP proxy. They have the
nasty habit of not filtering out FEAT responses they do not know about,
so once a client uses an advertised feature the proxy doesn't support,
the connection will fail.

In other words, the additional problems with firewalls and NAT devices
caused by LOCK completely defeat its intended purpose.

There is a good solution for this problem though, namely FTP over TLS.
Following the AUTH TLS command, most firewalls and NAT routers stop
interfering with the control connection. Perhaps mandate FTP over TLS
for LOCK?

2. What about Telnet signals (e.g. IP and Synch) on the NVT during a
transfer?

Suggestion: Not allowed during a running transfer.

3. How is the STAT command to be handled?

From RFC959: The command may be sent during a file transfer (along with
the Telnet IP and Synch signals--see the Section on FTP Commands) in
which case the server will respond with the status of the operation in
progress

There's no way to send the STAT reply during an ongoing download (e.g.
RETR) if LOCK is being used.

During an ongoing upload (e.g. STOR), how should STAT be handled?

Suggestion: No STAT during a running transfer allowed.

4. SIZE command

From RFC3659: The FTP command, SIZE OF FILE (SIZE), is used to obtain
the transfer size of a file from the server-FTP process.  This is the
exact number of octets (8 bit bytes) that would be transmitted over the
data connection should that file be transmitted.

I suggest that if LOCK is being used, it refers to the amount of octets
that would be enclosed in the boundary start end end, excluding the
boundary markers.

5. From the draft: "To turn off LOCK and return to the previous
behavior, a client can
   issue a PORT, PASV, or similar command."

Not precise enough. What is similar? PASS is similar to PASV, only one
different letter. I suggest explicitly naming all commands.

6. REST command. Same problem as with SIZE. I propose the restart offset
refers to the actual data payload excluding the boundary.

7. Transfer modes other than MODE S aren't mentioned. Explicitly state
that LOCK with other transfer modes is undefined.

Regards,
Tim Kosse

---

From: Ángel González <[email-masked]>
Date: Sun, 03 Feb 2013 23:09:23 +0100

> interfering with the control connection. Perhaps mandate FTP over TLS
Worth suggesting as a note, but I don't think it should be required.

> Suggestion: No STAT during a running transfer allowed.

I think the LOCK command should be changed (and probably renamed)
into a command multiplexing the connection.
The current LOCK command would be equivalent to multiplexing one
channel with the control connection. But it would add support to
send commands/signals/replies between chunks, plus doing several
transmissions in parallel.

---

From: Anthony Bryan <[email-masked]>
Date: Tue, 5 Feb 2013 09:30:56 -0500

Tim, thanks so much for the review!

here's the text I've used to address your comments.

let me know what you think, if we can refine it more.

> interfering with the control connection. Perhaps mandate FTP over TLS

   FTP secured with TLS [RFC4217] is RECOMMENDED when LOCK is in use as
   most firewalls and NATs will stop interfering with the control
   connection following the AUTH TLS command.

> Suggestion: Not allowed during a running transfer.

   The STAT command and Telnet IP and Synch signals are not allowed
   during a running transfer.

> Suggestion: No STAT during a running transfer allowed.

   The STAT command and Telnet IP and Synch signals are not allowed
   during a running transfer.

> that would be enclosed in the boundary start end end, excluding the

   When LOCK is in use, the response to a SIZE command refers to the
   amount of octets that would be enclosed in the boundary start and
   end, excluding the boundary markers.

> different letter. I suggest explicitly naming all commands.

   To turn off LOCK and return to the previous behavior, a client can
   issue a EPRT, EPSV, PASV, or PORT command.

> refers to the actual data payload excluding the boundary.

   When LOCK is in use, the restart offset of the REST command refers to
   the actual data payload excluding the boundary.

> that LOCK with other transfer modes is undefined.

   LOCK is undefined with other transfer modes, besides STREAM.