Subject: [art] RFC8089 file scheme and percent encoding

From: Christian Amsüss <[email-masked]>
Date: Mon, 23 Aug 2021 12:45:32 +0200

Hello ART people,

discussions around XDG's thumbnail specification[1] which builds on
URIs, mostly file:// URIs.

(Looks like this is the most recent venue to discuss the spec, please
redirect me otherwise).

As I understand RFC8089, it does not mandate, or even recommend, any
particular escaping of characters inside file names. This is relevant
for those octets where the encoder has a choice in encoding them, eg.
the colon (":") character inside filenames. (That character being
problematic on some platforms is immaterial to the discussion at hand;
same goes for ";", "+" etc).

(It's not explicit either about escaping the `#` and `?` characters, but
it doesn't need to -- for the `#` that's because the fragment is
independent of the scheme, and for `?` as well as `[` and `]` it's
implied by the path-absolute ABNF production).

In many protocols, this isn't an issue, because there is a URI producer
(often the server) that is somehow authoritative of the names. File
URIs, on the other hand, are commonly produced for any given local file
by different programs running on the same machine, and they need to
agree for some uses. (That's especially true for the thumbnail case,
where different URIs or URI references are hashed).

Is there any (other) recommendation on whether a file whose name
contains a reserved character should have it percent encoded, or a part
of 8089 which I missed?

I'd like to recommend in the XDG discussion that reserved characters be
not encoded unless syntactically necessary (ie. they are `?`, `#`), but
I'd prefer to have that recommendation backed by something they can cite
when explaining the rules.

Thanks
Christian

[1]: https://gitlab.freedesktop.org/xdg/xdg-specs/-/merge_requests/29#note_1037277

-- 
Es ist nicht deine Schuld, dass die Welt ist, wie sie ist -- es wär' nur deine
Schuld, wenn sie so bleibt.
(You are not to blame for the state of the world, but you would be if that
state persisted.)
  -- Die Ärzte

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 23 Aug 2021 12:59:00 +0200

> contains a reserved character should have it percent encoded, or a part

RFC 3986 defines that path segments need to be characters from the pchar production.
So, e.g., "/" / "?" / "#" / "[" / “]” are out (“/“ of course is the path segment delimiter) and need to be percent-encoded within a pathname.

Grüße, Carsten

---

From: Christian Amsüss <[email-masked]>
Date: Mon, 23 Aug 2021 13:20:08 +0200

Hello Carsten, ART,

On Mon, Aug 23, 2021 at 12:59:00PM +0200, Carsten Bormann wrote:
> So, e.g., "/" / "?" / "#" / "[" / “]” are out (“/“ of course is the
> path segment delimiter) and need to be percent-encoded within a
> pathname.

It defines a production, and thus forces things on some chars, but pchar
allows both `file:///a%40b.jpg` and `file:///[email-masked]`. Either is a valid
expression of a file named `/[email-masked]`, and mapping either back to POSIX
operations will find the same file, but the direction from a file path
to a URI is lacking guidance.

(Precisely, the ambiguious characters are sub-delims, ":" and "@").

BR
c

-- 
To use raw power is to make yourself infinitely vulnerable to greater powers.
  -- Bene Gesserit axiom

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 23 Aug 2021 13:56:12 +0200

On 23. Aug 2021, at 13:20, Christian Amsüss <[email-masked]> wrote:
> 
> (Precisely, the ambiguious characters are sub-delims, ":" and "@“).

You could percent-encode “a” as well, so that is “ambiguous” as well, for your definition of ambiguous.
Why would you?

Grüße, Carsten

---

From: Christian Amsüss <[email-masked]>
Date: Mon, 23 Aug 2021 14:22:47 +0200

> for your definition of ambiguous.

Any unreserved character can, under the rules of syntax-based
normalization of RFC3986 Section 6.2.2.2 alone, be normalized into the
unreserved character. The ambiguity is only between the normalized form
and some forms that are not normalized.

For subdelims, ":" and "@", there is no syntax-based normalization, and
while the schema implies some equivalence between the forms (because
they map to the same file), it gives no preference to the encoding.

> Why would you?

Frankly I wouldn't (because it's needless over-encoding, and on top of
it because I can't express that over-encoding in CoAP path components),
but I'm observing it in practice where people put things through generic
percent encoders (which'd do the right thing in other places).

If "There's no point in encoding these in file URIs" is the outcome of
this thread (possibly with a note that it's not explicit because nobody
ever thought anyone would do it), I'm fine with it. (It's not an erratum
to file then, is it? Or should I better file it anyway if only for
documentation?).

Thanks
c

---

From: Martin J. Dürst <[email-masked]>
Date: Thu, 26 Aug 2021 16:01:29 +0900

Hello Christian,

> and some forms that are not normalized.

Yes, and this is a generic property of URIs, not a property of any 
specific scheme (such as file:).

> ever thought anyone would do it), I'm fine with it.

There's definitely no needed (unless syntactically necessary as you say 
in your original post for '?' and '#'). But that just means that RFC 
3986 applies. See RFC 3986, Section 2, and in particular Section 2.4 
(https://datatracker.ietf.org/doc/html/rfc3986/#section-2.4).

> (It's not an erratum

I don't think there is any need for an erratum. That's just how URIs 
work. And file: URIs aren't an exception.

Regards,   Martin.

> Thanks
> c
> 
> 
> _______________________________________________
> art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/art
>

---

From: Christian Amsüss <[email-masked]>
Date: Thu, 26 Aug 2021 10:19:40 +0200

Hello Martin,

On Thu, Aug 26, 2021 at 04:01:29PM +0900, Martin J. Dürst wrote:
> Yes, and this is a generic property of URIs, not a property of any specific
> scheme (such as file:).

Sure; I only pointed that out to differentiate Carsten's "a" vs. "%61"
example from delimiters.

> There's definitely no needed (unless syntactically necessary as you say in
> your original post for '?' and '#'). But that just means that RFC 3986
> applies. See RFC 3986, Section 2, and in particular Section 2.4
> > [...]
> I don't think there is any need for an erratum. That's just how URIs work.
> And file: URIs aren't an exception.

File URIs are a bit special in that process of composing the URI from
component parts (mentioned in the Section 2.4) is not managed by a
single producer, but in that any application will turn a file name into
a file:// URI (except on Redox OS which uses URIs natively), and later
pass them around.

These different producers act with different escapings for lack of
guidance (where "minimally escaped" would suffice), and thus produce
different URIs for the same resource. On HTTP & co, that doesn't
happen[1] because there's a single source of authority (the server).

If two applications store metadata about file:///media/Steins;Gate/ and
file:///media/Steins%3BGate/, and that data isn't unified, who is to
change their ways, and why?

In an HTTP situation, the server operator would need to act, as it
serves the same content under aliased URIs (rather than redirecting to
the canonical one). But in a file system, we can barely blame the user
for combining applications with different file mappings on the same
machine.

Thanks
c

1: Well, it *does* happen when people write their links manually into
  HTML files, but then the server either willingly accepts URI aliasing,
  or it fixes the situation by providing redirects, eg. to add trailing
  slashes.

-- 
You don't use science to show that you're right, you use science to
become right.
  -- Randall Munroe

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 27 Aug 2021 11:49:34 +1000

[trimming to respond to just one point]

> change their ways, and why?

Well, to quote the book (RFC 3986, Section 2.2):

   URI producing applications should percent-encode data octets that
   correspond to characters in the reserved set unless these characters
   are specifically allowed by the URI scheme to represent data in that
   component.

';' is in the reserved set, and isn't specifically allowed by the file URI
scheme to represent data in the path component.

Cheers
-- 
  Matthew Kerwin
  https://matthew.kerwin.net.au/