Subject: [VCARDDAV] semi-colon in URIs / BNF

From: Jari Urpalainen <[email-masked]>
Date: Tue, 26 Aug 2008 16:32:14 +0300

Hi !

If i've understood it correctly URIs must appear as percent-encoded in
the current vcardrev i-d without the "standard" vcard value escaping,
that is, semi-colons are _not_ escaped in URIs like in an example of the
spec: 

TEL;TYPE=work,voice,pref,msg:tel:+1-555-555-5555;ext=5555

or is it allowed to escape the semi-colon since parsers can do the right
thing when they expose the real content ?

Also is my reading of the BNF right that text(-lists) must not contain
semi-colon, i.e. it must appear as escaped always ?

Also i'm a bit of confused about the text-list definition in section 5
versus the "formal" grammar in section 9.

So i'm asking this just to get the vcard parser to behave
"correctly" ...

br, Jari

_______________________________________________
VCARDDAV mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/vcarddav

---

From: Simon Perreault <[email-masked]>
Date: Tue, 26 Aug 2008 13:52:40 -0400

On Tuesday 26 August 2008 09:32:14 Jari Urpalainen wrote:
> semi-colons are _not_ escaped in URIs like in an example of the
> spec:
>

The latter. Semi-colons (and any other character for that matter) MAY be 
escaped.

> So i'm asking this just to get the vcard parser to behave

A text-list MAY contain unescaped semi-colons. Only backslashes, newlines, and 
commas MUST be escaped.

Note that a URI is *not* the same as a text-list. It MAY contain unescaped 
backslashes, newlines, and commas.

Did that answer your question?

-- 
Please try Numb, a STUN/TURN server implementation.
Free access at http://numb.viagenie.ca/.
_______________________________________________
VCARDDAV mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/vcarddav

---

From: Jari Urpalainen <[email-masked]>
Date: Wed, 27 Aug 2008 09:56:15 +0300

Thanks Simon for a prompt response, more inline

> The latter. Semi-colons (and any other character for that matter) MAY be 
ok, IMO only the latter should be allowed, since both commas and
semi-colons have a "reserved" meaning (separators) both in uris and
vcard property values. And definitely some normative text about this
behavior must exist in the i-d. 

If both modes were allowed it would be impossible to make a generic
value parser, i.e. the value parser should know in advance does it need
to produce multiple "values" or not (ok, compound types are the most
complex types, but parsing seems to be doable, however).   

> A text-list MAY contain unescaped semi-colons. Only backslashes, newlines, and 

right this is my confusion, text-list (originally from rfc2425 i assume)
abnf says this, but section 9 abnf for NICKNAME and CATEGORIES doesn't
allow unescaped semi-colons. And again this behavior is imo reasonable
for the value parser.

> backslashes, newlines, and commas.
well, rfc3986 _is_ pretty restrictive about allowed characters, and yes
commas and semi-colons are allowed, but many other characters must be
percent-encoded like newlines or backslashes (if really needed in the
uri).

> Did that answer your question?
So in summary, from the implementation pov if the precedence of colons
and semi-colons is maintained always (semi-colons separate bigger
entities than colons) and both are escaped as needed it is fairly easy
to make a "generic" value parser. If the precedence changes or unescaped
commas/semi-colons exist inconsistently, the value parsing is much
harder, uglier and more error prone. 

br Jari

---

From: Simon Perreault <[email-masked]>
Date: Wed, 27 Aug 2008 10:54:25 -0400

> behavior must exist in the i-d.

This is not strictly true. Commas do not have a reserved meaning in all vCard 
property values. For example, URI values is one case where commas have no 
special meaning. In text-list, they are special and are used to separate the 
components of the list.

Since it doesn't affect the URI value there shouldn't be any text about that.

> > A text-list MAY contain unescaped semi-colons. Only backslashes,
> > newlines, and commas MUST be escaped.
>

Section 9 is wrong. It's in our task list to clean it up. Unescaped 
semi-colons will be allowed.

> > Note that a URI is *not* the same as a text-list. It MAY contain
> > unescaped backslashes, newlines, and commas.
>

Of course, the set of allowed characters is limited by the syntax of a URI. My 
point was that vCard doesn't impose any other restrictions. Of course if URIs 
could contain newlines we'd have to escape them in vCard...

> harder, uglier and more error prone.

Here's the steps for unescaping:

1. Undo line folding for all value types.
2. Next, branch on the value type.
  - text-list: \\ \, \n
  - anything else: no escaping at all
3. Does the property use semi-colon delimiting?
  - yes: \;
  - no: no escaping

If you can think of ways we can make this simple, please tell us! :)

Simon

---

From: Jari Urpalainen <[email-masked]>
Date: Thu, 28 Aug 2008 11:12:34 +0300

inline

> Since it doesn't affect the URI value there shouldn't be any text about that.
please, even a note might suffice that it may be escaped or not. 

> semi-colons will be allowed.
ok, though not what i was anticipating/hoping for since i thought
section 9 is correct...

> If you can think of ways we can make this simple, please tell us! :)
indeed my no 1 requirement: you can use the very same value parser
despite the type, so no decision in phase 2 at all. 

so let's take still some concrete trivial parsing examples (i don't
claim every parser is doing this with the very same way, but i'm aware
how a quite popular oss submission does this or tries to do it with some
success).

First the most complex type, compound (or structure):
N:Stevenson;John;Philip,Paul,\;;Dr.;Jr.,M.D.,A.C.P.
So we have 5 components (or whatever is the name of this part of a
compound structure) "Stevenson", "John", "Philip,Paul,;",
"Dr.","Jr.,M.D.,A.C.P." and some of them have multiple values. So a
decent parser will extract all these 5 components and then parse those
multiple values within components. E.g. "additional names" component
values are then "Philip", "Paul" and ";". 

(ADDR type might be a better example here where you might have multiple
streets separated with commas, and still also some escaped commas, but
this is just an example...)

Yet you can have text lists which are separated by the comma character
with a really weird nickname:
NICKNAME:Jim;,Jimmie
if you throw this to the previous compound structure parser, it will get
definitely a wrong answer unless you escape the semi-colon: 
NICKNAME:Jim\;,Jimmie
when it finds one component with two values.

However, in the URI type, the semi-colon is indeed not that rare or
theoretical, e.g. even in the I-D example:
TEL;TYPE=work,voice,pref,msg:tel:+1-555-555-5555;ext=5555
So once again the previous parser won't do the right thing unless you
escape the semi-colon. With SIP URIs the very same thing may happen.

So basically if you'd require commas and semicolons to be escaped always
within values the parsing would be pretty painless (if parsing can ever
can be, since it is typically the most complex thing to implement
properly). That being said, if your value parser must interpret the type
before parsing you need some ugly logic based e.g. on the known
attribute names which will certainly lead to some "utter crap" comments
in the code ;-) which i'm trying to get rid of. Also IMHO it would also
make the spec easier to understand since you don't need to figure it out
when to escape or not. But this is not the first nor the last case when
discontent exists between specs/implementations, at least from my
part ;-), so at least now i understand the I-D intent of this issue.

thanks, Jari

---

From: Simon Perreault <[email-masked]>
Date: Thu, 28 Aug 2008 07:48:03 -0400

On Thursday 28 August 2008 04:12:34 you wrote:
> > This is not strictly true. Commas do not have a reserved meaning in all
> > vCard property values. For example, URI values is one case where commas
> > have no special meaning. In text-list, they are special and are used to
> > separate the components of the list.
> >

Well, that wouldn't be true. :) Commas MUST not be escaped in URI values.

Since my explanations obviously aren't clear, could you please propose some 
text that you think would be unambiguous?

> escape the semi-colon. With SIP URIs the very same thing may happen.

I disagree that semicolons should be escaped in call value types.

- Semicolons are only special in a subset of the properties using a text-list 
value type.
- Semicolon escaping is to them what URI escaping is to URI values, and what 
base64 encoding is to binary values. You *cannot* escape per-value-type 
processing/escaping.

> which i'm trying to get rid of. Also IMHO it would also 

I agree that the escaping rules should be made clearer. As usual, if you have 
some text in mind, please contribute. :)

---

From: Jari Urpalainen <[email-masked]>
Date: Fri, 29 Aug 2008 10:54:04 +0300

> Well, that wouldn't be true. :) Commas MUST not be escaped in URI values.
but you said in the original response that "Semi-colons (and any other
character for that matter) MAY be escaped." which contradicts with this.
surely in reality, it can only be either escaped or not, otherwise you
end up with double-escape troubles.

> some text in mind, please contribute. :)

in section 5 instead of the paragraph starting with "Compound type
values..."

Compound type values are delimited by a field delimiter, specified by
the SEMI-COLON character (ASCII decimal 59). The separated individual
components MAY have multiple values separated by a COMMA character. 
A SEMI-COLON or a COMMA in a component of a compound property value MUST
be escaped with a BACKSLASH character (ASCII decimal 92). 

In all other cases, a SEMI-COLON or a COMMA character in a property
value MUST NOT be escaped with a BACKSLASH character.

br, Jari

---

From: Jari Urpalainen <[email-masked]>
Date: Fri, 29 Aug 2008 11:43:52 +0300

> value MUST NOT be escaped with a BACKSLASH character.

oops, the last sentence certainly like:
In all other cases except in single-valued text types, a SEMI-COLON or a
COMMA character in a property value MUST NOT be escaped with a BACKSLASH
character.

Jari