Subject: [VCARDDAV] Proposal around escape character handling (2nd round)

From: Daisuke Miyakawa <[email-masked]>
Date: Tue, 13 Jul 2010 22:35:32 +0900

Hi group!

As you may know, I had proposals around escape characters.

Fortunately or unfortunately the thread became a bit too long for the other
people to look over.
Thus, I'd like to re-organize (and modify a bit) my proposals and ask your
opinions again.
Please let me know when I miss something important mentioned in the previous
thread (e.g. ignoring your opinion), though I carefully checked it.

****** What the current draft (rev12) defines:
Backslash, semicolon, comma, and new line MUST be encoded in accordance with
following rules:
- backslash <-> \\
- semicolon <-> \;
- comma <-> \,
- new line <-> \n or \N

Currently, how semicolon should be handled depends on which property has
it. According to Simon's description:
>* *The rule for comma, backslash, and newline are global and apply
> everywhere (see section 3.3). However, the rule for semicolon only
> applies to some properties which use the semicolon as separator. It is
> up to the people defining a particular X- property to decide whether
> they want to use the semicolon as separator.

****** Proposal 1 (new):
The one-to-one rules above MUST be applied to "all" the properties, even
including X- properties, for uniformity between properties.
In other words, semicolons MUST be escaped even when the property does not
allow multiple values (like (0, 1)).

In this proposal, how readers must/should act when ';' is given without
escape in those properties is undefined. I don't think "undefined" is a good
idea,
but I cannot think up better idea for mentioning it as a formal
specification.

****** Proposal 2 (new):
Add one additional one-to-one mapping.
- \t <-> TAB

Reason:
- This convention has been used to encode usual texts, not in vCard but in
the other text handlers (from C-language), I kind of thought it might be
better to add this
-- I felt vCard looks "exceptional" without this rule.
- I think white spaces should carefully be treated and \t is typical and
important for us to specially take care of.

I'd say that there are few opportunities where I've seen TAB in actual
vCard.
This proposal is just for keeping consistency between vCard 4.0 and the
other escaping rules used in other systems (like programming languages).
# For example, see http://www.python.org/dev/peps/pep-3138/

****** Proposal 3: (currently up to the group's decision)
\uNNNN <-> (a Unicode character with charcode 0xNNNN)
\U00NNNNNN <-> (a Unicode character with charcode 0xNNNNNN, where 0xNNNNNN
SHOULD be more than 0x10000)

The proposal above is mainly based on
http://www.python.org/dev/peps/pep-3138/
I modified this proposal a bit (from "\x" to "\u", "\U").

Reason:
Python, JavaScript and some other programming languages use \u \U format for
encoding Unicode, not \x (it is usually for 8bits).
This is not applied to Perl, where \x{NNNN}, 0xNNNN seem to be used.
I'm not sure about Ruby.
What I can tell is that programming languages actual support this kind of
escaping rules.

Theoretically we don't need to consider surrogate pairs like Python do, but
I think convention and uniformity are important from a practical view point.

Yea: 1 (only me)
Reason:
- The other actual implementations using Unicode (Java, Python, icu4c)
actually support this format.
- I prefer actual usability to theoretical simplicity, while I agree that
"theoretically" this spec is not needed.
-- My experience handling Unicode with surrogate pairs tell me that this
mapping is practically useful.
- I don't think \xNN is needed in vCard, as we don't need to take care of
readability of 8bits. We just need b encoding for it.

Nay: 4 (Simon, Cyrus, Julian, and Barry)
Simon's reply
> I disagree again. If you are using a given character in a sentence,
> whether it is visible or not, it is because you intend the recipient to
> read it. Otherwise, the character would not be useful and would not be
> present. For example, in this paragraph I used many spaces which are
> invisible and I don't think we would gain anything by replacing them
> with \x20 in a vCard. We are encoding user-readable text in vCard, not
> random bits.

Barry's reply:
> a single wire format is sufficient; display of non-ASCII characters is an
orthogonal issue

****** Proposal 4 (new):
When a vCard entry happened to have escaped characters undefined in vCard
4.0 spec, readers SHOULD just remove the backslash and append the wrongly
escaped characters as is.
(This is not "MUST" but "SHOULD", because actual astute vCard readers may
have to cope with wrong input composed by the other composer)

e.g. This is \a pen. -> This is a pen.
(Readers SHOULD NOT understand \a as an alert (like C-language requires) but
just a character 'a').

Jurian's reply related to this proposal
> on \ escaping: either allow \ in front of any character, or be *very*
clear that using it when it's not needed makes the vCard invalid (test
cases? validator service?)

I suppose just requesting readers to remove invalid backslash would suffice.

I agree that my proposals are a bit too dependent on practical view points.
Feel free to correct me if they have problems (from the view of the IETF way
or something I'm not familiar with).

Thanks,

-- 
Daisuke Miyakawa (宮川大輔)
[email-masked]

---

From: Simon Perreault <[email-masked]>
Date: Tue, 13 Jul 2010 09:49:11 -0400

Thanks for organizing your proposals. It makes it much easier to discuss
and track.

For the record, I'll be waiting for consensus to emerge on proposals 1,
2, and 4. I don't strongly care one way or another.

As for proposal 3, right now there seems to be consensus against it.
Unless the situation is reversed, it won't go into the spec.

Thanks,
Simon
-- 
NAT64/DNS64 open-source --> http://ecdysis.viagenie.ca
STUN/TURN server        --> http://numb.viagenie.ca
vCard 4.0               --> http://www.vcarddav.org

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 13 Jul 2010 10:59:42 -0400

Hi Daisuke,

--On July 13, 2010 10:35:32 PM +0900 Daisuke Miyakawa 
<[email-masked]> wrote:

> In other words, semicolons MUST be escaped even when the property does
> not allow multiple values (like (0, 1)).
>
>
> escape in those properties is undefined. I don't think "undefined" is a
> good idea,

; is used as a separator for "compound" values not multiple values (i.e. 
the "cardinality" (0, 1) has no bearing on that). In compound values it is 
vitally important to know where the field delimiters are (';') vs normal 
text occurrences of the field delimiter ('\;').

So there is a real problem here. For example, if in the future I define a 
new property FOO that uses a ;-delimited compound value, then I want to be 
sure that clients not aware of this property will properly "round-trip" it. 
So if I send such a client this:

FOO:delimited;text\;string

I want to be sure that when it parses and re-generates the vcard, the exact 
same value (octet-by-octet) comes back. In that case the client must not 
unescape - if it did then it would generate:

FOO:delimited;text;string

alternatively, if the client left the \; as-is and then re-generated it 
could end up with:

FOO:delimited;text\\;string

and that would be wrong! Now a smart client might spot the use of \; in the 
original value and somehow "mark" that as being compound and then apply 
compound text generation rules when re-generating (in which case it would 
hopefully generate the correct result). However, I am not sure we can rely 
on that - if we want to then we need text clearly explaining what has to 
happen. Failing that, there is no way to define a new property that uses a 
compound TEXT value and ensure that it is backwards compatible (without 
resorting to using something other than ; and , as delimiters).

Given all of that, I think we need text stating that new properties 
(registered, X-, vendor etc) MUST NOT use COMMA or SEMI-COLON text 
delimiting as existing clients may not roundtrip them. If a "structured" 
value is required, then a different delimiter has to be used. We may want 
to pick a specific character for that for IANA registered properties if we 
care.

> - This convention has been used to encode usual texts, not in vCard but
> in the other text handlers (from C-language), I kind of thought it might
> be better to add this

Please see my comments to your proposal #4.

Also, it needs to be absolutely clear, that if we adopt this, that a TAB 
character used for line-folding MUST NOT be encoded. i.e. this is 
definitely not allowed:

DESCRIPTION:Long line of text that is folded
\tright here

> \U00NNNNNN <-> (a Unicode character with charcode 0xNNNNNN, where
> 0xNNNNNN SHOULD be more than 0x10000)
>
>

I will again re-iterate my opposition to this. I think it adds too much 
extra complexity to parsing. We have support for utf-8 - let's leave it at 
that. I worry if we allow this, someone else will turn around and ask us to 
support &amp; style entities too!

> (Readers SHOULD NOT understand \a as an alert (like C-language requires)
> but just a character 'a').

Doing this would be at odds with your proposals #2 and #3. If we had 
allowed clients to do this in the past, then a client could have 
legitimately generated "\t" and expected that to be mapped to "t" (same for 
"\u", "x", "\U"). That would prevent us for extending the \ escape 
characters to have \t -> TAB. So, since I think we should leave our options 
open for extending \ escaping in the future, I believe we should not 
legitimize use of \ escaping for characters the spec does not explicitly 
allow.

The spec is very clear right now:

"In all other cases, escaping MUST NOT be used."

I think that meets Julian point of being clear on what is allowed and what 
is not.

-- 
Cyrus Daboo

---

From: Simon Perreault <[email-masked]>
Date: Tue, 13 Jul 2010 11:36:53 -0400

On 2010-07-13 10:59, Cyrus Daboo wrote:
> So there is a real problem here. For example, if in the future I define
> a new property FOO that uses a ;-delimited compound value, then I want
> to be sure that clients not aware of this property will properly
> "round-trip" it. So if I send such a client this:
> 

Right, there is a problem with the current text.

Assume we specify that semicolons that are not component separators MUST
be escaped. Then a client can correctly interpret this unknown property
as having two components. And it will always output it exactly the way
it received it. That fixes the problem, right?

If so, this would point toward the acceptance of proposal 1.

Simon

---

From: Florian Zeitz <[email-masked]>
Date: Tue, 13 Jul 2010 19:25:54 +0200

> If so, this would point toward the acceptance of proposal 1.
While I'm not strictly against this I wonder if this is really needed.
If we are working under the assumption that the client doesn't
understand the FOO property, can't we just say that it MUST NOT parse it
at all and replicate it exactly as received? (e.g. by storing it with a
verbatim-flag)
I don't see the point of knowing that something I don't understand has
two components...

--
Florian Zeitz

---

From: Mike Douglass <[email-masked]>
Date: Tue, 13 Jul 2010 14:59:38 -0400

While implementing calendar client support using carddav for attendee 
lookup we came across the inconsistencies in vcard KIND and attendee 
CUTYPE which I believe ought to be addressed.

KIND has

      KIND-param = "VALUE=text" / any-param
      KIND-value = "individual" / "group" / "org" / "location" / "thing"

CUTYPE has

        cutypeparam        = "CUTYPE" "="
                           ("INDIVIDUAL"   ; An individual
                          / "GROUP"        ; A group of individuals
                          / "RESOURCE"     ; A physical resource
                          / "ROOM"         ; A room resource
                          / "UNKNOWN"      ; Otherwise not known
                          / x-name         ; Experimental type
                          / iana-token)    ; Other IANA-registered
                                           ; type
        ; Default is INDIVIDUAL

So INDIVIDUAL and GROUP align OK
CUTYPE.RESOURCE = KIND.thing?

No CUTYPE equivalent for KIND.ORG
KIND.LOCATION is better than CUTYPE.ROOM (an event might take place on a 
sports field)

-- 

Mike Douglass                           [email-masked]
Senior Systems Programmer
Communication&  Collaboration Technologies      518 276 6780(voice) 2809
(fax)
Rensselaer Polytechnic Institute 110 8th Street, Troy, NY 12180

---

From: Daisuke Miyakawa <[email-masked]>
Date: Wed, 14 Jul 2010 12:43:17 +0900

As for the proposal 2, 3, 4, I'll refrain my opinion right now: there's no
strong opposition toward what you guys suggest =)

I'd like to focus on proposal 1: escaping semicolon, a field delimeter
2010/7/13 Cyrus Daboo <[email-masked]>
>
> Hi Daisuke,
>
> --On July 13, 2010 10:35:32 PM +0900 Daisuke Miyakawa <
new property FOO that uses a ;-delimited compound value, then I want to be
sure that clients not aware of this property will properly "round-trip" it.
So if I send such a client this:
> I want to be sure that when it parses and re-generates the vcard, the
exact same value (octet-by-octet) comes back. In that case the client must
could end up with:
> and that would be wrong! Now a smart client might spot the use of \; in
the original value and somehow "mark" that as being compound and then apply

That makes sense.

I think adding a rule migt be helpful:
- vCard readers MUST handle "\;" as a character ';' even when its property
name is unknown.
  That is, FOO, X-EXTRA-PROPERTY, or any other property names MUST NOT
affect how property values be treated.

>From the view of backward compatibility (2.1, 3.0), however, I agree that
this is not sufficient.
I guess a lot of vCard libraries would re-use implementations for vCard 3.0,
which does not have strict rule around the delimiter (;).
If they just use them, what you mention above would happen
(“delimited;text\;string” -> "delimited;text;string" OR
"delimited;text\\;string")

(registered, X-, vendor etc) MUST NOT use COMMA or SEMI-COLON text
delimiting as existing clients may not roundtrip them. If a "structured"
value is required, then a different delimiter has to be used. We may want to
pick a specific character for that for IANA registered properties if we
care.

Feasible enough. If we do so, it would be better to clearly define what
"new" properties are.

We have several properties.

name = "SOURCE" / "NAME" / "KIND" / "FN" / "N" / "NICKNAME"

/ "PHOTO" / "BDAY" / "DDAY" / "BIRTH" / "DEATH"

/ "ANNIVERSARY" / "SEX" / "ADR" / "LABEL" / "TEL" / "EMAIL"

/ "IMPP" / "LANG" / "TZ" / "GEO" / "TITLE" / "ROLE" / "LOGO"

/ "ORG" / "MEMBER" / "RELATED" / "CATEGORIES" / "NOTE"

/ "PRODID" / "REV" / "SOUND" / "UID" / "CLIENTPIDMAP" / "URL"

/ "VERSION" / "CLASS" / "KEY" / "FBURL" / "CALADRURI"

/ "CALURI" / "XML" / iana-token / x-name

I think N, ORG, and ADR must be considered as "old" registered properties,
which are able to use the delimiter.
Then, does it make sense for us to prohibit the use of the delimiter in the
other properties including iana-token and x-name?

Also, we will need to take care of SORT-AS parameter too.

Now we don't have SORT-STRING as a property name, but rev12 has SORT-AS,
which utilized the delimiter (semicolon).
I think we also need to define its behavior

N;SORT-AS="Koura;Osamu":Koura;Osamu;;

Thanks,

-- 
Daisuke Miyakawa (宮川大輔)
[email-masked]