Subject: [VCARDDAV] Questions about text handling in vCard 4.0 (rev 11)

From: Daisuke Miyakawa <[email-masked]>
Date: Mon, 5 Jul 2010 12:49:35 +0900

Hi vCard experts,

Now I'm confused with how to decode characters with backslash in a text
field, like '\t'.

4.1. in vCard 4.0 shows how to encode some characters like line break and
comma, but there's no mention about the following topics developer may
encounter.
- how a receiver accept string with backslash which is not determined in the
spec (e.g. How "\t\h\i\s\ \i\s\ \a\ \t\e\s\t\n" should be decoded if the
receiver happened to see it?)
- how to handle invisible/control characters valid in Unicode (e.g. tab,
full-width space used in Japanese, etc.) It can be held as is, but I think
it might be better to encode it to something like \xXXXX only when it is not
readable enough.
- How to handle ',', '\,', ';', and '\;' in X- properties (e.g.
"X-EXTRA-SORT-STRING:This,is\, a\;test;\, isn't\;it?"). What to do when some
sender wanted to use splitter (',' or ';') in vCard 4.0.

Thanks,

-- 
Daisuke Miyakawa (宮川大輔)
[email-masked]

---

From: Simon Perreault <[email-masked]>
Date: Mon, 05 Jul 2010 08:26:19 -0400

On 2010-07-04 23:49, Daisuke Miyakawa wrote:
> - how a receiver accept string with backslash which is not determined in
> the spec (e.g. How "\t\h\i\s\ \i\s\ \a\ \t\e\s\t\n" should be decoded if
> the receiver happened to see it?)

This means that the sender is buggy and is generating output with
undefined meaning. Whether and how the receiver could try to recover
from this output is also undefined.

> full-width space used in Japanese, etc.) It can be held as is, but I
> think it might be better to encode it to something like \xXXXX only when
> it is not readable enough.

Disagree. All characters must be left as is except for the ones that
must be escaped.

> "X-EXTRA-SORT-STRING:This,is\, a\;test;\, isn't\;it?"). What to do when
> some sender wanted to use splitter (',' or ';') in vCard 4.0.

The rule for comma, backslash, and newline are global and apply
everywhere (see section 3.3). However, the rule for semicolon only
applies to some properties which use the semicolon as separator. It is
up to the people defining a particular X- property to decide whether
they want to use the semicolon as separator.

Simon
-- 
NAT64/DNS64 open-source --> http://ecdysis.viagenie.ca
STUN/TURN server        --> http://numb.viagenie.ca
vCard 4.0               --> http://www.vcarddav.org

---

From: Daisuke Miyakawa <[email-masked]>
Date: Mon, 5 Jul 2010 22:00:32 +0900

2010/7/5 Simon Perreault <[email-masked]>

> from this output is also undefined.

I think "undefined" does not help actual receivers. They will have to cope
with real senders which often have some bugs. Almost all of the bugs are
unexpected, but some can be, like this time. At least, we are able to reduce
the possibility of ambiguity now.

When my developing a receiver for vCard 3.0, I found some really well-known
sender  (please let me make the application's name secret..) encoded texts
wrongly using '\', and I had to decide on how to handle it, as receiver's
behavior was "undefined" while actually it is easily detected.

I don't think just one tiny mistake in sender side should make receivers
confused.

> Disagree. All characters must be left as is except for the ones that

I'd like to know the rationale.

My understanding about the reason why not using QP is because it is harder
to read.
This time, on the other hand, we cannot see the value without encoding ("
\x3000 " is much easier to read than " "). Making the ambiguity around
spaces and control characters visible is feasible enough, I suppose.

> they want to use the semicolon as separator.

Understood.

> _______________________________________________
> VCARDDAV mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/vcarddav
>

-- 
Daisuke Miyakawa (宮川大輔)
[email-masked]

---

From: Simon Perreault <[email-masked]>
Date: Mon, 05 Jul 2010 09:13:03 -0400

On 2010-07-05 09:00, Daisuke Miyakawa wrote:
> I think "undefined" does not help actual receivers. They will have to
> cope with real senders which often have some bugs. Almost all of the
> bugs are unexpected, but some can be, like this time. At least, we are
> able to reduce the possibility of ambiguity now.
> 
> When my developing a receiver for vCard 3.0, I found some really
> well-known sender  (please let me make the application's name secret..)
> encoded texts wrongly using '\', and I had to decide on how to handle
> it, as receiver's behavior was "undefined" while actually it is easily
> detected.
> 

Can you please suggest text for what you have in mind? Probably a
sentence or two to be added at the end of 3.3...?

> This time, on the other hand, we cannot see the value without encoding
> (" \x3000 " is much easier to read than " 　 "). Making the ambiguity
> around spaces and control characters visible is feasible enough, I suppose.

I disagree again. If you are using a given character in a sentence,
whether it is visible or not, it is because you intend the recipient to
read it. Otherwise, the character would not be useful and would not be
present. For example, in this paragraph I used many spaces which are
invisible and I don't think we would gain anything by replacing them
with \x20 in a vCard. We are encoding user-readable text in vCard, not
random bits.

Thanks,
Simon

---

From: Daisuke Miyakawa <[email-masked]>
Date: Tue, 6 Jul 2010 13:53:39 +0900

> sentence or two to be added at the end of 3.3...?

Here's an arbitrary example.

NOTE:Some sender do like this\: yes, a colon is escaped.\N
 \tIn this line, a tab is wrongly escaped.\n;This semicolon is not escaped,
then Parsers would confuse;
 what this semicolon actually means in sender side apart from
specification\; the sender wanted to split
 this note into two while vCard 4.0 does not allow\, or just it is an
exception?

What I actually saw and got confused in vCard 3.0 was "\:".

What I may possibly encounter in the vCard 4.0 will be the combination
of "\;" and ";" in one sentence, while of course it is wrong in our
specification.

According to "Postel's Law", we should "be liberal in what you accept, and
conservative in what you send." This time, however, "how to be liberal in
what we accept" is not defined in the current draft,
while I agree "how to be conservative in what we send" is defined well.

> with \x20 in a vCard. We are encoding user-readable text in vCard, not

Well, sorry for confusing you.
The space is just an example, and I also don't think it is a good idea to
let senders to use \x method toward every characters including Ascii.

The problem here is we cannot estimate potential Unicode which harm actual
readability/edit-ability/xxx-ability caused by receiver/editor side's
limitation. Even readability is part of my concern.

Another example: a final fallback method users will use during reading and
editing vCard would be just opening the file and editing it manually. Then
how can they edit unknown characters like Chinese, Japanese, Korean, V...?

-- 
Daisuke Miyakawa (宮川大輔)
[email-masked]

---

From: Daisuke Miyakawa <[email-masked]>
Date: Tue, 6 Jul 2010 14:41:29 +0900

> What I actually saw and got confused in vCard 3.0 was "\:".

Ah, sorry. After looking at the section 2.4.2 in RFC 2426, COLON MUST be
escaped. My bad :(
Please ignore my saying around colon. But I am still interested in this
topic overall.

-- 
Daisuke Miyakawa (宮川大輔)
[email-masked]

---

From: Simon Perreault <[email-masked]>
Date: Tue, 06 Jul 2010 09:46:12 -0400

> Here's an arbitrary example.

I meant text that we should add to the vCard 4 specification and that
would address your concern...

> The problem here is we cannot estimate potential Unicode which harm
> actual readability/edit-ability/xxx-ability caused by receiver/editor
> side's limitation. Even readability is part of my concern.

What we're talking about here is how Unicode characters get encoded in
vCard. The current method is to encode them as-is. You are suggesting to
use \xNNNN, and your argument is readability. I think this argument is
weak because:

- Unicode characters may actually be *more* readable than \xNNNN notation.
- Do we really care about readability of vCard data that is going to be
displayed exactly the same to the user regardless of the encoding?

> Another example: a final fallback method users will use during reading
> and editing vCard would be just opening the file and editing it
> manually. Then how can they edit unknown characters like Chinese,
> Japanese, Korean, V...?

I don't see a problem. What do you mean by "unknown"? I can just use vi
on any UTF-8 file and edit it without any difficulty. One can do the
same with Microsoft Word or any other editor that understands UTF-8 I
would assume.

Simon

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 06 Jul 2010 10:10:42 -0400

Hi Simon,

> would address your concern...

Actually I do not think we should add any text that legitimizes incorrect 
generation of vCard data. The "be liberal" is a statement for implementors 
to act on, not for specifications themselves to describe.

Yes I too have seen broken escape sequences in vCard data and have adjusted 
our parser to be "liberal" and treat any backslash/character pair that is 
not explicitly defined in vCard as an "escape" for the character. i.e., 
just remove the backslash. That is an implementation decision and not 
something that should go in the spec.

> displayed exactly the same to the user regardless of the encoding?

I am with Simon on this. \xNNNN escaping is not needed.

> same with Microsoft Word or any other editor that understands UTF-8 I

+1

-- 
Cyrus Daboo

---

From: Daisuke Miyakawa <[email-masked]>
Date: Tue, 6 Jul 2010 23:25:36 +0900

> - Unicode characters may actually be *more* readable than \xNNNN notation.

Please see the font issue I described bellow. When no font is available, it
cannot be read at all.
Rather, it may corrupt the way other characters are shown.

> same with Microsoft Word or any other editor that understands UTF-8 I

Hmm, "unknown" was unclear.

If no font for the locale is available, it is "unknown" to the machine.
For example, Unicode 5.2 allows Mahjong tiles in text
http://www.unicode.org/charts/PDF/Unicode-5.1/U51-1F000.pdf

I don't think usual PC is able to render it

You can try the tiles http://0xcc.net/jsescape/

I can see \U0001F002 as just a square (🀂 looks almost same as □)

Unicode 6.0 will support emoji, pictgraphs used in Japan.
[?] can be sent in text format. I don't think usual editors support it at
first.
At that time, it will be displayed with unreadable something.

In my PC, I guess no Hebrew font, while Unicode has it. Fonts for all
characters in Unicode
cannot be expected even in usual PCs.

Unicode has 22bits and potentially it may grow. Relying on the assumption
all usual computers are able to
show them when seeing vCard is a bit difficult.

I agree that \xNNNN (or \xNNNNNNNN, \UNNNNNNNNN) does not solve this issue
completely, but
without taking care of it, I think usability will not improve enough.

One proposal I can do is that composer side are allowed (but not
recommended) to use the format only when they cannot emit the word (like
when users want to edit foreign friends name without an appropriate IME) but
know codepoint for that characters.
Receiver side MUST be able to decode \xNNNN to appropriate Unicode form.
This is a kind of dirty compromise but there's no technical difficulty nor
theoretically insufficiency. I suppose I can implement sender/receiver
easily.

Simon

-- 
Daisuke Miyakawa (宮川大輔)
[email-masked]

---

From: Daisuke Miyakawa <[email-masked]>
Date: Tue, 6 Jul 2010 23:28:14 +0900

One additional note. In Japanese mobile phones, NOTE field would be full of
Emojis, so this is not imaginary something but from actual use cases.

2010/7/6 Daisuke Miyakawa <[email-masked]>

-- 
Daisuke Miyakawa (宮川大輔)
[email-masked]

---

From: Simon Perreault <[email-masked]>
Date: Tue, 06 Jul 2010 10:38:29 -0400

On 2010-07-06 10:25, Daisuke Miyakawa wrote:
> when users want to edit foreign friends name without an appropriate IME)
> but know codepoint for that characters.
> This is a kind of dirty compromise but there's no technical difficulty
> nor theoretically insufficiency. I suppose I can implement
> sender/receiver easily.

I think you are confusing encoding with how a particular implementation
renders the text.

It is perfectly fine for an implementation to render text as \xNNNN when
e.g. a font is unavailable.

It is also perfectly fine for an implementation to take \xNNNN as input.

But what goes on the wire is pure UTF-8. How it gets rendered or how it
is input by a user is completely irrelevant for the vCard data format.

Simon

---

From: Daisuke Miyakawa <[email-masked]>
Date: Tue, 6 Jul 2010 23:42:45 +0900

Hi,

At first, thank you for your follow-ups!

2010/7/6 Cyrus Daboo <[email-masked]>

> Hi Simon,
>
>
> --On July 6, 2010 9:46:12 AM -0400 Simon Perreault <
> [email-masked]> wrote:
>
> not explicitly defined in vCard as an "escape" for the character. i.e., just
> remove the backslash. That is an implementation decision and not something
> that should go in the spec.

We should decide on whether or not the latest spec allows obvious ambiguity
and allows each developer to lose the original meaning, which means (at
least for me) developing regression tests will become harder, because I
cannot determine which kind of result is expected even after carefully
seeing the specification.

When multiple applications received/sent contact list, the meaning would
become ambiguous. I think we should avoid this kind of undefinedness if we
are already aware of it, as my understanding is that specification is for
removing that kind of ambiguity as much as possible at establishing stage.

If we can accept it, I have no further objection.

-- 
Daisuke Miyakawa (宮川大輔)
[email-masked]

---

From: Daisuke Miyakawa <[email-masked]>
Date: Tue, 6 Jul 2010 23:54:08 +0900

2010/7/6 Simon Perreault <[email-masked]>

> is input by a user is completely irrelevant for the vCard data format.

Well, I think we should also take care of actual use cases vCard is treated
by users, not only from the view of purity of UTF-8. I know \xNNNN makes the
text dirtier.

This is my understanding: If we have contact application with 100%
capability, we don't need to take care of \xNNNN topic at all. But my
(short) experience tells me that, really often, I had to look at bare data
with emacs, vi, or notepad.
At that stage, adding \xNNNNNNNN is really practically useful fallback for
reading/editing the data.

Feel free to correct me. I know my saying is really biased toward "how it
can be implemented and be used",
not "how it should be written as a specification".

Thanks!

-- 
Daisuke Miyakawa (宮川大輔)
[email-masked]

---

From: Daisuke Miyakawa <[email-masked]>
Date: Wed, 7 Jul 2010 12:09:12 +0900

Please let me add two actual cases we can refer and I'd like to know your
opinion about

1) Java
http://java.sun.com/javase/technologies/core/basic/intl/faq.jsp

s1=hello there
s2=\u3053\u3093\u306b\u3061\u306f

<http://java.sun.com/javase/technologies/core/basic/intl/faq.jsp>
Java has been aware of Unicode and allows both format, though they don't use
UTF-8 but UTF-16.
I usually don't see \u format since it is not readable, but I sometimes see
it in comments, which are appropriately handled by javadoc and emitted as
human-readable html.

2) ICU4C

I'm not familiar with ICU library at a whole, but I have experience looking
at its collator config.
For example, icu4c/data/coll/ja.txt says...

ja{
    Version{"2.0.41.26"}
    collations{
        standard{
            Sequence{
                " [strength 3 ] [hiraganaQ on ]"
                "&ヽ=ゝ"
...

 "<條<梛<梃<檮<梹<桴<梵<梠<梺<椏<梍<桾<椁<棊<椈<棘<椢<椦<棡<椌<棍<棔<棧<棕<椶<椒<椄<棗<棣<椥<棹<棠<棯<椨"
...
                "&←=￩"
                "&↑=￪"
                "&→=￫"
                "&↓=￬"
                "&│=￨"
                "&■=￭"
                "&○=￮"
                "&、=､"
                "&。=｡"
                "&「=｢"
                "&」=｣"
                "&ん<'\u0020'<'!'<'\"'<'#'<'
...

In zh.txt, I can see '\uE84D'.

Even ICU4C, a reliable library for Unicode developed by Unicode authority,
is allowing this kind of format (though it uses \u or \U, not \x), so I
think it is worth considering vCard's supporting this kind of format.

Thanks,

-- 
Daisuke Miyakawa (宮川大輔)
[email-masked]

---

From: Simon Perreault <[email-masked]>
Date: Wed, 07 Jul 2010 07:50:26 -0400

Group,

I'd like to hear what other people think of this encoding proposal. I
think all the arguments have been made. What do you think of it? So far
I don't think there is consensus either for or against because only
three people voiced their opinion.

Thanks,
Simon

---

From: Barry Leiba <[email-masked]>
Date: Thu, 8 Jul 2010 00:47:27 -0400

> three people voiced their opinion.

I agree with you and Cyrus.

Barry

---

From: Julian Reschke <[email-masked]>
Date: Thu, 08 Jul 2010 07:03:59 +0200

> three people voiced their opinion.

- a single wire format is sufficient; display of non-ASCII characters is 
an orthogonal issue

- on \ escaping: either allow \ in front of any character, or be *very* 
clear that using it when it's not needed makes the vCard invalid (test 
cases? validator service?)

Best regards, Julian