Subject: [Jcardcal] multiple property values vs structured property values.

From: Evert Pot <[email-masked]>
Date: Sun, 26 May 2013 20:11:46 +0200

Hi guys,

I'm currently working on both a jCal and jCard implementation.

With any luck, I'll have this ready by the end of the day, so they could be added to the "Implementation status" section (if this is helpful, let me know).

I just came across the 'structured property values' section. I realize there's been a discussion about this before, but I guess I wasn't really fully aware of the implications of the conclusion.

It bothers me a little bit that there's a difference between how CATEGORIES and properties like ADR are encoded.

These two are encoded as:

CATEGORIES:a,b -> "categories", {}, "text", "a", "b"
ADR:a;b;c;d;e -> "adr", {}, "text", ["a", "b", "c", "d", "e" ]

It has bothered me quite a bit that across the mimedir standards, sometimes a comma is used as a separator, and sometimes a semi-colon, requiring me to pretty much maintain a map of properties and their preferred separator.

I was a bit surprised to see that there's a similar inconsistency in jCard.

jCal, which also uses comma's sometimes (CATEGORIES) and semi-colons at other times (REQUEST-STATUS, GEO) does not have a difference in how these two are encoded. In each case, they are encoded as a multi-valued property.

From a pure implementor's standpoint, it's a bit annoying to have to maintain a whitelist for how values are encoded, because the 'TEXT' value-type is now not enough to figure this out.

But, if this discussion has already been concluded, I also don't wish to interfere with your progress. In which case: my apologies.

Cheers,
Evert

---

From: Simon Perreault <[email-masked]>
Date: Mon, 27 May 2013 10:28:02 +0200

Le 2013-05-26 20:11, Evert Pot a écrit :
> CATEGORIES:a,b -> "categories", {}, "text", "a", "b" ADR:a;b;c;d;e ->
> "adr", {}, "text", ["a", "b", "c", "d", "e" ]
>
> It has bothered me quite a bit that across the mimedir standards,
> sometimes a comma is used as a separator, and sometimes a semi-colon,
> requiring me to pretty much maintain a map of properties and their
> preferred separator.
>

Uh... It's not an inconsistency. vCard 4 defines CATEGORIES as a list of
text values and ADR as a structure where each member is a list of text
values. So you can have commas between semicolons in ADR. I don't see
any inconsistency here.

> jCal, which also uses comma's sometimes (CATEGORIES) and semi-colons
> at other times (REQUEST-STATUS, GEO) does not have a difference in
> how these two are encoded. In each case, they are encoded as a
> multi-valued property.

That would be a problem with jCal as far as I can see.

> From a pure implementor's standpoint, it's a bit annoying to have to
> maintain a whitelist for how values are encoded, because the 'TEXT'
> value-type is now not enough to figure this out.

You wouldn't have to maintain a whitelist if jCal is fixed, right?

Simon

---

From: Evert Pot <[email-masked]>
Date: Mon, 27 May 2013 13:08:20 +0200

> values. So you can have commas between semicolons in ADR. I don't see

Fair enough..

I guess I just see the distinction between the two as unneeded, and 'old cruft' dating all the way back to vCard 2.1, still making it's way into newer specs. Just like the requirement to encode empty parts in ADR and N, there's no true function anymore, it's simply like that because it always has.

Similarly I have a hard time accepting that the distinction between multiple-values and structured values serve a real-world purpose at this point. It's a new format, so there's no BC to consider.

> That would be a problem with jCal as far as I can see.

I can accept that. I noticed that there's overlap in contributors on both drafts, should I report it on the jCal list as well?

> You wouldn't have to maintain a whitelist if jCal is fixed, right?

When I'm decoding, you are right. I can make the assumption that if the first text-value is encoded as an array, I'm dealing with a structured value and use ; as a delimiter. If it's a string, and there's more than one, I can use , as a delimiter.

It's a bit more involved when I'm encoding jCard though. Logically it makes the most sense to represent both structured values, as well as multiple-values as an array, simply because I'd like users to be able to use the API as such:

$vCard->n = ["last", "first"]
$vCard->categories = ["work", "friends"]

I have a list of each vCard property, and what their default value type is (for these cases, it's TEXT for both).

So I will still have to maintain a list for properties that are considered 'structured', or 'multiple values'.

It's not a huge issue, because the way vCard 4.0 is written pretty much makes it a very bad idea to define any custom structured values:

   Some properties (e.g., N and ADR) comprise multiple fields delimited
   by a SEMICOLON character (U+003B).  Therefore, a SEMICOLON in a field
   of such a "compound" property MUST be escaped with a BACKSLASH
   character.  SEMICOLON characters in non-compound properties MAY be
   escaped.  On input, an escaped SEMICOLON character is never a field
   separator.  An unescaped SEMICOLON character may be a field
   separator, depending on the property in which it appears.

The implication of this language is:

- For any custom property, it's impossible to to determine if an unescaped semi-colon was intended as a field-separator, or as a literal.

But that's more of a vCard issue, rather than a jCard one ;)

Cheers, and let me know if I need to take action on the jCal list to bring encoding of TEXT-values in line with jCard.

Evert

---

From: Evert Pot <[email-masked]>
Date: Mon, 27 May 2013 13:57:41 +0200

>> You wouldn't have to maintain a whitelist if jCal is fixed, right?

For your reference, this is how I've implemented json-encoding of text values:

https://github.com/fruux/sabre-vobject/commit/7a7c599537bfe975b354dc8157217281aeea43c2

Cheers,
Evert

---

From: Philipp Kewisch <[email-masked]>
Date: Mon, 27 May 2013 14:40:55 +0200

On 5/27/13 1:08 PM, Evert Pot wrote:
>> Uh... It's not an inconsistency. vCard 4 defines CATEGORIES as a list 
>> of text values and ADR as a structure where each member is a list of 
>> text values. So you can have commas between semicolons in ADR. I 
>> don't see any inconsistency here. 
> Fair enough..
>
I guess it just a different concept. The comma-separated values are 
"multiple values for the property", while the semicolons are a "single 
structured value". In the real world, an ADR only has one value, which 
is structured while there can be multiple categories. If we were to drop 
the the structured values, then it wouldn't be consistent: an ADR 
property would always have multiple values. A simple parser might only 
be interested in the first real value and would have to special case for 
ADR because all values are important.

> I can accept that. I noticed that there's overlap in contributors on both drafts, should I report it on the jCal list as well?
Note that in draft-ietf-jcardcal-jcal-02, both REQUEST-STATUS and GEO 
uses a single array as the value:

        ["request-status", {}, "text",
           [
            "3.7",
            "Invalid Calendar User",
            "ATTENDEE:mailto:[email-masked]"
           ]
        ],

        ["geo", {}, "float", [ 37.386013, -122.082932 ] ]

And categories uses single values. REQUEST-STATUS and especially GEO are 
a bit unfortunate, this goes all the way back to rfc5545. We chose to 
special case REQUEST-STATUS since there is added value and there are 
similarities with jcard structured values. You can use the same parser 
for ADR as for REQUEST-STATUS.

For GEO, the real issue lies in rfc5545. The spec says that a GEO 
property has the type float, but the separator is not a comma (which 
would suggest multiple values) but a semicolon.
* If we chose a single string value for GEO, the parser would have to 
special case because the value is not a number but a string with two 
numbers.
* If we define it as a multi-value property (i.e use COMMA in jCal), the 
parser needs to special case because the original property uses a semicolon.

Therefore we chose the nearest logical step to define GEO as a 
"structured" value during conversion. I don't think I've copied this to 
jCal yet, but the same handling applies as for jCard strucutured 
properties: If the parser doesn't know about GEO, then it should convert 
it as an unknown property and keep the opaque string. If the parser 
knows about GEO, it will handle it as a structured property and make the 
first value an array with the two lat/lng components.

Does this sound general enough? In short I don't see any inconsistencies 
in jCal, given the added text about unknown structured values is added 
also to jCal.
> So I will still have to maintain a list for properties that are considered 'structured', or 'multiple values'.
Yes, I think this is needed anyway since the data itself doesn't hold 
this information. I think this is an implementation detail though. You 
could structure the API to generally assume that setting multiple values 
(comma separated) is possible. If you know it is structured, you instead 
only allow setting one value, which is a structured value. This way you 
have:

$vCard->n = [["last", "first"]]
$vCard->categories = ["work", "friends"]

And in the most complex case, for a structured value that has multiple 
values:

$vCard->adr = [["", "", ["left side", "first house"], ...]]
$vCard->fn = ["Philipp Kewisch"]

> Cheers, and let me know if I need to take action on the jCal list to bring encoding of TEXT-values in line with jCard.
This list will handle both jCard and jCal, so its fine.

---

From: Michael Angstadt <[email-masked]>
Date: Mon, 27 May 2013 12:24:56 -0400

On Mon, May 27, 2013 at 7:08 AM, Evert Pot <[email-masked]> wrote:
> For any custom property, it's impossible to to determine if an unescaped semi-colon was intended as a field-separator, or as a literal.

I think it says somewhere in the specs that the values of custom
properties (called "extended properties") are to be treated as plain
text values, unless the client knows how to parse them.  But I don't
see how this is an issue?  Of *course* it's impossible to know how to
parse an extended property's value.  They are, by definition, not part
of the standard!

On Mon, May 27, 2013 at 8:40 AM, Philipp Kewisch <[email-masked]> wrote:
> The comma-separated values are "multiple values for the property", while the
> semicolons are a "single structured value".

That's a good way of putting it.

On Mon, May 27, 2013 at 8:40 AM, Philipp Kewisch <[email-masked]> wrote:
> Therefore we chose the nearest logical step to define GEO as a "structured"
> value during conversion.

Just curious: Has the iCal group ever considered using a "geo" URI
(like in vCard 4)?

-Mike

On Mon, May 27, 2013 at 8:40 AM, Philipp Kewisch <[email-masked]> wrote:
> values. So you can have commas between semicolons in ADR. I don't see any
> inconsistency here.
>
> Fair enough..
>
> I guess I just see the distinction between the two as unneeded, and 'old
> cruft' dating all the way back to vCard 2.1, still making it's way into
> newer specs. Just like the requirement to encode empty parts in ADR and N,
> there's no true function anymore, it's simply like that because it always
> has.
>
> Similarly I have a hard time accepting that the distinction between
> multiple-values and structured values serve a real-world purpose at this
> point. It's a new format, so there's no BC to consider.
>
> structured value". In the real world, an ADR only has one value, which is
> structured while there can be multiple categories. If we were to drop the
> the structured values, then it wouldn't be consistent: an ADR property would
> always have multiple values. A simple parser might only be interested in the
> first real value and would have to special case for ADR because all values
> are important.
>
>
>
> I can accept that. I noticed that there's overlap in contributors on both
> drafts, should I report it on the jCal list as well?
>
> Note that in draft-ietf-jcardcal-jcal-02, both REQUEST-STATUS and GEO uses a
> single array as the value:
>
> And categories uses single values. REQUEST-STATUS and especially GEO are a
> bit unfortunate, this goes all the way back to rfc5545. We chose to special
> case REQUEST-STATUS since there is added value and there are similarities
> with jcard structured values. You can use the same parser for ADR as for
> REQUEST-STATUS.
>
> For GEO, the real issue lies in rfc5545. The spec says that a GEO property
> has the type float, but the separator is not a comma (which would suggest
> multiple values) but a semicolon.
> Therefore we chose the nearest logical step to define GEO as a "structured"
> value during conversion. I don't think I've copied this to jCal yet, but the
> same handling applies as for jCard strucutured properties: If the parser
> doesn't know about GEO, then it should convert it as an unknown property and
> keep the opaque string. If the parser knows about GEO, it will handle it as
> a structured property and make the first value an array with the two lat/lng
> components.
>
> Does this sound general enough? In short I don't see any inconsistencies in
> jCal, given the added text about unknown structured values is added also to
> jCal.
>
> When I'm decoding, you are right. I can make the assumption that if the
> first text-value is encoded as an array, I'm dealing with a structured value
> and use ; as a delimiter. If it's a string, and there's more than one, I can
> use , as a delimiter.
>
> It's a bit more involved when I'm encoding jCard though. Logically it makes
> the most sense to represent both structured values, as well as
> multiple-values as an array, simply because I'd like users to be able to use
> the API as such:
>
> Yes, I think this is needed anyway since the data itself doesn't hold this
> information. I think this is an implementation detail though. You could
> structure the API to generally assume that setting multiple values (comma
> separated) is possible. If you know it is structured, you instead only allow
> setting one value, which is a structured value. This way you have:
>
>
> Cheers, and let me know if I need to take action on the jCal list to bring
> encoding of TEXT-values in line with jCard.
>
> _______________________________________________
> jcardcal mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jcardcal
>

---

From: Evert Pot <[email-masked]>
Date: Mon, 27 May 2013 20:02:12 +0200

> parse an extended property's value.  They are, by definition, not part

..this is getting off-topic. Last email about this from me. If anyone would like to debate further, drop me a line.

There's indeed the concept of the 'unknown' value-type[1].
However this only applies to custom properties that do not have an explicit VALUE= parameter defined.
If VALUE=TEXT, that mechanism is not used.

So in the case of custom TEXT properties in vCard 4.0, there's effectively no way to know in advance how to interpret the value. The only proper solution is to give the user access to the raw string, or let the user per-custom property define how it should be interpreted.

Given this property:

X-PROP;VALUE=TEXT:foo;bar

There's two correct ways to re-encode this into json, namely:

[
  "x-prop", {}, 'text', 'foo;bar'
]

and:

[
  "x-prop", {}, 'text', ['foo', 'bar']
]

This is because vCard 4.0 states

   An unescaped SEMICOLON character may be a field
   separator, depending on the property in which it appears.[2]

vCard 3.0, iCalendar 2.0 and jCard don't have this issue.

So my point is, even though a parser may not be aware of a custom property, it should still know how to unambiguously decode it into primitives. Two comparisons:

In json, we always know how to unescape \", \n, or \uXXXX, regardless of the property.
In xml documents, even if your xml parser doesn't know any specific elements, you can still always easily infer the meaning of &amp; or &gt;.

In vCards everything after the : is a bit of a wild west in comparison :)

Evert

[1]: http://tools.ietf.org/html/draft-ietf-jcardcal-jcard-03#section-5
[2]: http://tools.ietf.org/html/rfc6350#section-3.4

---

From: Michael Angstadt <[email-masked]>
Date: Mon, 27 May 2013 17:05:57 -0400

On Mon, May 27, 2013 at 2:02 PM, Evert Pot <[email-masked]> wrote:
> The only proper solution is to give the user access to the raw string, or let the user per-custom property define how it should be interpreted.

That's exactly what I do in my vCard library.

> In vCards everything after the : is a bit of a wild west in comparison :)

True. :)  Although, I think it's safe to say that at least newlines
can be relied upon to be consistently escaped across all properties
(or else, the vCard would break).

-Mike

---

From: Evert Pot <[email-masked]>
Date: Wed, 29 May 2013 14:12:51 +0200

Hi Phillip,

I missed this response in the thread

On May 27, 2013, at 2:40 PM, Philipp Kewisch <[email-masked]> wrote:

[snip]

> Note that in draft-ietf-jcardcal-jcal-02, both REQUEST-STATUS and GEO uses a single array as the value:
> 
>            "ATTENDEE:
> mailto:[email-masked]
> "
>           ]
>        ],
> 
> And categories uses single values. REQUEST-STATUS and especially GEO are a bit unfortunate, this goes all the way back to rfc5545. We chose to special case REQUEST-STATUS since there is added value and there are similarities with jcard structured values. You can use the same parser for ADR as for REQUEST-STATUS.
> 
> For GEO, the real issue lies in rfc5545. The spec says that a GEO property has the type float, but the separator is not a comma (which would suggest multiple values) but a semicolon.
> * If we chose a single string value for GEO, the parser would have to special case because the value is not a number but a string with two numbers.
> * If we define it as a multi-value property (i.e use COMMA in jCal), the parser needs to special case because the original property uses a semicolon.
> 
> Therefore we chose the nearest logical step to define GEO as a "structured" value during conversion. I don't think I've copied this to jCal yet, but the same handling applies as for jCard strucutured properties: If the parser doesn't know about GEO, then it should convert it as an unknown property and keep the opaque string. If the parser knows about GEO, it will handle it as a structured property and make the first value an array with the two lat/lng components.
> 
> Does this sound general enough? In short I don't see any inconsistencies in jCal, given the added text about unknown structured values is added also to jCal.

This makes sense, and I missed the fact that GEO also was converted to a structured value. With that in mind, it sounds indeed to me that things are now fully self-consistent :)

> Yes, I think this is needed anyway since the data itself doesn't hold this information. I think this is an implementation detail though. You could structure the API to generally assume that setting multiple values (comma separated) is possible. If you know it is structured, you instead only allow setting one value, which is a structured value. This way you have:
> 
> And in the most complex case, for a structured value that has multiple values:
> 

This is not a totally optimal API for me, so I will end up maintaining a list of structured/multi-value properties. My intention is to give the library as much 'syntax sugar' as possible.

But at the very least I now know I can maintain a single list of structured vs. multi-valued properties and serialize/deserialize accordingly.

Cheers, and thanks,

Evert