Subject: [Cbor] 🔔 WGLC on draft-ietf-cbor-array-tags-03

From: Francesca Palombini <[email-masked]>
Date: Wed, 6 Mar 2019 12:53:07 +0000

CBOR WG,

The chairs believe the array-tags document is ready for WGLC: https://tools.ietf.org/html/draft-ietf-cbor-array-tags-03

The WGLC will end by *20th of March*, please make sure to send your comments to the list before then.

Best regards,
Francesca & Barry

On 05/03/2019, 23:58, "CBOR on behalf of Carsten Bormann" <[email-masked] on behalf of [email-masked]> wrote:

    -03 reflects the fact that IANA has made the allocations.
    
    From this author’s point of view, we are ready for WGLC.
    
    Grüße, Carsten

    > On Mar 5, 2019, at 23:55, [email-masked] wrote:
    > 
    > 
    > A New Internet-Draft is available from the on-line Internet-Drafts directories.
    > This draft is a work item of the Concise Binary Object Representation Maintenance and Extensions WG of the IETF.
    > 
    >        Title           : Concise Binary Object Representation (CBOR) Tags for Typed Arrays
    >        Authors         : Johnathan Roatch
    >                          Carsten Bormann
    > 	Filename        : draft-ietf-cbor-array-tags-03.txt
    > 	Pages           : 14
    > 	Date            : 2019-03-05
    > 
    > Abstract:
    >   The Concise Binary Object Representation (CBOR, RFC 7049) is a data
    >   format whose design goals include the possibility of extremely small
    >   code size, fairly small message size, and extensibility without the
    >   need for version negotiation.
    > 
    >   The present document makes use of this extensibility to define a
    >   number of CBOR tags for typed arrays of numeric data, as well as two
    >   additional tags for multi-dimensional and homogeneous arrays.  It is
    >   intended as the reference document for the IANA registration of the
    >   CBOR tags defined.
    > 
    > 
    > The IETF datatracker status page for this draft is:
    > https://datatracker.ietf.org/doc/draft-ietf-cbor-array-tags/
    > 
    > There are also htmlized versions available at:
    > https://tools.ietf.org/html/draft-ietf-cbor-array-tags-03
    > https://datatracker.ietf.org/doc/html/draft-ietf-cbor-array-tags-03
    > 
    > A diff from the previous version is available at:
    > https://www.ietf.org/rfcdiff?url2=draft-ietf-cbor-array-tags-03
    > 
    > 
    > Please note that it may take a couple of minutes from the time of submission
    > until the htmlized version and diff are available at tools.ietf.org.
    > 
    > Internet-Drafts are also available by anonymous FTP at:
    > ftp://ftp.ietf.org/internet-drafts/
    > 
    > _______________________________________________
    > I-D-Announce mailing list
    > [email-masked]
    > https://www.ietf.org/mailman/listinfo/i-d-announce
    > Internet-Draft directories: http://www.ietf.org/shadow.html
    > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
    > 
    
    _______________________________________________
    CBOR mailing list
    [email-masked]
    https://www.ietf.org/mailman/listinfo/cbor

---

From: Jeffrey Yasskin <[email-masked]>
Date: Thu, 7 Mar 2019 17:03:37 -0800

This generally looks good to me, with decreasing confidence as the
document gets farther from the simple typed-array use case in its
title. A couple comments below:

1. The document justifies itself with "performance" improvements, but
I believe the tags also save the space overhead of encoding a type for
each element or nested array.

2. It's not clear on the first reading that
https://tools.ietf.org/html/draft-ietf-cbor-array-tags-03#section-3.1
defines two different tags. I'd appreciate either a subsection for tag
1040 or parallel subsections for the two tags.

3. The definition of tag 40 ends with "Data in the Typed Array byte
string …", but the second sub-array doesn't need to be a Typed Array.

4. The definition of tag 1040 starts with "Note" making it look
non-normative. I'd rather have that paragraph define the meaning of
the data item, like "The data item is interpreted the same as in tag
40 except that the second sub-array holds elements with the first
dimension contiguous (column-major order)."

5. The reference to Section 7.1 of [TypedArrayUpdate] is out of date.

6. It's not clear to me that representing the source of the elements
in a uint8 array is worth the confusion of re-using the
is-little-endian bit. What's a use case? It exists in Javascript to
define how assignment works, but one does not assign to CBOR data.

7. What's the use case for the homogenous array tag? Tags, in general,
are mostly useful for generic parsers: do we have an example of one of
those that's going to add a new set of homogenous array types for its
consumers to have to check for?

Jeffrey

---

From: Laurence Lundblade <[email-masked]>
Date: Thu, 7 Mar 2019 19:15:58 -0800

Comments in my usual vein…

Because of the big/little endian option, it seems like this needs a discussion about design protocols with it, about preferred encoding / decoding and about interop.

Probably it should say decoders should support both big and little and the encoder can do what is natural. It seems like there might be an option for both sides to support only one endianness, which will likely often be little endian because it is common.

LL

> On Mar 6, 2019, at 4:53 AM, Francesca Palombini <[email-masked]> wrote:
> 
> CBOR WG,
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 8 Mar 2019 08:56:01 +0100

> each element or nested array.

Potentially.  (Depending on how many elements of the array are small enough to benefit from even shorter encoding.)  But that is indeed a motivation.

For this reader, space savings are part of “performance”.
Do we have to be more explicit about that?

> 5. The reference to Section 7.1 of [TypedArrayUpdate] is out of date.

I believe these are all good opportunities for improvements.  Will make issues.

> define how assignment works, but one does not assign to CBOR data.

It is not really where they came from, but how they are meant to be processed further.
So a JavaScript implementation would deserialize into a clamped array.

> consumers to have to check for?

A generic decoder might lay out a more compact array of integers instead of an array of CBOR data items when it sees this tag (and the first element is an integer).
I’m not sure there is an implementation out already that does this.

I think this becomes relevant at the encoder API only if there are natural native types being fed in that are homogeneous (and the application gives license to use the tag 41, but the typed array tags do not apply).

(Meta-comment: Maybe 40, 1040, and 41 should have gone into a separate document.
But it did feel expeditious to include them here.)

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Fri, 8 Mar 2019 07:43:31 -0800

In addition to what I said before, a deterministic encoding should be specified (formerly known as canonical) in the specification and that should probably be network / big endian.

To go on a bit more about detailed implementation...

In a native implementation there is also memory access alignment issue on the decode side, but not the encode side. Memory alignment is an issue on some Arm CPUs, though not on Intel CPUs as far as I know. So on some Arm CPUs, it won’t be possible to just return pointers to the bytes that came off the wire. Because non-string and non-aggregate CBOR data items can be 1, 2, 3, 5 and 9 bytes long, there can be no guarantee of alignment of the incoming encoded CBOR bytes. 
A portable decoder implementation will have to allocate aligned memory for the array and copy it. The decoder can byte-swap while it is copying so it lines up OK with the decoder having to do most of the work to achieve interop.

There’s no alignment issue on the encode side.

The only other thing I’ve encountered in CBOR that needs copying of data is indefinite length strings. 

LL

> On Mar 7, 2019, at 7:15 PM, Laurence Lundblade <[email-masked]> wrote:
>

---

From: Jim Schaad <[email-masked]>
Date: Sat, 9 Mar 2019 11:18:06 -0800

Here is my WGLC review of the document.

1.  In section 2.1 I had a brief moment of frisson on the use of the term binary16 rather than float16 in Table 1.  There is no need to change it, I just was momentarily jolted.

2.  In section 2.1 - Slightly odd to say that ll is lsb as it is two bits long.  No need to change it, but the RSE may notice it.

3.  In section 2.1 - suggest s/Only the big endian variant is used./Only the big endian variant is assigned a tag./

4. In section 2.1 - There should be a note about the fate of the tags that would be <float, signed>.  Are they reserved or open for assignment.

5. In section 2.1 - After reading the last sentence, I am no longer sure what the fate of the 8-bit little endian tags are.  It would appear that they are being assigned from this paragraph.

6. In section 3 - It would make thing clearer if the description here noted that Homogeneous Arrays are one-dimensional array.

7. There should be a security consideration in terms of the Homogeneous type in terms of a naïve implementation.  The description contains the text "The element type  of the array is thus determined by the application type of the first array element"  Since it is not clear that this choosing of the application type does or does not still require that each element of the array be correctly type checked while reading the output is done, this requirement needs to be highlighted in the event that an element which does not following the homogeneous type follows.  Additionally, it might need to be noted that this does not imply that the same elements are going to be present in each element in the array.

Jim

---

From: Jim Schaad <[email-masked]>
Date: Sat, 9 Mar 2019 11:39:07 -0800

> -----Original Message-----
> From: CBOR <[email-masked]> On Behalf Of Laurence Lundblade
> Sent: Friday, March 8, 2019 7:44 AM
> To: Francesca Palombini <[email-masked]>
> Cc: [email-masked]; [email-masked]
> Subject: Re: [Cbor] 🔔 WGLC on draft-ietf-cbor-array-tags-03
> 
> In addition to what I said before, a deterministic encoding should be
> specified (formerly known as canonical) in the specification and that should
> probably be network / big endian.

I don't know that I agree with this.  Most of the machines that I am working with today are little endian machines, from that point of view if such a thing is done then little endian makes more sense.  However the fact that these are different tags implies that there is no need to have a deterministic encoding for this.  The encoding is going to be deterministic based on which endianness is chosen and that is machine or program specific.  I would expect that many implementations would not bother with the big endian encode and might not do the decode as they would be very rarely found.

> In a native implementation there is also memory access alignment issue on
> the decode side, but not the encode side. Memory alignment is an issue on
> some Arm CPUs, though not on Intel CPUs as far as I know. So on some Arm
> CPUs, it won’t be possible to just return pointers to the bytes that came off
> the wire. Because non-string and non-aggregate CBOR data items can be 1, 2,
> 3, 5 and 9 bytes long, there can be no guarantee of alignment of the incoming
> encoded CBOR bytes.
> A portable decoder implementation will have to allocate aligned memory for
> the array and copy it. The decoder can byte-swap while it is copying so it lines
> up OK with the decoder having to do most of the work to achieve interop.

I am confused here about what you are saying here.  I think you mean that returning a pointer to data block in the encoded structure may be a problem if it is not aligned, but if that is what you are saying I really don't understand that from the above text.

Jim

> indefinite length strings.

That depends on the internal structure of binary data as well.  It may also need to be copied to get correct alignment.

> > Because of the big/little endian option, it seems like this needs a discussion
> about design protocols with it, about preferred encoding / decoding and
> about interop.
> >
> > Probably it should say decoders should support both big and little and the
> encoder can do what is natural. It seems like there might be an option for
> both sides to support only one endianness, which will likely often be little
> endian because it is common.
> >
> > LL
> >
> >
> >> On Mar 6, 2019, at 4:53 AM, Francesca Palombini
> >> The chairs believe the array-tags document is ready for WGLC:
> >> On 05/03/2019, 23:58, "CBOR on behalf of Carsten Bormann" <cbor-
> [email-masked] on behalf of [email-masked]> wrote:
> >>
> >>> This draft is a work item of the Concise Binary Object Representation
> Maintenance and Extensions WG of the IETF.
> >>>
> >>> Please note that it may take a couple of minutes from the time of
> >>> submission until the htmlized version and diff are available at
> tools.ietf.org.
> >>>
> >>> Internet-Draft directories: http://www.ietf.org/shadow.html or
> >>> ftp://ftp.ietf.org/ietf/1shadow-sites.txt
> >>>
> >>

---

From: Laurence Lundblade <[email-masked]>
Date: Sat, 9 Mar 2019 12:45:59 -0800

Hi Jim,

> On Mar 9, 2019, at 11:39 AM, Jim Schaad <[email-masked]> wrote:
> 
> 
> 

My understanding of deterministic encoding is something like this — For a given input, say the array [100,200,300,400], two independently written encoders should produce *exactly* the same encoded bits. A general test vector for this array would work correctly for both an encoder written in the Foo language on a big endian machine and the Bar language on a little endian machine.

There is also the COSE Sig_structure and similar use cases. Some encoded CBOR that is not transmitted must be constructed by the sender and receiver and it has to be exactly correct because it is being hashed.

If the implementor gets to choose big / little endian as they wish, we don’t get deterministic encoding like that. 

Personally, I don’t think deterministic encoding is that useful and would probably be happy seeing it go away completely. I suggested that in Bangkok, and the response was that some people (who weren’t in the room) really care about it. Here, I’m just pointing out, that if we’re doing deterministic encoding, we should do it for array tags too.

I think interop is the bigger issue here (and deterministic encoding != interop) . If everyone does what is natural for their CPU, we mostly have interop because little endian is the most common, but interop is not guaranteed because big endian CPUs exist. I think the spec should be tighter and should spell out what is to be done by a generic encode/decoder so that interop is guaranteed. Maybe frame it up in three options:

1) Specify rules for preferred encoders and decoders like this: decoders should support both big and little endian. Encoders should do network byte order, but can choose as they wish.  

2) Just outright say tagged arrays are always little endian. Drop the tagging of little vs big.

3) Just outright say tagged arrays are always network big endian. Drop the tagging of little vs big.

If most CPUs were big endian, the obvious choice would be 3). The problem with 2) is the mixed message where some parts of CBOR are big endian and other are little. The problem with 1) is that it is more complicated and implementors are more likely to get it wrong.

Personally, I kind of like 2).

> I am confused here about what you are saying here.  I think you mean that returning a pointer to data block in the encoded structure may be a problem if it is not aligned, but if that is what you a…

Yes, that is what I’m saying. Sorry I wasn’t more clear.

> That depends on the internal structure of binary data as well.  It may also need to be copied to get correct alignment.

For a rich, complete and true generic decoder, the obvious thing to do with array tags is return an array of integers in the native CPU format. If the CPU doesn’t support unaligned access, then that array of integers must be aligned.

I think the contents of a binary string are different. First of all, if it has internal structure, they should have encoded the internals with CBOR in the first place. :-). Second, there is nothing in the CBOR type information that tells a generic CBOR encoder if it needs to be aligned or not. I would think most of the time it does not, so aligning by default would be wasteful.

All that said, I don’t see that alignment has any affect on the text in the draft.  The main issue for me is the interop issue that arrises from the option to transmit either big or little endian.

LL

---

From: Jim Schaad <[email-masked]>
Date: Sat, 9 Mar 2019 14:07:22 -0800

> From: Laurence Lundblade <[email-masked]>
> Sent: Saturday, March 9, 2019 12:46 PM
> To: Jim Schaad <[email-masked]>
> Cc: Francesca Palombini <[email-masked]>;
> [email-masked]; [email-masked]
> >> specified (formerly known as canonical) in the specification and that
> >> should probably be network / big endian.
> >
> > I don't know that I agree with this.  Most of the machines that I am working
> with today are little endian machines, from that point of view if such a thing is
> done then little endian makes more sense.  However the fact that these are
> different tags implies that there is no need to have a deterministic encoding
> for this.  The encoding is going to be deterministic based on which
> endianness is chosen and that is machine or program specific.  I would expect
> that many implementations would not bother with the big endian encode
> and might not do the decode as they would be very rarely found.
> 
> My understanding of deterministic encoding is something like this — For a
> given input, say the array [100,200,300,400], two independently written
> encoders should produce *exactly* the same encoded bits. A general test
> vector for this array would work correctly for both an encoder written in the
> Foo language on a big endian machine and the Bar language on a little endian
> machine.
> 
> There is also the COSE Sig_structure and similar use cases. Some encoded
> CBOR that is not transmitted must be constructed by the sender and receiver
> and it has to be exactly correct because it is being hashed.
> 
> Personally, I don’t think deterministic encoding is that useful and would
> probably be happy seeing it go away completely. I suggested that in Bangkok,
> and the response was that some people (who weren’t in the room) really
> care about it. Here, I’m just pointing out, that if we’re doing deterministic
> encoding, we should do it for array tags too.
> 
> 
> I think interop is the bigger issue here (and deterministic encoding !=
> interop) . If everyone does what is natural for their CPU, we mostly have
> interop because little endian is the most common, but interop is not
> guaranteed because big endian CPUs exist. I think the spec should be tighter
> and should spell out what is to be done by a generic encode/decoder so that
> interop is guaranteed. Maybe frame it up in three options:
> 
> 1) Specify rules for preferred encoders and decoders like this: decoders
> should support both big and little endian. Encoders should do network byte
> order, but can choose as they wish.
> 
> If most CPUs were big endian, the obvious choice would be 3). The problem
> with 2) is the mixed message where some parts of CBOR are big endian and
> other are little. The problem with 1) is that it is more complicated and
> implementors are more likely to get it wrong.
> 

Given that I don't care about deterministic mapping from data model to CBOR encoding, I don't see an issue here.  I will always get the same CBOR bytes given the same binary data. 

> >> In a native implementation there is also memory access alignment
> >> issue on the decode side, but not the encode side. Memory alignment
> >> is an issue on some Arm CPUs, though not on Intel CPUs as far as I
> >> know. So on some Arm CPUs, it won’t be possible to just return
> >> pointers to the bytes that came off the wire. Because non-string and
> >> non-aggregate CBOR data items can be 1, 2, 3, 5 and 9 bytes long,
> >> there can be no guarantee of alignment of the incoming encoded CBOR
> bytes.
> >> A portable decoder implementation will have to allocate aligned
> >> memory for the array and copy it. The decoder can byte-swap while it
> >> is copying so it lines up OK with the decoder having to do most of the
> work to achieve interop.
> >
> > I am confused here about what you are saying here.  I think you mean that
> returning a pointer to data block in the encoded structure may be a problem
> if it is not aligned, but if that is what you are saying I really don't understand
> that from the above text.
> 

I would agree this is a valid issue to highlight.

Jim

> > That depends on the internal structure of binary data as well.  It may also
> need to be copied to get correct alignment.
> 
> For a rich, complete and true generic decoder, the obvious thing to do with
> array tags is return an array of integers in the native CPU format. If the CPU
> doesn’t support unaligned access, then that array of integers must be
> aligned.
> 
> I think the contents of a binary string are different. First of all, if it has internal
> structure, they should have encoded the internals with CBOR in the first
> place. :-). Second, there is nothing in the CBOR type information that tells a
> generic CBOR encoder if it needs to be aligned or not. I would think most of
> the time it does not, so aligning by default would be wasteful.
> 
> 
> All that said, I don’t see that alignment has any affect on the text in the draft.
> The main issue for me is the interop issue that arrises from the option to
> transmit either big or little endian.
> 
> LL
> 
> 
> >
> >
> >>
> >> LL
> >>
> >>
> >>
> >>> On Mar 7, 2019, at 7:15 PM, Laurence Lundblade
> >>> <[email-masked]>
> >> wrote:
> >>>
> >> about design protocols with it, about preferred encoding / decoding
> >> and about interop.
> >>>
> >> encoder can do what is natural. It seems like there might be an
> >> option for both sides to support only one endianness, which will
> >> likely often be little endian because it is common.
> >>>
> >>> LL
> >>>
> >>>
> >>>>> The Concise Binary Object Representation (CBOR, RFC 7049) is a
> >>>>> data format whose design goals include the possibility of
> >>>>> extremely small code size, fairly small message size, and
> >>>>> extensibility without the need for version negotiation.
> >>>>>
> >>>>> number of CBOR tags for typed arrays of numeric data, as well as
> >>>>> two additional tags for multi-dimensional and homogeneous arrays.
> >>>>> It is intended as the reference document for the IANA registration
> >>>>> of the CBOR tags defined.
> >>>>>
> >>>>>

---

From: Jeffrey Yasskin <[email-masked]>
Date: Wed, 13 Mar 2019 11:29:17 -0700

> should probably be network / big endian.

I think it's the job of the protocol to describe how it's encoded
deterministically, rather than this tag. For example, the choice of whether
or not to tag an array at all is a potential point for non-determinism to
enter, so the protocol needs to specify whether there's a tag, and if so,
which one. At that point, it's made the choice of little-vs-big endian, so
there's no more question for this document to answer.

Jeffrey

---

From: Jim Schaad <[email-masked]>
Date: Thu, 14 Mar 2019 18:48:56 -0700

I would say that this is analogous to the question of how date and time are encoded.  There are several different options and we don’t say which of them is deterministic for encoding.  We just say how to encode it once the value and tag are selected.

Jim

From: CBOR <[email-masked]> On Behalf Of Jeffrey Yasskin
Sent: Wednesday, March 13, 2019 11:29 AM
To: Laurence Lundblade <[email-masked]>
Cc: [email-masked]; [email-masked]; Francesca Palombini <[email-masked]>
Subject: Re: [Cbor] 🔔 WGLC on draft-ietf-cbor-array-tags-03

On Fri, Mar 8, 2019 at 7:43 AM Laurence Lundblade <[email-masked] <mailto:[email-masked]> > wrote:

I think it's the job of the protocol to describe how it's encoded deterministically, rather than this tag. For example, the choice of whether or not to tag an array at all is a potential point for non-determinism to enter, so the protocol needs to specify whether there's a tag, and if so, which one. At that point, it's made the choice of little-vs-big endian, so there's no more question for this document to answer.

Jeffrey