Subject: [Gen-art] Gen-art telechat review of draft-ietf-json-rfc4627bis-09.txt

From: Elwyn Davies <[email-masked]>
Date: Tue, 17 Dec 2013 13:23:55 +0000

I am the assigned Gen-ART reviewer for this draft. For background on
Gen-ART, please see the FAQ at
< http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq>.

Please wait for direction from your document shepherd
or AD before posting a new version of the draft.

Document: draft-ietf-json-rfc4627bis-09.txt
Reviewer: Elwyn Davies
Review Date: 17 December 2013
IETF LC End Date: 25 November 2013
IESG Telechat date: 19 December 2013

Summary:
Almost ready for publication as a standard track RFC.  Several of the 
issues identified in the last call/previous telechat review have been 
addressed but the authors have noted that the document update was agreed
on the basis that it would make minimal changes.  However, at least the 
first of the 'minor issues' called out represents undocumented 
functionality that could result in non-interoperable implemenattions if 
the unordered requirement is not honoured.  The second is an 
inconsistency and the third is a judgment call.

There are a couple of recommendations that I think would assist 
readability and reflect usual practice with such documents. 

Major issues:
None.

Minor issues:
s4: The unordered nature of the entries in an object is not mentioned. 
Presumably it should state that two objects with the same sets of 
name-value pairs ("members") in any order are logically equivalent.

s10: s9 allows there to be parsers that accept extensions.  s10 only 
allows generators to produce strict JSON.  Perhaps one should allow for 
strict generators and extended generators that match with extended parsers?

s12:
Since JSON text may potentially be parsed (effectively compiled) into what 
looks like a piece of executable binary, it is vital that parsers don't 
suffer from buffer overruns etc. This should probably be called out.

Nits/editorial comments:
General: Many RFCs that use ABNF specifications where parts of the ABNF 
are presented in separate sections also provide a section or appendix 
with the whole of the ABNF in one place.  Whilst this provides a very 
minor double maintenance problem, it does aid implementers and 
facilitates checking of the complete grammar to catch mismatches.

s6: To be absolutely definite, assert that numbers are represented in 
base 10.

s13: Might be useful to insert examples with
- an array with elements that are not the same type
- equivalent objects with the members in different orders.
These could also show off escape sequences and maximally compact formats 
(using the equivalent object example to show that the white space is 
irrelevant).

_______________________________________________
Gen-art mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/gen-art

---

From: Jari Arkko <[email-masked]>
Date: Tue, 17 Dec 2013 12:01:30 -0500

Elwyn: Thank you very much for your review. Tim, Pete - any thoughts on Elwyn's comments?

Jari

---

From: Tim Bray <[email-masked]>
Date: Tue, 17 Dec 2013 10:23:03 -0800

I was actually puzzling about where to send them, I guess
draft-ietf...bis.all will do the right thing.

> > name-value pairs ("members") in any order are logically equivalent.

Introduction: “An object is an unordered collection of zero or more
name/value pairs”.  Is it worth calling this out again in section 4?

None of the JSON drafts has ever said a word about equivalence of the
constructs represented in JSON texts, and I’m reluctant to step on that
slippery slope, since we have excellent interoperability in the field.

> > strict generators and extended generators that match with extended

I think we’d have to take this up in the working group and get a sense of
the consensus; it’d be a big discussion. I’m happy with the way it is -
maximally Postel’s law, so to speak.  I think that if anyone wanted to do
either an extension or even a subset of JSON, I’d really want it to have a
distinct new name.  It seems good that the meaning of JSON is quite fixed
and, once again, I’d point to the excellent interoperability we’ve had in
practice.

> > suffer from buffer overruns etc. This should probably be called out.

I don’t really accept the “executable” premise; JSON is usually compiled
into lists and hash-tables and strings and numbers.  There are a lot of
different bugs you could stumble into while parsing JSON, and it’s not
obvious to me that buffer-overrun stands out among them; but maybe I’m
missing something obvious.

> > facilitates checking of the complete grammar to catch mismatches.

I think I’ll toss this one to my chairs & AD.  I chaired another WG where
we did in fact aggregate the formal specification (in that case a RelaxNG
schema, but similar at a high level).  I think my feeling is that JSON is
such a simple language that the value-add may not be that great (and once
again, in practice interoperability has been excellent) - but if the
chairs/AD think this is a good idea, I have no objection.

> > s6: To be absolutely definite, assert that numbers are represented in

If there are no objections from the chairs/AD, I’m fine with this.  We’re
late in the process, what’s the right way to go about it?

> > (using the equivalent object example to show that the white space is

Hah, there was such an example, it was like [ "13", 14 ] and it looked so
weird the WG asked me to fix it up.  But something like [ "Thomas", 3.7,
true ] would make the point.  Once again, I have no objection if our
chairs/AD are fine with it.

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 17 Dec 2013 10:35:30 -0800

> Elwyn: Thank you very much for your review. Tim, Pete - any thoughts on Elwyn's comments?

Wearing my co-chair hat, I'll take a shot at answering them, given that they have to do with both the WG charter and WG consensus.

>> name-value pairs ("members") in any order are logically equivalent.

This is not a "minor issue". Our charter says that the WG "will keep changes to a minimum", and we have discussed almost every issue in this light. While some people would agree with Elwyn that entries in an object are unordered, many would disagree because JSON can be parsed by streaming parsers. There were multiple threads on the mailing list about this topic. The wording in the current draft comes after multiple consensus calls on the topic. Adding a requirement that we now declare the entries unordered goes against the charter and the consensus of the WG.

>> allows generators to produce strict JSON.  

Both of these are carried over from RFC 4627 unchanged.

>> Perhaps one should allow for 

This would be a change from RFC 4627 that the WG didn't even consider, given our charter requirement.

>> suffer from buffer overruns etc. This should probably be called out.

In keeping with trying to keep changes to a minimum, the WG chose not to list ever possible thing that parsers could do wrong that could be perceived as security considerations. If we add this one, there would be a tendency to add many other, hopefully obvious items such as "don't allow numbers to have exponents that are too large" and so on.

>> facilitates checking of the complete grammar to catch mismatches.

This has not been listed as a problem since RFC 4627 was published.

>> s6: To be absolutely definite, assert that numbers are represented in 

To date, no implementer has said that they were fooled by the lack of representation.

>> (using the equivalent object example to show that the white space is 

For a new format, additional examples might be useful. This hasn't been perceived as useful by any other reviewer so far.

--Paul Hoffman

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 17 Dec 2013 20:45:57 +0000

Mainly a response on the 'unordered' issue.  I'll leave the other ones 
as things for
consideration between AD and the WG.  See below:

Regards,
Elwyn

> This is not a "minor issue". Our charter says that the WG "will keep changes to a minimum", and we have discussed almost every issue in this light. While some people would agree with Elwyn that ent…
In that case you have an inconsistency, since the Introduction says that 
the entries
*are* unordered.  I can't immediately see why using a streaming parser 
interacts
with the unordered issue but I am willing to be enlightened.  I was 
assuming that
the effect should be that the internal representation generated after 
parsing should be the same whatever order the entries come in.

/E

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 17 Dec 2013 13:10:10 -0800

On Dec 17, 2013, at 12:45 PM, Elwyn Davies <[email-masked]> wrote:

> In that case you have an inconsistency, since the Introduction says that the entries
> *are* unordered.

They *are* unordered. However, your message said that the document should state a specific definition of that: "two objects with the same sets of name-value pairs ("members") in any order are logically equivalent". We could not get consensus on a  specific definition of "unordered".

>  I can't immediately see why using a streaming parser interacts
> with the unordered issue but I am willing to be enlightened.  

A streaming parser doesn't keep track of what name-value pairs it has already seen. They allow objects to have name-value pairs where the name is the same but the value is different. Lots of people thought this was a terrible idea, and many thought it was just fine. The wording in the document reflects the WG consensus on this difference of view.

> I was assuming that
> the effect should be that the internal representation generated after parsing should be the same whatever order the entries come in.

That is a reasonable assumption, but it turns out there are many different reasonable assumptions.

However, I can't tell if you consider this a "minor issue" or you really want to force the WG to come up with a consensus definition for "unordered". If the former, great; if the latter, I would want to hear from the IESG what they consider we did wrong with the earlier consensus process and what process we would need to resolve this.

--Paul Hoffman

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 17 Dec 2013 21:31:37 +0000

Hi, Paul.

> However, I can't tell if you consider this a "minor issue" or you really want to force the WG to come up with a consensus definition for "unordered". If the former, great; if the latter, I would wa…
Just to be quite clear, I can't and don't want to force the WG to do 
anything - that's not how gen-art works!  The object of the exercise is 
to help Jari do his job and in the process suggest things that might 
make the doc easier for non specialists in the subject.

However, it seems to me that the inability to come up with a view as to 
what unordered actually means could well lead to interoperability 
issues. Whether this is significant is clearly not my call, but we 
should also be clear that the issue exists.

/E

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 17 Dec 2013 14:59:12 -0800

On Dec 17, 2013, at 1:31 PM, Elwyn Davies <[email-masked]> wrote:

> However, it seems to me that the inability to come up with a view as to what unordered actually means could well lead to interoperability issues. Whether this is significant is clearly not my call, but we should also be clear that the issue exists.

Do you believe that those interoperability issues are not covered by the text in Section 4?

   An object whose names are all unique is interoperable in the sense
   that all software implementations which receive that object will
   agree on the name-value mappings.  When the names within an object
   are not unique, the behavior of software that receives such an object
   is unpredictable.  Many implementations report the last name/value
   pair only; other implementations report an error or fail to parse the
   object; other implementations report all of the name/value pairs,
   including duplicates.

--Paul Hoffman

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 17 Dec 2013 23:58:30 +0000

>     object; other implementations report all of the name/value pairs,
It may be that we are counting the traditional angels, but yes. This 
doesn't say
anything about what order, or lack of order, the entries with different 
names  should occur. Having several of the same name as long as they 
appear together would conform to an ordered set.

Would it cause problems to add "irrespective of the order that the 
name/value pairs appear in the JSON string" to the end of the first 
sentence in the above?

/Elwyn

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 17 Dec 2013 16:13:48 -0800

On Dec 17, 2013, at 3:58 PM, Elwyn Davies <[email-masked]> wrote:

> Would it cause problems to add "irrespective of the order that the name/value pairs appear in the JSON string" to the end of the first sentence in the above?

Define "problems". Seriously. As I said earlier, we took two runs at getting better consensus in this area, and failed. Is it a "problem" for us to take another month trying to do so? If that's what we need to do, we can do it. I propose that the likelihood of this is either low (because people are just as dug in as they have been for months) or high (because people are now exhausted with the topic).

I'm still confused about why you keep asking. You have said that this is "minor", and that we shouldn't be forced into anything, but you keep coming back on this topic. Can you be a bit clearer on why this is so important to you, given that the document already has a paragraph about the interoperability issues? If we're completely missing something, we should work on it; but I'm hearing very mixed messages from you about the topic.

--Paul Hoffman

---

From: Elwyn Davies <[email-masked]>
Date: Wed, 18 Dec 2013 00:52:32 +0000

On Tue, 2013-12-17 at 16:13 -0800, Paul Hoffman wrote:
> Define "problems". Seriously. As I said earlier, we took two runs at
> getting better consensus in this area, and failed. Is it a "problem"
> for us to take another month trying to do so? If that's what we need
> to do, we can do it. I propose that the likelihood of this is either
> low (because people are just as dug in as they have been for months)
> or high (because people are now exhausted with the topic).
> 
> I'm still confused about why you keep asking. You have said that this
> is "minor", and that we shouldn't be forced into anything, but you
> keep coming back on this topic. Can you be a bit clearer on why this
> is so important to you, given that the document already has a
> paragraph about the interoperability issues? If we're completely
> missing something, we should work on it; but I'm hearing very mixed
> messages from you about the topic.
> 
> --Paul Hoffman

I am coming at it from the perspective of a user.  When I was first
doing things with JSON, I could not find anything that told me either
that array elements could be anything or that object order didn't
mattter.  This wasted time for me and affected the code I was writing.

There are quite a lot of potential users.

/Elwyn

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 17 Dec 2013 17:04:15 -0800

On Dec 17, 2013, at 4:52 PM, Elwyn Davies <[email-masked]> wrote:

> mattter.  

You noted earlier the issue with "array elements could be anything", and based on that, we added words to that effect. We cannot say "object order didn't matter" because it does for some parsers. 

> This wasted time for me and affected the code I was writing.

Correct. If RFC 4627 had been written more carefully, this would not be a problem. It wasn't written carefully in a number of areas that affect interoperability, as we note (as politely as possible) throughout the document at hand.

It sounds like you are chafing against the charter, which prohibits us from making significant changes that will break implementations that made different guesses about what the sloppy parts of RFC 4627 meant. Jari can decide if that is significant enough to send the document back to the WG for this (and at least three other areas) where we were forced to come up with "if you do X you will most likely be interoperable" wording, and have us make breaking changes to the new document. I hope that doesn't happen.

> There are quite a lot of potential users.

Certainly. And that is the motivator for the wording that we added, instead of just moving RFC 4627 to standards track with no warning about the sloppiness.

--Paul Hoffman

---

From: Jari Arkko <[email-masked]>
Date: Thu, 19 Dec 2013 16:02:02 +0200

Elwyn: thank you for your careful review (as always!) of specifications. And thank you Tim and Paul considering the input. 

But maybe a little bit of explanation about the role of Gen-ART review is in order. The IETF produces very high-quality specifications. The way that we achieve that is through multiple mechanisms, from having the expert people to careful WG processes and broad review. The Gen-ART reviews are exactly that, reviews. I use them as a way to decide if there are serious enough problems in the document that it warrants me to read it in more detail and/or possibly raise an issue in the IESG review. But they are also reviews, and have components (e.g. editorial suggestions) that document authors and working groups may consider and take into account. And often do. And the reviewers, being responsible, usually are willing to clarify and discuss their comments; it is not that they necessarily are personally after anything, but they are providing input to you. It is a valuable service to us all. But the WG is still in charge of the operations, or if, there's a serious enough problem, then I will raise a Discuss.

In this case, I do agree with the first editorial suggestion of being explicit about the base (even if it is clearer in the ABNF). But it is your job to determine if you think the change is worthwhile and correct.

Then there are issues that we might raise that are truly problematic, and some that are potentially problematic. I think the ordering issue could be seen in the latter category (and I have a similar user perspective as Elwyn). But if the WG/chair has a good explanation of the situation (e.g., we've discussed this and the current text is the compromise that was reached), that is often fine. I think so too in this case. Then again I actually read Elwyn's comment in a slightly different light. Calling for clarity on whether there's an agreement on the ordered/unordered nature of the set. Obviously, bits are in a particular order on the wire, but the argument you WG has had is probably on the semantics of what that means; and this is a software issue. Given two same but differently ordered inputs, does the software produce "equivalent" object. Obviously there are multiple ways of implementing this, and I fully understand why the question is non-trivial for the WG.

Personally, I would note the lack of universal agreement on this on the document. Maybe worthwhile some discussion on the upcoming IESG telechat… but at the same time I at least am not interested in pushing the WG to *solve* the agreement or even to entertain lengthy debates. Running code trumps other concerns in this case, IMO. 

Jari

---

From: Tim Bray <[email-masked]>
Date: Thu, 19 Dec 2013 08:08:35 -0800

We could a one-sentence paragraph to the end of 4: “Implementations also
differ as to whether or not they make the order of object members available
to receiving software.”

On Thu, Dec 19, 2013 at 6:02 AM, Jari Arkko <[email-masked]> wrote:

> Elwyn: thank you for your careful review (as always!) of specifications.
> And thank you Tim and Paul considering the input.
>
> But maybe a little bit of explanation about the role of Gen-ART review is
> in order. The IETF produces very high-quality specifications. The way that
> we achieve that is through multiple mechanisms, from having the expert
> people to careful WG processes and broad review. The Gen-ART reviews are
> exactly that, reviews. I use them as a way to decide if there are serious
> enough problems in the document that it warrants me to read it in more
> detail and/or possibly raise an issue in the IESG review. But they are also
> reviews, and have components (e.g. editorial suggestions) that document
> authors and working groups may consider and take into account. And often
> do. And the reviewers, being responsible, usually are willing to clarify
> and discuss their comments; it is not that they necessarily are personally
> after anything, but they are providing input to you. It is a valuable
> service to us all. But the WG is still in charge of the operations, or if,
> there's a serious enough problem, then I will raise a Discuss.
>
> In this case, I do agree with the first editorial suggestion of being
> explicit about the base (even if it is clearer in the ABNF). But it is your
> job to determine if you think the change is worthwhile and correct.
>
> Then there are issues that we might raise that are truly problematic, and
> some that are potentially problematic. I think the ordering issue could be
> seen in the latter category (and I have a similar user perspective as
> Elwyn). But if the WG/chair has a good explanation of the situation (e.g.,
> we've discussed this and the current text is the compromise that was
> reached), that is often fine. I think so too in this case. Then again I
> actually read Elwyn's comment in a slightly different light. Calling for
> clarity on whether there's an agreement on the ordered/unordered nature of
> the set. Obviously, bits are in a particular order on the wire, but the
> argument you WG has had is probably on the semantics of what that means;
> and this is a software issue. Given two same but differently ordered
> inputs, does the software produce "equivalent" object. Obviously there are
> multiple ways of implementing this, and I fully understand why the question
> is non-trivial for the WG.
>
> Personally, I would note the lack of universal agreement on this on the
> document. Maybe worthwhile some discussion on the upcoming IESG telechat…
> but at the same time I at least am not interested in pushing the WG to
> *solve* the agreement or even to entertain lengthy debates. Running code
> trumps other concerns in this case, IMO.
>
> Jari
>
> > You noted earlier the issue with "array elements could be anything", and
> based on that, we added words to that effect. We cannot say "object order
> didn't matter" because it does for some parsers.
> >
> > Correct. If RFC 4627 had been written more carefully, this would not be
> a problem. It wasn't written carefully in a number of areas that affect
> interoperability, as we note (as politely as possible) throughout the
> document at hand.
> >
> > It sounds like you are chafing against the charter, which prohibits us
> from making significant changes that will break implementations that made
> different guesses about what the sloppy parts of RFC 4627 meant. Jari can
> decide if that is significant enough to send the document back to the WG
> for this (and at least three other areas) where we were forced to come up
> with "if you do X you will most likely be interoperable" wording, and have
> us make breaking changes to the new document. I hope that doesn't happen.
> >
> > Certainly. And that is the motivator for the wording that we added,
> instead of just moving RFC 4627 to standards track with no warning about
> the sloppiness.
> >
> > --Paul Hoffman
>
>

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 19 Dec 2013 08:15:49 -0800

On Dec 19, 2013, at 8:08 AM, Tim Bray <[email-masked]> wrote:

> We could a one-sentence paragraph to the end of 4: “Implementations also differ as to whether or not they make the order of object members available to receiving software.”

That seems to be a statement that I think the WG would agree is true and fits into the "tell folks about interoperability issues" category without saying what developers should/SHOULD do.

Jari, would that meet your goal for the DISCUSS?

--Paul Hoffman

---

From: Tim Bray <[email-masked]>
Date: Thu, 19 Dec 2013 08:16:08 -0800

Or to be more in tune with the language in the rest of the spec.  “JSON
parsing library implementations have been observed to differ as to whether
or not they make information as to the ordering of object members
available; therefore, implementations whose behavior does not depend on
object ordering will be interoperable in the sense that they will not be
affected by these differences.”

On Thu, Dec 19, 2013 at 8:08 AM, Tim Bray <[email-masked]> wrote:

>> > You noted earlier the issue with "array elements could be anything",
>> and based on that, we added words to that effect. We cannot say "object
>> order didn't matter" because it does for some parsers.
>> >

---

From: Jari Arkko <[email-masked]>
Date: Thu, 19 Dec 2013 18:17:35 +0200

I would be very happy with this. And again, I only wanted to briefly discuss this, so I don't want to hold the document even if you do not add something. But I like the addition, and it also seems to match my understanding of the real situation. 

Jari

On Dec 19, 2013, at 6:16 PM, Tim Bray <[email-masked]> wrote:

> Or to be more in tune with the language in the rest of the spec.  “JSON parsing library implementations have been observed to differ as to whether or not they make information as to the ordering of object members available; therefore, implementations whose behavior does not depend on object ordering will be interoperable in the sense that they will not be affected by these differences.”
> 
>

---

From: Elwyn Davies <[email-masked]>
Date: Thu, 19 Dec 2013 18:45:04 +0000

Hi.

That seems to be a reasonable outcome... sorry it took a while to get
there and thanks for your forbearance.

Regards,
Elwyn