Subject: [alto] Modeling JSON based protocols using YANG

From: Xiao SHI <[email-masked]>
Date: Mon, 20 Oct 2014 02:05:14 -0400

Hi folks,

As a few of us were working on modeling the ALTO protocol using YANG, we
were pondering on a more general question: can YANG model all JSON based
protocols? What is the condition for a JSON based protocol (at least the
message format) to have an syntactically equivalent, hence interoperable,
YANG model with JSON encoding? Alternatively, in order to interoperate,
semantic equivalence might be sufficient, is there any condition for
semantic equivalence?

tl;dr: A JSON based protocol carried by HTTP can have syntactically
equivalent YANG model, if and only if all the keys in the key-value pairs
in the JSON message are pre-defined keywords in the protocol.

We've come up with a few ideas, and a few things below are work in
progress, but we would love your feedback!

Thank you!

Xiao

======================

1. Introduction.

JavaScript Object Notation (JSON) has been a popular choice as the message
encoding for many network protocols such as the Application Layer Traffic
Optimization (ALTO) protocol, the Content Delivery Networks Interconnection
(CDNi) protocol, etc.

Meanwhile, there are broad interests in the networking community to use
YANG to define data model so that one can use YANG related tools such as
OpenDayLight controller. Although YANG itself is XML based, there have been
efforts to model JSON content using YANG [draft-ietf-netmod-yang-json-01].

A natural question rises: can YANG model all JSON based protocols? What is
the condition for a JSON based protocol (at least the message format) to
have an syntactically equivalent, hence interoperable, YANG model with JSON
encoding? Alternatively, in order to interoperate, semantic equivalence
might be sufficient, is there any condition for semantic equivalence?

2. Claim

A JSON based protocol carried by HTTP can have syntactically equivalent
YANG model, if and only if:

(1) the message encoding condition is met;

(2) the uri condition is met.

2.1. The message encoding condition

The JSON message encoding MUST not contain a variable as a key in a JSON
object key-value pair. In other words, the keys in the JSON message must be
an already defined constant (or keyword) in the message format of the
protocol.

For example, if I have a protocol where “network-map”, “src”, and “dsts”
are defined keywords, JSON text A meets the condition whereas B does not
because “PID1” and “PID2” are not protocol keywords (they are resource IDs).

A:

{

 “network-map”: [

   {

     “src”: “PID1”,

     “dsts”: [“PID2”, “PID3”]

   },

   {

     “src”: “PID2”,

     “dsts”: [“PID3”, “PID4”]

   }

 ]

}

B:

{

 “network-map”: [

   “PID1: [“PID2”, “PID3”],

   “PID2”: [“PID3”, “PID4”]

 ]

}

>From this we know that since ALTO protocol uses encoding B, there cannot be
a syntactically equivalent YANG model.

2.2 The URI condition

Some of the YANG-related protocols might have URI constraints, e.g.
RESTCONF. For now, we assume either that the JSON-based protocol URI could
be conformed to RESTCONF compliant uri, or that the server could have a
routing mapping between the protocol compliant uri and the RESTCONF
compliant uri, hence this condition would not be an issue, which allows us
to focus on the message encoding condition.

3. Proof

3.1. The message encoding condition is necessary.

We first note that this condition is a necessary condition for a JSON based
protocol to have a syntactically equivalent YANG model by proving its
contrapositive.

If one of the keys in the key-value pair in the JSON document is not
pre-defined, the corresponding XML tags will not be pre-defined keywords.
Therefore, it would not possible to model it in YANG without using
YANG’s anyxml
statement (which allows arbitrary XML content).

However, using the anyxml statement would defeat our purpose of modeling
the data as it allows arbitrary XML content, and will not be helpful in the
subsequent parsing process.

3.2. The message encoding condition is sufficient.

We prove this by providing a translation procedure from a JSON message that
is compliant with the protocol we are trying to model, to a custom java
class that can be used for Jackson data binding, then to a YANG model.

We note that the middle step of translating to and from the custom parser
class is not necessary, but it will be useful.

3.2.1. motivation

JSON data binding is the process of binding data structures (objects,
arrays, etc.) in JSON to the appropriate data structures in the server
(e.g. java classes, database tables, etc.) in parsing the JSON text. In
order to process JSON messages in a meaningful manner, data binding is
necessary. Even if the binding is not explicit, the server would need to do
it eventually. For example, one can read JSON content in a stream without
binding it to the java classes, but eventually in order to make sense of
the data, the server would eventually have to organize it, which is roughly
analogous to data binding upfront. Popular choices for JSON parsing and
data binding include jackson and gson.

We use Jackson full data binding as our approach. Full data binding binds
JSON content into plain old java objects (POJOs), i.e. this custom parser
class can neither extend nor implement any other class. Jackson uses
ObjectMapper with the custom parser class to parse JSON content into this
class.

3.2.2. The message encoding condition

The message encoding condition is that all keys in each key-value pair in
the JSON text must be pre-defined keywords. As the keys will become either
class names and instance variable names, or be keys in the java maps, it is
easy to see that the condition is equivalent to “there exists a full data
binding in Jackson custom parser class without using any map structures
(Map<String, ?>).”

3.2.3. Proof

We provide a recursive binding process from a JSON object to the Jackson
custom parser class to be used by Jackson ObjectMapper.

Type determine_type(value) {

 if (value is of a primitive type, i.e. string, number, boolean, null) {

   return the corresponding java primitive type;

 }

 if (value is a JSON object) {

   return build_parser_class(value).class;

 }

 if (value is an array) {

   return ArrayList<T> where T=determine_type(value[0]);

 }

 // should not reach here.

}

Class build_parser_class(JSONObject obj) {

 create custom class C;

 for each key/value pair in the obj {

   add instance variable v in C;

   the name of variable v <- key; // (__known__ a/c to our assumption)

   the type of variable v <- determine_type(value);

 }

 return C.class;

}

Naming:

--change everything into CamelCase (i.e. remove dashes, etc.)

--for instance variables, use “my” prefix, (e.g. myVariable, myNetworkMap,
etc.)

--for the custom class name, if the object is an element of the array, use
“Element” suffix.

This is just one convention so that the next step proceeds smoothly. As
long as this naming translation is consistent with the naming stage in the
next step, it will work just fine.

Example (a modified ALTO protocol network map example):

JSON object:

{

 “meta”: {

   “resource-id”: “my-default-map”,

   “tag”: “aab875ef69c87d012”

 },

 “network-map”: [

   {

     “src”: “PID1”,

     “dsts”: [“PID1”, “PID2”, “PID3”]

   },

   {

     “src”: “PID2”,

     “dsts”: [“PID1”, “PID3”]

   },

   {

     “src”: “PID3”,

     “dsts”: [“PID2”, “PID3”]

   }

 ]

}

Result of build_parser_class(obj):

Class JSONObject {

 Meta myMeta;

 ArrayList<NetworkMapElement> myNetworkMap;

}

Class Meta {

 String myResourceId;

 String myTag;

}

Class NetworkMapElement {

 String mySrc;

 ArrayList<String> myDsts;

}

Now given the Jackson Parser Java Class, to get a syntactically equivalent
YANG model:

YANGModel build_yang_model(Class C) {

 for each instance variable (Type, Name) {

   if (Type is primitive type: string, number, boolean, null) {

     add the following to the YANG module:

     “leaf Name { type <YANG equivalent of Type>; }”

   }

   if (Type is an ArrayList<TypeElement>) {

     if (TypeElement is primitive type) {

       add the following to the YANG module:

       “leaf-list Name { type <YANG equivalent of TypeElement>; }”

     } else {

       // TypeElement is a custom parser class

       add the following to the YANG module:

       “list Name { <result from build_yang_model<TypeElement.class>> }”

     }

   }

   if (Type is a custom parser class) {

     add the following to the YANG module:

     “container Name { <result from build_yang_model<Type.class>> }”

   }

 }

}

Result from the previous example:

container meta {

 leaf resource-id {

   type string;

 }

 leaf tag {

   type string;

 }

}

list network-map {

 leaf src {

   type string;

 }

 leaf-list dsts {

   type string;

 }

}

This does validate the JSON document with XML-JSON encoding. For your
reference, this is the XML document which validates:

<?xml version="1.0" encoding="UTF-8" ?>

<meta>

 <resource-id>my-default-map</resource-id>

 <tag>aab875ef69c87d012</tag>

</meta>

<network-map>

 <src>PID1</src>

 <dsts>PID1</dsts>

 <dsts>PID2</dsts>

 <dsts>PID3</dsts>

</network-map>

<network-map>

 <src>PID2</src>

 <dsts>PID1</dsts>

 <dsts>PID3</dsts>

</network-map>

<network-map>

 <src>PID3</src>

 <dsts>PID2</dsts>

 <dsts>PID3</dsts>

</network-map>

This proves that the message encoding condition is a sufficient condition
for the JSON object to have a YANG model.

Note the model generated is very crude and lose almost all constraints and
all inheritance features (if any), because it focuses on the syntax and is
essentially converted from an JSON object compliant with a protocol instead
of from the protocol itself. Hence this result is more useful in
determining which JSON based protocols cannot have a syntactically
equivalent YANG model, than in generating a good YANG model.

3.3. Conclusion

Our claim holds. A JSON based protocol carried by HTTP can have
syntactically equivalent YANG model, if and only if all the keys in the
key-value pairs in the JSON message are pre-defined keywords.

4. Semantic equivalence

For JSON based protocols that don’t satisfy the message encoding condition,
it is still possible to have a semantically equivalent YANG model. All that
is required for the protocol compliant clients and the YANG model compliant
server to interoperate is an adapter which does the following:

1) translate FROM YANG server compliant response msg TO alto compliant
response msg

2) translate FROM alto compliant request msg TO YANG server compliant
request msg

4.1. Claim

This adapter needs to be protocol-aware.

Ideally, given any YANG model, we would like to be able to automatically
(or at least mechanically) generate this message adapter, which means not
looking at the protocol or its compliant msgs. However, without knowing the
specific protocol that we are working with (i.e. human intervention, i.e.
looking at the protocol compliant msgs), such an adapter cannot be
auto-generated.

4.2. Proof by Indistinguishability

Suppose both the YANG server compliant msg m_y and the actually protocol
compliant msg m_p are in JSON (or have been encoded into JSON). Looking at
the differences between the two messages, call these differences {d1, d2,
..., dn}. The goal for the auto-generated adapter would be to identify and
eliminate these differences. Construct a new JSON msg m' where all but one
difference di is the same as m_p and di is the same as the m_y. Without
looking at the protocol (or m_p), the auto-generated adapter would not be
able to distinguish between m' and m_p in its translation process, which
means, it won't be able to tell whether it should change di or not. Hence,
such an adapter must be protocol-aware.

A good example is the dependent-vtag in the ALTO protocol:

"dependent-vtag" : [

 {

   "resource-id" : "my-network-map",

   "tag" : "abcd1234"

 }

]

It was specified this way in the alto protocol. However, it could
conceivably be the case that it was originally the following map structure,
and was converted into the above encoding because of the map->list+key
issue. (This case is actually one of the few differences in the m_y and m_p
where the adapter does not need to convert it back to a map structure.)

"dependent-vtag" : {

 "my-network-map" : {

   "tag" : "abcd1234"

 }

}

Without knowing the protocol, there is no way to tell.

5. Ramifications

We now understand the basic condition for a JSON based protocol to have a
YANG Model. For the protocols that don’t meet this condition, there can be
a semantic equivalent YANG model, but there won’t be a generic process of
generating the adapter for all such protocols.

---

From: Xiao SHI <[email-masked]>
Date: Mon, 20 Oct 2014 09:57:18 -0400

Hi Lada,

Thank you so much for your feedback!

The motivation here is to produce a YANG model given a defined protocol,
e.g. ALTO, CDNi, etc. This way, we may incorporate future protocols into
ODL controllers or use other YANG related tools to handle those protocols.
Hence the objective of the modeling is the protocol, not the messages. I am
not suggesting to use YANG as a schema language.

The conditions discussed in this process will be helpful in the following
sense:
1) disprove the possibility to get a syntactically equivalent YANG model
from a JSON based protocol;
2) generate a Jackson JSON parser class and YANG model from the protocols
that meet the criteria;
3) for designers of future protocols who are thinking of using YANG related
tools, set a guideline of what their messages should look like (or MUST not
contain);
4) for the YANG/NETCONF WG, describing the compatibility issues that they
might be interested in tackling for expanding YANG's usage.

For this example, "foo": [ { "bar": 1 }, 2 ] is also not easy to do
data-binding in the parser, because the elements of the array are not of
the same type, which is indeed an assumption that I left out.

For the {"foo”:null} example, can't we model it using type empty? That's
equivalent to the XML element <foo /> right?

Cheers,
Xiao

On Mon, Oct 20, 2014 at 9:23 AM, Ladislav Lhotka <[email-masked]> wrote:

> Xiao SHI <[email-masked]> writes:
>
> > Hi folks,
> >
> >
> Generating a YANG data model from existing instance data seems backwards
> to me, although Examplotron (http://examplotron.org) does something quite
> similar. In any case, it might be quite challenging because YANG isn't
> intended as a general schema language. Your observation above is true
> but it is just one aspect of the problem. For instance
>
> "foo": [ { "bar": 1 }, 2 ]
>
> or
>
> "foo": null
>
> cannot be modelled in YANG using the encoding of
> draft-ietf-netmod-yang-json.
>
> Lada
>
> >
> > _______________________________________________
> > netmod mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/netmod
>
> --
> Ladislav Lhotka, CZ.NIC Labs
> PGP Key ID: E74E8C0C
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 20 Oct 2014 15:23:07 +0200

or

"foo": null

Lada

---

From: Xiao SHI <[email-masked]>
Date: Mon, 20 Oct 2014 11:33:56 -0400

Hi Andy,

I agree with you. Just to clarify, what I am trying to do is to use YANG
data structures for the protocol whose message encoding is JSON. I am not
trying to use JSON as a data structure. Does this make sense?

Best,
Xiao

On Mon, Oct 20, 2014 at 11:29 AM, Andy Bierman <[email-masked]> wrote:

>
>
> On Mon, Oct 20, 2014 at 6:23 AM, Ladislav Lhotka <[email-masked]> wrote:
>
> I think perhaps RESTCONF is overloading JSON and this has led to some
> confusion.
> RESTCONF uses JSON strictly as an encoding format, not a data structure.
> JSON is just
> the message encoding. The corresponding instance document must conform to
> the YANG definitions (not just be valid JSON).
>
> A protocol should pick between YANG data structures and JSON data
> structures,
> and not try to do both.
>
>
>
>
>> Lada
>>
>>
> Andy
>
>
>> >

---

From: Xiao SHI <[email-masked]>
Date: Mon, 20 Oct 2014 22:29:16 -0400

Hi Lada,

I agree with your statement that YANG is designed to model datastore and
the protocol messages.

Right now the question I am trying to answer is this scenario: we have the
protocols (hence the protocol messages) first, and would like to know
whether we would be able to use YANG to model the content messages (in
order to use the YANG tools for this protocol, etc.). And the question is
what are the conditions for JSON msgs to be able to be modeled by YANG.

For JSON arrays:
0. a standalone non-object JSON value is not model-able in YANG:
examples:
--"foo"
--14
--[1,2,3]
which translates to the following in XML:
<0>1</0>
<1>2</1>
<2>3</2>

This is why your example "foo": [ { "bar": 1 }, 2 ] cannot be
modeled--because 2 is a standalone value and {"bar": 1} is an object,
otherwise there are following ways to model an array with a mix of elements:

1. an array with only built-in types or derived types (standalone JSON
values except for arrays and objects) can be modeled in YANG as the
following:
"foo" : [1, "abc", enum1]

leaf-list foo {
  type union {
    type int64;
    type string;
    type enumeration {
      enum enum1;
    }
  }
}

2. an array of all objects
(1) This structure is model-able as well, right?
"foo": [{"a": 1}, {"a": 1, "b": 2}]

list foo {
  leaf a {
    type int;
  }
  leaf b {
    type int;
  }
}

(2) how about this?
"foo": [{"a": 1, "b": 2, "c": 3}, {"c": 1, "b":3, "y": 2}]

If the JSON-XML translate it to the following:
        <foo>
<a>1</a>
<b>2</b>
<c>3</c>
</foo>
<foo>
<c>1</c>
<b>3</b>
<y>2</y>
</foo>
We can still model this using the choice statement, right?

If the above statements about whether these JSON texts are model-able are
correct, essentially the only JSON arrays that are not model-able are (a)
those mixed with objects and a standalone non-object JSON value; and (b)
those whose elements is an array. And this gap is caused by the difference
between leaf-list modeling capabilities of handling arrays with non-object,
non-array, standalone JSON values, and the list+choice modeling
capabilities of modeling arrays with objects. Does this sound plausible?

Cheers,
Xiao

On Mon, Oct 20, 2014 at 11:56 AM, Ladislav Lhotka <[email-masked]> wrote:

> Hmm, YANG was designed for modelling datastores and contents of protocol
> messages, not protocols as such.
>
> >
> <foo/> corresponds to
> "foo": [null]
> Section 6.9 in draft-ietf-netmod-yang-json-01 gives the reasoning,
> essentially it is because some software isn't able to distinguish
>
> "foo": null
>
> from "foo" being absent.
>
> Lada
>
> >
> > Cheers,
> > Xiao
> >
> >> > A natural question rises: can YANG model all JSON based protocols?
> What
> >> is
> >> > object key-value pair. In other words, the keys in the JSON message
> must
> >> be
> >> >>From this we know that since ALTO protocol uses encoding B, there
> cannot
> >> be
> >> > compliant uri, hence this condition would not be an issue, which
> allows
> >> us
> >> > the data as it allows arbitrary XML content, and will not be helpful
> in
> >> the
> >> > necessary. Even if the binding is not explicit, the server would need
> to
> >> do
> >> > class names and instance variable names, or be keys in the java maps,
> it
> >> is
> >> > all inheritance features (if any), because it focuses on the syntax
> and
> >> is
> >> > looking at the protocol or its compliant msgs. However, without
> knowing
> >> the
> >> > compliant msg m_p are in JSON (or have been encoded into JSON).
> Looking
> >> at
> >> > issue. (This case is actually one of the few differences in the m_y
> and
> >> m_p
> >> > YANG Model. For the protocols that don’t meet this condition, there
> can
> >> be

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 21 Oct 2014 08:00:46 +0200

> I agree with your statement that YANG is designed to model datastore and

YANG was designed to model datastores. It was _not_ designed to model
arbtirary protocol messages. Here is the abstract of RFC 6020:

   YANG is a data modeling language used to model configuration and
   state data manipulated by the Network Configuration Protocol
   (NETCONF), NETCONF remote procedure calls, and NETCONF notifications.

I think this describes the purpose quite clearly. I understand that
people like to expand the scope in several directions and I think it
is good if people experiment with new ideas. However, one important
goal is also to keep YANG highly useful for the original purpose it
was created for.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Mon, 20 Oct 2014 08:29:00 -0700

Andy

> > For example, if I have a protocol where "network-map", "src", and "dsts"
> > because "PID1" and "PID2" are not protocol keywords (they are resource
> IDs).
> >
> > A:
> >
> > {
> >
> >  "network-map": [
> >
> >    {
> >
> >      "src": "PID1",
> >
> >      "dsts": ["PID2", "PID3"]
> >
> >    },
> >
> >    {
> >
> >      "src": "PID2",
> >
> >      "dsts": ["PID3", "PID4"]
> >
> >    }
> >
> >  ]
> >
> > }
> >
> > B:
> >
> >
> > {
> >
> >  "network-map": [
> >
> >    "PID1: ["PID2", "PID3"],
> >
> >    "PID2": ["PID3", "PID4"]
> >
> >  ]
> >
> > }
> >
> > YANG's anyxml
> > easy to see that the condition is equivalent to "there exists a full data
> > (Map<String, ?>)."
> > --for instance variables, use "my" prefix, (e.g. myVariable,
> myNetworkMap,
> > etc.)
> >
> > "Element" suffix.
> >  "meta": {
> >    "resource-id": "my-default-map",
> >
> >    "tag": "aab875ef69c87d012"
> >
> >  },
> >
> >  "network-map": [
> >
> >    {
> >
> >      "src": "PID1",
> >
> >      "dsts": ["PID1", "PID2", "PID3"]
> >
> >    },
> >
> >    {
> >
> >      "src": "PID2",
> >
> >      "dsts": ["PID1", "PID3"]
> >
> >    },
> >
> >    {
> >
> >      "src": "PID3",
> >
> >      "dsts": ["PID2", "PID3"]
> >
> >    }
> >
> >  ]
> >
> > }
> >
> >      "leaf Name { type <YANG equivalent of Type>; }"
> >
> >    }
> >
> >        "leaf-list Name { type <YANG equivalent of TypeElement>; }"
> >
> >      } else {
> >
> >        "list Name { <result from build_yang_model<TypeElement.class>> }"
> >
> >      }
> >
> >    }
> >
> >      "container Name { <result from build_yang_model<Type.class>> }"
> >
> >    }
> >
> >  }
> >
> > }
> >
> > For JSON based protocols that don't satisfy the message encoding
> condition,
> > YANG Model. For the protocols that don't meet this condition, there can
> be
> > a semantic equivalent YANG model, but there won't be a generic process of

---

From: Andy Bierman <[email-masked]>
Date: Mon, 20 Oct 2014 08:45:23 -0700

> trying to use JSON as a data structure. Does this make sense?
yes -- so you want to write the YANG module that corresponds to the
existing JSON usage. Not extend YANG to support all possible JSON usage.
Good.

Andy

>>> What is
>>> must be
>>> cannot be
>>> allows us
>>> in the
>>> to do
>>> it is
>>> and is
>>> knowing the
>>> Looking at
>>> and m_p
>>> > YANG Model. For the protocols that don't meet this condition, there
>>> can be

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 20 Oct 2014 17:56:23 +0200

<foo/> corresponds to

"foo": [null]

"foo": null

from "foo" being absent.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 21 Oct 2014 11:29:28 +0200

Hi Xiao,

see inline.

> modeled--because 2 is a standalone value and {"bar": 1} is an object,

Yes.

> "foo" : [1, "abc", enum1]

It should be "foo" : [1, "abc", "enum1"]

>       enum enum1;

Right.

> "foo": [{"a": 1}, {"a": 1, "b": 2}]

Yes.

> We can still model this using the choice statement, right?

Right.

> capabilities of modeling arrays with objects. Does this sound

It does, I am not aware of any other issues.

Cheers, Lada