Subject: Re: [Cbor] CDDL Module Design

From: Jim Schaad <[email-masked]>
Date: Wed, 1 Jul 2020 07:30:57 -0700

Adding new issues to the list:

1.  Are there any items for which the order of evaluation of the modules is
going to be important.  I think the answer is yes - consider adding elements
to a group and then using the group in an array.

2.  Are there places where the current definitions of anything get messed up
with multiple modules and ordering.  I am thinking of cuts in maps (where I
can't remember which hill things died on) or maybe the newly proposed
.feature control.

Jim

> -----Original Message-----
> From: CBOR <[email-masked]> On Behalf Of Jim Schaad
> Sent: Tuesday, June 30, 2020 10:17 PM
> To: 'Michael Richardson' <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [Cbor] CDDL Module Design
> 
> 
> 
> > -----Original Message-----
> > From: Michael Richardson <[email-masked]>
> > Sent: Tuesday, June 30, 2020 8:54 PM
> > To: Jim Schaad <[email-masked]>
> > Cc: [email-masked]
> > Subject: Re: [Cbor] CDDL Module Design
> >
> >
> > Jim Schaad <[email-masked]> wrote:
> >     > 1.  What is the naming scheme to be used?  Both XML Schema and
> > ASN.1 use
> >     > unique identifiers for a module.  The problem with this approach
> > is
> that a
> >     > revised module has a new identifier even if all of the changes
were
> >     > preserving of the old module.  This can also present problems if
two
> >     > different versions of the same module are used as it can be
> difficult to
> >     > know which version of a module is being provided.
> >
> > Doesn't the version problem also include the "DLL hell" problem?
> > That is:
> >      module A includes module B.1 which includes C.1
> >             A includes module D.1 which includes C.2
> >
> > and who knows if C.1 and C.2 can be used at the same time?
> >
> > So I totally agree that this is a problem.
> >
> >     > I think therefore it
> >     > would be nice to have a naming system that recognizes version
> numbers,
> > but
> >     > can be used with a "latest version" mode.  One possible way to
> > do
> this
> > would
> >     > be to use a query parameter to get a specific version of a
> > module
> while
> >     > without the query parameter the compiler grabs the most recent
> version it
> >     > has.
> >
> > I think that this is probably a very elegant solution.
> >
> >     > 2.  Do want the naming scheme to be used to access a module name
> from
> > the
> >     > web, and to create a server to host all of the IETF based CDDL
> modules?
> >
> > A think that I dislike about the ENTITY stuff in XML is that the URLs
> > are
> way
> > too specific.   Do you want avoid that?  Will it be some URN space?
> 
> I was thinking about the question of doing a URN instead of a URL - hence
the
> term naming scheme.  The interesting question is how should the naming be
> done and what short of mapping is needed.  One of the nice things about
DOIs
> is that they can be used as an indirect reference with a well known server
to
> get actual URLs
> 
> >
> >     > 3.  For any number of reasons we need to be able to state that
> > some symbols
> >     > are going to be imported from a different module.  There are
> > some
> issues
> >     > around doing a couple of other things:
> >     > *  If all symbols are imported then you either need to have a
> > module
> based
> >     > naming system or you need to ensure that there are no collisions.
> >     > *  If only specific symbols are imported, then they could either
> > be
> a) made
> >     > to be unique by renaming or b) made to be unique in the current
> module
> >     > simply by how the module is written
> >
> > Can you give an example of (b)?
> > I think that Python mostly got this process right.
> 
> (b) is just - don't define any conflicts between what I am importing and
what I
> have written in the text.  If I am going to use COSE_Signed as an import,
then I
> really should define a different symbol in the current CDDL file by that
name.
> 
> >
> >     > 4.  ASN.1 has a method to say that only specific symbols can be
> exported
> >     > from a module.  I have never used this capability and I am not
> > clear
> that it
> >     > would be useful to have here.  Should we support it anyway?
> >
> > No. It just leads to stupid work arounds.
> > _ for things that should not be exported, but not enforced.
> 
> I tend to agree with this.  One of the interesting questions would be
transitive
> imports though.  If module A imports COSE_Signed from module B can module
> B import COSE_Signed from module B or must it reference module A.
> I remember the first time I figured out that ASN.1 allows this and to be
frank I
> found it somewhat irritating.  It does allow moving symbols between
modules,
> but that has the potential to break other modules suddenly.
> 
> >
> >     > 5.  How should the system be setup to be able to recognize this
> capability.
> >     > The suggestion in the freezer document currently is to use a
> > pragma system.
> >     > Introducing a pragma system can easily be seen as a step to
> > allow
> for
> >     > compilers of CDDL to be written.   The A2C compiler that I wrote
> used --#
> >     > and #-- as the start and stop delimiters for pragmas as "--" is
the
> >     > start/stop comment character.  (ASN.1 is not line based so there
> > is
> no
> >     > comment till a return.)  A similar approach can easily be used
here.
> >
> > I prefer that the statement not be a pre-processor-like hack.
> 
> The interesting question then, is how much do we want to change the CDDL
> language.  If we do pragmas this is a standalone document.  If we do
changes in
> the grammar then it is harder to make this standalone.
> 
> >
> >     > 6.  If we use a pragma naming system, do we want to setup in
> > advance
> for
> > it
> >     > to have namespaces so that a single module could be used with
> different
> >     > compilers.  I ended up using a different string  than the OSS
> compiler did
> >     > ("--<" ">--") to be able to do this but having a namespace could
> also
> >     > support this.
> >
> > I don't understand.
> 
> Suppose I write a CDDL compiler - called my_great_cddl_compiler and
Carsten
> writes one called no_so_good_cddl_compiler.  I could reference pragmas as
> ME:UseInt32 and CB:UseInt32 as separate pragmas in the same file and each
> compiler would recognize that one that it implements and do the right
thing
> (which is to use some int sized at least 32 bits?)  They could have
different
> behavior for the same name.
> 
> >
> >     > 7.  What set of pragmas are we looking at to start with.  XML
> > Schema
> has
> >     > some attributes that can be used for controlling the compiler.
> > One
> of this
> >     > is a location hint attribute which can be used for giving
alternate
> >     > locations to use for finding a module (such as a location on my
> machine).
> >     > There might be some other pragmas that might be useful to
> > specify as well.
> >     > It might be easier to specify co-occurrence constraints using
> pragmas
> > rather
> >     > than trying to put it directly into the language.
> >
> > This is useful at the top-level file.
> > Particularly for a group of people who are working together.
> > For "location on my machine", I think that I'd rather be providing a
> command
> > flag, but I don't object to having ../.. paths or
> https://my.little.server.example/
> > in a file.
> 
> Another one at the top level might be to use one of the more interesting
things
> that the OSS compiler does is to allow one to say - This is the list of
symbols to
> emit which can be used rather than either emit everything or emit the
first
> symbol in the file which made me do some really strange things in the COSE
> CDDL.
> 
> Jim
> 
> 
> >
> > --
> > Michael Richardson <[email-masked]>, Sandelman Software Works
> > - = IPv6 IoT consulting =-
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 1 Jul 2020 16:49:36 +0200

Hi Jim,

> to a group and then using the group in an array.

What do you mean by “adding elements”?

CDDL was designed to be insensitive to the order of rules, with the exception of the special meaning of the first rule.
I would expect that we compose at the level of rules.

However, prioritized choice and  “/=“/"//=“ conspire against that original design.  So, yes, where a choice is composed from multiple “/=“/"//=“ rules, the order is indeed relevant.

> ..feature control.

Ordering within a group (different from the above) is important.
(That order controls cuts and other forms of prioritized choice, even if the group creates a map that by definition is not order-sensitive.)
But composition from separate rules needs to be explicit here, e.g. in SDF:

sdfinfo = {
 title: text
 version: text
 copyright: text
 license: text
 EXTENSION-POINT
}

EXTENSION-POINT = ( * text => any ) 

Putting the extension point first would enable the use of values that do not match `text` in title, version, copyright, and license entries.
But the specifier put EXTENSION-POINT last explicitly, so there is no such danger.

I can’t imagine at the moment how .feature would introduce new order sensitivities.

Grüße, Carsten

---

From: Jim Schaad <[email-masked]>
Date: Wed, 1 Jul 2020 07:49:32 -0700

> From: CBOR <[email-masked]> On Behalf Of Carsten Bormann
> Sent: Tuesday, June 30, 2020 10:33 PM
> we have had a long discussion about namespacing in the SDF work, and came
> up with:
> https://www.ietf.org/id/draft-onedm-t2trg-sdf-00.html#name-names-and-
> namespaces
> 
> Now SDF is not CDDL, but I think there are lessons to be learned.
> 
> For a while, we were thinking about using URIs to find referenced parts of a
> specification.  This is comparable to the way Java does its module naming; you
> essentially need to put the code into the place that is implied by its name.
> 
> Instead, we have decided to go for a push model, which is essentially the way
> C++, C#, and other languages maintain their namespaces.  A module declares
> into which namespaces its names go.  There is no automatic finding of
> modules, but there also is no pain about who can contribute to a namespace.

There is also nothing to prevent the reuse of the same name in the in the same namespace by multiple contributors.  

> 
> > On 2020-07-01, at 02:52, Jim Schaad <[email-masked]> wrote:
> >
> > Here is a topic that we can potentially discuss tomorrow if there is
> > extra time on the agenda.
> >
> > When looking at a CDDL module design, there are a set of questions
> > that I think need to be addressed:
> >
> > 1.  What is the naming scheme to be used?  Both XML Schema and ASN.1
> > use unique identifiers for a module.  The problem with this approach
> > is that a revised module has a new identifier even if all of the
> > changes were preserving of the old module.  This can also present
> > problems if two different versions of the same module are used as it can be
> difficult to
> > know which version of a module is being provided.   I think therefore it
> > would be nice to have a naming system that recognizes version numbers,
> 
> (I think you are talking about revision numbers.
> Version numbers are often not what one wants; instead one should use feature
> identifiers.)

I am happy with the term revision number here.  Without having a better description of what you think is the difference, I am not sure that I see a significant one.  I do have a potential problem with the use of feature identifiers as we would not be tagging it with two different meanings that need to be distinguished.  At one point were we going to use this for differences in the CDDL language and now you are also using it for differences in CDDL sources.

> 
> > but
> > can be used with a "latest version" mode.  One possible way to do this
> > would be to use a query parameter to get a specific version of a
> > module while without the query parameter the compiler grabs the most
> > recent version it has.
> >
> > 2.  Do want the naming scheme to be used to access a module name from
> > the web, and to create a server to host all of the IETF based CDDL modules?
> 
> That was what we decided we didn’t want to do in SDF.
> 
> > 3.  For any number of reasons we need to be able to state that some
> > symbols are going to be imported from a different module.
> 
> Note how C++ doesn’t really do that.
> (Well, if you declare a name and don’t define it, it needs to be external.)

And yet C++ does end up with header files which need to be included (or duplicated) in order to have the same consistent view of what any class does.   It also will end up with problems if two different source files declare (and define) a single name in the same namespace.

> > There are some issues
> > *  If all symbols are imported then you either need to have a module
> > based naming system or you need to ensure that there are no collisions.
> 
> I would like to separate the external naming system from the internal one we
> use in a spec.
> 
> Note how a spec already today uses symbols from the prelude and from the
> spec itself.  (This of course works “automagically” only because the prelude is
> fixed and you simply cannot use the symbols that the prelude squats on.)
> 
> > *  If only specific symbols are imported, then they could either be a)
> > made to be unique by renaming or b) made to be unique in the current
> > module simply by how the module is written
> 
> Yes.
> 
> > exported from a module.  I have never used this capability and I am
> > not clear that it would be useful to have here.  Should we support it anyway?
> 
> I think there might be a place for an advisory-level setting (as in “these names
> will normally be imported if this module is imported whole-sale”).  Experience
> shows that there are good reasons to violate any “encapsulation” attempt, so I
> would not disallow importing further names from a module explicitly.
> 
> > 5.  How should the system be setup to be able to recognize this capability.
> > The suggestion in the freezer document currently is to use a pragma system.
> > Introducing a pragma system can easily be seen as a step to allow for
> > compilers of CDDL to be written.   The A2C compiler that I wrote used --#
> > and #-- as the start and stop delimiters for pragmas as "--" is the
> > start/stop comment character.  (ASN.1 is not line based so there is no
> > comment till a return.)  A similar approach can easily be used here.
> 
> This is the approach I would prefer.
> 
> However, there also is semantics imparted by using a symbol.
> In actual CDDL usage, the symbols used become annotations in the post-
> validation instance.  These annotations will need to be namespaced, and
> possibly be linked to RDF names (which are URIs not intended for
> dereferencing).
> 
> > 6.  If we use a pragma naming system, do we want to setup in advance
> > for it to have namespaces so that a single module could be used with
> > different compilers.  I ended up using a different string  than the
> > OSS compiler did ("--<" ">--") to be able to do this but having a
> > namespace could also support this.
> 
> Indeed, we could play around with more than one module/namespace
> management system, as long as there is a way to generate a basic CDDL
> specification from each.
> 
> > 7.  What set of pragmas are we looking at to start with.  XML Schema
> > has some attributes that can be used for controlling the compiler.
> > One of this is a location hint attribute which can be used for giving
> > alternate locations to use for finding a module (such as a location on my
> machine).
> 
> Tempting, but this leads us back into pull land.
> 
> > There might be some other pragmas that might be useful to specify as well.
> > It might be easier to specify co-occurrence constraints using pragmas
> > rather than trying to put it directly into the language.
> 
> I think that all constraints need to be part of the language.
> 
> Grüße, Carsten
> 
>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 1 Jul 2020 16:54:47 +0200

> I prefer that the statement not be a pre-processor-like hack.

The C/C++ preprocessor may be considered a hack because it operates at a lexical level.  We should not make that mistake.  Apart from that, it does useful things.

Or, in other words, your hack may be my perfect layering.

I would prefer it if a CDDL compiler (using that word in its english sense) could spit out something that looks like CDDL 1.0 and has the same semantics (well, maybe except for semantic names, which don’t exist in CDDL 1.0).

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Wed, 01 Jul 2020 11:30:39 -0400

> The C/C++ preprocessor may be considered a hack because it operates at
    > a lexical level.  We should not make that mistake.  Apart from that, it
    > does useful things.

    > Or, in other words, your hack may be my perfect layering.

I agree.
As has been pointed out in
   https://www.goodreads.com/book/show/44919.Working_Effectively_with_Legacy_Code

the preprocessor is a really good tool for inserting shims.
But, gosh, I'm hoping that this won't be a requirement for CDDL?

    > I would prefer it if a CDDL compiler (using that word in its english
    > sense) could spit out something that looks like CDDL 1.0 and has the
    > same semantics (well, maybe except for semantic names, which don’t
    > exist in CDDL 1.0).

Hmm. yes. Okay.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Jim Schaad <[email-masked]>
Date: Wed, 1 Jul 2020 10:43:25 -0700

> From: Carsten Bormann <[email-masked]>
> Sent: Wednesday, July 1, 2020 7:50 AM
> > 1.  Are there any items for which the order of evaluation of the
> > modules is going to be important.  I think the answer is yes -
> > consider adding elements to a group and then using the group in an array.
> 
> However, prioritized choice and  “/=“/"//=“ conspire against that original
> design.  So, yes, where a choice is composed from multiple “/=“/"//=“ rules, the
> order is indeed relevant.

This is the case that I was thinking of.  

Module A
group = (
   element1 : string
)

Module B
group //= ( element2 : int )

Module C
group //= (element3 : float )

Module D
myArray = [ group ]

> 
> > 2.  Are there places where the current definitions of anything get
> > messed up with multiple modules and ordering.  I am thinking of cuts
> > in maps (where I can't remember which hill things died on) or maybe
> > the newly proposed ..feature control.
> 
> (That order controls cuts and other forms of prioritized choice, even if the
> group creates a map that by definition is not order-sensitive.) But composition
> from separate rules needs to be explicit here, e.g. in SDF:
> 
> sdfinfo = {
>  title: text
>  version: text
>  copyright: text
>  license: text
>  EXTENSION-POINT
> }
> 
> Putting the extension point first would enable the use of values that do not
> match `text` in title, version, copyright, and license entries.

The question comes if you add more elements to the end of the map.  We have had this discussion during the cut process and I think that I lost that round but I would need to do a deep read on this to figure that out. Things become interesting if you do

sdfinfo = {
  title: text,
  version: text,
  copyright: text,
  license: text,
  More-attributes,
  EXTENSION-POINT
}

More-attributes = ()

> I can’t imagine at the moment how .feature would introduce new order

I am not sure that I do either, it just is something that raises questions about what happens if I add new fields to a map.   In the above, does the .feature on EXTENSION-POINT always come into effect if I add something to More-attributes in a different module?

Jim

> >>> A think that I dislike about the ENTITY stuff in XML is that the
> >>> URLs are
> >> way
> >> term naming scheme.  The interesting question is how should the
> >> naming be done and what short of mapping is needed.  One of the nice
> >> things about
> > DOIs
> >> imports though.  If module A imports COSE_Signed from module B can
> >> module B import COSE_Signed from module B or must it reference module
> A.
> >> The interesting question then, is how much do we want to change the
> >> CDDL language.  If we do pragmas this is a standalone document.  If
> >> we do
> > changes in
> >> ME:UseInt32 and CB:UseInt32 as separate pragmas in the same file and
> >> each compiler would recognize that one that it implements and do the
> >> right
> > thing
> >> symbol in the file which made me do some really strange things in the
> >> COSE CDDL.
> >>
> >> Jim
> >>
> >>
> >>>
> >>> --

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 1 Jul 2020 22:52:35 +0200

Hi Jim,

let me pick on your example some more :-)

> group //= (element3 : float )

Right.  Note that `group` is a group choice, so only one (and exactly one) of the alternatives can be chosen.  As the types “element1", “element2", “element3” (yes, the bare words turn into strings) do not overlap, the sequence in the choice is irrelevant.  If they did overlap, the first match would win, and the implied cut makes sure the alternatives are not tried.

So this is different from “adding entries to the group”, which would allow multiple of the entries to go into the array.

> The question comes if you add more elements to the end of the map.  We have had this discussion during the cut process and I think that I lost that round but I would need to do a deep read on this…

I would expect you are trying to build a group choice that is called More-attributes, so that we then can say:

More-attributes //= (foo: “bar”)

Starting that choice with () doesn’t work very well — the empty group always matches, and because of prioritized choice, nothing of the other alternatives is tried.

So let’s start it with something that never matches.  There is no idiomatic way to do this in CDDL yet, so let me invent one:

More-attributes = (1 .eq 2 => 3)

(This stochastically breaks the little brain of the CDDL tool, but let’s ignore that for now; maybe there are better ways to build impossible group choices.)

Now I can add 
More-attributes //= (foo: “bar”)
And
More-attributes //= (3: 4)

For good measure.

The way this is used in the maps means that only exactly one of the choices is ever added, which is not what I would call “adding entries"; if you want to be able to add from all of them (and optionally so), use

 * More-attributes,

in the map.

This is all a bit tedious, why it is not a very idiomatic way to build up maps.

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Wed, 01 Jul 2020 17:40:39 -0400

> the preprocessor is a really good tool for inserting shims.

The word I wanted was "seams", not shims.
C gives you pre-processor seams, and link seams.
Java lets you have neither.

A seam is a way to replace some function, usually something low-level like
"open(2)" or "socket(2)" so that in a unit test you can feed in test data
rather than live data.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 1 Jul 2020 23:57:26 +0200

> On 2020-07-01, at 23:40, Michael Richardson <[email-masked]> wrote:
> 
> 

We call that a “mock” (see [1] for another take of this, which involves “fake”s).  (Which may be implemented as a shim, i.e., recurring on the real function, which when made into an art form is also known as “advising” or “advice” [2].  Or as a “stub”.)

Real programming involves the use of linker scripts, where you often can rearrange some of the symbols or map one to the other.  I think it would be good if we have some isolation between the world of external names (and name references) and the internal symbols in a specification.  If this is not to become too tedious, some optional automation of setting up the mapping (and a way to gradually escape that automation) is desirable.

Hacking some of this functionality together at the lexical level (as the preprocessor makes you do) usually provides leaky abstractions, where the leak eventually bites you.  That’s why CDDL has generics, not macros (not that macros would be wrong, if done hygienically instead of lexically).

Grüße, Carsten

[1]: https://www.infoworld.com/article/3143104/my-two-cents-on-fakes-stubs-mocks-and-shims.html
[2]: https://en.wikipedia.org/wiki/Advice_(programming)