Subject: Re: [6tisch] #39 (architecture): Ralph's INT AREA review

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 26 Nov 2015 16:45:21 +0000

Dear all:

I created issue 39 to address Ralph's comments; The changes are quite extensive, but I tried to maintain the content to the level of 08.
The result is already visible in https://bitbucket.org/6tisch/draft-ietf-6tisch-architecture/src/063a04bd273af1cc7e1ae67d0e26e8f9ef28b24c/draft-ietf-6tisch-architecture-09.txt?at=master
And will be discussed tomorrow at the interim call per the published agenda.

The proposed new structure looks like as; per Ralph's suggestion I grouped sections 3, 4 and 5 into a high level architecture and actually made smaller more specific subsections:

1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  2
   2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . . . .  4
   3.  High Level Architecture  . . . . . . . . . . . . . . . . . . .  4
     3.1.  6TiSCH Stack . . . . . . . . . . . . . . . . . . . . . . .  4
     3.2.  TSCH: A Deterministic MAC Layer  . . . . . . . . . . . . .  5
     3.3.  Scheduling TSCH  . . . . . . . . . . . . . . . . . . . . .  6
     3.4.  Routing and Forwarding Over TSCH . . . . . . . . . . . . .  8
     3.5.  A Non-Broadcast Multi-Access Radio Mesh Network  . . . . .  9
     3.6.  A Multi-Link Subnet Model  . . . . . . . . . . . . . . . . 11
     3.7.  Join Process and Registration  . . . . . . . . . . . . . . 12
    3.8.  Dependencies on Work In Progress . . . . . . . . . . . . . 12
   4.  Deeper Dive  . . . . . . . . . . . . . . . . . . . . . . . . . 14
     4.1.  6LoWPAN (and RPL)  . . . . . . . . . . . . . . . . . . . . 14
       4.1.1.  RPL Leaf Support in 6LoWPAN ND . . . . . . . . . . . . 14
       4.1.2.  RPL Root And 6LBR  . . . . . . . . . . . . . . . . . . 15
     4.2.  TSCH and 6top  . . . . . . . . . . . . . . . . . . . . . . 15
       4.2.1.  6top . . . . . . . . . . . . . . . . . . . . . . . . . 15
         4.2.1.1.  Hard Cells . . . . . . . . . . . . . . . . . . . . 16
         4.2.1.2.  Soft Cells . . . . . . . . . . . . . . . . . . . . 16
       4.2.2.  6top and RPL Objective Function operations . . . . . . 16
       4.2.3.  Network Synchronization  . . . . . . . . . . . . . . . 17
       4.2.4.  SlotFrames and Priorities  . . . . . . . . . . . . . . 18
       4.2.5.  Distributing the reservation of cells  . . . . . . . . 19
     4.3.  Communication Paradigms and Interaction Models . . . . . . 21
     4.4.  Schedule Management Mechanisms . . . . . . . . . . . . . . 22
       4.4.1.  Static Scheduling  . . . . . . . . . . . . . . . . . . 22
       4.4.2.  Neighbor-to-neighbor Scheduling  . . . . . . . . . . . 22
       4.4.3.  Remote Monitoring and Schedule Management  . . . . . . 23
       4.4.4.  Hop-by-hop Scheduling  . . . . . . . . . . . . . . . . 26
     4.5.  Forwarding Models  . . . . . . . . . . . . . . . . . . . . 26
       4.5.1.  Track Forwarding . . . . . . . . . . . . . . . . . . . 26
        4.5.1.1.  Transport Mode . . . . . . . . . . . . . . . . . . 28
         4.5.1.2.  Tunnel Mode  . . . . . . . . . . . . . . . . . . . 29
         4.5.1.3.  Tunnel Metadata  . . . . . . . . . . . . . . . . . 29
       4.5.2.  Fragment Forwarding  . . . . . . . . . . . . . . . . . 30
       4.5.3.  IPv6 Forwarding  . . . . . . . . . . . . . . . . . . . 31
     4.6.  Centralized vs. Distributed Routing  . . . . . . . . . . . 32
       4.6.1.  Packet Marking and Handling  . . . . . . . . . . . . . 32
   5.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 33
   6.  Security Considerations  . . . . . . . . . . . . . . . . . . . 33
     6.1.  Join Process Highlights  . . . . . . . . . . . . . . . . . 33
   7.  Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . 35
     7.1.  Contributors . . . . . . . . . . . . . . . . . . . . . . . 35
     7.2.  Special Thanks . . . . . . . . . . . . . . . . . . . . . . 36
     7.3.  And Do not Forget  . . . . . . . . . . . . . . . . . . . . 36
   8.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 36
     8.1.  Normative References . . . . . . . . . . . . . . . . . . . 36

More below:

Summary

-------

This document is not ready for publication.  It needs to address several technical and editorial issues before publication.  In my opinion, the document:

* misses the intent of an "architecture" document

> I read that the actionable is the below, for which the above is an introduction

* mixes high-level architecture with more complete design or specification content

> I separated them now as section 3 and 4 and moved some text to adapt.

* misses some architecture components

> There is now a clearer section "3.8.  Dependencies on Work In Progress". Text was added on the deterministic work.

* is incomplete and/or has been submitted for publication prior to completion of the architecture design

> I removed all references to volumes. The WG will withhold the document till is is complete or the WG aborts, whichever comes first.

Substantive/technical issues

-----------------------------

This document seems to be a work in progress.  There are several references to "first volume of the 6TiSCH architecture".  In my opinion, it should be possible to write a description of the architecture in a single document.  If not, then there should be a plan for what aspects of the architecture will go into other volumes.  This document can't be assessed for completeness without some overview of the entire architecture.  It might be that the WG would be better served by publishing the key components of this document in a more dynamic way, such as in a wiki, and then publish the final architecture when the component protocols and specifications are complete.

> We are now following your recommendation, sections 3.3 and 3.4

In my opinion, this document contains aspects of an architecture document, a requirements document and a design specification.  However, it is missing some key aspect of each kind of document.  For example, section 8 gives what I consider to be a description of the communication paradigms that are part of the architecture.  The communication paradigms are described (for the most part) in the abstract, without specifying the design of how those paradigms are implemented.

> That text was expanded

> The following table was added to clarify what works with what that was also a comment by Kris about the apparent combinatory expansion which is fact is not there):

+---------------------+------------+-----------------------------------+

|   Forwarding Model  |  Routing   |          Scheduling               |

+=====================+============+===================================+

|G-MPLS Track Fwrding |     PCE    |Remote Monitoring and Schedule Mgt |

+---------------------+------------+-----------------------------------+

|                     |            |   Static (Minimal Configuration)  |

+  classical IPv6     +     RPL    +-----------------------------------+

|         /           |            |   Neighbor-to-Neighbor (SF0)      |

+ 6LoWPAN Fragment F. +------------+-----------------------------------+

|                     |Reactive P2P|        Hop-by-Hop (TBD)           |

+---------------------+------------+-----------------------------------+

Section 6, on the other hand, gives specific design details that would be better expressed in a design or specification document.

> I removed the text that looked like requirement or design (regarding 6lo which is now handling it) and pointed at the 6BBR draft that is pending adoption.

Similarly, section 10 specifies the current, preliminary design for the join process, rather than an architecture for security that describes all of the required security functions and how they relate to each other.

> That we still have to work out as a WG but we are chartering for it.

Several key pieces of the design and architecture are missing.  While I understand that this is the "first volume" of a suite of architecture documents, it seems to me that decisions about:

* How do applications interact with the network to request deterministic behavior of a datagram, a flow, a bundle of flows?

* How does the network report back to an application in the case where the deterministic behavior can't be met, or in the case where the network status has changed and an existing reservation can no longer be met?

* How is centralized track computation performed?

* How will the PCE/NME interact with other, autonomous functions such as the routing protocol?  Or, will the PCE/NME control all forwarding?

> Clarified relation with the detnet work and provided details in "4.4.3.  Remote Monitoring and Schedule Management". Note that there are indeterminations so I can only state expectations or goals.

  With respect to Centralized routing and scheduling, the 6TiSCH

  Architecture is (expected to be) be an extension of the detnet work

   Deterministic Networking Architecture [I-D.finn-detnet-architecture],

> Also added a general architecture picture inherited from RFC 7426

need to be made before decisions about the following can be made:

* using a hierarchical multi-link subnet architecture

* management protocols

* routing protocols

* scheduling protocols

* fragment forwarding

> these are agreements that the group made over the last 2 years, and the architecture tracked. Mostly cast in stone by now, but for the fragment thing that is still an open issue with 6lo.

What, precisely, are the requirements?  I see this text:

   traffic that is highly sensitive to jitter, quite

   sensitive to latency, and with a high degree of operational

   criticality so that loss should be minimized at all times.

What are the time scales for jitter and latency - nanoseconds, microseconds, milliseconds?  What are acceptable loss rates?  What are the tradeoffs between loss and jitter/latency?

> I agree that text about that would be informative. Typical control loops are in the order of a second, but a tight schedule could run down to a few Hundredths of seconds if energy is not an issue. Actual numbers for the process control industry are given in section 2.1 of https://tools.ietf.org/html/draft-ietf-roll-rpl-industrial-applicability-02 which is referenced. Should we have more in this particular document?

What are the requirements for mobility?  Do those requirements mandate the hierarchical multi-link subnet architecture?

> I agree that text about that would be informative there too. There is usually no physical mobility in the classical sense, at least when deterministic scheduling is involved. But the radio conditions may make it so that a device reparents elsewhere, ending up attached to a different backbone router. There's also the case of a fully machine that is moved elsewhere and reconnects. So the answer is yes. But note that the RPL mesh is NBMA so already a multilink subnet though homogeneous.

I'm not a security person, but it seems to me that relying solely on L2 security as described in section 10 is inadequate.  The details of the join procedure belong in another document.

>We'll be working on it with the new charter but I cannot contribute text at this point. We have great discussion on the ML though.

Editorial issues

----------------

I think the abstract should read something like:

   This document describes a network architecture that provides

   low-latency, low-jitter and high-reliability packet delivery.  It

   combines a high speed powered backbone and subnetworks using IEEE

   802.15.4 time-slotted channel hopping (TSCH) to meet the

   requirements of industrial deterministic applications.

> OK but for "industrial" which is not the scope for 6TiSCH. What about

"   requirements of LowPower wireless deterministic applications." ?

In general, try to avoid time-dependent references.  For example:

   TSCH was introduced with the IEEE802.15.4e

   [IEEE802154e] amendment and will be wrapped up in the next revision

   of the IEEE802.15.4 standard.

only makes sense at the time the document is published.

> OK the sentence now reads

                                                  TSCH was initially introduced with the

IEEE802.15.4e amendment [IEEE802154e]of the IEEE802.15.4 standard

and constituted a part of the standard from that day

Also try to avoid "new"; for example, change:

> removed all irrelevant occurences of "new"

   At the same time, a new breed of Time Sensitive Networks is being

   developed to enable traffic that is highly sensitive to jitter, quite

   sensitive to latency, and with a high degree of operational

   criticality so that loss should be minimized at all times.

to:

   Time Sensitive Networks enable traffic that is highly sensitive to

   jitter, quite sensitive to latency, and with a high degree of

   operational criticality so that loss should be minimized at all

   times.

> adopted as is

What is the "different time scale" for TSN and TSCH?  The document gives no sense about the relative requirements and operational characteristics of TSN and TSCH.

> added "several orders of magnitude". 6TiSCH subdivises the line rate that is originally at 128kbps so we can end up with an effective rate in kbps, compared to the Ethernet that is typically deployed at 100Mbps but could run faster. Control loops are operated 2 to 3 orders of magnitude faster on Ethernet. These are really different worlds and different applications, but a determiitsic Ethernet backbone may be use to federate a 6TiSCH network.

Explain in the Introduction (if I have this right) that the 6tisch architecture uses route computation to allocate and schedule cells to IPv6 packets.

> Unsure of the expected wording. We have routing and sceduling. In the Neighbor-to-Neighbor scheduling case, (SF0) Routing will cause traffic to follow certain routes, and dynamic scheduling will adapt the number of time slots. In the Remote Monitoring and Schedule Management case, the PCE is aware of a flows and creates a Track with timeslots that can serve those flows.

Rather than go into detail, for example, about using routing protocols to distribute ND registrations, explain that the multi-link subnet architecture requires extensions to NDP because not all hosts in a subnet can communicate with each other directly.

> added:

         6TiSCH nodes are not necessarily reachable from one another at Layer-2

         and an LLN may span over multiple links. This effectively forms an

         homogeneous non-broadcast multi-access (NBMA) subnet, which is beyond

         the scope of existing IPv6 ND methods. Extensions to IPv6 ND have to be

         introduced.

I can't find the term "mote" anywhere else in the document, nor is it a widely used term, so the parenthetical "(also called motes)" is unneeded.

> no more mote

I think the reference to 6tisch-terminology should come first in section 2.  It took me a while to discover the importance of that document in understanding this architecture document.

> done

The citation of "Multi-link Subnet Support in IPv6" [I-D.ietf-ipv6-multilink-subnets] in the terminology section seems inappropriate, as the document expired 13 years ago and the concept was explicitly deprecated in RFC 4903.

> removed that reference; that was a kudos to the original work which may not work in general but can still find applicability such as here.

The overview in section 4 is helpful but provides some redundant details and leaves out some others.  In particular, what are the roles of the PCE and NME, and what communication is required among them and the various forwarding nodes to complete the schedule computation and distribute the schedules?  What are the architectural components of route computation and update?

> This is being refined. The abstract components are not preented in more details in section "4.4.3.  Remote Monitoring and Schedule Management"

It would help the reader understand the motivation for the multi-volume architecture to move the first paragraph of section 5.1 to the Introduction.

> Abandonned the multi volume approach

Can sections 3, 4 and 5 be merged?  They all appear to explain aspects of an overview of the architecture.

> done

Thanks a lot Ralph!

Pascal