Subject: [Teas-ns-dt] Pull-request #8: Reza’s proposed text addition to the controller section

From: Jari Arkko <[email-masked]>
Date: Sun, 23 Feb 2020 20:09:37 +0000

This email concerns Pull-request #8: Reza’s proposed text addition to
the controller section. This review is again a personal review, not
holding any hats, just providing my opinion.

This is overall good text and relatively ready for inclusion.

I really liked the way that the section explained not just
the initial mapping but the mapping of telemetry etc. Good work!

There are some places where one can cut a bit unnecessary text, and a few
sentences where a more matter-of-fact style would work better in an
IETF document. I also think we need to be careful in not making the
controller section too long. It may be useful to add some specific
discussions elsewhere under the considerations section of the
framework document. And some discussions may not  be needed
in this document, in my opinion, such as the part about closed loop
optimizations - that's an important topic but we don't need to talk about
it to explain the framework. And I don't think we want to get to the
discussion of e2e slices. Transport slices are components of those,
but we can leave  it at that.

I also wondered how well the controller and mapping sections
complement each other; the editors should probably do a pass to ensure
that each section has the right content. Eric, John: feel free to move
text around if it fits better the other section!

I left a few TODOs in the text that I'd propose we use. There were
a couple of unclear points, mostly related to concepts not introduced
earlier and/or whose need to be included was unclear to me.

I'd use this text, edited from Reza's original:

# Transport Slice Controller (TSC)

    The transport slice controller takes abstract requests for
    transport slices and implements them using a suitable underlying
    technology.  A transport slice controller is the key building
    block for control and management of the transport slice.  It
    provides the creation/modification/deletion, monitoring and
    optimization of transport Slices in a multi-domain, a
    multi-technology and multi-vendor environment.

    The controller provides the following functions:

    * Provides a technology-agnostic NBI for creation/modification/
      deletion of the transport slices.  The attributes of this
      interface will be discussed in section sssss.  In summary, the
      API exposed by this NBI communicates the endpoints of the
      transport slice, transport slice SLO, various policies, and
      provides a way to monitor the slice.

    * Using the network abstract topology, it provides mechanism to find
      the Service endpoints (these are technology specific).
      (TODO: this seems unclear. What endpoints are these, are why
      are they different from endpoints discussed earlier?)

    * Provides "Mapping Functions" for realization of transport slices.
      In other words, it will use the mapping functions which map
      technology-agnostic NBI request to technology-specific SBIs.

    * On its southbound interface, the controller monitors Telemetry
      data (e.g.  OAM results, Statistics, States etc.) for all
      technology-specific services/paths/tunnels which are used to
      realize the transport slice.

    * Aggregates all the Telemetry data of underlying realization of a
      transport slice (i.e. services/paths/tunnels) and calculate the
      current "Transport Slice SLO" and exposes them to "Higher level
      system" through its NBI.  It also raises a notification in case
      the transport slice is out of SLO.

    * Provides the abstract network topology and in specific the "inter-
      domain" links between multiple network domains.
      (TODO: This is unclear, at least to me.)

---

From: Eric Gray <[email-masked]>
Date: Mon, 24 Feb 2020 13:35:57 +0000

Jari, et al,

My annotations below.  The “key” is as follows:

Red text, highlighted in Yellow should not be included (as is) at all.
Text highlighted in Cyan needs significant rewording to be included.

See Purple (Bold/Italic) text for explanation.

Note: an ongoing issue – raised on the list earlier – with at least a few of these contributions has to do with the “presumption” of the existence of an SBI that the TSC maps explicitly to.  The issue with this is that – if we do not allow at least a possibility that the TSC is an entirely proprietary implementation – then we are definitely getting ahead of ourselves.

We are (not yet) being asked to define a “standard TSC.”

Consequently, it is certainly within the realm of possibility that any such “proprietary implementation” will use the NBI of lower-level entities (a YANG model for virtual or physical devices, for example) it is a long way too presumptuous to assume that the TSC will offer an corresponding SBI – even for this case.  It is also likely that a proprietary TSC implementation may manage some subset of logical/physical devices using some non-standard interface – or might even be collocated with one or more logical/physical network entities.

From: Teas-ns-dt <[email-masked]> On Behalf Of Jari Arkko
Sent: Sunday, February 23, 2020 3:10 PM
To: Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]>
Cc: [email-masked]
Subject: [Teas-ns-dt] Pull-request #8: Reza’s proposed text addition to the controller section

EG > We should not include text that attempts to point to other text that does not yet exist, and this is a bullet and should not include a “summary.”

EG > Obviously.  IMO, the requesting application (higher-level controller, for example) MUST specify the end-points for the transport service, at least using some abstraction (e.g. – name), and the precise technology of the connecting interface.  This is fairly obvious as the request is based on some knowledge about what is connecting to the Transport Slice at each end that the TSC controller cannot know a priori.  Since this information MUST be in the “API” request, there should be nothing for the TSC to “discover.”

EG > This “restatement” adds no value, and refers to a presumed “SBI” – which we should not include here (see general observation at the top of this mail).

EG > Again refers to a presumed SBI.  This might be restated as a function the TSC provides relative to the devices and connections used in its abstract topology.

EG > The highlighted text needs re-wording; the TSC will – under no circumstances – be expected to “calculate [any] SLO” but would instead be given the parameters that MUST be met.  Similarly, the TSC MUST be given “threshold values” at which a notification to the “higher level system” (application, etc.) would be provided – or no such notification should be expected.  The TSC itself is almost certainly not in a position to determine this information, and a notification of an out of SLO condition is too late to be useful (it is not typically something a service provider wants, simply because – instead of proactively warning when the out of SLO condition may soon arise – it instead provides a notification when the provider is obligated to pay a penalty for being out of contract).

EG > As currently worded, it is more than unclear – it is wrong. While it is true that the TSC will likely create an abstract topology, to meet the requested connectivity and service requirements, an NBI would describe only the connectivity and service requirements.  In order for the user of the NBI to construct an “abstract topology” it would need to be privy to an inappropriate amount of information about the underlying transport network.  Note that the NBI would need to be able to inform the requesting higher-level entity when it cannot provide some service or connectivity requested in sufficient detail to allow the higher-level entity the flexibility to make a modified request – but the level of detail MUST NOT include specific topological restrictions (beyond “requested connectivity to X” cannot be provided, or “service Y cannot be provided in connecting end-point ‘p’ to end-point ‘q’).  If the intent in this wording is that – by “topology” – what is meant is “the connectivity and service requirements” then this should be explicitly stated.

---

From: Eric Gray <[email-masked]>
Date: Mon, 24 Feb 2020 15:34:11 +0000

I provide suggested re-wording below.

From: Teas-ns-dt <[email-masked]> On Behalf Of Eric Gray
Sent: Monday, February 24, 2020 8:36 AM
To: Jari Arkko <jari.arkko=[email-masked]>; Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]>
Cc: [email-masked]
Subject: Re: [Teas-ns-dt] Pull-request #8: Reza’s proposed text addition to the controller section

Jari, et al,

Note: As discussed during the meeting today, we agree that – as long as we are referring to a logical south-bound interface (that may or may not actually exist), then we can use “SBI” as a way to avoid having to refer to an unnamed concept along the lines of “whatever it is that the TSC maps service and connectivity requests to virtual or physical devices and connections.”  We need to verify that this is consistent with the wording in the definitions draft.

From: Teas-ns-dt <[email-masked]<mailto:[email-masked]>> On Behalf Of Jari Arkko
Sent: Sunday, February 23, 2020 3:10 PM
To: Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [Teas-ns-dt] Pull-request #8: Reza’s proposed text addition to the controller section

Rewording:

  *   Provides a technology-agnostic NBI for creation/modification/deletion of transport slices. The NBI communicates the endpoints of the transport slice, transport slice SLO, and applicable policies, and provides a way to monitor slices.

Rewording:

  *   Provides requested services and endpoint connectivity.

Rewording:

  *   Provides a mapping function for the realization of transport slices.

EG > Again refers to a presumed SBI.  This might be restated as a function the TSC provides relative to the devices and connections used in its abstract topology. Note: even if we assume that SBI is a logical concept that may or may not exist, referring tot the TSC’s “southbound interface” – as a singular thing – begs the concept to have a “singular” real definition.  To illustrate this point, what does it mean to refer to the TSC southbound interface in the case where no such interface actually exists?

Rewording:

  *   The controller collects Telemetry data (e.g. OAM results, Statistics and State information, etc.)for all technology-specific services, paths, tunnels – etc. – that are used to realize the transport slice.

Rewording:

  *   Evaluates transport slice performance in comparison with the SLO parameters extant for the transport slice and exposes this aggregate information to the requesting higher-level system (or application through its NBI.  This may include notifications requested based on thresholds, policies, etc.

I would omit this bullet as it merely restates what has already been said in terms of “realization of transport slices.”

---

From: "Rokui, Reza (Nokia - CA/Ottawa)" <[email-masked]>
Date: Tue, 25 Feb 2020 01:49:26 +0000

See inline for my comments.

Note that my responses are for final Green re-wording statements which goes to the draft.

Reza

From: Eric Gray <[email-masked]>
Date: Monday, February 24, 2020 at 6:34 PM
To: Eric Gray <eric.gray=[email-masked]>, Jari Arkko <jari.arkko=[email-masked]>, Reza Rokui <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: RE: Pull-request #8: Reza’s proposed text addition to the controller section

Jari, et al,

[Reza] Agreed

Rewording:

[Reza] Agreed

Rewording:

[Reza] Modified rewording:

  *   Using the abstract topology of the interconnected networks and provided transport slice endpoints , it will find the Service endpoints.

Explanation:
The concept is shown in our  original Definition draft which later has been removed due to number of pages of the draft to be included in framework draft (or other drafts).
This picture is for 5G use-case but demonstrates the idea.
In summary, the transport slice request is between “transport slice endpoints” gNB1, gNB2, UPF1, UPF2 and UPF3 where the realization of this transport slice is between “service endpoints” ER1, ER2 and ER3.
So, it is clear that the “transport slice endpoints” are different from  “service endpoints”.

                        | Create Transport slice 20 between “transport slice endpoints” gNB1 & gNB2
                        | to UPF1 & UPF2 & UPF3  with SLO  latency
                        | 10 ms or better
                        v
       +---------------------------------------------+
       |Operator-Y Transport Slice Controller (TSC)  |
       +---------------------------------------------+
                        | Implement (Realize) transport slice 20
                        | between “Service endpoints” ER1, ER2 and ER3 with SLA latency
                        | 10 ms  or  better
                        v                          +----+
                      .----.                       +UPF1|
    [gNB1] +----+    (      )---.                 /+--=-+
         \ |    |===========================[ER2]+
          \| ER1|  (      Network       )          +----+
          /|    |===========================[ER3]+-|UPF2|
         / +----+    `----------------'         +  +----+
    [gNB2]                                       \
                                                  \+----+
                                                   +UPF3|
                                                   +----+
    Legends
      === : Tunnels & Services
      ER: Edge Router

Rewording:

[Reza] Agreed

Rewording:

[Reza] Agreed

Rewording:

[Reza] Agreed with a following modification

  *   Using the collected Telemetry data for all technology-specific services, paths, tunnels, evaluates transport slice performance in comparison with the SLO parameters extant for the transport slice and exposes this aggregate information to the requesting higher-level system (or application through its NBI.  This may include notifications requested based on thresholds, policies, etc.

[Reza] Agreed

---

From: Jari Arkko <[email-masked]>
Date: Tue, 25 Feb 2020 08:20:29 +0000

FWIW, I’m happy with the conclusions your Reza and Eric drew. But this one requires further thinking:

Rewording:

Explanation:

Jari: I think this is still confusing, at least to me. From what I can understand there needs to be two things:

  *   At the NBI, the requester needs to tell the controller what endpoints it wants the transport slice to be connecting. This I think we already say. So, for instance, I want this virtual ethernet in this computer connected with a transport slice Slice1 at 10 gbit/s with another virtual ethernet in that other computer.
  *   The controller needs to maintain an internal mapping of how a given slice is realized, so that, for instance, Slice1 in the first computer corresponds to VLAN 123 on physical interface ethx on that computer, switch configuration for VLAN 123, and again VLAN 123 and physical interface ethy on the second computer.

But I’m confused whether your text above talks about the first or the second bullet. In fact it seems to talk about some other function, where the controller somehow analyses abstract topologies to find some endpoints… why? Sorry, maybe not enough caffeinated drinks this morning but I don’t understand ☹

Maybe this would work better:

The controller maintains the connections of the transport slice to the endpoints specified by the consumer. In addition, it will keep track of how communication from those endpoints is mapped to specific underlying components of the realization of the transport slice (such as VPN tunnel endpoints or MPLS labels).

Jari

---

From: Eric Gray <[email-masked]>
Date: Tue, 25 Feb 2020 13:40:12 +0000

Jari,

              Thanks for summing up the one remaining issue I (also) have with Reza’s proposed text – i.e. – exactly what does it mean to say that the TSC “will find the Service endpoints?”

Reza,

              If you’re okay with the wording proposed by Jari, we should move forward based on that wording.

              I will go into some detail on you proposed text in reply to your earlier mail because it contains your figure that I suspect contains the bit I am not understanding.

--
Eric

From: Jari Arkko <[email-masked]>
Sent: Tuesday, February 25, 2020 3:20 AM
To: Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]>; Eric Gray <[email-masked]>; [email-masked]
Subject: Re: Pull-request #8: Reza’s proposed text addition to the controller section
Importance: High

Rewording:

Explanation:

Jari

---

From: "Rokui, Reza (Nokia - CA/Ottawa)" <[email-masked]>
Date: Tue, 25 Feb 2020 13:44:59 +0000

Jari,
My response is inline.

Reza

From: Jari Arkko <[email-masked]>
Date: Tuesday, February 25, 2020 at 11:20 AM
To: Reza Rokui <[email-masked]>, Eric Gray <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: Pull-request #8: Reza’s proposed text addition to the controller section

Rewording:

Explanation:

[Reza] It is mainly option 2 above. I am fine with your text Jari but it definitely needs more explanations somewhere either in this draft or other drafts. This is basically related to Section-6 of the original Definition draft which was removed. At that time we wanted to include them in Framework draft.

Jari

---

From: Eric Gray <[email-masked]>
Date: Tue, 25 Feb 2020 14:35:33 +0000

Reza,

              Thanks for this reply.  See my comment(s) about your explanation, below.  Note that I have cut out the pieces where we have agreement…

--
Eric

From: Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]>
Sent: Monday, February 24, 2020 8:49 PM
To: Eric Gray <[email-masked]>; Eric Gray <eric.gray=[email-masked]>; Jari Arkko <jari.arkko=[email-masked]>; [email-masked]
Cc: Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]>
Subject: Re: Pull-request #8: Reza’s proposed text addition to the controller section
Importance: High

Reza

--- [ SNIP ] ---

Rewording:

Explanation:

I strongly suspect that this figure is at the crux of the confusion.
In one possible view, one might assume that the higher-level controller (or application) asks the TSC to connect a gNB to a UPF.  In this view, the TSC needs to determine which ER is connected to the correct gNB and which ER is connected to the correct UPF – hence the need to “find the Service endpoints” (because this view assume the service end-points are the edge routers).

This becomes slightly more understandable, if we assume that the higher-level controller actually asks for a Transport Slice connection from one IP (Ethernet, Geneve, etc.) addressed end-point to another – without specifying what these end points are as logical RAN/Core Mobility functions (TMI for the TSC).

But that is simply one view, and – IMO – a view that borrows complexity.

In another view (or way of looking at it), the higher-level entity asks the TSC to connect two or more addressable entities providing both connectivity and SLO parameters, where (at least from the perspective of the higher-level entity, the service terminates at each of the addressable entities included in the request to the TSC.  In this view, the links connecting any ER in the transport service  to the specific entities are part of the requested service – hence any edge router involvement is within the service, not a termination point for the service.  Also, in this view, “finding” the appropriate edge router is not any different from finding the path (or route) for the connection.  This applies during creation of, maintenance or and deletion of the Slice – hence this view is actually the simplest one to use in discussion.  Also note that this view may be significantly more correct, in any case where any specific connected entity may be reached via more than one link.

Treating the slice end points as if they are distinct from the service endpoints is an arbitrary, complicating and possibly outright incorrect distinction.

Finally, I don’t really care about the “abstract topology” part but I do not see that it adds any value.

  --- [ SNIP ] ---

---

From: Eric Gray <[email-masked]>
Date: Tue, 25 Feb 2020 14:40:18 +0000

I agree that more explanation is needed somewhere.  For one thing, we need to agree on the definition for “Service Endpoints” (and whether ot not they are – or can be – distinct from slice end points.

I suspect that this needs to be handled in the definitions draft.  See also my detailed comments in separate mail – which address a disconnect in the figure you used in your explanation…

From: Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]>
Sent: Tuesday, February 25, 2020 8:45 AM
To: Jari Arkko <[email-masked]>; Eric Gray <[email-masked]>; [email-masked]
Cc: Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]>
Subject: Re: Pull-request #8: Reza’s proposed text addition to the controller section
Importance: High

Jari,
My response is inline.

Reza

From: Jari Arkko <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, February 25, 2020 at 11:20 AM
To: Reza Rokui <[email-masked]<mailto:[email-masked]>>, Eric Gray <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: Pull-request #8: Reza’s proposed text addition to the controller section

Rewording:

Explanation:

Jari

---

From: John E Drake <[email-masked]>
Date: Tue, 25 Feb 2020 14:57:34 +0000

Hi,

Snipped, comments inline

Yours Irrespectively,

John

In another view (or way of looking at it), the higher-level entity asks the TSC to connect two or more addressable entities providing both connectivity and SLO parameters, where (at least from the perspective of the higher-level entity, the service terminates at each of the addressable entities included in the request to the TSC.  In this view, the links connecting any ER in the transport service  to the specific entities are part of the requested service – hence any edge router involvement is within the service, not a termination point for the service.  Also, in this view, “finding” the appropriate edge router is not any different from finding the path (or route) for the connection.  This applies during creation of, maintenance or and deletion of the Slice – hence this view is actually the simplest one to use in discussion.  Also note that this view may be significantly more correct, in any case whe
re any specific connected entity may be reached via more than one link.

[JD]  I completely agree with Eric on this.  Transport endpoints are between CEs, and any PE functions, e.g., VPNs, service chains, or 5G specific functions, are part of the service offered to them.  This has been the IETF model for over twenty years.

[JD]  I also agree with Eric on this.

  --- [ SNIP ] ---

Juniper Business Use Only

---

From: "Dongjie (Jimmy)" <[email-masked]>
Date: Wed, 26 Feb 2020 12:32:34 +0000

Agree that some explanation about the terms would be needed.

It seems a little odd that the “transport slice endpoints” refers to the nodes “gNB, UPF…” which are outside the transport domain, while the “service endpoints” are actually the nodes which are the border nodes of the transport network.

Best regards,
Jie

From: Teas-ns-dt [mailto:[email-masked]] On Behalf Of Eric Gray
Sent: Tuesday, February 25, 2020 10:40 PM
To: Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]>; Jari Arkko <[email-masked]>; [email-masked]
Subject: Re: [Teas-ns-dt] Pull-request #8: Reza’s proposed text addition to the controller section

From: Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]<mailto:[email-masked]>>
Sent: Tuesday, February 25, 2020 8:45 AM
To: Jari Arkko <[email-masked]<mailto:[email-masked]>>; Eric Gray <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Cc: Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]<mailto:[email-masked]>>
Subject: Re: Pull-request #8: Reza’s proposed text addition to the controller section
Importance: High

Jari,
My response is inline.

Reza

Rewording:

Explanation:

Jari

---

From: John E Drake <[email-masked]>
Date: Wed, 26 Feb 2020 19:20:27 +0000

Hi,

As I indicated in an email yesterday, I agreed with Eric’s point :

In other words a transport slice endpoint is always a tenant device (or in IETF terminology a ‘customer equipment (CE)’ device) and the services provided to that transport slice endpoint (e.g., VPNs, service chains, or 5G specific functions) are part of the service definition for that transport slice endpoint.  So, the two types of endpoints defined in section 5.2.1 are actually within the transport slice provider’s network and are not visible outside of it, and section 5.2.1 is incorrect as currently written.

These functions may either be located entirely within the ‘provider edge (PE)’ device (IETF terminology) to which a transport slice endpoint is attached or distributed between the PE and other nodes within the tenant slice provider’s network.  (As an aside I don’t think the functionality associated with ‘transport type endpoint’ actually exists because at a very minimum a packet needs to be tagged with the tenant identifier (or in IETF terminology the ‘VPN identifier’.)

To answer Jimmy’s point, below,  we could state that a given tenant has the option of supplementing the services provided by a transport slice with services of its own.  However, the devices providing these supplemental services are attached to a transport slice as transport slice endpoints and that the transport slice provider is completely unaware of them.  In IETF terminology a device providing such services is a ‘Customer-Provider Edge (C-PE)’ device.

Yours Irrespectively,

John

Juniper Business Use Only
From: Teas-ns-dt <[email-masked]> On Behalf Of Dongjie (Jimmy)
Sent: Wednesday, February 26, 2020 7:33 AM
To: Eric Gray <eric.gray=[email-masked]>; Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]>; Jari Arkko <[email-masked]>; [email-masked]
Subject: Re: [Teas-ns-dt] Pull-request #8: Reza’s proposed text addition to the controller section

Best regards,
Jie

From: Teas-ns-dt [mailto:[email-masked]] On Behalf Of Eric Gray
Sent: Tuesday, February 25, 2020 10:40 PM
To: Rokui, Reza (Nokia - CA/Ottawa) <[email-masked]<mailto:[email-masked]>>; Jari Arkko <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Teas-ns-dt] Pull-request #8: Reza’s proposed text addition to the controller section

Jari,
My response is inline.

Reza

Rewording:

Explanation:

Jari