Subject: Re: [Raw] new draft on segment routing approach to TSN

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 26 Feb 2021 08:39:54 +0000

Hello Yaakov

If it was me I’d split the draft to take the deeper explanation of the why in a framework informational piece and leave the minimum in the standard track document.

Think that the implementer is a person of the art, and that the IESG will challenge every word in the standard track specification...

I d welcome text that explains that the PSE has to look at not only where to forward but also when to forward, in the RAW architecture/ framework draft.

And yes I often use train station and bus lines and connection to explain deterministic...

Take care,

Pascal

Le 26 févr. 2021 à 05:32, Yaakov Stein <[email-masked]> a écrit :

Pascal,

Please accept my apologies for having misunderstood your intention regarding “green waves”.

I originally read it as two separate statements, namely

  1.  people won’t understand the “green wave” analogy
  2.  people in general think that TSN is magic and no packet ever waits
while I now understand that you intended one statement

  1.  people will misunderstand the analogy to green waves and believe that I am implying that no packet ever waits.

My reply to your true intention is that even in those places where traffic lights are coordinated
it doesn’t mean that every car is first to go through every light or that you don’t have to slow down before lights.
It merely implies that there is high correlation between car arrival times and green light periods.
This is what Qbv and calendaring are supposed to accomplish,
and if people will be misled then perhaps I need to find another analogy.

Perhaps something with bus arrival times is more appropriate where people queue up to enter the bus.
The reason I wanted to avoid that one is that buses don’t stick to schedule.
Since they wait longer when there happen to be more passengers getting on,
they then meet even more passengers at the next stop, while the following bus meets fewer, resulting in bunching.

Y(J)S

From: Pascal Thubert (pthubert) <[email-masked]>
Sent: 25/02/2021 23:27
To: Yaakov Stein <[email-masked]>
Cc: Tianran Zhou <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: new draft on segment routing approach to TSN

Hello Yaakov

Please see below:

Le 25 févr. 2021 à 21:33, Yaakov Stein <[email-masked]<mailto:[email-masked]>> a écrit :

Pascal,

Wow, there is a lot to respond to.

Thanks for the pointers. I was not aware of this work.
However, I believe this proposal is both different and can be complementary to the ones you mention.

Indeed; the questions that this raises are:

- which WG, so I added RAW to the candidate list in this thread

- how to package the complementary proposals in how many draft

Your excellent work focuses on SR signaling in the forwarding plane with strict intermediate latencies at each stage. It would be grand to neatly insert this in a SRv6 together with other information that the RAW PSE processes in its forwarding/scheduling decision.

There are multiple dimensions in which the scope of your work could be enlarged; one of them would be to use a similar signaling but with a best effort objective, which is what I discussed. This could easily be combined if we decide to, so the same signaling serves both use cases. And maybe more, to be discussed.

I was going to add a section on how to reproduce Qbv behavior using a stack of deadlines (the argument is a bit tricky, but straightforward).
I don’t necessarily think that is a good idea, but it should be there to counter claims that SRTSN only reduces the expectation
but can’t provide an absolute upper bound on latency (when based on EDF).

Very cool;

I like to use the term latency, but sometimes use delay to avoid overusing it.

Your decision, at least as long as it is personal submission ; but having a clear terminology is useful. Literature value is of lesser importance here I guess. Latency is what we use in DetNet and RAW. Delay often refers to extra, possibly unwanted time, like a queueing delay. At least you’d make my reader life easier by defining your terminology and sticking to it quite strictly : )

I thought that “green wave” would be understood by those who drive cars in cities where traffic light coordination is common.

Yes, and I think you were clear enough; but then only a few deterministic flows really operate like that. Another flow that use the same outgoing port will have to suffer some delay in a time-triggered queue till the port is free, the delay being taken from the end to end latency budget. My suggestion was to reconsider the text to avoid giving the impression that all flows benefit from that green wave.

I attempted to describe a data structure for EDF to be used instead of a queue,

Neat

but never implied that “queueing” (meaning packets waiting to be transmitted) doesn’t occur.
If that were the case there would not be a need for any data structure at all.

Cool

(Another thing I didn’t get into was the trade-offs among the different options,
 from worst case complexity and difficulty to implement in hardware PoVs.)

I’d love to read that too

I took a quick look at US 9602420 and will study it further.

It’s not as deep as an academic paper but a good hint at how you can turn schedule into QoS or another technique to reorder the set of packets that compete for the port.

However the purpose of this draft is not to mandate any specific scheduling mechanism,
rather to show how such mechanisms can be integrated and hopefully more readily distributed.

Usually you’re expected to provide the meaning of the signaling and the visible behavior of the node; the way the node implements the does not need to be spec but some hints on how that can be achieved are welcome, possibly in annex.

Incidentally I am working on a joint path+deadline optimization algorithm which only requires partial information.

I’d love to chat in this; missing the face to face meetings! Could it be that we do SF in summer?

Proofs of absolute upper bounds on latency are nice, but generally come at the price of either

This is exactly why we did what we did; it also comes at the cost of gathering very specific info on the flows and not all flows can do that, think CBR.

  1.  significantly reduced bandwidth efficiency (like TDM, or equivalently time gating as in Qbv or FlexE),
  2.  unscalable computation
  3.  relatively loose deadlines to being with (like Andrews Zhang)

I need to read it, sorry I was not aware enough of that line of work.

  1.
  2.  assumptions that aren’t necessarily obeyed in real life (like a lot of the network calculus proofs).
I would be happy to discuss your variant to see in which class it falls.

Same here!

Pascal

Y(J)S

From: Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>>
Sent: 25/02/2021 19:40
To: Tianran Zhou <[email-masked]<mailto:[email-masked]>>; Yaakov Stein <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: new draft on segment routing approach to TSN

CAUTION: External sender. Do not click links or open attachments unless you know the content is safe.
Hi Yaakov and all:

Whereever Yaakov decides to place it I’ll be there supporting the work. The draft itself is incredibly well-written and information-rich.
Note that there’s also work in RAW that mentions SR operation DetNet related operations (draft-pthubert-raw-architecture<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-pthubert-raw-architecture-05&data=04%7C01%7Cyaakov_s%40rad.com%7C0f58b1362ad8426a72fc08d8d9d41ec1%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498852383325595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FVosVlYVBfSoHawDj8o6MnU%2B8boPbF7b90oC982lsm4%3D&reserved=0>). RAW has vested interest in intelligent forwarding decision, that would be the trademark vs. DetNet. With this draft, the forwarding is not based on Qbv schedule but the forwarder has some latitude as long as it matches the hop deadline. So RAW may be a good place.
And then there’s draft-chen-detnet-sr-based-bounded-latency<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-chen-detnet-sr-based-bounded-latency-01&data=04%7C01%7Cyaakov_s%40rad.com%7C0f58b1362ad8426a72fc08d8d9d41ec1%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498852383335588%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6pvuDE5KNHlhm%2Bu8Tj3z72EyR343MKzO8aEuLp%2BOiu0%3D&reserved=0>. Ideally all these related items would progress in the same room.

Also a few notes on the draft itself:
- maybe use latency instead of delay; it would be nice to maybe define delay as something else, e.g., the delay representing the time the packet spends queued in one hop vs. the latency that is end to end?
- not sure the term green wave is well understood by the public here; the draft gives the impression that the TSN path is faster than the best effort and involves no queueing. For the most part that is untrue; the latency is bounded but for most flows it is longer than best effort. Best effort can be really fast with passthrough in an empty network. The problem is the long tail and possibly congestion loss. For TSN, there can be very special flows that will traverse the city with all the lights green, but usually there’ll be queuing. The difference is that the queueing latency is constant and the overall latency is withing bounds.
- Time triggered is not the only TSN operation. I wonder what the draft would become with asynchronous shaper in mind. We designed (and as I must announce, patented as US9602420<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatents.google.com%2Fpatent%2FUS9602420&data=04%7C01%7Cyaakov_s%40rad.com%7C0f58b1362ad8426a72fc08d8d9d41ec1%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498852383345583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9gwFvvZITbWsRvAA12JmP1zzTwyLuI9DVA9ph2fuEH4%3D&reserved=0>) a system very similar to the one proposed in the draft, but that is designed to adapt QoS depending on whether the packet is early or late vs. its schedule, and not tagging the schedule in the since the latency is considered end to end not hop by hop. The use case is slightly different since we apply this without a global controller and a provable guarantees all flows will meet the deadline – so not really detnet-, but more like a best effort that all flows meet their deadline in a stochastic environment. If Yaakov is interested, we can contribute on that aspect.

Good luck with the draft,

Pascal

From: detnet <[email-masked]<mailto:[email-masked]>> On Behalf Of Tianran Zhou
Sent: jeudi 25 février 2021 9:14
To: Yaakov Stein <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Detnet] new draft on segment routing approach to TSN

Hi Yaakov,

This is an interesting topic.
After a quick review, there are several questions as follows:
1. It’s clear to me to have a deadline for each packet. So that router can schedule the packet based on the urgency. But what’s the motivation to split the end to end deadline to several local ones?
2. How to divide an end to end deadline into several local deadlines? Is there any example algorithm that could be used by the controller?
3. As far as I know, most devices do not support edf. I am not sure whether your proposal based on edf could really be useful.

Cheers,
Tianran

From: Pce [mailto:[email-masked]] On Behalf Of Yaakov Stein
Sent: Tuesday, February 23, 2021 9:14 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: [Pce] new draft on segment routing approach to TSN

All,

I would like to call your attention to a new ID https://www.ietf.org/archive/id/draft-stein-srtsn-00.txt<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Farchive%2Fid%2Fdraft-stein-srtsn-00.txt&data=04%7C01%7Cyaakov_s%40rad.com%7C0f58b1362ad8426a72fc08d8d9d41ec1%7Cf9047108cc2c4e4897a343fad1b3bf9d%7C1%7C0%7C637498852383345583%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xs2g3OnkF18VI6ZFHO5LIg5EmJW0m%2BLUHzflSq0I1DY%3D&reserved=0>
which describes using a stack-based approach (similar to segment routing) to time sensitive networking.
It furthermore proposes combining segment routing with this approach to TSN
resulting in a unified approach to forwarding and scheduling.

The draft is information at this point, since it discusses the concepts and does not yet pin down the precise formats.

Apologies for simultaneously sending to 3 lists,
but I am not sure which WG is the most appropriate for discussions of this topic.

  *   DetNet is most relevant since the whole point is to control end-to-end latency of a time-sensitive flow.
  *   Spring is also directly relevant due to the use of a stack in the header and the combined approach just mentioned.
  *   PCE is relevant to the case of a central server jointly computing an optimal path and local deadline stack.
I’ll let the chairs decide where discussions should be held.

Y(J)S