Subject: [Detnet] pulling packets

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 13 Nov 2018 05:10:24 +0000

Dear all

This video https://youtu.be/BO0QhaxBRr0 has a great illustration of the UNI packet pull method that we use in section "5.1.  Local Loop Flow Control" of https://tools.ietf.org/html/draft-thubert-tsvwg-detnet-transport-01 to pull packets from the detnet unaware host from the detnet ingress box. The problem for detnet is that shaping in the detnet ingress can create bloat and uncontrolled end to end latency so we want to control the maximum amount of pending frames to keep the detnet ingress busy and push back to the L4/5 for anything above that.

I hope you'll love the video as much as I did;

Pascal

---

From: "Gengxuesong (Geng Xuesong)" <[email-masked]>
Date: Wed, 14 Nov 2018 07:57:49 +0000

Dear Pascal,

I consider the problem of "DetNet transport layer" all over again, and finally I make myself confused. So I send this email to search for help. Please correct me if you find anything wrong below:

1.       TSN doesn't need transport layer, because the talker and listener are TSN aware, and they can send the packets in some particular time slot based on the traffic specification;

2.       DetNet does need transport layer, because talker and listener are not under control;

3.       TCP can't be used for DetNet , because TCP tries to find the "fair bandwidth" for the flows and it makes the flow rate changes all the time, while the sending rate of DetNet is supposed to be constant, and every packet should show up in an expected time slot;

4.       UDP can't be used for DetNet, because there is no rate control. If the talker sends the packets in the wrong pattern, UDP can do nothing.

5.       So we need "DetNet transport layer", which can :a. Send the packet in a deterministic pattern; b. If anything goes wrong, make it go back to the correct pattern.

6.       This, finally, makes the talker "DetNet aware"(especially point a above), which is just the same as what happens in TSN.

If this logic chain works, what we actually requires is just a DetNet talker rather than a new transport layer.

Best Regards
Xuesong

From: detnet [mailto:[email-masked]] On Behalf Of Pascal Thubert (pthubert)
Sent: Tuesday, November 13, 2018 1:10 PM
To: [email-masked]; Black, David <[email-masked]>
Subject: [Detnet] pulling packets

Dear all

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 14 Nov 2018 09:52:54 +0000

Hello Xuesong:

We do layers so as to communalise services so every applications does not need to reinvent its own.

The fact that every industrial application that adaptateur from serial but to IP had to implement a jitter elimination buffer is an indication that we failed to provide just that at a lower layer.

Same goes for voice and video. But didn’t you see it on occasions? Those jitter elimination techniques create a latency in the discussion or a delay in the tv stream that you can actually observe.

Now let us look at the detnet service layer and compare with traditional Layer 4 functions.

 The service layer provides a flow control, in this particular case a rate-based one. This is similar to functionality found in DC-TCP or in a long gone past, IBM’s Rapid Transport Protocol, though we probably need only a simplified version of that.

The service layer performs reordering and duplication elimination functions. TCP does that too.

The service layer distributes packets and replicas over multiple paths (detnet transports). There’s a hint of that in MP-TCP though we need a powerful addition for replication and coding.

The detnet service layer is above the detnet transport layer which happens to be layer 3.

All in all, this tells me that the detnet transport layer operates at layer 4.

I agree with you that there is an exact same need for TSN. Just like there is a need for IS-IS at both layers.

The IEEE may very well provide the same features in the talker, and we may very well be successful at sharing specifications just like we did for IS-IS. But for us I see the detnet service layer packaged as a Layer 4 that the rest of the IETF will call a Transport. Which makes the name collision very unfortunate, by the way.

Note that my draft also discusses a L2 tokenized loop to avoid the buffer bloat at the detnet ingress, which is the piece that is analogous to the solution shown in the video.

Take care,

Pascal

Le 14 nov. 2018 à 14:58, Gengxuesong (Geng Xuesong) <[email-masked]<mailto:[email-masked]>> a écrit :

Dear Pascal,

I consider the problem of “DetNet transport layer” all over again, and finally I make myself confused. So I send this email to search for help. Please correct me if you find anything wrong below:

1.       TSN doesn’t need transport layer, because the talker and listener are TSN aware, and they can send the packets in some particular time slot based on the traffic specification;

3.       TCP can’t be used for DetNet , because TCP tries to find the “fair bandwidth” for the flows and it makes the flow rate changes all the time, while the sending rate of DetNet is supposed to be constant, and every packet should show up in an expected time slot;

4.       UDP can’t be used for DetNet, because there is no rate control. If the talker sends the packets in the wrong pattern, UDP can do nothing.

5.       So we need “DetNet transport layer”, which can :a. Send the packet in a deterministic pattern; b. If anything goes wrong, make it go back to the correct pattern.

6.       This, finally, makes the talker “DetNet aware”(especially point a above), which is just the same as what happens in TSN.

Best Regards
Xuesong

From: detnet [mailto:[email-masked]] On Behalf Of Pascal Thubert (pthubert)
Sent: Tuesday, November 13, 2018 1:10 PM
To: [email-masked]<mailto:[email-masked]>; Black, David <[email-masked]<mailto:[email-masked]>>
Subject: [Detnet] pulling packets

Dear all

This video https://youtu.be/BO0QhaxBRr0 has a great illustration of the UNI packet pull method that we use in section “5.1.  Local Loop Flow Control” of https://tools.ietf.org/html/draft-thubert-tsvwg-detnet-transport-01 to pull packets from the detnet unaware host from the detnet ingress box. The problem for detnet is that shaping in the detnet ingress can create bloat and uncontrolled end to end latency so we want to control the maximum amount of pending frames to keep the detnet ingress busy and push back to the L4/5 for anything above that.

I hope you’ll love the video as much as I did;

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 14 Nov 2018 09:59:11 +0000

Typos below, some thanks to my intelligent smart phone, and some to confused-me.

Please read:

All in all, this tells me that the detnet service layer operates at layer 4

Regards,

Pascal

Le 14 nov. 2018 à 16:53, Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>> a écrit :

Hello Xuesong:

Take care,

Pascal

Dear Pascal,

Best Regards
Xuesong

Dear all

Pascal
_______________________________________________
detnet mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/detnet

---

From: "Black, David" <[email-masked]>
Date: Wed, 14 Nov 2018 15:52:30 +0000

Bottom Line:  I would suggest focusing discussion of a DetNet “transport protocol” on what has to be done via tunnels to make PREOF work for the IP solution (data plane), and would suggest using separate drafts for this and local loop traffic shaping.

Details inline …

Thanks, --David

From: Pascal Thubert (pthubert) [mailto:[email-masked]]
Sent: Wednesday, November 14, 2018 4:59 AM
To: Gengxuesong (Geng Xuesong)
Cc: [email-masked]; Black, David
Subject: Re: [Detnet] pulling packets

[EXTERNAL EMAIL]
Please report any suspicious attachments, links, or requests for sensitive information.
Typos below, some thanks to my intelligent smart phone, and some to confused-me.

Please read:

Regards,

Pascal

Same goes for voice and video. But didn’t you see it on occasions? Those jitter elimination techniques create a latency in the discussion or a delay in the tv stream that you can actually observe.
[David>] When things work, jitter elimination is common behavior.  When they don’t work, i.e., something doesn’t arrive in time, both retransmit behavior and reaction to missing data are app-specific.

 The service layer provides a flow control, in this particular case a rate-based one. This is similar to functionality found in DC-TCP or in a long gone past, IBM’s Rapid Transport Protocol, though we probably need only a simplified version of that.
[David>] It’s important to distinguish flow control, which can be provide on a link-by-link basis from congestion control, which is a superset of flow control and can only be provided end-to-end.  As DetNet eliminates all congestion, congestion control is not required.  Specific example – DC-TCP is driven by ECN congestion signals (CE marks) coming back from the network, but if a DetNet node ever marks a DetNet packet as CE, something has gone seriously awry.

The service layer performs reordering and duplication elimination functions. TCP does that too.
[David>] Yes … but … TCP elimination of duplicate packets is intended only to deal with “stupid network tricks” …

The service layer distributes packets and replicas over multiple paths (detnet transports). There’s a hint of that in MP-TCP though we need a powerful addition for replication and coding.
[David>] DetNet differs because the replication and distribution happen at nodes in the network, not at end systems.  For any TCP-like protocol (reliable delivery with congestion control), packet duplication is always a manifestation of a network “bug,” i.e., divergence from ideal network behavior.  For the RE in DetNet PREOF, packet duplication is a network “feature” ;-).  As of right now, packet duplication and elimination, plus associated reordering (PREOF)  is not part of the IP solution – an additional encapsulation would likely be needed to support that functionality, and the result which would look a lot like a DetNet transport tunnel between nodes in the DetNet network.  If anyone wants to work on this, reading draft-ietf-intarea-tunnels is RECOMMENDED.

All in all, this tells me that the detnet transport service layer operates at layer 4.

I agree with you that there is an exact same need for TSN. Just like there is a need for IS-IS at both layers.
[David>] Many of the IP networks over which DetNet operates will not be using IS-IS for IP (layer 3) routing – there are a number of alternatives.

The IEEE may very well provide the same features in the talker, and we may very well be successful at sharing specifications just like we did for IS-IS. But for us I see the detnet service layer packaged as a Layer 4 that the rest of the IETF will call a Transport. Which makes the name collision very unfortunate, by the way.
[David>] Bottom Line:  I would suggest focusing discussion of a DetNet “transport protocol” on what has to be done via tunnels to make PREOF work for the IP solution (data plane), and would suggest using separate drafts for this and local loop traffic shaping.

Take care,

Pascal

Best Regards
Xuesong

Dear all

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 14 Nov 2018 16:40:31 +0000

I agree with the bottom line David,

that was the intention; though I’m unclear if and where I can publish the lower layer pull thing. I wrote it as an eye opener, hoping people would see the issue. I retain Lou’s suggestion that PCF is a beginning for it though it is just a faucet with no control of what goes though when we want a just in time per packet pull.

For your points below, I mentioned IS-IS as an illustration that we can have a same operation at both layers, that is not so unusual. I’m not saying detnet need IS-IS, that was really not the point.

As for the functions I just point out the likeness to illustrate that the detnet service layer is akin though far from identical to existing variations of TCP. Seems we agree on this likeness. I do not read that you oppose my conclusion that the detnet service layer can be implemented as a detnet « transport protocol ».

For now I need to read on the links you suggested.

Many thanks for all this!

Pascal

Le 14 nov. 2018 à 22:52, Black, David <[email-masked]<mailto:[email-masked]>> a écrit :

Details inline …

Thanks, --David

Cc: [email-masked]<mailto:[email-masked]>; Black, David

[EXTERNAL EMAIL]

Please read:

Regards,

Pascal

Take care,

Pascal

Best Regards
Xuesong

Dear all

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 14 Nov 2018 16:46:13 +0000

Please read PFC for priority flow control;
I did too much wireless this week :)

Regards,

Pascal

Le 14 nov. 2018 à 23:41, Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>> a écrit :

Pascal

Details inline …

Thanks, --David

[EXTERNAL EMAIL]

Please read:

Regards,

Pascal

Take care,

Pascal

Best Regards
Xuesong

Dear all

Pascal

---

From: "Black, David" <[email-masked]>
Date: Thu, 15 Nov 2018 01:13:23 +0000

Pascal,

Thanks for the response – I want to follow up on some of Xuesong’s comments that may shed some light on the lower layer pull functionality:

> 3.  TCP can’t be used for DetNet, because TCP tries to find the “fair bandwidth” for the flows and it makes the flow rate changes
> all the time, while the sending rate of DetNet is supposed to be constant, and every packet should show up in an expected time slot;

Not exactly – in this “game”, DetNet gets to go first, and enforces the time slot behavior.  TCP sees what DetNet provides – it looks like a network path with a max bandwidth that is limited by the reservation for the DetNet flow, a rather stable round-trip-time and pacing/shaping behavior (on the originating host, preferably).  There are some details to be worked out around what happens when TCP tries to overrun the DetNet flow reservation, as that could build up a large queue on the host, but if the reservation is sized to be larger than the TCP throughput, TCP should work as the host queue should drain.

> 4.  UDP can’t be used for DetNet, because there is no rate control. If the talker sends the packets in the wrong pattern, UDP can do nothing.
That would be a “feature” here :-) because DetNet is taking care of the details ;-).  In contrast, TCP-like retransmit behavior may not be what’s wanted in a hard real-time system.

> 5.  So we need “DetNet transport layer”, which can :a. Send the packet in a deterministic pattern; b. If anything goes wrong, make it go back to the correct pattern.
Every DetNet router does this. A pacer/shaper should deal nicely with packets being sent in the wrong pattern.   This is preferably located on the originating host, as that’s a better place to build a queue than the first hop switch/router.  Again if the reservation is sized to be larger than the traffic, the originating host queue should drain.

In the reverse direction, where the traffic (TCP or UDP or …) exceeds the reservation, something on the host needs to figure out what’s going on in order to throttle TCP and/or the application, as the usual network congestion feedback techniques (loss, ECN CE marks) don’t work for DetNet traffic.   This is where a network pull mechanism may be a useful part of the solution, as getting the application to understand that it’s overrunning its reservation is preferable to having  the DetNet network start dropping (or CE-marking) packets if a DetNet flow overruns its reservation in order to get the application or transport protocol (e.g., TCP) to slow down.

As for the notion that PREOF for DetNet IP is a DetNet “transport protocol” – I suppose that’s true, but I would suggest use of DetNet “PREOF tunnel protocol” as clearer terminology.

Thanks, --David

From: Pascal Thubert (pthubert) [mailto:[email-masked]]
Sent: Wednesday, November 14, 2018 11:46 AM
To: Black, David
Cc: Gengxuesong (Geng Xuesong); [email-masked]
Subject: Re: [Detnet] pulling packets

[EXTERNAL EMAIL]

Regards,

Pascal

that was the intention; though I’m unclear if and where I can publish the lower layer pull thing. I wrote it as an eye opener, hoping people would see the issue. I retain Lou’s suggestion that PCFPFC is a beginning for it though it is just a faucet with no control of what goes though when we want a just in time per packet pull.
For your points below, I mentioned IS-IS as an illustration that we can have a same operation at both layers, that is not so unusual. I’m not saying detnet need IS-IS, that was really not the point.

Pascal

Details inline …

Thanks, --David

[EXTERNAL EMAIL]

Please read:

Regards,

Pascal

Take care,

Pascal

Best Regards
Xuesong

Dear all

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 15 Nov 2018 04:23:50 +0000

Hello David

Please see below

Le 15 nov. 2018 à 08:13, Black, David <[email-masked]<mailto:[email-masked]>> a écrit :

Pascal,

I think Bob Briscoe presented  on Sunday and very clearly what happens with classical TCP - a seesaw pattern that fails to utilize the available bandwidth. This coincides with my own observations when I implemented CIR for IBM’s frame relay box a few years back.

Bob also presented the benefits of DC TCP in that space. I understand that his point matches your words in a corridor discussion we had. I also understand that in the case of the fixed bandwidth. A rate adaptation is a complex and undesired function.

Attempting to achieve the service layer rate enforcement with a window-based operation confuses a function that we have and its derivative that we want, and causes additional buffers and latency between the egress of the window based service layer and the ingress of the load enforcer - the first detnet router.

Conclusion is that we want a simple rate enforcement function in the layer-4 « detnet transport « . Could we call it dettrans to avoid confusion ?

Sorry I’m embarking now...

Take care

Pascal

Thanks, --David

From: Pascal Thubert (pthubert) [mailto:[email-masked]]
Sent: Wednesday, November 14, 2018 11:46 AM
To: Black, David
Cc: Gengxuesong (Geng Xuesong); [email-masked]<mailto:[email-masked]>
Subject: Re: [Detnet] pulling packets

[EXTERNAL EMAIL]

Regards,

Pascal

Pascal

Details inline …

Thanks, --David

[EXTERNAL EMAIL]

Please read:

Regards,

Pascal

Take care,

Pascal

Best Regards
Xuesong

Dear all

Pascal

---

From: Norman Finn <[email-masked]>
Date: Thu, 15 Nov 2018 05:15:48 +0000

1. I think there  is a use for a "Sort of like TCP, but is suitable for use as a DetNet flow".

2. You can run a number of TCP flows over a pipe offering the DetNet service.  More flows will use more of the bandwidth.  Leftover bandwidth is available to best effort flows, but not to other DetNet flows.  This is just like TCP over an Ethernet.  Very useful when allocating "hard" partitioned bandwidth.

2. You can run one TCP flow as a DetNet flow.  You can run several of them.  Each flow wastes approx. 25% of its bandwidth, which is not reservable by other DetNet flows.  On the other hand, you can control exactly how much bandwidth is available to each TCP flow, and all of the unused bandwidth is available to best-effort TCP flows.  This seems theoretically useful, but I'm not sure that the extra cost over DSCP is worth the difference in service compared to DSCP.

-- Norm

________________________________
From: detnet [[email-masked]] on behalf of Pascal Thubert (pthubert) [[email-masked]]
Sent: Wednesday, November 14, 2018 8:23 PM
To: Black, David
Cc: Gengxuesong (Geng Xuesong); [email-masked]
Subject: Re: [Detnet] pulling packets

Hello David

Please see below

Pascal,

Take care

Pascal

Thanks, --David

[EXTERNAL EMAIL]

Regards,

Pascal

Pascal

Details inline …

Thanks, --David

[EXTERNAL EMAIL]

Please read:

Regards,

Pascal

Take care,

Pascal

Best Regards
Xuesong

Dear all

Pascal

---

From: "Gengxuesong (Geng Xuesong)" <[email-masked]>
Date: Thu, 15 Nov 2018 07:34:42 +0000

Hi David,

Thank you for your explanation, and they are very helpful.
Please find my comments inline…

All in all: some text is needed here, even if no new transport protocol is expected. How to make DetNet work together with TCP or UDP is  not so straight forward.

Best Regards
Xuesong

From: Black, David [mailto:[email-masked]]
Sent: Thursday, November 15, 2018 9:13 AM
To: Pascal Thubert (pthubert) <[email-masked]>
Cc: Gengxuesong (Geng Xuesong) <[email-masked]>; [email-masked]
Subject: RE: [Detnet] pulling packets

Pascal,

<Xuesong>  Thank you for pointing out the complex issues here :

1.       TCP window can be different from the reserved bandwidth:

a.       If TCP window>(reserved bandwidth=sending rate), everything seems good;

b.      If TCP window<(reserved bandwidth=sending rate), the reserved bandwidth can’t be guaranteed for the talker any more. [problem]

c.       If (TCP window=sending rate)>reserved bandwidth, the talker will push too many packets into the network, which can't be handled by the DetNet network.

2.       Whether retransmission is definitely not needed in DetNet.

And I’m not sure whether I understand  “if the reservation is sized to be larger than the traffic, the originating host queue should drain” correctly. Do you mean that if case 1.b listed above happens, the sending rate should go along with the reserved bandwidth rather than the TCP window? If so, I think there should be should mechanism/rules to tell the talkers what should they do.

<Xuesong> I would like to point out another option here: maybe we CAN use ECN in DetNet. If the ‘packet pulling’ mechanism is too far away from the current TCP mechanism, whether we can consider to allow the ingress to use ECN as an indication of misbehavior of the talker.

Thanks, --David

[EXTERNAL EMAIL]

Regards,

Pascal

Pascal

Details inline …

Thanks, --David

[EXTERNAL EMAIL]

Please read:

Regards,

Pascal

Take care,

Pascal

Best Regards
Xuesong

Dear all

Pascal

---

From: "Gengxuesong (Geng Xuesong)" <[email-masked]>
Date: Thu, 15 Nov 2018 07:38:55 +0000

Hi Lou

I agree with you. I listed one possibility in the last email responding to David:

“I would like to point out another option here: maybe we CAN use ECN in DetNet. If the ‘packet pulling’ mechanism is too far away from the current TCP mechanism, whether we can consider to allow the ingress to use ECN as an indication of misbehavior of the talker.”

Best Regards
Xuesong

From: Lou Berger [mailto:[email-masked]]
Sent: Thursday, November 15, 2018 3:14 PM
To: Norman Finn <[email-masked]>; Pascal Thubert (pthubert) <[email-masked]>; Black, David <[email-masked]>
Cc: Gengxuesong (Geng Xuesong) <[email-masked]>; [email-masked]
Subject: Re: [Detnet] pulling packets

----------

> 1. I think there  is a use for a "Sort of like TCP, but is suitable for use as a DetNet flow".

I go one step further and say that there is any use for a TCP congestion control algorithm that is DetNet aware.

Lou

> Le 15 nov. 2018 à 08:13, Black, David <[email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>> a écrit :
>
> Pascal,
>
> Cc: Gengxuesong (Geng Xuesong); [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>
> Le 14 nov. 2018 à 23:41, Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>> a écrit :
> Le 14 nov. 2018 à 22:52, Black, David <[email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>> a écrit :
> Cc: [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>; Black, David
> Le 14 nov. 2018 à 16:53, Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>> a écrit :
> Hello Xuesong:
>
> The service layer performs reordering and duplication elimination functions.. TCP does that too.
> Le 14 nov. 2018 à 14:58, Gengxuesong (Geng Xuesong) <[email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>> a écrit :
> Dear Pascal,
>
> To: [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>; Black, David <David.Black@dell..com<mailto:[email-masked]<mailto:David.Black@dell..com%3cmailto:[email-masked]>>>
> [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>
> [email-masked]<mailto:[email-masked]<mailto:[email-masked]%3cmailto:[email-masked]>>
> ----------

---

From: Lou Berger <[email-masked]>
Date: Thu, 15 Nov 2018 16:12:06 +0900

So I continue to strongly disagree with preof as a transport protocol.  It 
*is* a service provided by DetNet which, just like any other 1+1 service, 
is provided by many other network technologies - that are unfortunately 
referred to as transport *network* technologies.

I do think there is room for a  customized or specific transport protocol, 
and expect this work to take place in the tsvwg.  In discussions with David 
we identify that there might be a need for a new detnet service parameter 
which indicates if the detnet user traffic was not congestion aware, in 
which case DetNet networks would be required to drop such traffic at the 
detnet edge - thus ensuring that the traffic would not eacape the domain.

I will also note that this requirement is inherently met by the current 
preof mechanism size it is strictly limited to an Mpls DetNet/te domain.

Lou

----------
On November 15, 2018 1:41:16 AM "Pascal Thubert (pthubert)" 
<[email-masked]> wrote:

> that was the intention; though I’m unclear if and where I can publish the 
> lower layer pull thing. I wrote it as an eye opener, hoping people would 
> see the issue. I retain Lou’s suggestion that PCF is a beginning for it 
> though it is just a faucet with no control of what goes though when we want 
> a just in time per packet pull.
>
> For your points below, I mentioned IS-IS as an illustration that we can 
> have a same operation at both layers, that is not so unusual. I’m not 
> saying detnet need IS-IS, that was really not the point.
>
> As for the functions I just point out the likeness to illustrate that the 
> detnet service layer is akin though far from identical to existing 
> variations of TCP. Seems we agree on this likeness. I do not read that you 
> oppose my conclusion that the detnet service layer can be implemented as a 
> detnet « transport protocol ».
>
> Le 14 nov. 2018 à 22:52, Black, David 
> <[email-masked]<mailto:[email-masked]>> a écrit :
>
> Bottom Line:  I would suggest focusing discussion of a DetNet “transport 
> protocol” on what has to be done via tunnels to make PREOF work for the IP 
> solution (data plane), and would suggest using separate drafts for this and 
> local loop traffic shaping.
>
> Details inline …
>
> Thanks, --David
>
> Le 14 nov. 2018 à 16:53, Pascal Thubert (pthubert) 
> <[email-masked]<mailto:[email-masked]>> a écrit :
> Hello Xuesong:
>
> The fact that every industrial application that adaptateur from serial but 
> to IP had to implement a jitter elimination buffer is an indication that we 
> failed to provide just that at a lower layer.
>
> Same goes for voice and video. But didn’t you see it on occasions? Those 
> jitter elimination techniques create a latency in the discussion or a delay 
> in the tv stream that you can actually observe.
> [David>] When things work, jitter elimination is common behavior.  When 
> they don’t work, i.e., something doesn’t arrive in time, both retransmit 
> behavior and reaction to missing data are app-specific.
>
>  The service layer provides a flow control, in this particular case a 
>  rate-based one. This is similar to functionality found in DC-TCP or in a 
>  long gone past, IBM’s Rapid Transport Protocol, though we probably need 
>  only a simplified version of that.
> [David>] It’s important to distinguish flow control, which can be provide 
> on a link-by-link basis from congestion control, which is a superset of 
> flow control and can only be provided end-to-end.  As DetNet eliminates all 
> congestion, congestion control is not required.  Specific example – DC-TCP 
> is driven by ECN congestion signals (CE marks) coming back from the 
> network, but if a DetNet node ever marks a DetNet packet as CE, something 
> has gone seriously awry.
>
> The service layer distributes packets and replicas over multiple paths 
> (detnet transports). There’s a hint of that in MP-TCP though we need a 
> powerful addition for replication and coding.
> [David>] DetNet differs because the replication and distribution happen at 
> nodes in the network, not at end systems.  For any TCP-like protocol 
> (reliable delivery with congestion control), packet duplication is always a 
> manifestation of a network “bug,” i.e., divergence from ideal network 
> behavior.  For the RE in DetNet PREOF, packet duplication is a network 
> “feature” ;-).  As of right now, packet duplication and elimination, plus 
> associated reordering (PREOF)  is not part of the IP solution – an 
> additional encapsulation would likely be needed to support that 
> functionality, and the result which would look a lot like a DetNet 
> transport tunnel between nodes in the DetNet network.  If anyone wants to 
> work on this, reading draft-ietf-intarea-tunnels is RECOMMENDED.
>
> [David>] Many of the IP networks over which DetNet operates will not be 
> using IS-IS for IP (layer 3) routing – there are a number of alternatives.
>
> The IEEE may very well provide the same features in the talker, and we may 
> very well be successful at sharing specifications just like we did for 
> IS-IS. But for us I see the detnet service layer packaged as a Layer 4 that 
> the rest of the IETF will call a Transport. Which makes the name collision 
> very unfortunate, by the way.
> [David>] Bottom Line:  I would suggest focusing discussion of a DetNet 
> “transport protocol” on what has to be done via tunnels to make PREOF work 
> for the IP solution (data plane), and would suggest using separate drafts 
> for this and local loop traffic shaping.
>
> Note that my draft also discusses a L2 tokenized loop to avoid the buffer 
> bloat at the detnet ingress, which is the piece that is analogous to the 
> solution shown in the video.
>
> Take care,
>
> Pascal
>
> Le 14 nov. 2018 à 14:58, Gengxuesong (Geng Xuesong) 
> <[email-masked]<mailto:[email-masked]>> a écrit :
> Dear Pascal,
>
> I consider the problem of “DetNet transport layer” all over again, and 
> finally I make myself confused. So I send this email to search for help. 
> Please correct me if you find anything wrong below:
>
> 1.       TSN doesn’t need transport layer, because the talker and listener 
> are TSN aware, and they can send the packets in some particular time slot 
> based on the traffic specification;
>
> 3.       TCP can’t be used for DetNet , because TCP tries to find the “fair 
> bandwidth” for the flows and it makes the flow rate changes all the time, 
> while the sending rate of DetNet is supposed to be constant, and every 
> packet should show up in an expected time slot;
>
> 4.       UDP can’t be used for DetNet, because there is no rate control. If 
> the talker sends the packets in the wrong pattern, UDP can do nothing.
>
> 5.       So we need “DetNet transport layer”, which can :a. Send the packet 
> in a deterministic pattern; b. If anything goes wrong, make it go back to 
> the correct pattern.
>
> 6.       This, finally, makes the talker “DetNet aware”(especially point a 
> above), which is just the same as what happens in TSN.
>
> If this logic chain works, what we actually requires is just a DetNet 
> talker rather than a new transport layer.
>
>
> Best Regards
> Xuesong
>
>
> To: [email-masked]<mailto:[email-masked]>; Black, David 
> <[email-masked]<mailto:[email-masked]>>
> This video https://youtu.be/BO0QhaxBRr0 has a great illustration of the UNI 
> packet pull method that we use in section “5.1.  Local Loop Flow Control” 
> of https://tools.ietf.org/html/draft-thubert-tsvwg-detnet-transport-01 to 
> pull packets from the detnet unaware host from the detnet ingress box. The 
> problem for detnet is that shaping in the detnet ingress can create bloat 
> and uncontrolled end to end latency so we want to control the maximum 
> amount of pending frames to keep the detnet ingress busy and push back to 
> the L4/5 for anything above that.
>
> [email-masked]

---

From: "Gengxuesong (Geng Xuesong)" <[email-masked]>
Date: Thu, 15 Nov 2018 08:10:52 +0000

Hi Pascal,

(Sorry that I responded to the latest email first.)

Thank you for your response, and I agree with you. I think David’s suggestion is great, maybe the draft can be split and keep the ‘flow rate control’ part independently. I think this work is important for applying DetNet in a real network, and more considerations could be added into it.

Best Regards
Xuesong

From: Pascal Thubert (pthubert) [mailto:[email-masked]]
Sent: Wednesday, November 14, 2018 5:53 PM
To: Gengxuesong (Geng Xuesong) <[email-masked]>
Cc: [email-masked]; Black, David <[email-masked]>
Subject: Re: pulling packets

Hello Xuesong:

Take care,

Pascal

Best Regards
Xuesong

Dear all

Pascal

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 15 Nov 2018 09:11:57 +0100

How are you going to calculate the latency for elastic traffic
without a specified DetNet flow envelope ?

Aka: As i said in my prior mail to Michael's review, i don't think
we have working models for elastic traffic in DetNet and even if
we had, i think they would be sufficiently more complex thart it would
make sense to define them as an add-on.

This proposal from Pascal is nice and useful. To me it's some
form of add-on, where i am not sure i havd the right name for it right now,
but i don't think it requires a change to the underlying DDetNet model
of envelope defined flows. It primarily allows TCP flows to be mapped
to DetNet.

Cheers
   Toerless

On Thu, Nov 15, 2018 at 04:12:06PM +0900, Lou Berger wrote:
> *is* a service provided by DetNet which, just like any other 1+1 service, is
> provided by many other network technologies - that are unfortunately
> > that was the intention; though I???m unclear if and where I can publish
> > the lower layer pull thing. I wrote it as an eye opener, hoping people
> > would see the issue. I retain Lou???s suggestion that PCF is a beginning
> > for it though it is just a faucet with no control of what goes though
> > when we want a just in time per packet pull.
> > 
> > have a same operation at both layers, that is not so unusual. I???m not
> > As for the functions I just point out the likeness to illustrate that
> > the detnet service layer is akin though far from identical to existing
> > variations of TCP. Seems we agree on this likeness. I do not read that
> > you oppose my conclusion that the detnet service layer can be
> > implemented as a detnet « transport protocol ».
> > 
> > Bottom Line:  I would suggest focusing discussion of a DetNet
> > ???transport protocol??? on what has to be done via tunnels to make
> > PREOF work for the IP solution (data plane), and would suggest using
> > separate drafts for this and local loop traffic shaping.
> > 
> > Details inline ???
> > 
> > Thanks, --David
> > 
> > The fact that every industrial application that adaptateur from serial
> > but to IP had to implement a jitter elimination buffer is an indication
> > that we failed to provide just that at a lower layer.
> > 
> > Same goes for voice and video. But didn???t you see it on occasions?
> > Those jitter elimination techniques create a latency in the discussion
> > or a delay in the tv stream that you can actually observe.
> > they don???t work, i.e., something doesn???t arrive in time, both
> > retransmit behavior and reaction to missing data are app-specific.
> > 
> > long gone past, IBM???s Rapid Transport Protocol, though we probably
> > need  only a simplified version of that.
> > [David>] It???s important to distinguish flow control, which can be
> > provide on a link-by-link basis from congestion control, which is a
> > superset of flow control and can only be provided end-to-end.  As DetNet
> > eliminates all congestion, congestion control is not required.  Specific
> > example ??? DC-TCP is driven by ECN congestion signals (CE marks) coming
> > back from the network, but if a DetNet node ever marks a DetNet packet
> > as CE, something has gone seriously awry.
> > 
> > [David>] Yes ??? but ??? TCP elimination of duplicate packets is
> > intended only to deal with ???stupid network tricks??? ???
> > 
> > (detnet transports). There???s a hint of that in MP-TCP though we need a
> > [David>] DetNet differs because the replication and distribution happen
> > at nodes in the network, not at end systems.  For any TCP-like protocol
> > (reliable delivery with congestion control), packet duplication is
> > always a manifestation of a network ???bug,??? i.e., divergence from
> > ideal network behavior.  For the RE in DetNet PREOF, packet duplication
> > is a network ???feature??? ;-).  As of right now, packet duplication and
> > elimination, plus associated reordering (PREOF)  is not part of the IP
> > solution ??? an additional encapsulation would likely be needed to
> > support that functionality, and the result which would look a lot like a
> > DetNet transport tunnel between nodes in the DetNet network.  If anyone
> > wants to work on this, reading draft-ietf-intarea-tunnels is
> > RECOMMENDED.
> > 
> > I agree with you that there is an exact same need for TSN. Just like
> > there is a need for IS-IS at both layers.
> > using IS-IS for IP (layer 3) routing ??? there are a number of
> > alternatives.
> > 
> > The IEEE may very well provide the same features in the talker, and we
> > may very well be successful at sharing specifications just like we did
> > for IS-IS. But for us I see the detnet service layer packaged as a Layer
> > 4 that the rest of the IETF will call a Transport. Which makes the name
> > collision very unfortunate, by the way.
> > ???transport protocol??? on what has to be done via tunnels to make
> > PREOF work for the IP solution (data plane), and would suggest using
> > separate drafts for this and local loop traffic shaping.
> > 
> > Note that my draft also discusses a L2 tokenized loop to avoid the
> > buffer bloat at the detnet ingress, which is the piece that is analogous
> > to the solution shown in the video.
> > 
> > Take care,
> > 
> > Pascal
> > 
> > I consider the problem of ???DetNet transport layer??? all over again,
> > and finally I make myself confused. So I send this email to search for
> > help. Please correct me if you find anything wrong below:
> > 
> > 1.       TSN doesn???t need transport layer, because the talker and
> > listener are TSN aware, and they can send the packets in some particular
> > time slot based on the traffic specification;
> > 
> > 3.       TCP can???t be used for DetNet , because TCP tries to find the
> > ???fair bandwidth??? for the flows and it makes the flow rate changes
> > all the time, while the sending rate of DetNet is supposed to be
> > constant, and every packet should show up in an expected time slot;
> > 
> > 4.       UDP can???t be used for DetNet, because there is no rate
> > control. If the talker sends the packets in the wrong pattern, UDP can
> > do nothing.
> > 
> > 5.       So we need ???DetNet transport layer???, which can :a. Send the
> > packet in a deterministic pattern; b. If anything goes wrong, make it go
> > back to the correct pattern.
> > 
> > 6.       This, finally, makes the talker ???DetNet aware???(especially
> > point a above), which is just the same as what happens in TSN.
> > 
> > This video https://youtu.be/BO0QhaxBRr0 has a great illustration of the
> > UNI packet pull method that we use in section ???5.1.  Local Loop Flow
> > Control??? of
> > https://tools.ietf.org/html/draft-thubert-tsvwg-detnet-transport-01 to
> > pull packets from the detnet unaware host from the detnet ingress box.
> > The problem for detnet is that shaping in the detnet ingress can create
> > bloat and uncontrolled end to end latency so we want to control the
> > maximum amount of pending frames to keep the detnet ingress busy and
> > push back to the L4/5 for anything above that.
> > 
> > I hope you???ll love the video as much as I did;
> > 
> > Pascal

-- 
---
[email-masked]

---

From: Mach Chen <[email-masked]>
Date: Thu, 15 Nov 2018 08:41:51 +0000

I agree with that PREOF should not be part of the “DetNet transport protocol”,  the “DetNet transport protocol” should focus on how to handle the mismatch between the sending traffic and the promised traffic pattern.

If the talker/host is TSN/DetNet aware, is the “DetNet transport protocol”  also necessary? In other words, does the “DetNet transport protocol”  only apply to the case where the host/talker is none TSN/DetNet aware?

Best regards,
Mach

From: detnet [mailto:[email-masked]] On Behalf Of Lou Berger
Sent: Thursday, November 15, 2018 3:12 PM
To: Pascal Thubert (pthubert) <[email-masked]>; Black, David <[email-masked]>
Cc: Gengxuesong (Geng Xuesong) <[email-masked]>; [email-masked]
Subject: Re: [Detnet] pulling packets

So I continue to strongly disagree with preof as a transport protocol.  It *is* a service provided by DetNet which, just like any other 1+1 service, is provided by many other network technologies - that are unfortunately referred to as transport *network* technologies.

I do think there is room for a  customized or specific transport protocol, and expect this work to take place in the tsvwg.  In discussions with David we identify that there might be a need for a new detnet service parameter which indicates if the detnet user traffic was not congestion aware, in which case DetNet networks would be required to drop such traffic at the detnet edge - thus ensuring that the traffic would not eacape the domain.

I will also note that this requirement is inherently met by the current preof mechanism size it is strictly limited to an Mpls DetNet/te domain.

Lou

On November 15, 2018 1:41:16 AM "Pascal Thubert (pthubert)" <[email-masked]<mailto:[email-masked]>> wrote:
I agree with the bottom line David,

Pascal

Details inline …

Thanks, --David

[EXTERNAL EMAIL]

Please read:

Regards,

Pascal

Take care,

Pascal

Best Regards
Xuesong

Dear all

Pascal
_______________________________________________
detnet mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/detnet
_______________________________________________
detnet mailing list
[email-masked]<mailto:detnet%40ietf.org>
https://www.ietf.org/mailman/listinfo/detnet

---

From: Lou Berger <[email-masked]>
Date: Thu, 15 Nov 2018 16:14:19 +0900

----------
On November 15, 2018 2:16:35 PM Norman Finn <[email-masked]> 

> 1. I think there  is a use for a "Sort of like TCP, but is suitable for use 

I go one step further and say that there is any use for a TCP congestion 
control algorithm that is DetNet aware.

Lou

> 2. You can run a number of TCP flows over a pipe offering the DetNet 
> service.  More flows will use more of the bandwidth.  Leftover bandwidth is 
> available to best effort flows, but not to other DetNet flows.  This is 
> just like TCP over an Ethernet.  Very useful when allocating "hard" 
> partitioned bandwidth.
>
> 2. You can run one TCP flow as a DetNet flow.  You can run several of them. 
>  Each flow wastes approx. 25% of its bandwidth, which is not reservable by 
> other DetNet flows.  On the other hand, you can control exactly how much 
> bandwidth is available to each TCP flow, and all of the unused bandwidth is 
> available to best-effort TCP flows.  This seems theoretically useful, but 
> I'm not sure that the extra cost over DSCP is worth the difference in 
> service compared to DSCP.
>
> -- Norm
>
> Le 15 nov. 2018 à 08:13, Black, David 
> Thanks for the response – I want to follow up on some of Xuesong’s comments 
> that may shed some light on the lower layer pull functionality:
>
>> 3.  TCP can’t be used for DetNet, because TCP tries to find the “fair 
>> bandwidth” for the flows and it makes the flow rate changes
>> all the time, while the sending rate of DetNet is supposed to be constant, 
>> and every packet should show up in an expected time slot;
>
> Not exactly – in this “game”, DetNet gets to go first, and enforces the 
> time slot behavior.  TCP sees what DetNet provides – it looks like a 
> network path with a max bandwidth that is limited by the reservation for 
> the DetNet flow, a rather stable round-trip-time and pacing/shaping 
> behavior (on the originating host, preferably).  There are some details to 
> be worked out around what happens when TCP tries to overrun the DetNet flow 
> reservation, as that could build up a large queue on the host, but if the 
> reservation is sized to be larger than the TCP throughput, TCP should work 
> as the host queue should drain.
>
> I think Bob Briscoe presented  on Sunday and very clearly what happens with 
> classical TCP - a seesaw pattern that fails to utilize the available 
> bandwidth. This coincides with my own observations when I implemented CIR 
> for IBM’s frame relay box a few years back.
>
> Bob also presented the benefits of DC TCP in that space. I understand that 
> his point matches your words in a corridor discussion we had. I also 
> understand that in the case of the fixed bandwidth. A rate adaptation is a 
> complex and undesired function.
>
> Attempting to achieve the service layer rate enforcement with a 
> window-based operation confuses a function that we have and its derivative 
> that we want, and causes additional buffers and latency between the egress 
> of the window based service layer and the ingress of the load enforcer - 
> the first detnet router.
>
> Conclusion is that we want a simple rate enforcement function in the 
> layer-4 « detnet transport « . Could we call it dettrans to avoid confusion ?
>
>> 4.  UDP can’t be used for DetNet, because there is no rate control. If the 
>> talker sends the packets in the wrong pattern, UDP can do nothing.
> That would be a “feature” here :-) because DetNet is taking care of the 
> details ;-).  In contrast, TCP-like retransmit behavior may not be what’s 
> wanted in a hard real-time system.
>
>> 5.  So we need “DetNet transport layer”, which can :a. Send the packet in a 
>> deterministic pattern; b. If anything goes wrong, make it go back to the 
>> correct pattern.
> Every DetNet router does this. A pacer/shaper should deal nicely with 
> packets being sent in the wrong pattern.   This is preferably located on 
> the originating host, as that’s a better place to build a queue than the 
> first hop switch/router.  Again if the reservation is sized to be larger 
> than the traffic, the originating host queue should drain.
>
> In the reverse direction, where the traffic (TCP or UDP or …) exceeds the 
> reservation, something on the host needs to figure out what’s going on in 
> order to throttle TCP and/or the application, as the usual network 
> congestion feedback techniques (loss, ECN CE marks) don’t work for DetNet 
> traffic.   This is where a network pull mechanism may be a useful part of 
> the solution, as getting the application to understand that it’s 
> overrunning its reservation is preferable to having  the DetNet network 
> start dropping (or CE-marking) packets if a DetNet flow overruns its 
> reservation in order to get the application or transport protocol (e.g., 
> TCP) to slow down.
>
> As for the notion that PREOF for DetNet IP is a DetNet “transport protocol” 
> – I suppose that’s true, but I would suggest use of DetNet “PREOF tunnel 
> protocol” as clearer terminology.
>
> Thanks, --David
>
> Le 14 nov. 2018 à 23:41, Pascal Thubert (pthubert) 
> see the issue. I retain Lou’s suggestion that PCFPFC is a beginning for it 
> <David.Black@dell..com<mailto:[email-masked]>>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 15 Nov 2018 21:32:51 +0000

Hello Xuesong

There seems to be a confusion here:

The frame pulling mechanism is a layer-2 thing, over the UNI interface. It is used to strictly control the amount of queuing and thereby latency at the detnet ingress router. Lou and I discussed that PFC is a beginning of a solution for this but far too brutal in my view; I’d like a PFC with a controlled credit so that the latency can be bounded and so that we are protected against ingress congestion loss.

Our TCP discussion is another thing. The bottom line there is that TCP’s window based flow control is counter productive; a pure rate enforcement would do a much better job. The dettrans draft argues in addition that there’s a need for blocking to optimize the filling of the packets; that is due to the expected DetNet SLA expressed in a max number of packets of a max size per unit of time.

Maybe a reread of the draft would clarify ?

Regards,

Pascal

Le 15 nov. 2018 à 08:39, Gengxuesong (Geng Xuesong) <[email-masked]<mailto:[email-masked]>> a écrit :

Hi Lou

Best Regards
Xuesong

From: Lou Berger [mailto:[email-masked]]
Sent: Thursday, November 15, 2018 3:14 PM
To: Norman Finn <[email-masked]<mailto:[email-masked]>>; Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>>; Black, David <[email-masked]<mailto:[email-masked]>>
Cc: Gengxuesong (Geng Xuesong) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Detnet] pulling packets

----------

Lou

> From: detnet [[email-masked]<mailto:[email-masked]>] on behalf of Pascal Thubert (pthubert) [[email-masked]<mailto:[email-masked]>]

---

From: "Gengxuesong (Geng Xuesong)" <[email-masked]>
Date: Fri, 16 Nov 2018 02:00:47 +0000

Hi Pascal

Thank you for you clarification, I think I have a better understand of your proposal about the packet pulling and TCP modification now. I also agree that PFC is an option to enable communication in the UNI interface. While as I know, PFC is mostly enabled in DC, which is not available for all the network. And PFC if for pushing packets back when congestion happens rather than pulling them out (?).
And what I am trying to say is another story. Maybe we can put these two mechanisms together, all in layer 4. ECN can be a tool to control the queuing of ingress, which will send message to the talker by ACK.

Best Regards
Xuesong

From: Pascal Thubert (pthubert) [mailto:[email-masked]]
Sent: Friday, November 16, 2018 5:33 AM
To: Gengxuesong (Geng Xuesong) <[email-masked]>
Cc: Lou Berger <[email-masked]>; Norman Finn <[email-masked]>; Black, David <[email-masked]>; [email-masked]
Subject: Re: [Detnet] pulling packets

Hello Xuesong

Regards,

Pascal

Best Regards
Xuesong

----------

Lou

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Fri, 16 Nov 2018 09:30:14 +0000

Hello Lou

You certainly have a point. I indicated ways in which PREOF resembles operations we traditionally place at Layer 4. The most important thing in my view being that PREOF enqueues packets to different IP addresses when distributing an application flow over multiple IP-based DetNet transports layers. One may argue that HIP does also something alike in a shim layer between 3 and 4.

Now, there’s also a list of things by which PREOF differs from those found in our traditional L4 protocols. The most important thing is that PREOF is not running only to the end points. There are excursions inside the network at the detnet relays. Placing it at L4 was a convenience to write a single story in one draft in draft-thubert-tsvwg-detnet-transport. If we decide it is a shim at the upper Layer 3 that works too.

In any fashion this work has ramifications at all 3 layers. The question is how to organize it over the different areas and SDOs.

All the best,

Pascal

From: detnet <[email-masked]> On Behalf Of Lou Berger
Sent: jeudi 15 novembre 2018 14:12
To: Pascal Thubert (pthubert) <[email-masked]>; Black, David <[email-masked]>
Cc: Gengxuesong (Geng Xuesong) <[email-masked]>; [email-masked]
Subject: Re: [Detnet] pulling packets

Lou

Pascal

Details inline …

Thanks, --David

[EXTERNAL EMAIL]

Please read:

Regards,

Pascal

Take care,

Pascal

Best Regards
Xuesong

Dear all

Pascal

---

From: "Black, David" <[email-masked]>
Date: Fri, 16 Nov 2018 17:00:56 +0000

My take is that it could make a lot of sense to work on a protocol that provides PREOF for the IP solution.  That protocol will likely involve a tunnel of some form – the Transport Area does care quite a bit about tunnels (many of which involve UDP – see RFC 8085), but we don’t refer to every tunnel as a “transport protocol” – using a more precise term like “IP PREOF tunnel” might reduce the confusion here, as I think an IP-in-IP tunnel with a shim header is a likely approach (outer IP header contains IP address of DetNet node that performs duplicate elimination, inner IP header contains IP destination address of packet after elimination).

Thanks, --David

From: detnet [mailto:[email-masked]] On Behalf Of Pascal Thubert (pthubert)
Sent: Friday, November 16, 2018 4:30 AM
To: Lou Berger; Black, David
Cc: Gengxuesong (Geng Xuesong); [email-masked]
Subject: Re: [Detnet] pulling packets

[EXTERNAL EMAIL]
Hello Lou

All the best,

Pascal

From: detnet <[email-masked]<mailto:[email-masked]>> On Behalf Of Lou Berger
Sent: jeudi 15 novembre 2018 14:12
To: Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>>; Black, David <[email-masked]<mailto:[email-masked]>>
Cc: Gengxuesong (Geng Xuesong) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Detnet] pulling packets

Lou

Pascal

Details inline …

Thanks, --David

[EXTERNAL EMAIL]

Please read:

Regards,

Pascal

Take care,

Pascal

Best Regards
Xuesong

Dear all

Pascal