Subject: Re: [MBONED] deploying internet-wide multicast (RE: deprecating MSDP)

From: Stig Venaas <[email-masked]>
Date: Tue, 7 Nov 2017 15:13:27 -0800

Hi

I don't think AMT needs to be more reliable than regular multicast. If
you are concerned about Wi-Fi, then by using AMT to the device, the
packets will be unicast packets, so the regular Wi-Fi handling of
unicast packets (including retransmissions) should be sufficient. Do
you need additional retransmissions?

Stig

On Thu, Sep 7, 2017 at 8:16 PM, Mikael Abrahamsson <[email-masked]> wrote:
> On Thu, 7 Sep 2017, Manfredi, Albert E wrote:
>
>> users is via unicast, and the stream protocol depends on the user's
>
>
> Here is another thing. Most of the traffic to user devices traverse at least
> one wifi hop. The unfortunate truth is that multicast doesn't work well on
> todays wifi environment (I raised this issue within the IETF close to two
> years ago, and we're still nowhere near anything I would say even remotely
> looks like a widespread solution).
>
> So yes, AMT tunnels to unicast everything from the ISP to the devices might
> work. They would have to be initiated by the device/app itself (to aid
> initial deployment).
>
> 6to4 was decided to be a really bad way of doing IPv6 (public relays nobody
> knew how well they worked, they were badly maintained etc).
>
> AMT looks like it was designed the same way. Now, with AMT you don't have
> the problem with return path uncertainty that 6to4 had, so that is less of a
> problem.
>
> What I see lacking here is reliable transport. If we're going to do this,
> the client is going to have to be able to request retransmits to work around
> unreliable delivery of packets.
>
> How do we do that? 5 second buffer in the AMT relay (or a device sitting
> next to the relay) that the client can request re-transmits from? It would
> need to be stateful somehow. We'd also need some kind of identifier/sequence
> number in each AMT packet so the end device can tell that a packet was
> missing, and can request a retransmit of this.
>
> Thoughts?
>
> --
> Mikael Abrahamsson    email: [email-masked]
>
> _______________________________________________
> MBONED mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mboned

---

From: Stig Venaas <[email-masked]>
Date: Tue, 7 Nov 2017 15:23:38 -0800

Hi

There may be some room for improvement for 4610, but I think it's a
bad idea to encapsulate data packets in TCP. I don't get why you would
need that reliability. Does it need to be more reliable than regular
multicast? However, it might be a good idea to use TCP to avoid
periodic signaling and send just a single SA message without data.
Basically doing a single NULL-register by FHR to RP when the source
becomes active, and relaying this to other RPs. And of course,
withdrawing the SA when source is no longer active. Alternatively
doing this just between the Anycast-RPs, not involving FHR to RP.

Stig

On Mon, Sep 11, 2017 at 9:54 AM, Toerless Eckert <[email-masked]> wrote:
> Right. So it would not enable new content owners to come in and get
> an easier market entry to reach customers for live streaming... sitting
> in front of STBs.
>
> On Mon, Sep 11, 2017 at 06:36:38PM +0200, Mikael Abrahamsson wrote:
>> On Mon, 11 Sep 2017, Mikael Abrahamsson wrote:
>>
>> >There is URL I know of.
>>
>> There is *no* URL that I know of.
>>
>> This isn't volume business afaik. I don't remember which one it was,
>> but it was basically in cooperation with one of the larger typical
>> "channel package providers" you might get on coax TV. One of Canal
>> Digital, VIASAT or someone like that.
>>
>> --
> ---
> [email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 8 Nov 2017 02:29:03 +0100

Thanks Mikael.  Would be great to see if we can close any gaps
to get towards more deployments!

When you say 6to4 failed and AMT is similar, i assume that there is more
to it than the reliability aspect you discuss afterwards. Would be good
to get more insight into any other aspects you think killed 6to4 and that
we may want to reconsider for AMT.

As Stig mentioned, there is fairly good retransmission across WiFi
for unicast packets, so merely by relying on AMT you get that benefit.
I have not seen good analysis whether or not this can be deemded to
be sufficient. I wold guess that it is buffering far less traffic than
a few seconds, so i am not discounting your approach, but it would be
great to have explicit testing data to make the case for what you
propose.

Lets assume get the data to show WiFi unicast retransmission itself
is insufficient and unicast is only reliable enough because of the
combination of TCP and WiFi unicast retransmission:

I hesitate to invent new retransmit mechanisms. Should we not rather
use an existing reliable transport such as TCP or SCTP ? Admittedly,
RTP also implemented its own retransmit schemes for the same reasons,
i wonder how good that experience was. 

The main issue IMHO is that i would like for clients to be able to
connect to arbitrary remote AMT relays because different relays can
support different content. We can do this with todays AMT approach.
And its left up to the client library to determine which relay to connect
to for which content. So no network config needed if we do a direct AMT
tunnel from client to relay.

If we want WiFi-reliability from home gateway to client then we can
easily loose this flexibility. Either the tunnel is reliable from the
remote relay to the client, so that you retransmit across the whole path
when the wifi blurbs. Thats IMHO terrible. Or you put the relay into the
home gateway, and then you need some scheme for how the home gateway
builds another AMT tunnel to the remote relay. And a scheme how the
client can indciate for each content the remote relay address.

SO, if you ask me, i would just try to tunnel the original AMT tunnel
between client and remote AMT relay into a TCP tunnel between client
and home gateway. Not sure what the easiest reused tunnel method is.
I would love SOCKS, but it does not define tunneling of UDP over TCP
(just over UDP). One could use just that "whatever" VPN tunnel that
can use TCP. OpenConnect for example falls back to TCP.

I have always missed a simple, non-encrypted/strong-authenticated
TCP tunnel option in network products *sigh*.

Cheers
    Toerless

On Fri, Sep 08, 2017 at 05:16:00AM +0200, Mikael Abrahamsson wrote:
> Here is another thing. Most of the traffic to user devices traverse
> at least one wifi hop. The unfortunate truth is that multicast
> doesn't work well on todays wifi environment (I raised this issue
> within the IETF close to two years ago, and we're still nowhere near
> anything I would say even remotely looks like a widespread
> solution).
> 
> So yes, AMT tunnels to unicast everything from the ISP to the
> devices might work. They would have to be initiated by the
> device/app itself (to aid initial deployment).
> 
> 6to4 was decided to be a really bad way of doing IPv6 (public relays
> nobody knew how well they worked, they were badly maintained etc).
> 
> AMT looks like it was designed the same way. Now, with AMT you don't
> have the problem with return path uncertainty that 6to4 had, so that
> is less of a problem.
> 
> What I see lacking here is reliable transport. If we're going to do
> this, the client is going to have to be able to request retransmits
> to work around unreliable delivery of packets.
> 
> How do we do that? 5 second buffer in the AMT relay (or a device
> sitting next to the relay) that the client can request re-transmits
> from? It would need to be stateful somehow. We'd also need some kind
> of identifier/sequence number in each AMT packet so the end device
> can tell that a packet was missing, and can request a retransmit of
> this.
> 
> Thoughts?
> 
> -- 

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 8 Nov 2017 02:45:47 +0100

Improving on 4610 is a bit like proposing fixes for 

Be happy that you didn't have to sit through the years of IETF discussing
the MSDP encasulated packets. Which is the dicsussion that ultimately
made MSDP fall froms tandard to experimental. Its really like discussing how
to best deal with horse manure. You really need to accept that there are no
good solutions other than changing your motor. Aka: the problem is
the data triggered PIM-SM (S,G) state building, not MSDP. MSDP just
tries to carry the horse manure over longer trips ;-).

I would just want to engineer multicast state building on top of reliable transport,
so i do not have to deal with all the complexities of even predicting the
behavior under loss. Let alone optimizing/perfecting.

Just make all RP and LHR BGP speakers (RPs = RR) and use a BGP address family
to carry (S,G) SA state. Probably only small changes over how MVPN is done
with BGP. Just need to decide on how to do (*,G). 

Cheers
    Toerless

-- 
---
[email-masked]

---

From: <[email-masked]>
Date: Wed, 8 Nov 2017 10:04:46 +0800 (CST)

Hi Toerless,

It may has been solved in RFC7716. :)

Best Regards,
Sandy

------------------原始邮件------------------
发件人： <[email-masked]>;
收件人： <[email-masked]>;
抄送人： <[email-masked]>;
日 期 ：2017年11月08日 09:46
主 题 ：Re: [MBONED] deprecating MSDP
Improving on 4610 is a bit like proposing fixes for

Cheers
Toerless

--
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 8 Nov 2017 03:17:37 +0100

Or at least a good starting point, yes. In doubt an SP solution for PEs is
likely designed to require a lot of configuration, so something more easily
self-configuring may be required for enterprises or other environments.
But thats just my prejudice speaking. Have to read through it in more detail.

Thanks!
    Toerles

On Wed, Nov 08, 2017 at 10:04:46AM +0800, [email-masked] wrote:
> ------------------????????????------------------
> ???????????? <[email-masked]>;
> ???????????? <[email-masked]>;
> ???????????? <[email-masked]>;
> ??? ??? ???2017???11???08??? 09:46
> ??? ??? ???Re: [MBONED] deprecating MSDP

-- 
---
[email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 8 Nov 2017 10:50:34 +0100 (CET)

> I don't think AMT needs to be more reliable than regular multicast. If

Deploying multicast for linear TV without FEC is an operational nightmare 
that never stops. You need to constantly measure customer experience 
(preferrably in the STBs, all of this with proprietary solutions afaik) or 
try to measure it in your network (using expensive equipement).

I recommend all my enemies to deploy this at large scale.

> unicast packets (including retransmissions) should be sufficient. Do you 
> need additional retransmissions?

Unicast will be far superior to multicast over wifi, but it's probably 
still not enough to give a good enough customer experience without FEC.

FEC in this aspect means the application deteriorates gracefully under 
packet loss, by smearing out infomation across packets and degrading 
quality without for instance magenta squares gliding along the screen 
until the next full screen update (that might be a second into the 
future (or more)).

This "FEC" can also be (as some have done), by the devices receiving 
multicast packets and then requesting (via unicast) from a server of some 
kind, to get the packets it lost. The few solutions available when I was 
involved in this before, were all proprietary. They were also used to 
speed up channel changes by providing full screen update immediately, 
before same information came via the multicast stream.

In packet networks, packets are lost. It's a fact. They're designed to 
behave this way. From what I can see, there are two ways to handle this. 
Either we come up with a standards based retransmission mechanism for 
multicast streams that anyone can use, or we punt the problem to the 
application layer, and let them figure it out. We need to be perfectly 
clear to everybody what our choice is.

---

From: Leonard Giuliano <[email-masked]>
Date: Wed, 8 Nov 2017 06:37:22 -0800

Indeed, RFC7716 is a good start.  And 
draft-zzhang-bess-bgp-multicast-01.txt goes even further to completely 
replace PIM with BGP as a multicast routing protocol.

On Wed, 8 Nov 2017, Toerless Eckert wrote:

| Or at least a good starting point, yes. In doubt an SP solution for PEs is
| likely designed to require a lot of configuration, so something more easily
| self-configuring may be required for enterprises or other environments.
| But thats just my prejudice speaking. Have to read through it in more detail.
| 
| Thanks!
|     Toerles
| 
| On Wed, Nov 08, 2017 at 10:04:46AM +0800, [email-masked] wrote:
| > Hi Toerless,
| > 
| > >Just make all RP and LHR BGP speakers (RPs = RR) and use a BGP address family
| > to carry (S,G) SA state. Probably only small changes over how MVPN is done
| > with BGP. Just need to decide on how to do (*,G).
| > 
| > It may has been solved in RFC7716. :)
| > 
| > Best Regards,
| > Sandy
| > 
| > ------------------????????????------------------
| > ???????????? <[email-masked]>;
| > ???????????? <[email-masked]>;
| > ???????????? <[email-masked]>;
| > ??? ??? ???2017???11???08??? 09:46
| > ??? ??? ???Re: [MBONED] deprecating MSDP
| > Improving on 4610 is a bit like proposing fixes for
| > 
| > Be happy that you didn't have to sit through the years of IETF discussing
| > the MSDP encasulated packets. Which is the dicsussion that ultimately
| > made MSDP fall froms tandard to experimental. Its really like discussing how
| > to best deal with horse manure. You really need to accept that there are no
| > good solutions other than changing your motor. Aka: the problem is
| > the data triggered PIM-SM (S,G) state building, not MSDP. MSDP just
| > tries to carry the horse manure over longer trips ;-).
| > 
| > I would just want to engineer multicast state building on top of reliable transport,
| > so i do not have to deal with all the complexities of even predicting the
| > behavior under loss. Let alone optimizing/perfecting.
| > 
| > Just make all RP and LHR BGP speakers (RPs = RR) and use a BGP address family
| > to carry (S,G) SA state. Probably only small changes over how MVPN is done
| > with BGP. Just need to decide on how to do (*,G).
| > 
| > Cheers
| > Toerless
| > 
| > On Tue, Nov 07, 2017 at 03:23:38PM -0800, Stig Venaas wrote:
| > > Hi
| > >
| > > There may be some room for improvement for 4610, but I think it's a
| > > bad idea to encapsulate data packets in TCP. I don't get why you would
| > > need that reliability. Does it need to be more reliable than regular
| > > multicast? However, it might be a good idea to use TCP to avoid
| > > periodic signaling and send just a single SA message without data.
| > > Basically doing a single NULL-register by FHR to RP when the source
| > > becomes active, and relaying this to other RPs. And of course,
| > > withdrawing the SA when source is no longer active. Alternatively
| > > doing this just between the Anycast-RPs, not involving FHR to RP.
| > >
| > > Stig
| > >
| > >
| > > On Mon, Sep 11, 2017 at 9:54 AM, Toerless Eckert <[email-masked]> wrote:
| > > > Right. So it would not enable new content owners to come in and get
| > > > an easier market entry to reach customers for live streaming... sitting
| > > > in front of STBs.
| > > >
| > > > On Mon, Sep 11, 2017 at 06:36:38PM +0200, Mikael Abrahamsson wrote:
| > > >> On Mon, 11 Sep 2017, Mikael Abrahamsson wrote:
| > > >>
| > > >> >There is URL I know of.
| > > >>
| > > >> There is *no* URL that I know of.
| > > >>
| > > >> This isn't volume business afaik. I don't remember which one it was,
| > > >> but it was basically in cooperation with one of the larger typical
| > > >> "channel package providers" you might get on coax TV. One of Canal
| > > >> Digital, VIASAT or someone like that.
| > > >>
| > > >> --
| > > >> Mikael Abrahamsson    email: [email-masked]
| > > >
| > > > --
| > > > ---
| > > > [email-masked]
| > > >
| > > > _______________________________________________
| > > > MBONED mailing list
| > > > [email-masked]
| > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_mboned&d=DwICAg&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=iw2TU3OZ0CDpCbqeV23zdah2FoG9Do-zEmGgWTaavDg&m=rQDiyMm1c_DRTBSb1SUMotzGXWVNBJzio6Mq8C1Bfos&s=fkvyOdq3F14iAD1iYb5R_OgEXUNZetf_0PA3BmeiEoU&e=
| > 
| > --
| > ---
| > [email-masked]
| > 
| > _______________________________________________
| > MBONED mailing list
| > [email-masked]
| > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_mboned&d=DwICAg&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=iw2TU3OZ0CDpCbqeV23zdah2FoG9Do-zEmGgWTaavDg&m=rQDiyMm1c_DRTBSb1SUMotzGXWVNBJzio6Mq8C1Bfos&s=fkvyOdq3F14iAD1iYb5R_OgEXUNZetf_0PA3BmeiEoU&e=
| 
| 
| 
| -- 
| ---
| [email-masked]
| 
| _______________________________________________
| MBONED mailing list
| [email-masked]
| https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_mboned&d=DwICAg&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=iw2TU3OZ0CDpCbqeV23zdah2FoG9Do-zEmGgWTaavDg&m=rQDiyMm1c_DRTBSb1SUMotzGXWVNBJzio6Mq8C1Bfos&s=fkvyOdq3F14iAD1iYb5R_OgEXUNZetf_0PA3BmeiEoU&e=
|

---

From: Stig Venaas <[email-masked]>
Date: Wed, 8 Nov 2017 07:51:17 -0800

Hi

I only meant that with AMT and Wi-Fi, the experience should hopefully
be equivalent to regular native multicast. I agree you often need FEC
to protect against loss, and also in some cases some kind of
retransmission or unicast assistance. Assuming there is no loss in the
core until an AMT relay is reached, then I suppose an AMT specific
solution could work, but I would like a generic solution that can work
for native multicast. I don't remember exactly what was done for
reliable multicast in the IETF now. Anyone remember better than me? I
remember proposals including how to do retransmissions.

Stig

On Wed, Nov 8, 2017 at 1:50 AM, Mikael Abrahamsson <[email-masked]> wrote:
>> packets will be unicast packets, so the regular Wi-Fi handling of unicast
>> packets (including retransmissions) should be sufficient. Do you need
>> additional retransmissions?
>
>
> Unicast will be far superior to multicast over wifi, but it's probably still
> not enough to give a good enough customer experience without FEC.
>
> packet loss, by smearing out infomation across packets and degrading quality
> without for instance magenta squares gliding along the screen until the next
> full screen update (that might be a second into the future (or more)).
>
> involved in this before, were all proprietary. They were also used to speed
> up channel changes by providing full screen update immediately, before same
> information came via the multicast stream.
>
> application layer, and let them figure it out. We need to be perfectly clear
> to everybody what our choice is.
>
>
> --

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 8 Nov 2017 18:22:23 +0000

Hi,

FLUTE and NORM are the reliable multicast transports I’ve seen:
https://tools.ietf.org/html/rfc6726
https://tools.ietf.org/html/rfc5740

AFAIK neither has wide usage, but there’s implementations:
http://mad.cs.tut.fi/download.html
https://www.nrl.navy.mil/itd/ncs/products/norm

If I understand correctly, Cablelabs’s IP Multicast ABR system uses a sort of bastardized version of NORM with a path for out-of-band unicast repair that doesn’t let receivers prevent the transmit window from sliding forward, since the streaming video use case can’t use the norm approach of letting receivers slow down the transmit:
https://apps.cablelabs.com/specification/?category=VIDEO&subcat=IP%20MULTICAST&query=&doctype=&content=false&archives=false

There’s some more speculative ongoing work I know of that might be relevant.
https://tools.ietf.org/html/draft-pardue-quic-http-mcast-01
https://tools.ietf.org/html/draft-ietf-tsvwg-rlc-fec-scheme-01

I think FLUTE or NORM implementations could be made to work today for a non-time-critical download service, if there were multicast connectivity.

I agree with Mikael that there’s more work to do before ordinary implementations of open standards can be used to operate a video service over multicast IP. I do want to operate a multicast video service, but not without FEC or some kind of repair.

I also agree that AMT can be a helpful way to get multicast IP over wifi as unicast at layer 2 (though I would rather see native multicast to the home and either AMT relay or DMS (or both) at the wifi router, instead of AMT relays in the ISP, because multicast can help a lot on the access network for PON or CMTS, though I do take the point that in some cases relay in the ISP is more practical to deploy).

But I agree with Stig that it’s not very helpful to make AMT reliable, as compared to making a sufficiently reliable multicast transport protocol that can work for streaming video.

-Jake

On 11/8/17, 7:53 AM, "Stig Venaas" <[email-masked]> wrote:

Hi

Stig

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 8 Nov 2017 10:32:03 -0800

Note I use LISP all the time to get multicast over Wifi via https://datatracker.ietf.org/doc/draft-ietf-lisp-signal-free-multicast/.

If anyone wants to try it, please send me a private note.

Dino

> On Nov 8, 2017, at 10:22 AM, Holland, Jake <[email-masked]> wrote:
> 
> Hi,
>

---

From: Greg Shepherd <[email-masked]>
Date: Wed, 8 Nov 2017 10:54:33 -0800

On Wed, Nov 8, 2017 at 10:32 AM, Dino Farinacci <[email-masked]> wrote:

> Note I use LISP all the time to get multicast over Wifi via
> https://datatracker.ietf.org/doc/draft-ietf-lisp-signal-free-multicast/.
>

Exactly. ANY unicast encap of multicast over wifi will gain from the wifi
loss retransmission.

Greg

> > If I understand correctly, Cablelabs’s IP Multicast ABR system uses a
> sort of bastardized version of NORM with a path for out-of-band unicast
> repair that doesn’t let receivers prevent the transmit window from sliding
> forward, since the streaming video use case can’t use the norm approach of
> letting receivers slow down the transmit:
> > https://apps.cablelabs.com/specification/?category=VIDEO&
> subcat=IP%20MULTICAST&query=&doctype=&content=false&archives=false
> >
> > I think FLUTE or NORM implementations could be made to work today for a
> non-time-critical download service, if there were multicast connectivity.
> >
> >
> > I agree with Mikael that there’s more work to do before ordinary
> implementations of open standards can be used to operate a video service
> over multicast IP. I do want to operate a multicast video service, but not
> without FEC or some kind of repair.
> >
> > I also agree that AMT can be a helpful way to get multicast IP over wifi
> as unicast at layer 2 (though I would rather see native multicast to the
> home and either AMT relay or DMS (or both) at the wifi router, instead of
> AMT relays in the ISP, because multicast can help a lot on the access
> network for PON or CMTS, though I do take the point that in some cases
> relay in the ISP is more practical to deploy).
> >
> > But I agree with Stig that it’s not very helpful to make AMT reliable,
> as compared to making a sufficiently reliable multicast transport protocol
> that can work for streaming video.
> >
> > -Jake
> >
> unicast
> still
> quality
> speed

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 8 Nov 2017 21:34:33 +0100

On Wed, Nov 08, 2017 at 10:50:34AM +0100, Mikael Abrahamsson wrote:
> Deploying multicast for linear TV without FEC is an operational
> nightmare that never stops.

I'd limit this to "over non-sp-controlled networks" (e.g.: home). It
works fine since a decade in e.g.: cable networks from headend to fiber/coax
edge for example.

> You need to constantly measure customer
> experience (preferrably in the STBs, all of this with proprietary
> solutions afaik) or try to measure it in your network (using
> expensive equipement).

What standard mechanisms are used to diagnose unicast/ABR into STBs
such as self-unfriendlyness congestion issues ?

But pointing to others aside: Should definitely have a well defined standards & diagnostics
stack profile for multicast TV reception. Not sure if this is a job for Mboned or
some ART group though.

> I recommend all my enemies to deploy this at large scale.

Any published experience from the last 6? years of T-Entertain? IP multicast service ?

[...]
> Unicast will be far superior to multicast over wifi, but it's
> probably still not enough to give a good enough customer experience
> without FEC.
> 
> FEC in this aspect means the application deteriorates gracefully
> under packet loss, by smearing out infomation across packets and
> degrading quality without for instance magenta squares gliding along
> the screen until the next full screen update (that might be a second
> into the future (or more)).

Slick brand marketing!

   "Real-time magenta subscriber diagnostics of wifi problems".

> multicast packets and then requesting (via unicast) from a server of
> some kind, to get the packets it lost. The few solutions available
> when I was involved in this before, were all proprietary. They were
> also used to speed up channel changes by providing full screen
> update immediately, before same information came via the multicast
> stream.

The MSTV solution was proprietary. IETF did standardize RTP extensions
to allow for standardized implementation of these mechanisms: RTCP
retransmission requests to be used both for randomn loss (eg: by DSL/ATM
link) as well as "retransmiss" from last IDR frame after channel change.
Look for RFCs with Ali Begen on it. 

The availability of this standardized RTP/RTCP option of course is
a possible challenger for trying to redo this with AMT or similar.
Especially if we can not figure out how to support the channel change
retransmission equall well. 

These retransmission mechanisms are really orthogonal to FEC but can of
course be combined.

Fixed rate FEC is pretty good for predictable BER cases but not a good
choice for highly irregular and bursty packet loss like on WiFi.
Which is why WiFi itself opted for a retransmission scheme. 

> In packet networks, packets are lost. It's a fact. They're designed
> to behave this way.

Even unicast reliable transport is built with a certain
expectation of reliability in the network layer. One rule is
that network layer protocols must not have repeatable predictable loss.
This was why we dragged on forever with the MSDP data-in-SA discussion.

In reality, the more important loss limit is IMHO the amount of
random (non-congestion caused) loss that transport protocols will support
without assuming congestion - aka: don't downspeed.

If i remember correctly, TCP starts to speed down if it sees loss
while still repairing another loss. So maybe 1 loss per RTT is ok.
<= 200 msec e2e RTT for internet traffic, 20 TCP connections in parallel
== < 1 packet/10 msec random loss.

> From what I can see, there are two ways to
> handle this. Either we come up with a standards based retransmission
> mechanism for multicast streams that anyone can use, or we punt the
> problem to the application layer, and let them figure it out. We
> need to be perfectly clear to everybody what our choice is.

First of all, there are the above mentioned RT(C)P retransmission
schemes, so the question is why those are insufficient:

- automatic feature on home-gateways: could we make these
  RTP retransmissions work automatic on a home-gateway, eg:
  if native IP multicast was used. For random WiFi loss i think
  we can. For fast channel change, i think the RT(C)P feature
  on the home gateway would need to punt the retransmission requests
  back to a cache in the SP network.

- automatic feature on home-gateways in the face of AMT into the
  client device. No change if home gateway is relay. More difficult
  when it is not.

- Use of non-RTP based multicast. Relevant or not ?  If it is relevant
  then we'd have to resolve to some non-rtp based mechanism and maybe some
  scheme like i mentioned in prior email wih TCP might be the best solution.

Live and let Die with RTP [Y/N] ?

Cheers
    Toerless

-- 
---
[email-masked]

---

From: "Morten V. Pedersen" <[email-masked]>
Date: Wed, 8 Nov 2017 21:59:15 +0100

Hi,
In general providing reliable multicast over WiFi is a challenging 
problem. We did our own implementation of such a transport some years 
back. We have a bit of public information on it here:
http://www.steinwurf.com/products/score.html

Multicast over WiFi is very challenging also due to the way 802.11 
treats multicast traffic:
https://www.ietf.org/id/draft-mcbride-mboned-wifi-mcast-problem-statement-01.txt

In my experience it is hard to avoid some form of FEC (erasure coding) 
if the multicast needs to be able to scale up to a substantial amount of 
receivers.

All the best,
Morten

On 11/08/2017 07:54 PM, Greg Shepherd wrote:
>
>
> On Wed, Nov 8, 2017 at 10:32 AM, Dino Farinacci <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     https://datatracker.ietf.org/doc/draft-ietf-lisp-signal-free-multicast/
>     <https://datatracker.ietf.org/doc/draft-ietf-lisp-signal-free-multicast/>.
>
> Exactly. ANY unicast encap of multicast over wifi will gain from the 
> wifi loss retransmission.
>
> Greg
>
>     > On Nov 8, 2017, at 10:22 AM, Holland, Jake <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >
>     > Hi,
>     >
>     <https://tools.ietf.org/html/rfc6726>
>     <https://tools.ietf.org/html/rfc5740>
>     >
>     <http://mad.cs.tut.fi/download.html>
>     <https://www.nrl.navy.mil/itd/ncs/products/norm>
>     >
>     > If I understand correctly, Cablelabs’s IP Multicast ABR system
>     uses a sort of bastardized version of NORM with a path for
>     out-of-band unicast repair that doesn’t let receivers prevent the
>     transmit window from sliding forward, since the streaming video
>     use case can’t use the norm approach of letting receivers slow
>     down the transmit:
>     >
>     <https://apps.cablelabs.com/specification/?category=VIDEO&subcat=IP%20MULTICAST&query=&doctype=&content=false&archives=false>
>     >
>     <https://tools.ietf.org/html/draft-pardue-quic-http-mcast-01>
>     <https://tools.ietf.org/html/draft-ietf-tsvwg-rlc-fec-scheme-01>
>     >
>     > I think FLUTE or NORM implementations could be made to work
>     today for a non-time-critical download service, if there were
>     multicast connectivity.
>     >
>     >
>     implementations of open standards can be used to operate a video
>     service over multicast IP. I do want to operate a multicast video
>     service, but not without FEC or some kind of repair.
>     >
>     > I also agree that AMT can be a helpful way to get multicast IP
>     over wifi as unicast at layer 2 (though I would rather see native
>     multicast to the home and either AMT relay or DMS (or both) at the
>     wifi router, instead of AMT relays in the ISP, because multicast
>     can help a lot on the access network for PON or CMTS, though I do
>     take the point that in some cases relay in the ISP is more
>     practical to deploy).
>     >
>     > But I agree with Stig that it’s not very helpful to make AMT
>     reliable, as compared to making a sufficiently reliable multicast
>     transport protocol that can work for streaming video.
>     >
>     > -Jake
>     >
>     > On 11/8/17, 7:53 AM, "Stig Venaas" <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >
>     > Hi
>     >
>     > On Wed, Nov 8, 2017 at 1:50 AM, Mikael Abrahamsson
>     <[email-masked] <mailto:[email-masked]>> wrote:
>     nightmare
>     probably still
>     <mailto:[email-masked]>
>     >
>     > [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/mboned>
>     >
>     >
>     > [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/mboned>
>
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/mboned>
>
>
>
>

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 8 Nov 2017 22:07:19 +0100

The CableLabs solution does not carry IP multicast across the home network.
It makes the home gateway an HTTP proxy where IP multicast + NORM is used
on the uplink to retrieve the content in a scalable fashion between content
server and gateways - via a fairly well controlled SP network only.
The amount of loss that NORM has to cope with should be very small, but of
course it needs to be 0 to support the use case, thats why you still need a
reliable transport protocol.

I was always hoping that there was still a chance to build
the transport solution for rate adaptive video on top of RTP because it
would create a lot less ABR problems like overhead and delay (talk about sports
and delay). I thought there are some solution architectures on the mobile side
that may use RTP, but not quite sure. I've helped build solutions that
do use rate adaptive RTP and i thought that worked very well (low latency et.al.).

For walled garden IPTV use cases, the cablelabs approach should be quite attractive 
you just need to eliminate all the cable specific pieces you don't need, which
shouldn't be difficult. You just won't get a good score on the soccer-goal-delay rankings. 

If we're talking federated or OTT video which is what AMT was looking to support,
then this cablelabs approach direction would rather be an alternative: Just put
an HTTP cache on your home-gateway  to enable local retransmissions from cache
on the gateway and rely on TCP (HTTP(s)) from gateway to client. No multicast
needed.

Cheers
    Toerless

On Wed, Nov 08, 2017 at 06:22:23PM +0000, Holland, Jake wrote:
> Hi,
> 
> FLUTE and NORM are the reliable multicast transports I???ve seen:
> AFAIK neither has wide usage, but there???s implementations:
> If I understand correctly, Cablelabs???s IP Multicast ABR system uses a sort of bastardized version of NORM with a path for out-of-band unicast repair that doesn???t let receivers prevent the transmit window from sliding forward, since the streaming video use case can???t use the norm approach of letting receivers slow down the transmit:
> There???s some more speculative ongoing work I know of that might be relevant.
> I agree with Mikael that there???s more work to do before ordinary implementations of open standards can be used to operate a video service over multicast IP. I do want to operate a multicast video service, but not without FEC or some kind of repair.
> 
> But I agree with Stig that it???s not very helpful to make AMT reliable, as compared to making a sufficiently reliable multicast transport protocol that can work for streaming video.
> 
> -Jake
> 

-- 
---
[email-masked]

---

From: "Holland, Jake" <[email-masked]>
Date: Wed, 8 Nov 2017 22:00:00 +0000

+1 on hoping for RTP. And maybe we should cc payload.

I still don’t know enough about RTP or video coding to be certain I’m making sense, but from what I understand, the adaptive RTP solutions today have sender back off under congestion? It seems to me that would need to change for multicast.

I think the Scalable Video Coding that’s part of VP9 (and optional in h.264) is a really promising approach for doing ABR with multicast RTP. In theory, if it were supported and the sender were sending the right way, in response to congestion indications like loss or CE markings, the client could leave (S,G)s corresponding to higher-resolution enhancement layers and still keep the lower-resolution video layers at a bandwidth level that works well for his path. If you got it to work well, maybe the purple squares or video skips could be at more like the frequency of the player buffering you’d get in unicast, and mostly only happening when network conditions change.

As I understand it, that’s not possible today, but if I’m reading the latest diffs correctly, support for the concept has been added to the vp9 payload draft, so it might become possible if somebody takes up the work?
https://tools.ietf.org/html/draft-ietf-payload-vp9-04#section-4.5

I’m not sure about the reliability, because it seems like you’d still get those purple squares on lost packets, but you could at least reduce the incidence of lost packets due to congestion, and maybe that would make it possible for a little bit of FEC to keep up under any reasonably stable network. But I imagine that improving the reliability will be another project of its own, and there’s a bad tradeoff with the delay. But maybe it’s possible to get reliability with less delay than reliable unicast segmented video has to pay.

This is precisely what I mean by “there’s work to do before normal implementations of open standards can do this”. To me it looks probably achievable but not imminent.

-Jake

On 11/8/17, 1:07 PM, "Toerless Eckert" <[email-masked]> wrote:

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 8 Nov 2017 23:26:52 +0100

On Wed, Nov 08, 2017 at 10:00:00PM +0000, Holland, Jake wrote:
> I still don???t know enough about RTP or video coding to be certain I???m making sense, but from what I understand, the adaptive RTP solutions today have sender back off under congestion? It seems to me that would need to change for multicast.

No, you do not slow down the sender.

RTP schemes are just like ABR schemes, eg: multiple bitrates availalable,
receiver "joins" to the bitrate(s) it can receive without congestion.

Key difference to ABR schemes is that you can docode after usually
every packet you receive as opposed to ABR where you can decode only after
the whole block is received (2 secs or longer usually). And with RTP
you can much easier work in content aware redundancy schemes. E.g.: in ABR
you need 100% reliability and therefore use retransmission like NORM/FLUTE
to get all pakets in a block. In RTP you can decide frame-by-frame
whether a loss of data from that frame results in a visual degradation
that demands retransmisssion or whethre you'd rather not retransmit but
for example interpolate it. And the encoding of different frames can be
done to support such more flexible & dynamic quality vs. delay vs.
burstyness/retransmission choices.

> I think the Scalable Video Coding that???s part of VP9 (and optional in h.264) is a really promising approach for doing ABR with multicast RTP. In theory, if it were supported and the sender were sending the right way, in response to congestion indications like loss or CE markings, the client could leave (S,G)s corresponding to higher-resolution enhancement layers and still keep the lower-resolution video layers at a bandwidth level that works well for his path. If you got it to work well, maybe the purple squares or video skips could be at more like the frequency of the player buffering you???d get in unicast, and mostly only happening when network conditions change.

No idea if VP9 is introducing something new or if this is all just what
we had in MPEG SVC for a decade. I'd hope that they simply copied the
best options and made the IPR free. I'd be surprised though if they had
specific support for some of the low-latency variable loss support details
that commercial MPEG solutions have built.

> As I understand it, that???s not possible today, but if I???m reading the latest diffs correctly, support for the concept has been added to the vp9 payload draft, so it might become possible if somebody takes up the work?
> I???m not sure about the reliability, because it seems like you???d still get those purple squares on lost packets, but you could at least reduce the incidence of lost packets due to congestion, and maybe that would make it possible for a little bit of FEC to keep up under any reasonably stable network. But I imagine that improving the reliability will be another project of its own, and there???s a bad tradeoff with the delay. But maybe it???s possible to get reliability with less delay than reliable unicast segmented video has to pay.

In general you have frames that are used as references by other frames and
those that are not. Loss in the latter ones can to a good extend be
compensated for by interpolation between the next and prior frame if
you allow for a few frames playout delay and have enogh CPU to do
motion interpolation. Reference frames on the other hand should more
likely be protected by enough FEC so they almosst never have loss. Else
you need low-latency retransmissions like from a nearby cache on the
gateway (assuming last-hop loss on wifi).

> This is precisely what I mean by ???there???s work to do before normal implementations of open standards can do this???. To me it looks probably achievable but not imminent.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Wed, 8 Nov 2017 23:13:30 +0000

-----Original Message-----
From: MBONED [mailto:[email-masked]] On Behalf Of Toerless Eckert

> receiver "joins" to the bitrate(s) it can receive without congestion.

This might be splitting hairs, but in effect, with RTP, you can be slowing down or speeding up the sender. Even if the sender, or intermediate mixer, is no more advanced than just making various streams available for a given content, depending on the feedback from RTCP, the sender will be sending a more or a less compressed stream, to that unicast destination. Which is an adjustment of the sender's bits/sec, nicht war?

(The effect can be striking, if you have a slow interface like ADSL. I've seen it start at poor quality, then ramp up in a few seconds, in very obvious increments, and very consistently.)

RTP multicast can't be this responsive to individual destinations, however RTCP feedback is usable there too.

But mostly, I don't think RTP cares exactly how one uses the RTCP feedback. If you want to send the feedback as a control to the MPEG encoder, for the encoder to decide how much to compress the signal in real time, RTP wouldn't even know. That's entirely up to the implementation, I would think.

Here's from RFC 3550:

2.4 Layered Encodings

   Multimedia applications should be able to adjust the transmission
   rate to match the capacity of the receiver or to adapt to network
   congestion.  Many implementations place the responsibility of rate-
   adaptivity at the source.  This does not work well with multicast
   transmission because of the conflicting bandwidth requirements of
   heterogeneous receivers.  The result is often a least-common
   denominator scenario, where the smallest pipe in the network mesh
   dictates the quality and fidelity of the overall live multimedia
   "broadcast".

   Instead, responsibility for rate-adaptation can be placed at the
   receivers by combining a layered encoding with a layered transmission
   system.  In the context of RTP over IP multicast, the source can
   stripe the progressive layers of a hierarchically represented signal
   across multiple RTP sessions each carried on its own multicast group.
   Receivers can then adapt to network heterogeneity and control their
   reception bandwidth by joining only the appropriate subset of the
   multicast groups.

6. RTP Control Protocol -- RTCP

   The RTP control protocol (RTCP) is based on the periodic transmission
   of control packets to all participants in the session, using the same
   distribution mechanism as the data packets.  The underlying protocol
   MUST provide multiplexing of the data and control packets, for
   example using separate port numbers with UDP.  RTCP performs four
   functions:

   1. The primary function is to provide feedback on the quality of the
      data distribution.  This is an integral part of the RTP's role as
      a transport protocol and is related to the flow and congestion
      control functions of other transport protocols (see Section 10 on
      the requirement for congestion control).  The feedback may be
      directly useful for control of adaptive encodings [18,19], but
      experiments with IP multicasting have shown that it is also
      critical to get feedback from the receivers to diagnose faults in
      the distribution.  Sending reception feedback reports to all
      participants allows one who is observing problems to evaluate
      whether those problems are local or global.  With a distribution
      mechanism like IP multicast, it is also possible for an entity
      such as a network service provider who is not otherwise involved
      in the session to receive the feedback information and act as a
      third-party monitor to diagnose network problems.  This feedback
      function is performed by the RTCP sender and receiver reports,
      described below in Section 6.4.

Bert

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 9 Nov 2017 03:03:09 +0100

Bert,

Not sure i recognize the hair you're trying to split.
Is this about me using the word "sender" when it should have been "codec/encoder" ?

Aka: As soon as you want to distribute content to large set of receivers reachable via
different speeds, you will figure out a sete of streams of diffeernt bitrates layered
or not and the encoding parameters of these streams will not depend on near term
dynamic changes of the set of receivers or their receive bandwidth. Instead only
those reeivers will perform rate adaptation by receiving a subset of streams. Thats
what i called "no you do not slow down the sender".

Of course, if your overall set of receivers starts to expose an overall shift in 
required bitrates, like when everybody wants to watch movies on wrist watches or
everybody gets FTTH, then you will readjust the bitrates of your streams and/or their
number or dependency composition. 

Did i find the hair ?

Cheers
    Toerless

-- 
---
[email-masked]

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Thu, 9 Nov 2017 03:22:36 +0000

-----Original Message-----
From: Toerless Eckert [mailto:[email-masked]] 

> receiver "joins" to the bitrate(s) it can receive without congestion.

I'm saying, depending how this is done, in principle you can be slowing down or speeding up the stream dynamically, to each device, when unicast.

> Aka: As soon as you want to distribute content to large set of receivers
> reachable via different speeds, you will figure out a sete of streams of
> diffeernt bitrates layered or not and the encoding parameters of these
> streams will not depend on near term dynamic changes of the set of receivers
> or their receive bandwidth.

Okay. But RTP/RTCP doesn't restrict you to just using that layered approach, or having the destination device do the hard work. It looks like, at least in the US, that the large majority of media streams are from distributed servers to individual user devices, in *unicast* sessions. Multiple thousands of streams, from each server, obviously more thousands when in SD quality than HD. Furthermore, that vast majority are also "on demand" as opposed to "live." Most people are not watching IP streams by appointment, let's say. In any event, live or not, most of these streams are unicast, at least in the US.

In these unicast cases, it's also possible to use this method, explained in RFC 3550, Sections 6 and 10:

   [RTCP feedback] is an integral part of the RTP's role as
   a transport protocol and is related to the flow and congestion
   control functions of other transport protocols (see Section 10 on
   the requirement for congestion control).  The feedback may be
   directly useful for control of adaptive encodings [18,19], ...

That last sentence above, before the RFC begins talking about multicast. Then more detail in Section 10. After stating that RTP is often inelastic, it does say the following:

   Therefore, congestion control SHOULD be
   defined in each RTP profile as appropriate.  For some profiles, it
   may be sufficient to include an applicability statement restricting
   the use of that profile to environments where congestion is avoided
   by engineering.  For other profiles, specific methods such as data
   rate adaptation based on RTCP feedback may be required.

Again, the last sentence in the quote. The encoder can be quite advanced, and RTP will support it.

> Did i find the hair ?

Fast (Deutsch fast).

Bert

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 9 Nov 2017 06:44:40 +0100 (CET)

On Wed, 8 Nov 2017, Toerless Eckert wrote:

> works fine since a decade in e.g.: cable networks from headend to 
> fiber/coax edge for example.

"works fine" is interesting choice of words.

My main point is that with Internet access, your customers won't really 
notice if you're dropping 1/1000 to 1/10000 packets. TCP starts to behave 
noticably much worse for normal consumers somewhere around 0.1-0.5% packet 
loss in my experience. With multicast TV without any "trickery" (just put 
those DVB cells into packets and feed them into the STB) you need more 
along the line of 1/million packet loss (one MPEG artifact per 2 hours or 
so) for customers to not get upset.

This is several orders of magnitude more sensitive compared to any other 
application your residential access network will have to work with, it's 
also probably two orders of magnitude more demanding regarding packet loss 
compared to most business SLAs as well.

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 9 Nov 2017 10:24:34 +0100

Albert,

ABR does not rely on RTP. There is no per-stream re-encoding of video
done in unicast streaming.

You could implement RTP unicast streaming from the same multi-bitrate content
source used by ABR by using the RTCP feedback from the receiver, determining a
good bitrate from that and then selecting the best available bitrate
stream and RTP packetizing it. And with RTP you wouldn't get the same
playout delay as with ABR. On the other hand, upspeeding is easier with
the ABR scheme: Because you burst every segment, you can easier measure
the actual available bitrate with every segment. If you stream real-time
with RTP its very hard to to measure how much more available bitrate you have.
If you do not have ECN, the only way is to get loss and then rely on FEC
or retransmissions. But of course as soon as you accept retransmissions,
you are in territory where TCP may have been the easier choice. 

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 9 Nov 2017 17:49:13 +0100

Thanks, Mikael

+1 on all your statements, but just because a requirement (zero loss for
IPTV) is uncommon does not mean its difficult or novel.

"works fine" was about lossless IP multicast in networks (up to controllable
subscriber edge).  That was in production in early 2000 for financials,
military and big distributed app, so it was never one of IP multicasts problem
after 2008 with IPTV/linear-tv. And now detnet wants to take these services
in SP networks out of those niche markets into mainstream.

But lets set that discus aside, because i did not mean to derail from
your main point, which is the loss @home/WiFi that SPs can not manage at
network layer:

For any SP who wants to have existing IP multicast realtime walled garden
linear TV reliable across a home/wifi, we can easily add AMT over TCP as a
simple RFC. Mandate AMT/TCP relay on home gateway, mandate sock buffers
good enough for 5 seconds of playout, and be done with 98% of your wifi 
problems IMHO (one outage every 4 hours ? ;-). Thats easy enough written
up so that it wouldn't matter if we only get adoption by the selct few
"non-ABR crowd".

The main detail enhancement we need to include IMHO is that AMT
(S,G) joins should be allowed to carry the "real relay"
address, so that the home-GW can either join native walled-garden IPTV
multicast directly or become gateway for another AMT tunnel to the
over-the-top-content.

If anybody thinks he could work on getting this implemented in home
gateway and client-app, i would be happy to help writing this up as a draft. 

Cheers
    Toerless

On Thu, Nov 09, 2017 at 06:44:40AM +0100, Mikael Abrahamsson wrote:
> >I'd limit this to "over non-sp-controlled networks" (e.g.: home).
> >It works fine since a decade in e.g.: cable networks from headend
> >to fiber/coax edge for example.
> 
> My main point is that with Internet access, your customers won't
> really notice if you're dropping 1/1000 to 1/10000 packets. TCP
> starts to behave noticably much worse for normal consumers somewhere
> around 0.1-0.5% packet loss in my experience. With multicast TV
> without any "trickery" (just put those DVB cells into packets and
> feed them into the STB) you need more along the line of 1/million
> packet loss (one MPEG artifact per 2 hours or so) for customers to
> not get upset.
> 
> This is several orders of magnitude more sensitive compared to any
> other application your residential access network will have to work
> with, it's also probably two orders of magnitude more demanding
> regarding packet loss compared to most business SLAs as well.
> 
> -- 

-- 
---
[email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 9 Nov 2017 18:43:05 +0100 (CET)

> IPTV) is uncommon does not mean its difficult or novel.

Compared to setting up a network that has 0.02% packet loss compared to 
one 0.00001% packet loss, I would not say the latter is easy. It takes a 
completely different effort compared to the 0.02% one.

If you have 100.000 customers and 500 of them want linear TV, it might be 
a lot easier for you to not support native multicast at all in your 
network and just run AMT tunnels with retransmits (of some kind). That way 
you can get away with 0.02% packet loss network. I would of course like to 
have the AMT relays report in retransmits because that's valuable 
operational data.

> up so that it wouldn't matter if we only get adoption by the selct few

Agreed.

> multicast directly or become gateway for another AMT tunnel to the

Right.