Subject: Re: [MBONED] deprecating MSDP

From: Toerless Eckert <[email-masked]>
Date: Mon, 4 Sep 2017 16:12:35 +0200

Depending to some degree to the vendor implementation,
I was suggesting to use MSDP for anycast-RP over RFC4610 because it
had evolved a lot earlier and has a lot more bells and whistles that are
IMHO highly relevant for business criticial anycast PIM-SM deployments.

Alas of course, MSDP is only IPv4, and we should have something equally
good for IPv6 as MSDP has been for IPv4 - for anycast-RP. 

So, IMHO, we should close the gaps of RFC4610 over MSDP: Give
it the option to use TCP (arguably in line with PORT), have a
config/ops model equivalent to the MSDP MIB (subest required for
anycast-RP), and IMHO introduce the equivalent to the MSDP cache
for troubleshooting and per-neighbor state limiters and total state
limiters.

IMHO thats whats needed to deprecate MSDP completely. Shorter term
we can of course have some deployment guide that calls MSDP deprecated,
but not change the doc status to historic/deprecated.

Toerless

On Wed, Jul 26, 2017 at 09:43:37AM -0700, Leonard Giuliano wrote:
> Dale, 
> 
> Great questions and thanks for bringing up, as we'd love feedback from the 
> I2 community- it's probably the largest user of interdomain MSDP and 
> the centerpiece of this conversation.
> 
> draft-ietf-mboned-interdomainpeering-bcp is scoped to just focus on SSM.  
> The plan is for draft-acg-mboned-multicast-models to pick up from there 
> and make the strong recommendation to use SSM, not ASM, for interdomain 
> mcast deployment.  I am not sure about moving MSDP to historic and what 
> that would practically mean, but in IMO, MSDP is somewhat of a scapegoat 
> here.  The real culprit is ASM (or rather, network-based source 
> discovery); MSDP is merely driving the getaway vehicle, albeit with a 
> reckless and dangerous driving style.  Eliminate network-based source 
> discovery for interdomain mcast and the vast majority of the complexity 
> and problems of mcast go away (RPs, PIM registers, RPTs, SPT switchover, 
> MSDP, data-driven state creation, etc).
> 
> We are scoping this to *interdomain*, as it has been mentioned that within 
> a domain there might be reasonable uses for ASM.  And for that matter, 
> what you do in your own house is your business, but once you leave your 
> home and interact with your neighbors and share the roads with other 
> drivers, more stringent rules and guidelines are appropriate for the sake 
> of safety and order.  So we are not saying anything about MSDP for 
> *intradomain* use (for now).  Aside: there is of course RFC4610, but 
> Anycast PIM is pretty much identical to internal MSDP- same behavior by a 
> different name.
> 
> So recommending SSM (and not ASM) for interdomain has overwhelming 
> support, and is basically a no-brainer from a network perspective.  But 
> the big open question is if it's feasible/meaningful given the state of 
> apps today? That is, for those users of interdomain mcast on I2, are the 
> apps they use SSM aware?  Is there a good way to find this out from the I2 
> community?
> 
> -Lenny
> 
> On Tue, 25 Jul 2017, Dale W. Carder wrote:
> 
> | 
> | As was noted in the minutes, there is perhaps a significant set of the
> | multicast community (including ourselves) seriously considering 
> | turning off inter-domain MSDP for good.
> | 
> | Related, I am curious what are the barriers to moving MSDP to historic 
> | status?  In RFC4611, that document differentiates intra-domain vs 
> | inter-domain MSDP.  For that matter, it does a pretty good job at
> | explaining the state of MSDP, but doesn't necessarily reflect what to 
> | do in deployments today.  
> | 
> | That seems to me to be different than where draft-acg-mboned-multicast-models
> | is headed?  
> | 
> | draft-ietf-mboned-interdomainpeering-bcp clearly is not advocating for
> | more inter-domain MSDP.
> | 
> | Dale
> | 
> | _______________________________________________
> | MBONED mailing list
> | [email-masked]
> | https://www.ietf.org/mailman/listinfo/mboned
> | 
> 
> _______________________________________________
> MBONED mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mboned

-- 
---
[email-masked]

---

From: Leonard Giuliano <[email-masked]>
Date: Tue, 5 Sep 2017 14:20:50 -0700

IMO, closing the gap between RFC4610 and MSDP would be a step in the wrong 
direction.  Reinventing MSDP and calling it another name will solve 
nothing.  "A source discovery protocol by any other name would smell as 
sweet..."  If someone really wants to run Anycast RP in v6 within his 
domain, RFC4610 should be good enough.

Again, the problem isn't MSDP, the problem is ASM.  Tweaking RFC4610 to 
match the microscopic (and generally irrelevant) benefits MSDP might have 
will just create yet another overly complex mcast protocol to address the 
corner cases of a handful of ill-conceived network deployments.  And have 
the same result- no one deploys it bc of all this unnecessary complexity.

If the goal is to change course and get folks to deploy mcast, we should 
make things unambiguously simple- use SSM in the network layer and let the 
some other layer handle source discovery.

On Mon, 4 Sep 2017, Toerless Eckert wrote:

| 
| Depending to some degree to the vendor implementation,
| I was suggesting to use MSDP for anycast-RP over RFC4610 because it
| had evolved a lot earlier and has a lot more bells and whistles that are
| IMHO highly relevant for business criticial anycast PIM-SM deployments.
| 
| Alas of course, MSDP is only IPv4, and we should have something equally
| good for IPv6 as MSDP has been for IPv4 - for anycast-RP. 
| 
| So, IMHO, we should close the gaps of RFC4610 over MSDP: Give
| it the option to use TCP (arguably in line with PORT), have a
| config/ops model equivalent to the MSDP MIB (subest required for
| anycast-RP), and IMHO introduce the equivalent to the MSDP cache
| for troubleshooting and per-neighbor state limiters and total state
| limiters.
| 
| IMHO thats whats needed to deprecate MSDP completely. Shorter term
| we can of course have some deployment guide that calls MSDP deprecated,
| but not change the doc status to historic/deprecated.
| 
| Toerless
| 
| On Wed, Jul 26, 2017 at 09:43:37AM -0700, Leonard Giuliano wrote:
| > Dale, 
| > 
| > Great questions and thanks for bringing up, as we'd love feedback from the 
| > I2 community- it's probably the largest user of interdomain MSDP and 
| > the centerpiece of this conversation.
| > 
| > draft-ietf-mboned-interdomainpeering-bcp is scoped to just focus on SSM.  
| > The plan is for draft-acg-mboned-multicast-models to pick up from there 
| > and make the strong recommendation to use SSM, not ASM, for interdomain 
| > mcast deployment.  I am not sure about moving MSDP to historic and what 
| > that would practically mean, but in IMO, MSDP is somewhat of a scapegoat 
| > here.  The real culprit is ASM (or rather, network-based source 
| > discovery); MSDP is merely driving the getaway vehicle, albeit with a 
| > reckless and dangerous driving style.  Eliminate network-based source 
| > discovery for interdomain mcast and the vast majority of the complexity 
| > and problems of mcast go away (RPs, PIM registers, RPTs, SPT switchover, 
| > MSDP, data-driven state creation, etc).
| > 
| > We are scoping this to *interdomain*, as it has been mentioned that within 
| > a domain there might be reasonable uses for ASM.  And for that matter, 
| > what you do in your own house is your business, but once you leave your 
| > home and interact with your neighbors and share the roads with other 
| > drivers, more stringent rules and guidelines are appropriate for the sake 
| > of safety and order.  So we are not saying anything about MSDP for 
| > *intradomain* use (for now).  Aside: there is of course RFC4610, but 
| > Anycast PIM is pretty much identical to internal MSDP- same behavior by a 
| > different name.
| > 
| > So recommending SSM (and not ASM) for interdomain has overwhelming 
| > support, and is basically a no-brainer from a network perspective.  But 
| > the big open question is if it's feasible/meaningful given the state of 
| > apps today? That is, for those users of interdomain mcast on I2, are the 
| > apps they use SSM aware?  Is there a good way to find this out from the I2 
| > community?
| > 
| > -Lenny
| > 
| > On Tue, 25 Jul 2017, Dale W. Carder wrote:
| > 
| > | 
| > | As was noted in the minutes, there is perhaps a significant set of the
| > | multicast community (including ourselves) seriously considering 
| > | turning off inter-domain MSDP for good.
| > | 
| > | Related, I am curious what are the barriers to moving MSDP to historic 
| > | status?  In RFC4611, that document differentiates intra-domain vs 
| > | inter-domain MSDP.  For that matter, it does a pretty good job at
| > | explaining the state of MSDP, but doesn't necessarily reflect what to 
| > | do in deployments today.  
| > | 
| > | That seems to me to be different than where draft-acg-mboned-multicast-models
| > | is headed?  
| > | 
| > | draft-ietf-mboned-interdomainpeering-bcp clearly is not advocating for
| > | more inter-domain MSDP.
| > | 
| > | Dale
| > | 
| > | _______________________________________________
| > | MBONED mailing list
| > | [email-masked]
| > | https://www.ietf.org/mailman/listinfo/mboned
| > | 
| > 
| > _______________________________________________
| > MBONED mailing list
| > [email-masked]
| > https://www.ietf.org/mailman/listinfo/mboned
| 
| -- 
| ---
| [email-masked]
|

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Tue, 5 Sep 2017 21:45:53 +0000

-----Original Message-----
From: MBONED [mailto:[email-masked]] On Behalf Of Leonard Giuliano

> Tweaking RFC4610 to match the microscopic (and generally irrelevant)
> benefits MSDP might have will just create yet another overly complex
> mcast protocol to address the corner cases of a handful of ill-conceived
> network deployments.  And have the same result- no one deploys it bc of
> all this unnecessary complexity.

+1.

There should be, and apparently there is, no need for a special-purpose, multicast-only, source discovery mechanism.

I'm not as hard over as many others seem to be, on "SSM only." Because as multicast is actually implemented, usefully implemented, it is in individual networks, such as enterprise networks or internal platform networks, in which ASM is often the best choice. For instance, when one needs redundancy of sources.

I should also point out that multicast is seen, by the security establishment, as a potential security threat. Therefore, inter-domain multicast is not encouraged. Quite the contrary, it is blocked be default. I'm also informed, second hand I'll admit, that ISPs and CDNs do not much favor multicast, for their own technical and economic reasons. I suppose one reason might be the proliferation of streaming media protocols these days, specific to different devices, which defeats the purpose of multicast.

Maybe multicast will remain a niche solution, very useful where it's required, but also deliberately limited in scope. Which is okay, in my book.

Bert

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 6 Sep 2017 08:33:48 +0200 (CEST)

On Tue, 5 Sep 2017, Manfredi, Albert E wrote:

> default. I'm also informed, second hand I'll admit, that ISPs and CDNs 
> do not much favor multicast, for their own technical and economic 
> reasons. I suppose one reason might be the proliferation of streaming 
> media protocols these days, specific to different devices, which defeats 
> the purpose of multicast.

There are lots of benefits to using multicast, it's just that there are 
also drawbacks. You can run an imperfect network (0.1% packet loss) any 
few will notice as long you run unicast. As soon as you introduce 
multicast, most applications that an ISP might be interested in (generally 
linear TV), now all of a sudden you need a network that has 1 in million 
packet loss, you run into gnarly fault finding cases, you need network 
verification tools to make sure you're assuring quality over time, etc.

Running a unicast-only network is quite forgiving, running multicast for 
what people typically use multicast for in an ISP network, is a lot less 
forgiving. Upside is typically small, unless you're providing TV for a 
profit (I don't know of any other typical residential ISP revenue stream 
for multicast), it's just not worth it.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 6 Sep 2017 03:00:28 -0700

> Running a unicast-only network is quite forgiving, running multicast for what people typically use multicast for in an ISP network, is a lot less forgiving. Upside is typically small, unless you're providing TV for a profit (I don't know of any other typical residential ISP revenue stream for multicast), it's just not worth it.

This is why running multicast over a unicast-only overlay has the best of both worlds and reduces the cost of running multicast because you only have to debug the very edges of the network. And the closer you put the encap/decap points to sources and receivers reduces further the debug surface.

Dino

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 7 Sep 2017 14:58:24 +0200

> domain, RFC4610 should be good enough.

My argument is that RFC4610 alone is not good enough for a business
criticial deployment of ASM because it misses all the necessary specs
for reliability , hardening and troubleshooting.

> the same result- no one deploys it bc of all this unnecessary complexity.

Nothing what i proposed is IMHO specific to RFC4610 or MSDP. It's instead
what you'd need to have for any network layer protocol of similar
complexity to make it reliable, hardened and troubleshootable. Most
of these operational aspects have in the part left to vendors proprietary
approaches and IMHO thats been one of the biggest issues of success
of more complex solutions in networks.

Wrt to ASM being the problem: Imagine we only do SSM in the network,
and a distributed application layer solution would do source discovery.
Guess how that would be built. Definitely a lot easier troubleshotable
than RFC4610. Much more like what i am suggesting. Definitely on top of
TCP or better and not UDP. 

> some other layer handle source discovery.

Sure. ALl i am saying is that ASM without MSDP and with just RFC4610
is less operationally complete. I am not arguing right now how important
it would be to fix that, i would just resist documents stating that
ASM intradomain with just RFC4610 is in good shape.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 7 Sep 2017 15:45:00 +0200

Great uplevel, Mikael

IMHO its still worthwhile to brainstorm what an initial IP multicast service
"across the internet" would need to have any change to be adopted/used,
even if only to understand that the problem is not just solved by
"Just use SSM".

here's my best theory what could work:

Get an initial SSM service deployed into Tier 1 SP, available to 10++
million subs:

  - Avoids initial considerations for interdomain transit
  - Creates criticial mass to encourage app developers to support it.

Apps "just" need to support SSM API and congestion control aka RFC8085.
Eg: fountain/network codecs. SSM is easy, the rate adaption is quite advanced.

Operators can not trust that apps perform complint to RFC8085. Most
easily, initial SSM multicast would be in a scavenger class. Makes
all bandwidth used by it free for the SP. Hopefully good enough to make
marketing folks in SPs interested.

(S,G) state in network need to be controlled. On receiver edge,
receivers must only be permitted to join to few # of active (S,G) state.
Else networks can be killed with receiver (S,G) state. This ultimately is
introducing a requirement for some active (S,G) awareness into receiver
edge devices.

Senders should be be a chargeable service option. IMHO charging
for senders hould be handled like SMS was handled in Germany in the
90'th: Make it virtually free first and raise the bar once the junkies,
oops: subscribers can not live without it. Might want to charge some
entrance fee even day 1 to get only senders that have a real business
inererst and large number of receivers (instead of a repetition of
"cameras all over the world").

Cheers
    Toerless

On Wed, Sep 06, 2017 at 08:33:48AM +0200, Mikael Abrahamsson wrote:
> >default. I'm also informed, second hand I'll admit, that ISPs and
> >CDNs do not much favor multicast, for their own technical and
> >economic reasons. I suppose one reason might be the proliferation
> >of streaming media protocols these days, specific to different
> >devices, which defeats the purpose of multicast.
> 
> There are lots of benefits to using multicast, it's just that there
> are also drawbacks. You can run an imperfect network (0.1% packet
> loss) any few will notice as long you run unicast. As soon as you
> introduce multicast, most applications that an ISP might be
> interested in (generally linear TV), now all of a sudden you need a
> network that has 1 in million packet loss, you run into gnarly fault
> finding cases, you need network verification tools to make sure
> you're assuring quality over time, etc.
> 
> Running a unicast-only network is quite forgiving, running multicast
> for what people typically use multicast for in an ISP network, is a
> lot less forgiving. Upside is typically small, unless you're
> providing TV for a profit (I don't know of any other typical
> residential ISP revenue stream for multicast), it's just not worth
> it.
> 
> -- 

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 7 Sep 2017 16:00:01 +0200

> This is why running multicast over a unicast-only overlay has the best of both worlds and reduces the cost of running multicast because you only have to debug the very edges of the network. And the…

God point. Reducing the number of points in the network where you have to
deal with multicast replication state is a good thing. I don't think we have
a good complete BCP for how to best manae multicast state in SP networks. See
my prior email about amount of SSM sender and receiver state as well. 

Note though, that this is orthogonal to what Mikael mentioned. Current
IPTV style apps without any transport reliability (FEC, fountain codecs,
etc. pp) will also fail miserably when they are tunneled across unicast.
The only place where unicast tunneling magically solves the reliabilty issues
is on WiFi ;-)

Btw: just seeing that no document claims to be updating RFC3170. Does that not say everything about the state of IP multicast apps ? ;-))

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 7 Sep 2017 16:09:13 +0200 (CEST)

On Thu, 7 Sep 2017, Toerless Eckert wrote:

> IMHO its still worthwhile to brainstorm what an initial IP multicast 
> service "across the internet" would need to have any change to be 
> adopted/used, even if only to understand that the problem is not just 
> solved by "Just use SSM".

Indeed not solved by "just use SSM". It is still my firm opinion that 
there will be no "IP multicast service across the Internet" unless someone 
comes up with serious money and throws it ISPs way to enable this. And I 
am talking about ALL the ISPs which carry this, not just one. I don't see 
this happening anytime soon.

> marketing folks in SPs interested.

There are a lot of complications beneath this "highish level" overview.

> Senders should be be a chargeable service option. IMHO charging for 
> senders hould be handled like SMS was handled in Germany in the 90'th: 
> Make it virtually free first and raise the bar once the junkies, oops: 
> subscribers can not live without it. Might want to charge some entrance 
> fee even day 1 to get only senders that have a real business inererst 
> and large number of receivers (instead of a repetition of "cameras all 
> over the world").

I attended a one-day conference around 2002 or something, that had a talk 
about "multicast business models" and money flows etc. A lot of problems 
were outlined. These are still true today.

---

From: Leonard Giuliano <[email-masked]>
Date: Thu, 7 Sep 2017 11:33:40 -0700

| Indeed not solved by "just use SSM". It is still my firm opinion that there
| will be no "IP multicast service across the Internet" unless someone comes up
| with serious money and throws it ISPs way to enable this. And I am talking
| about ALL the ISPs which carry this, not just one. I don't see this happening
| anytime soon.

FWIW, we wouldn't have the web or netflix or youtube or any of the main 
things that people use the Internet for if such a similar barrier were 
considered for these services.  Content drives deployment.  If mcast is 
able to deliver some really compelling content that isn't available/viable 
via unicast, then it will find a way to get itself deployed.  VR perhaps?  
Anyway, the bar for this to happen is actually much lower now that we have 
AMT, as it only takes one/a few SPs to deploy Internet Mcast to deliver 
the content to end users, as opposed to ~ALL of them as was previously 
required.

FWIW2, all of the exact same arguments for/against mcast can be made for 
IPv6, and IPv6 seems to gaining a bit of momentum and steam after a few 
decades of similar deployment malaise.  Perhaps v6 can provide a teachable 
moment for mcast?

As for business models, anyone seeking massive $$$ for direct revenue from 
Internet mcast will most likely never find it.  I think mcast is best 
viewed like DNS or BGP (or IPv6)- infrastructure that is difficult to 
charge directly for, but enables higher layer services for which there is 
a panoply of viable business models.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 7 Sep 2017 22:24:37 +0200 (CEST)

On Thu, 7 Sep 2017, Leonard Giuliano wrote:

> decades of similar deployment malaise.  Perhaps v6 can provide a 
> teachable moment for mcast?

Agreed. I have worked in IPv6 deployment and advocacy for 10 years. IPv6 
solves a problem (adressing), and yet it was really hard to start getting 
it deployed. This is now happening. I don't see the same driver for 
multicast. IPv6 solves the problem of the new entrant into the market who 
doesn't have enough IPv4 addresses. It doesn't solve many problems for the 
incumbents, thus there is no real driver for them to deply IPv6. This is 
now happening anyway, because even they see that they'll have problems if 
IPv6 is not deployed.

Multicast is an optimization over unicast which mostly benefits the 
head-end. At the same time, the world is moving towards more and more 
"on-demand" where multicast has less to offer than it did 15 years ago 
when more content was linear TV and everybody was watching the same thing 
at the same time.

I just don't see the business model for multicast the way Toerless 
described it. I don't see it 15 years ago, I don't see it now.

Having the ISP make money off of linear TV that's provided in cooperation 
with some "cable company". That's obvious. ISP gets paid by "cable 
company" for distribution. Or it does it itself as hygene factor to 
attract more customers.

Now, what would the content be that would have the same "pull", so that 
customers would not buy from ISP if they didn't at least have that 
content/service available as an option? I can't think of one.

> a panoply of viable business models.

Can you give a few examples of these "panoply of viable business models" 
for Internet-wide multicast? I can't think of a single one.

I have deployed multicast in largish ISP networks twice in my life. It's 
painful. It's also expensive because of all the verification needed to be 
done initially and over time, to make sure your customers get all the 
packets.

>From cost/benefit perspective, I just don't see this flying. I would like 
to see it fly, I have always been fascinated by multicast ever since I 
watched a NASA multicast stream (shuttle launch) on my computer in my dorm 
back in 1996 or something. Still... I just don't see what would make ISPs 
do what Toerless described.

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Thu, 7 Sep 2017 20:53:58 +0000

-----Original Message-----
From: MBONED [mailto:[email-masked]] On Behalf Of Mikael Abrahamsson

> Multicast is an optimization over unicast which mostly benefits
> the head-end. At the same time, the world is moving towards more
> and more "on-demand" where multicast has less to offer than it
> did 15 years ago when more content was linear TV and everybody
> was watching the same thing at the same time.

I agree completely, and there is more. One of the benefits of Internet delivery of TV is precisely because, as a two-way medium, it inherently supports on demand. Given the option, much more often than not, people prefer to watch TV when it is convenient for them, not by appointment with a head end (in the US, as of mid-2015, only 39% of TV content was consumed by appointment; the percentage might be even lower today). Broadcast delivery, meaning "by appointment," was the only way of distributing TV content decades ago. People got used to it. But when options became available, like VCRs, people began to discover they prefer watching when it is convenient.

And there is still more. Because now different consumer streaming devices have conjured up their own little walled garden, such as iOS, Android, Roku, Amazon Fire, and so forth, and much of the content is viewed on demand, the solution is to deploy servers at the edges of networks, especially for the more popular content. So the delivery to users is via unicast, and the stream protocol depends on the user's device. And furthermore, these tend to be self-adjusting protocols, which probe the end to end link quality on a continuing basis. Not ideal for multicast.

But for some niche applications, multicast is the best answer by far. Such examples as distribution of navigation and system status data, in a control system network.

> Having the ISP make money off of linear TV that's provided in
> cooperation with some "cable company". That's obvious. ISP gets
> paid by "cable company" for distribution.

Even that is not entirely clear to me. Yes, for live material, the ISP may prefer multicast. But even here, some ISPs prefer to be able to count the number of individual unicast sessions, and charge accordingly. Even live material can be distributed to edge servers out of band, and then unicast to end users, using the specific protocol for that user's device.

Bert

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 8 Sep 2017 05:16:00 +0200 (CEST)

On Thu, 7 Sep 2017, Manfredi, Albert E wrote:

> users is via unicast, and the stream protocol depends on the user's

Here is another thing. Most of the traffic to user devices traverse at 
least one wifi hop. The unfortunate truth is that multicast doesn't work 
well on todays wifi environment (I raised this issue within the IETF close 
to two years ago, and we're still nowhere near anything I would say even 
remotely looks like a widespread solution).

So yes, AMT tunnels to unicast everything from the ISP to the devices 
might work. They would have to be initiated by the device/app itself (to 
aid initial deployment).

6to4 was decided to be a really bad way of doing IPv6 (public relays 
nobody knew how well they worked, they were badly maintained etc).

AMT looks like it was designed the same way. Now, with AMT you don't have 
the problem with return path uncertainty that 6to4 had, so that is less of 
a problem.

What I see lacking here is reliable transport. If we're going to do this, 
the client is going to have to be able to request retransmits to work 
around unreliable delivery of packets.

How do we do that? 5 second buffer in the AMT relay (or a device sitting 
next to the relay) that the client can request re-transmits from? It would 
need to be stateful somehow. We'd also need some kind of 
identifier/sequence number in each AMT packet so the end device can tell 
that a packet was missing, and can request a retransmit of this.

Thoughts?

---

From: Leonard Giuliano <[email-masked]>
Date: Fri, 8 Sep 2017 13:03:08 -0700

| Agreed. I have worked in IPv6 deployment and advocacy for 10 years. IPv6
| solves a problem (adressing), and yet it was really hard to start getting it
| deployed. This is now happening. I don't see the same driver for multicast.
| IPv6 solves the problem of the new entrant into the market who doesn't have
| enough IPv4 addresses. It doesn't solve many problems for the incumbents, thus
| there is no real driver for them to deply IPv6. This is now happening anyway,
| because even they see that they'll have problems if IPv6 is not deployed.

Well, advocates of v6 say it's needed to solve the address space problem, 
whereas skeptics say NAT already solved it.  Likewise, advocates of mcast 
say it solves the problem of duplicated traffic, whereas skeptics say CDNs 
already solved it.  Advocates of v6 suggest that an abundance of address 
space will make possible all sorts of new capabilities that were 
previously not possible or even thought of (IOT, smart appliances, etc).  
Likewise, advocates of mcast suggest that more efficient distribution of 
multidestinational traffic will democratize the streaming of live content 
and make possible all sorts of new *OTT* content that was previously not 
possible or even thought of (VR, etc).  That is, some faith in the unseen 
is a strong driver for both.

Not trying to weigh in on either side (at least of the v6 debate) other 
than to say these are enabling infrastructure technologies that share many 
of the same same arguments for/against.

| Multicast is an optimization over unicast which mostly benefits the head-end.
| At the same time, the world is moving towards more and more "on-demand" where
| multicast has less to offer than it did 15 years ago when more content was
| linear TV and everybody was watching the same thing at the same time.

Indeed, we live in an on-demand world, and that has greatly diminished the 
need for mcast.  But live linear video is not quite dead yet.  Plenty of 
people really do still watch TV.  And some content is simply preferred 
live (sporting events).  And perhaps the most common use of time-shifted 
video is the personal DVR- this uses live linear video (and saves it for 
personal playback later).

With >70% of Internet traffic carrying video, and traffic rates 
skyrocketing unabated- and that's without a ton of live linear content, 
which most consume on the plain-old TV (non Internet) system- I would 
suggest there's now more than ever a thirst for more efficient 
distribution of video content.  Indeed, the GGIE BOF in Chi IETF at least 
highlighted the challenges of video over the Internet (though I can't say 
I quite agreed with the solutions suggested there).

| Can you give a few examples of these "panoply of viable business models" for
| Internet-wide multicast? I can't think of a single one.

Think of any of the *OTT* business models of the unicast Internet: 
advertising, subscriptions, metering, selling customer data to dubious 
entities, etc.  All of these are just as applicable in the mcast world.  
And remember, just bc the content may be delivered via mcast, doesn't mean 
things like ad insertion and audience metering can't be done via unicast 
in the app layer in the background.

---

From: "Manfredi, Albert E" <[email-masked]>
Date: Fri, 8 Sep 2017 21:37:20 +0000

-----Original Message-----

> Indeed, we live in an on-demand world, and that has greatly diminished
> the need for mcast.  But live linear video is not quite dead yet.
> Plenty of people really do still watch TV.

True, but what content is actually preferred live has become the minority - sports and news. The trend has been downward rather steadily, for what amount of TV content is consumed "live." Mostly, I think, it's just that old habits die hard. People need to start using OTT methods, before they find this second nature. And this is what seems to be happening. 

> And perhaps the most common use of time-shifted video is the personal
> DVR- this uses live linear video (and saves it for personal playback
> later).

Best I have been able to find on this is that as of the beginning of 2015, use of in-home storage, for time shift viewing, accounted for about half of what we might generically call "on demand" viewing. In the US, back then, the stat I saw was 47% of on demand viewing was from streaming, the rest from PVR. But the trend was downward for PVR in-home recording, also due to the cord cutting phenomenon. (Most people in the US are still tied to cable or satellite TV, but that number going steadily down. The only practical way of in-home recording, when tied to a set-top box, becomes via that STB with built-in PVR. With cord cutting, that box goes away. Cord cutters use streaming, not so much out there buying their own PVRs anymore.)

So, Internet streaming is increasingly used for on demand. By today, I would be surprised if streaming didn't account for well over half of on demand viewing. Which is consistent with all other examples of stored content, migrating to the so-called "cloud."

Which means, in practice, that a scalable solution for on demand viewing is needed regardless. Also, a practical solution to accommodate the multiple streaming protocols needed. This is where the CDNs and the servers at the edges come in. But yes, for those major sporting events that many people really do watch simultaneously, multicast streaming through ISP networks would be one interesting option. For this, though, the WiFi problem needs a good solution, and if you really want targeted ads, during these very popular sporting events, that too needs a unicast solution.

So, another option is to use the same edge server infrastructure, as used for on demand. Deliver the live content to these edge servers out of band (e.g. via satellite), which is how on demand content is frequently delivered now, to edge servers. This way, no WiFi multicast problem in the user premises, and no problem accommodating the different type of streaming protocols or the targeted ads.

Best I can tell, multicast is definitely required, but it may never amount to a large fraction of Internet traffic. Except maybe in specialty networks.

Bert

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 11 Sep 2017 13:53:07 +0200

> described it. I don't see it 15 years ago, I don't see it now.

I don't think i said something about the business model, i was just
describing the minimum technical requirements for multicast outside of
closed-group services (eg: L3VPN) in SPs. 

You mentioned that there are difficulties underneath my high level
description. I would love to hear what those are to see if there are
standards gaps we could close (but see below firt for one discussion ;-).

I still think IP multicast is a business opportunity, but i have given
up for a long time now that it will happen with SPs.

The two business models that IMHO came closest where:

a) Regulator mandated equal access for competiors to replication as
   used in SPs own IPTV services. AFAIK, this never came to fruition
   anywhere.

b) OTT providers willing to pay for SP based replication to 
   deal with peak utilization. This one has not come true yet
   either but was one reason for businesses to invest into AMT.

In both cases, competitive and net-neutrality considerations make
these approaches complex.

> cooperation with some "cable company". That's obvious. ISP gets paid
> by "cable company" for distribution. Or it does it itself as hygene
> factor to attract more customers.

Are you describing DTs (and others) existing IPTV business model or something
half-way to my (a) ? (aka: not regulator enforced) ?

> Now, what would the content be that would have the same "pull", so
> that customers would not buy from ISP if they didn't at least have
> that content/service available as an option? I can't think of one.

With Satellite (e.g.: Astra), i can buy bandwidth and broadcast
whatever program i want - and hundreds of millions of STBs will be
able to receive it (instead of "this video requires internet explorer 5.7").
And Astra does not act as a controller for the content (at least not when
encrypted). AFAIK, there is no equivalent offer not even from a single ISP in Europe.

The fact that there is no offered IP equvialent to this is quite perplexing.
Given how much many billions of seed subsidies have been tanked into many
other technologies across europe, maybe the EU should try to get this
going via some initial subsidies.

> I have deployed multicast in largish ISP networks twice in my life.
> It's painful. It's also expensive because of all the verification
> needed to be done initially and over time, to make sure your
> customers get all the packets.

Most QoS/QoE (Quality of Service / Experience) issues exist in unicast as
well, (e.g.: UDP VoD without rate adaptation), or non-existing bandwidth
guarantees for TCP videos (e.g.: Netflix sucks on friday evenings).

In TCP you need to trust sender and receiver to perform
appropriate congestion control. And there are lot of hacks in ISP
to throttle non conforming traffic. Given how slowly this
has all evolved in unicast, one can of course see how difficult it
is for this industry to make something new as rate-adaptive multicast
work fairly.

> to see it fly, I have always been fascinated by multicast ever since
> I watched a NASA multicast stream (shuttle launch) on my computer in
> my dorm back in 1996 or something. Still... I just don't see what
> would make ISPs do what Toerless described.

See above. This being mboned (ops), and not mc-business, it should
be fine for us to figure out what missing pieces we should work on to
remove any technical obstacles.

Wrt to business: Lets see how slicing in mobile networks will evolve.
A lot of use cases there are closed group services. Aka: like L3VPN,
the paying customer could force the ISP to deliver what they
want (eg: IP multicast). But unlike wireline networks, those
closed groups could hopefully include residential users that just buy
into some slicing service when they buy an app or service that comes
with the app.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 11 Sep 2017 15:06:00 +0200 (CEST)

On Mon, 11 Sep 2017, Toerless Eckert wrote:

> Are you describing DTs (and others) existing IPTV business model or 
> something half-way to my (a) ? (aka: not regulator enforced) ?

I am not describing anything related to any specific ISP.

I was talking generally. I know multiple ISPs that have linear TV 
multicast in their network. I know 0 who have this "over the Internet". 
It's always some special port somewhere to someones headend.

> With Satellite (e.g.: Astra), i can buy bandwidth and broadcast whatever 
> program i want - and hundreds of millions of STBs will be able to 
> receive it (instead of "this video requires internet explorer 5.7"). And 
> Astra does not act as a controller for the content (at least not when 

That's because Astra has line-of-sight to these hundreds of millions of 
STBs. The equipment is also very well specified (DVB) and standardized 
(well, perhaps not the encryption part, still still seems to be provider 
specifics).

> The fact that there is no offered IP equvialent to this is quite 
> perplexing. Given how much many billions of seed subsidies have been 
> tanked into many other technologies across europe, maybe the EU should 
> try to get this going via some initial subsidies.

The IP model is to do this over unicast, and use lots of bandwidth for it. 
Astra satellite is probably in the X times gigabit range of bandwidth. 
Largish ISPs are probably 2-3 magnitudes above that. One runs a network 
that is optimized for unicast, the other one runs equipment that is 
optimized for broadcast.

> Most QoS/QoE (Quality of Service / Experience) issues exist in unicast 
> as well, (e.g.: UDP VoD without rate adaptation), or non-existing 
> bandwidth guarantees for TCP videos (e.g.: Netflix sucks on friday 
> evenings).

Indeed, but your customers won't call and complain because they had a 
little bit more buffering when loading youtube (or they might not even 
notice it). They will call when they have magenta blobs floating around on 
their TV because you lost a packet somewhere.

> See above. This being mboned (ops), and not mc-business, it should be 
> fine for us to figure out what missing pieces we should work on to 

Then my point is that we need to invent AMT tunnels with re-transmits, and 
make sure we have everything so applications can initiate these tunnels 
and start receiving data. That way we can overlay over existing networks 
and as this traffic starts to increase, people can start running native 
multicast instead, further and further out towards the edge of their 
networks. These AMT tunnels also need to have some kind of signalling so 
you can have multiple AMT tunnels and know whereto subscribe S,G for what 
groups.

Incremental deployment is the key here in my opinion.

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 11 Sep 2017 15:52:41 +0200

On Mon, Sep 11, 2017 at 03:06:00PM +0200, Mikael Abrahamsson wrote:
> multicast in their network. I know 0 who have this "over the
> Internet". It's always some special port somewhere to someones
> headend.

I'd already be happy to hear about examples where that headend is
run by somebody else than the ISP (or its subsidiaries, co-owned
business entities), and that somebody else is controlling the content.
That would be progress in the business models.

> >perplexing. Given how much many billions of seed subsidies have
> >been tanked into many other technologies across europe, maybe the
> >EU should try to get this going via some initial subsidies.
> 
> The IP model is to do this over unicast, and use lots of bandwidth
> for it. Astra satellite is probably in the X times gigabit range of
> bandwidth. Largish ISPs are probably 2-3 magnitudes above that. One
> runs a network that is optimized for unicast, the other one runs
> equipment that is optimized for broadcast.

One continues to ignore exploiting opportunities for multicasting ? ;-))

> >Most QoS/QoE (Quality of Service / Experience) issues exist in
> >unicast as well, (e.g.: UDP VoD without rate adaptation), or
> >non-existing bandwidth guarantees for TCP videos (e.g.: Netflix
> >sucks on friday evenings).
> 
> Indeed, but your customers won't call and complain because they had
> a little bit more buffering when loading youtube (or they might not
> even notice it). They will call when they have magenta blobs
> floating around on their TV because you lost a packet somewhere.

The visual experience with rate adapting multicast and network/fountain
codecs is pretty much the same as for unicast/ABR.

> >be fine for us to figure out what missing pieces we should work on
> >to remove any technical obstacles.
> 
> Then my point is that we need to invent AMT tunnels with
> re-transmits, and make sure we have everything so applications can
> initiate these tunnels and start receiving data. That way we can
> overlay over existing networks and as this traffic starts to
> increase, people can start running native multicast instead, further
> and further out towards the edge of their networks. These AMT
> tunnels also need to have some kind of signalling so you can have
> multiple AMT tunnels and know whereto subscribe S,G for what groups.
> 

Without multi-layered rate-adaptive encoding, TCP in AMT will not help,
and after a good standard for multi-layer rate-adaptive encoding,
the incremental deployment option is IMHO not AMT but a proxy:

Getting an open source reference rate-adaptive multicast video
sender/receiver stack (with join/leave to different (S,G) bitrates
according to loss) is IMHO the same problem as good TCP + MPEG/H26X stack
for unicast. We have those now, but how long did those take ?

I know of good commercial or prototpye stacks that do this well
(i actually helped to develop one), but with those stacks stuck in
the proprietary product offerings of those vendors, they are IMHO not
sufficient to create a larger market for multicast technologies.

In this decade (or next), i can not imagine that a few proprietary options
can actually push the needle in this market. Instead, this is something
for middleware developers.  Aka: somebody give a few hundred K to Mozilla 
and let them develop such a stack.

Once there is an open source reference, it can be codified into a standard
(profile of a bunch of existing session/app-layer protocols). And
then it would be a good product offering to build multicast->unicast
proxies - where the unicast side is TCP - indistinguishable from
existing ABR to the receiver - and the proxy does the actual multicast
join/leaves to the different bitrates. 

Yes, you could equally build AMT with TCP, but TCP pushes such an
AMT tunnel implementation out of traditional router forwarding paths.
And once you use a forwarding plane that can do TCP, then you can
equally do the above mentioned proxy function because thats just
a bit more work than an AMT proxy - eg: couple the measured
TCP send-rate to upstream join/leaves.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 11 Sep 2017 16:37:25 +0200 (CEST)

On Mon, 11 Sep 2017, Toerless Eckert wrote:

> I'd already be happy to hear about examples where that headend is run by 
> somebody else than the ISP (or its subsidiaries, co-owned business 
> entities), and that somebody else is controlling the content. That would 
> be progress in the business models.

Yes, this is available. The major channel distribution companies in Sweden 
can offer this themselves or through their partners. As an ISP you order a 
fiber to them, get it lit up, and then you get multicast packets. The same 
entitity will also do EPG etc to the set-top boxes. So as an ISP, you 
don't have to do much if you don't want to.

> codecs is pretty much the same as for unicast/ABR.

Ok. All IPTV I have seen had 0 tolerance to packet loss.

> and after a good standard for multi-layer rate-adaptive encoding, the 
> incremental deployment option is IMHO not AMT but a proxy:

I am not talking about packet loss induced by congestion. I am talking 
about packet loss induced by CRC errors on packets (and similar kinds of 
errors).

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 11 Sep 2017 18:14:54 +0200

On Mon, Sep 11, 2017 at 04:37:25PM +0200, Mikael Abrahamsson wrote:
> >business entities), and that somebody else is controlling the
> >content. That would be progress in the business models.
> 
> Yes, this is available. The major channel distribution companies in
> Sweden can offer this themselves or through their partners. As an
> ISP you order a fiber to them, get it lit up, and then you get
> multicast packets. The same entitity will also do EPG etc to the
> set-top boxes. So as an ISP, you don't have to do much if you don't
> want to.

Interesting. Any URL where i could find more details ?
Also: i forgot to include the element that the content owner could get
this set up. The way you describe it, it's something where the ISP decides
which content it wants to accept ?

> >The visual experience with rate adapting multicast and
> >network/fountain codecs is pretty much the same as for
> >unicast/ABR.
> 

Yes. IPTV is as much of an evolution to TV as SIP was to telephony.

In a world where "use" is a permissible glorification of "abuse",
the best thing about both of them is that they use IP.

> >Without multi-layered rate-adaptive encoding, TCP in AMT will not
> >help, and after a good standard for multi-layer rate-adaptive
> >encoding, the incremental deployment option is IMHO not AMT but a
> >proxy:
> 
> I am not talking about packet loss induced by congestion. I am
> talking about packet loss induced by CRC errors on packets (and
> similar kinds of errors).

A bunch of the IPTV solutions have dealt with that problem. Especially
because the L2 FEC on xDSL is not good enough. There where a couple
of RFCs introduced back in the days when the first IPTV solutions came
along, specifically for RTCP based retransmission requests. Those
would then be handled by a cache sitting eg: near the DSLAM. 
Burst versions of those retransmission requests are also used to support fast
channel change, eg: get the last GOP so you can immediately start
streaming instead of having to wait for the next I-frame (especially
when GOP sizes of 10 seconds are desired). Vendor solutions like
those from Microsoft used proprietary mechanisms achieving the same
goals. Not sure if they implement the standards now.

In any case, that type of proxy-cache functionality could easily
be added to an AMT relay/gateway. The only linkage required would
be to just permit those RTCP/RTP unicast packets through the
AMT tunnel. Is it better than doing TCP ?? You get the option
for fast channel change... TCP gives you ability to make
non-RTP packets reliable but with less controllable latency.
Both would IMHO require non-ASIC forwarding.

Easier yet, just use good FEC like fountain codecs. Network Codecs
should be fine for this too (easier, less IPR encumbered).

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 11 Sep 2017 18:35:27 +0200 (CEST)

On Mon, 11 Sep 2017, Toerless Eckert wrote:

> Interesting. Any URL where i could find more details ? Also: i forgot to 
> include the element that the content owner could get this set up. The 
> way you describe it, it's something where the ISP decides which content 
> it wants to accept ?

It was cooperation between the content distributor and the ISP. The ISP 
wanted to checkbox having "TV" in their offering, and the content 
distributor wanted to reach more customers.

There is URL I know of.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 11 Sep 2017 18:36:38 +0200 (CEST)

> There is URL I know of.

There is *no* URL that I know of.

This isn't volume business afaik. I don't remember which one it was, but 
it was basically in cooperation with one of the larger typical "channel 
package providers" you might get on coax TV. One of Canal Digital, VIASAT 
or someone like that.

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 11 Sep 2017 18:54:11 +0200

Right. So it would not enable new content owners to come in and get
an easier market entry to reach customers for live streaming... sitting
in front of STBs. 

On Mon, Sep 11, 2017 at 06:36:38PM +0200, Mikael Abrahamsson wrote:
> This isn't volume business afaik. I don't remember which one it was,
> but it was basically in cooperation with one of the larger typical
> "channel package providers" you might get on coax TV. One of Canal
> Digital, VIASAT or someone like that.
> 
> -- 

-- 
---
[email-masked]