Subject: Re: [arch-d] possible new IAB programme on Internet resilience

From: Dan York <[email-masked]>
Date: Wed, 1 Jan 2020 18:44:21 +0000

Stephen,

Lucy very nicely captured a concern I’d had… upon which I expanded a bit below..

On Dec 31, 2019, at 2:27 PM, Lucy Lynch <[email-masked]<mailto:[email-masked]>> wrote:

On Fri, 20 Dec 2019, Stephen Farrell wrote:

[1] https://github.com/intarchboard/resilience/

Circling back to the top here.

I think this is a fine topic for an IAB program and I took the
draft charter to encompass resilience as both a technical and a
design problem.

I also agree that this seems to be a great topic for an IAB program.

I am particularly interested in this statement:
----
Definition of resilience:

1]  the capability of a strained body to recover its size and shape
   after deformation caused especially by compressive stress
2]  an ability to recover from or adjust easily to misfortune or change

This program is mostly interested in definition #2.
----
I actually have my own concerns related to #1 as well and would hope that this program might consider the warping of the overall Internet model to accommodate currents trends or business practices.

As an example - an Internet optimized for the web may not be the same
internet that supports real time data collection and shared computation in the context of big science. How do we avoid closing out capabilities as we optimize for others? Narrowing of choices looks like a path to a limited and more brittle model to me.

I agree with Lucy’s example… and would also note that other sources of “compressive stress” could be the increasing movement of most all real-time communication via voice and video to be over the Internet, and most recently the very large movement of streaming online gaming, which has very different characteristics and stress factors (as noted in recent discussions in the new MOPS working group and the BOF before it).

I like Lucy’s phrase "the warping of the overall Internet model to accommodate current trends or business practices,” particularly when some of those current business practices may involve connecting networks not only to the public Internet, but also to private, proprietary, globe-spanning networks.

For example, at Amazon’s recent re:Invent conference they promoted a way to directly connect enterprise data centers to Amazon’s global AWS network (“Outposts”) and also a way to connect telco points-of-presence to Amazon’s network (“Wavelength”). My understanding is that Microsoft has similar functionality for Azure (“Stack”, I believe) and Google either has or is working on something similar for their Google Cloud Platform. Similarly, large entities such as Facebook and Netflix have built their own global, private networks that interconnect to local data centers (where those data centers are also connected to the public Internet).

All of these separate, private, global networks are designed to help speed access to content, applications, etc., through caching, “edge” computing, and other technologies. Thinking as a network engineer about running applications in various cloud providers, I can see the value that could be obtained by these connections. And some of these providers are typically promoting these services as providing a low-latency alternative to sending traffic across the public Internet.

Going back to the draft charter text ( https://github.com/intarchboard/resilience/ ) , I note this:

> One fundamental pattern contributing to Internet resilience is diversity: for example, diversity of physical links, of peer networks, of paths through the network. Lack of diversity is a key challenge for Internet resilience.

What if there winds up being a lack of diversity of paths through the “open” and “public” Internet? What if increasingly traffic winds up traveling through these proprietary global networks (to which you need to pay to connect and through that gain permission to send traffic - and only to that company’s platforms)?

Given that the Internet has always been a “network of networks”, there have been (and still are) multiple large, global networks to which you could connect your network and data centers. You may, in fact, connect to multiple of those large, global ISPs to have a higher degree of “resilience” for your network. The difference to me is that in connecting to those network providers (and paying to do so), you are connecting to the open, public Internet.

In contrast, connecting to these newer private networks gives you only access to the company’s cloud or content platform. It’s not the whole Internet.

So regarding definition #2, how does this evolution in network connectivity impact the overall resilience of the Internet to recover from issues?

I think it’s a very interesting question to consider as part of this program.

My 2 cents,
Dan

P.S. And yes, I’m well aware that some large enterprises also operate their own private, global networks / WANs to interconnect their own data centers - and have done so for many years. I think the difference is one of scale. The new “cloud” providers are operating networks significantly larger than any individual enterprise - and are also encouraging enterprises to move away from operating their own networks and to instead move their networking to these newer networks.

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 2 Jan 2020 08:12:19 +1300

Hi Dan,

> What if there winds up being a lack of diversity of paths through the “open” and “public” Internet? What if increasingly traffic winds up traveling through these proprietary global networks (to whi…

Is this really new, from a technical viewpoint? It reminds me very much of the early 1990s, when policy based BGP4 routing first became a thing, and acceptable use policies were applied by NSFNET, ESNET, and their equivalents in Europe and Asia. That was all about money, of course, except that it was public money.

Regards
   Brian

On 02-Jan-20 07:44, Dan York wrote:
>> On Dec 31, 2019, at 2:27 PM, Lucy Lynch <[email-masked] <mailto:[email-masked]>> wrote:
>>
> _______________________________________________
> Architecture-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/architecture-discuss
>

---

From: Dan York <[email-masked]>
Date: Wed, 1 Jan 2020 20:11:33 +0000

Brian,

On Jan 1, 2020, at 2:12 PM, Brian E Carpenter <[email-masked]<mailto:[email-masked]>> wrote:

Is this really new, from a technical viewpoint?

I’m not sure. I haven’t yet worked enough personally with these newer networks to be able to fully answer that question.  My knowledge comes from reading articles, going through tutorials, listening to talks / podcasts, and experimenting to the degree I can. But I don’t operate a network or data center where I could join in to one (or more) of them… so I have to defer to those who have that kind of more direct experience.

It reminds me very much of the early 1990s, when policy based BGP4 routing first became a thing, and acceptable use policies were applied by NSFNET, ESNET, and their equivalents in Europe and Asia. That was all about money, of course, except that it was public money.

Interesting thought. I guess the difference I could see was that in those days the networks were all using the same open standards-based protocols, even with AUP’s being applied. They were all weaving together a common network fabric, from what I saw as a network user at that time.

With these newer private networks, they may - or may not - be using the same protocols that the public Internet is using. You may or may not know. (And from a purely operational point-of-view, you may not care… a main reason for using the private network is to have low-latency connectivity to applications or content.)

In at least one of the cases (Amazon Outposts), my understanding is that the company comes and installs hardware in your data center that then connects your network back to Amazon’s AWS network. Is it all running open Internet protocols? Or are there proprietary protocols (or “extensions”) that are making the connectivity that much faster/better?  (And I honestly do not know.)  Do you only have connectivity to their platform(s)? Or do you have wider connectivity?

To me, this is part of the question - do these newer, global, private networks *increase* the overall resilience of the open, public Internet? Or do they *decrease* the overall resilience of the open, public Internet? Or do they not really have any effect? Or is it too soon to tell?

Interesting times,
Dan

---

From: Jared Mauch <[email-masked]>
Date: Wed, 1 Jan 2020 15:48:13 -0500

M

> On Jan 1, 2020, at 2:12 PM, Brian E Carpenter <[email-masked]> wrote:
> 
> Hi Dan,
> 

Many of us operate networks to meet our business goals. I think the issues that arise from that become relevant if one or more players requires certain networks or those one or two players are damaged or fail catastrophically. 

(Operational note: as we have privatized our own traffic we have found the public carriers no longer have the capability and capacity to handle the traffic if we place it back on their networks).

---

From: Jeff Tantsura <[email-masked]>
Date: Wed, 1 Jan 2020 15:40:39 -0800

While I share Lucy’ points, wrt connectivity of Outpost/similar platforms back to the centralized DC‘s - noone is reinventing the wheel(there’s no value), outer encap is IP,  do we really care what’s inside?

Regards,
Jeff

> On Jan 1, 2020, at 12:11, Dan York <[email-masked]> wrote:
> 
>  Brian,
>

---

From: Guntur Wiseno Putra <[email-masked]>
Date: Fri, 3 Jan 2020 09:10:36 +0700

Dear architecture-discuss,

"Communication" may be understood as "society" and Internet is about
networks of human communications...

If it matters --and perhaps others: (1) UN's inititives for Sustainable
Development Goals by supports of ICTs --including Internet & (2)
engineering Internet's (architectural) resiliency (*), one closed enough in
J. Hillier's "Strategic Navigation: a Multiplanar Methodology for Strategic
Spatial Planning" I mentioned above is "sustainability" referring to
longer-term plans.

"Multiplanar theory offers the potential for multiple plans (Hillier 2007,
2011) :

– several (or one collectively preferred) trajectories or “visions” of the
longer-term future, including concepts towards which human and non-human
actants “desire” to navigate, such as sustainability (Deleuze and
Guattari’s planes of immanence) ;

– shorter-term, location-specific detailed plans and projects with
collaboratively determined tangible goals, for example, for city centre
regeneration, provision of affordable housing and so on (Deleuze and
Guattari’s planes of organisation)".
(*) Apologize: I rendering "resilience" without accessing Stephen's
document directly at github --there was an icon feature I could not access
by clicking (the triple lines in up-right corner) and I wandered if there
were guidances to use the website...

Regard,
Guntur Wiseno Putra

Pada Kamis, 02 Januari 2020, Jeff Tantsura <[email-masked]>
menulis:

> While I share Lucy’ points, wrt connectivity of Outpost/similar platforms
> back to the centralized DC‘s - noone is reinventing the wheel(there’s no
> value), outer encap is IP,  do we really care what’s inside?
>
> Regards,
> Jeff
>
> What if there winds up being a lack of diversity of paths through the
> “open” and “public” Internet? What if increasingly traffic winds up
> traveling through these proprietary global networks (to which you need to
> pay to connect and through that gain permission to send traffic - and only
> to that company’s platforms)?
>
>
> I’m not sure. I haven’t yet worked enough personally with these newer
> networks to be able to fully answer that question.  My knowledge comes from
> reading articles, going through tutorials, listening to talks / podcasts,
> and experimenting to the degree I can. But I don’t operate a network or
> data center where I could join in to one (or more) of them… so I have to
> defer to those who have that kind of more direct experience.
>
> It reminds me very much of the early 1990s, when policy based BGP4 routing
> first became a thing, and acceptable use policies were applied by NSFNET,
> ESNET, and their equivalents in Europe and Asia. That was all about money,
> of course, except that it was public money.
>
>
> Interesting thought. I guess the difference I could see was that in those
> days the networks were all using the same open standards-based protocols,
> even with AUP’s being applied. They were all weaving together a common
> network fabric, from what I saw as a network user at that time.
>
> With these newer private networks, they may - or may not - be using the
> same protocols that the public Internet is using. You may or may not know.
> (And from a purely operational point-of-view, you may not care… a main
> reason for using the private network is to have low-latency connectivity to
> applications or content.)
>
> In at least one of the cases (Amazon Outposts), my understanding is that
> the company comes and installs hardware in your data center that then
> connects your network back to Amazon’s AWS network. Is it all running open
> Internet protocols? Or are there proprietary protocols (or “extensions”)
> that are making the connectivity that much faster/better?  (And I honestly
> do not know.)  Do you only have connectivity to their platform(s)? Or do
> you have wider connectivity?
>
> To me, this is part of the question - do these newer, global, private
> networks *increase* the overall resilience of the open, public Internet? Or
> do they *decrease* the overall resilience of the open, public Internet? Or
> do they not really have any effect? Or is it too soon to tell?
>
> Interesting times,
> Dan
>
>

---

From: Guntur Wiseno Putra <[email-masked]>
Date: Fri, 3 Jan 2020 21:19:40 +0700

Dear architecture-discuss
(Stephen and Others),

When we think about "cyberspace", and also "real space", of communications
networks, of societies, and about engineering Internet architecturally,
this quotation concerning with "design" --as the term has so far been part
of this thread of discussion-- may be inspiring:

Navigating strategically across multiple planes requires practitioners to
sense and discern connections and patterns in what is taking place, to try
to understand the underlying dynamics and interdependencies between
elements, to appreciate the diverse possibilities of what is happening and
what might happen, and to respond by designing actions which align with the
intentions and values of the agreed longer-term strategic trajectory (Hames
2007a, p. 114), but which are contextually appropriate, not copy/pastes of
other, previous or “best” practices (Hillier 2011, p. 508).

(being cited from J. Hillier's ".. Strategic Spatial Planning" I mentioned
in the previous message)

Note: I have just realized that the 1st message of this thread of
discussion --Stephen's one-- was sent by [email-masked]. Has
been there a transfer from @ietf to @iab in managing the
architecture-discuss list --while my accepted registration to
architecture-discuss@ietf was in March 2019...?

Regard,
Guntur Wiseno Putra

Pada Jumat, 03 Januari 2020, Guntur Wiseno Putra <[email-masked]>
menulis:

>> It reminds me very much of the early 1990s, when policy based BGP4
>> routing first became a thing, and acceptable use policies were applied by
>> NSFNET, ESNET, and their equivalents in Europe and Asia. That was all about
>> money, of course, except that it was public money.
>>
>>

---

From: Guntur Wiseno Putra <[email-masked]>
Date: Fri, 3 Jan 2020 21:19:40 +0700

Regard,
Guntur Wiseno Putra

---

From: Guntur Wiseno Putra <[email-masked]>
Date: Fri, 3 Jan 2020 21:44:47 +0700

Note: I have just realized that the 1st message of this thread of
discussion --Stephen's one-- was sent by [email-masked]. Has
been there a transfer from @ietf to @iab in managing the
architecture-discuss list --while my accepted registration to
architecture-discuss@ietf was in March 2019...? I make also BCC for this
message to [email-masked]...

Regard,
Guntur Wiseno Putra

Pada Jumat, 03 Januari 2020, Guntur Wiseno Putra <[email-masked]>
menulis:

> discussion --Stephen's one-- was sent by [email-masked].
> Has been there a transfer from @ietf to @iab in managing the
>> "Multiplanar theory offers the potential for multiple plans (Hillier
>> 2007, 2011) :
>>
>> – several (or one collectively preferred) trajectories or “visions” of
>> the longer-term future, including concepts towards which human and
>> non-human actants “desire” to navigate, such as sustainability (Deleuze and
>>> While I share Lucy’ points, wrt connectivity of Outpost/similar
>>> platforms back to the centralized DC‘s - noone is reinventing the
>>> wheel(there’s no value), outer encap is IP,  do we really care what’s
>>> inside?
>>>
>>> Regards,
>>> Jeff
>>>
>>> Interesting thought. I guess the difference I could see was that in
>>> those days the networks were all using the same open standards-based
>>> protocols, even with AUP’s being applied. They were all weaving together a
>>> common network fabric, from what I saw as a network user at that time.
>>>

---

From: Guntur Wiseno Putra <[email-masked]>
Date: Fri, 3 Jan 2020 21:44:47 +0700

Regard,
Guntur Wiseno Putra

---

From: Stephen Farrell <[email-masked]>
Date: Fri, 3 Jan 2020 20:13:13 +0000

Hi all,

Thanks for the ~70 messages on this over the
holidays - I think that demonstrates a clear and
thoughtful interest in the topic, which is great.
One concern is that there were only a few people who
said they'd do work (as opposed to comment on work),
so we might not know for a while if this'll take off
(and be useful) or not.  I think that's ok, but feel
free to send mail saying you would do work if that's
the case.

Meanwhile, my summary of the thread so far is below
- please correct me if I've gotten anything badly
wrong. (I'll offer this sub-thread to the IAB as a
summary of the discussion - many people, IAB members
included, are busy digesting things other than email
over the holidays:-)

Lastly, the initial idea for this originated from
Terry Manderson and not from within the IAB - Terry
told it to me over some beers at the very end of
IETF105 in Montreal and I related it to the IAB, who
were supportive.  Now that we've seen that people
don't think the idea is totally crazy, I think it's
a good time to ack Terry for that. Terry's not
responsible for any badness in charter-text of
course - that'd be down to my or collective IAB
ineptitude:-)

Cheers, S.

My notes on the thread [1] so far, roughly in order
of how concrete the subsequent action seems (to me)
to be.

(1) Various folks noted that robustness is highly
related to resilience, and that both may be
terms-of-art for some people. I thought the
message from Laurent [2] quoting Jeurgen
Schoenwaelder captured that nicely. My guess is that
a bit of charter editing to try embed that
distinction may be worthwhile to avoid potential
confusion.

(2) Marc [3] suggested that another example (e.g.
DDoS) might be useful to add so it's more clear that
diversity is not a panacea. Seems reasonable.

(3) Lucy [4] and others asked about sense#1 of the
quoted definition of resilience.  Seems to me the
high level bit of the answer is "yes, that could be
in scope" though of course work only happens when
people volunteer to do it, (and then actually do
it:-), so "in scope" is not at all the same as
saying that something will happen. I think the
"mostly" in the charter text is fine for that, but
changing to "mostly, but not exclusively," would
maybe be good.  (As an aside, that sub-thread
describing various things as "compression" is
interesting, maybe even cute:-)

(4) Toerless [5] asked about "membership" of the
programme, raising both the issues of potential
cliques and EAR related reasons why
smokey-back-rooms are even less desirable today
compared to a few years ago. I think that's one
where the IAB should probably chat - it may indicate
a need to change our conception of programmes and
membership a bit, both to be more open and to handle
those EAR-related issues.  (Personally, I'd like if
we could find a modus-operandi that's fully open,
but that can also recognise a smaller set of folks
as being active participants that the IAB think
might usefully contribute.)

(4) In a long sub-thread various people wondered
whether the proposed programme
would/could/should/should-not discuss various policy
or layer-9 issues, such as centralisation, (that
also had a bit of "#include <doh-discussion>"). I
guess we'll have to see what the IAB make of that
but FWIW, my take is that it generally makes sense
for IAB programmes to not be limited to
consideration of layers 1-8, but that no IAB
programme ought be seen as the exclusive venue for
any discussion, and nor does any IAB programme bind
the IETF to anything.  If the IAB agreed with that,
I think that'd maybe mean that the technical
consequences of such issues are in scope for
discussion, but with two caveats: 1) we need to keep
in mind that saying something is in-scope for a
programme does not mean that anything will happen -
as always that depends on someone doing (good) work
on a topic, writing that up for others to discuss,
and actually finishing the work and 2) there's still
a preference for IETF work to be done in the IETF,
so when/if anything in this space gets to be very
concrete (i.e. protocol specific or BCP-like) it'd
need to be decided in the IETF and will not be
decided in/by this IAB programme. I don't think any
charter-text change is needed for any of that.

(5) There was a mention of things being done in
ICANN, ITU or IGF rather than in an IAB programme. I
don't think those ideas received much support that I
could see, so don't think the charter needs changes
for this.  (In saying that, I'm not making any
comment on whether or not it'd be a good or bad idea
for those bodies to take up any related
discussions.)

[1]
https://mailarchive.ietf.org/arch/msg/architecture-discuss/pZR9EYOhTIir0m_YDXQkUGPXwfk
[2]
https://mailarchive.ietf.org/arch/msg/architecture-discuss/iFQVe_1QdiT7JfQqxt9T_b3bp8w
[3]
https://mailarchive.ietf.org/arch/msg/architecture-discuss/pYATTKqH_Avi7D1tOGjz7XD1RHQ
[4]
https://mailarchive.ietf.org/arch/msg/architecture-discuss/qo4evXDIVVDhgNKQ0XzzSMsXwpM
[5]
https://mailarchive.ietf.org/arch/msg/architecture-discuss/6dqFdDrq8DXICx0hACx_rwob6nU

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 13 Jan 2020 15:32:00 +0100

On Wed, Jan 01, 2020 at 06:44:21PM +0000,
 Dan York <[email-masked]> wrote 
 a message of 480 lines which said:

> For example, at Amazon’s recent re:Invent conference they promoted a
> way to directly connect enterprise data centers to Amazon’s global
> AWS network (“Outposts”) and also a way to connect telco
> points-of-presence to Amazon’s network (“Wavelength”). My
> understanding is that Microsoft has similar functionality for Azure
> (“Stack”, I believe) and Google either has or is working on
> something similar for their Google Cloud Platform. Similarly, large
> entities such as Facebook and Netflix have built their own global,
> private networks that interconnect to local data centers (where
> those data centers are also connected to the public Internet).

Some people here are probably aware of it, but I think it is an
opportunity to mention Geoff Huston's articles about the "death of
transit" <http://www.potaroo.net/ispcol/2017-03/gilding.html>
<https://2017.apricot.net/program/schedule/#/day/9/apnic-panel---forces-shaping-the-network>. I
don't agree with him but these articles seem relevant for this
specific part of the discussion.

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 13 Jan 2020 16:37:41 +0100

On Tue, Dec 31, 2019 at 07:27:17PM +0000, Lucy Lynch wrote:
> internet that supports real time data collection and shared computation in
> the context of big science. How do we avoid closing out capabilities as we
> optimize for others? Narrowing of choices looks like a path to a limited and
> more brittle model to me.

Indeed. With the new unique value propositions of new applications
like mobile phones and video streaming, the network layer quality and
availability acceptable to users deteriorated quite a bit over pre-IP
solutions.

There was a lot of growth of more network service reliability/quality
demanding applications, but it all happened in vertical industries and
mostly private networks and is IMHO still heavily discriminated against
by the way the IETF operates.

The IETF only has "Internet best (lousy) effort" as the gold standard: Every
new standard is forced to argue how it would behave well if
unintentionally leaking onto the BE Internet, no new standard is harrassed
to support better quality/reliability network layer services (IntServ,
DiffServ, DetNet, etc. pp). 

To me, this is really like asking new transportation standards to explain how
they are compatible with horse manure because horses are still the
golden standard for transportation.  So we continuously reinforce this historic
lowest common denominator and do not encourage really systematically 
the value proposition of better quality/resilience network infrastructures.

Toerless

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 15 Jan 2020 10:07:55 +0900

Replying to one point here ...

> DiffServ, DetNet, etc. pp).

This was probably true when I joined the IESG in 2013, which seems like
just yesterday in IETF timelines, but after the MPLS-in-UDP kerfluffle, the
TSV ADs and RTG ADs agreed that for specifications that might not behave
well on the open Internet, what we would ask was that these proposals
include a description of what safety mechanisms needed to be in place if
the protocol being described WAS running over the open Internet.

We put together a design team to figure out what to do, and that eventually
led to Transport Circuit Breakers, as in https://tools.ietf.org/html/rfc8084,
also BCP 208, which built on https://tools.ietf.org/html/rfc7510#section-5,
an early example of the kind of "congestion considerations" I'm talking
about, and produced by that design team.

I agree that this is hard, but it is possible to do better than best effort.

Best,

Spencer

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 15 Jan 2020 17:13:29 +0100

> I agree that this is hard, but it is possible to do better than best effort.

If i may stay to me naming best effort (BE) Internet traffic as horses then i
agree that this was very good work to ensure horses will not be harmed
on 2030 internet highways. Alas, i can not really see what else IETF is doing 
for a 2030 internet higway to support anything more than horses.

[rant]
I really wished the IETF would wake up to the need to not only bother about
horses and work more on better than BE modes of transport for those future networks.
CC work is still regurgitating the "all flows are equal" mantra and "we don't
define exactly what equal means", and we know forever this is not sufficient.
DetNet is already a decade behind IEEE TSN without any ability to build products,
IP doesn't even have all the pieces to support it (only MPLS), switched ethernets 
networks widely replacing IP networks in growth markets such as metropolitan
networks, any form of traffic differentiation is immediately faced with more
"net neutrality" misinterpretation in the IETF than encouragement to actually
work on it, and the expectation of the IETF for new work to adopt itself
to the historic structure of the IETF, its magically closed IAB, fine
grained subdivision of work across WGs and the like is just mindboggingly
unhelpful to strategic future work.
[/rant]

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Jared Mauch <[email-masked]>
Date: Wed, 15 Jan 2020 12:25:29 -0500

> On Jan 13, 2020, at 9:32 AM, Stephane Bortzmeyer <[email-masked]> wrote:
> 

I think it’s perhaps less about resilience and more about diversity and some natural monopolies that will always exist.  I’ve always said things like “physical access = root”, and the same is true with many of these things.  You may not have diversity until you reach the major aggregation points, which in my home are either Southfield, MI or Chicago.  It’s extremely unlikely my packets go anywhere else due to the underlying physical topology and costs.

We have seen large volumes of data be privatized as a result, including the network at my employer.

- Jared