Subject: [Lsr] Thoughts on the area proxy and flood reflector drafts.

From: Christian Hopps <[email-masked]>
Date: Sat, 6 Jun 2020 07:18:52 -0400

[ all the following is as a WG member ]

I've been thinking a lot about the proposed flooding reduction drafts currently vying for adoption by the WG. I've been doing this thinking in the context of wearing an operator hat (i.e., trying to leverage my prior experience working for DT on Terastream). In the abstract the Terastream architecture presented a good way to visualize and compare the benefits of using one of these solutions w/o getting too complex. A simplified model of this design can be seen as a horseshoe of horseshoes. the Major horseshoe is L2 and the minor horseshoes are L1. Each L1 area has 2 L2 routers for redundancy (I'll consider more though), and all L2 routers are full-mesh connected to support that redundancy.

Telco

But let's map this to a more DC centric view (I guess?) where each L1 area now has 10 L2 routers instead of 2 (i.e., 10%, but that could be change that later if need be).

Natural Design

Now for whatever reason some operators do not want to provision high-bandwidth transit links between their L2 routers in their L1 areas. This is critically important b/c otherwise you would simply use the above Natural Design. I'd like to better understand why that isn't just the answer here.

Anyway, forging ahead, here's what we get with unchanged IS-IS to support "use everything for transit"

All In

So the 800 L2 LSPs, and the impact on flooding dynamics, are what these drafts are trying to reduce or avoid.

Area Proxy

First I'll look at area proxy. This seems a fairly simple idea, basically it's taking the now L1L2 areas and advertising them externally as a single LSP so the impact is very similar to if they were L1 only. This maps fairly closely to the Telco and Natural Design from above. Each L1 router in the Telco design would have 100 LSPs The L12 routers would have 100 L1 + 16 L2 LSP. In the Natural Design each L1 router has 100 L1 and each L12 router would have 100 L1 and 80 L2. With Area Proxy each router  has 100 L1 and 100 "Inner L2 LSPs" and 80 "Outer LSPs" + 8 "Outer L2 LSPs"

The key thing to note here is that if you double the number of areas you only add to the Outside LSP and Proxy count just as it would scale in the Natural Design, so going from 8 to 16 areas here adds 80 more "Outside LSPs" and 8 more L2 Proxy LSPs for a total of 276 L2 LSPs even though you've added 800 more routers to your network.

Flood Reflectors

I'm less sure I understand this draft so please forgive me if I get this wrong. After reading this draft a few times I believe it is basically providing "L2 virtual links" over an L1 area between the area's L2 edge routers and then using a "flood reflector" to reduce the number of required "virtual links" by creating a hub-and-spoke topology of them.

The draft does a bit of hand-waving at "judicious scoping of reachability" being used in place of tunneling. I could guess at what this might mean, but I shouldn't have to; the draft should spell it out so it can be judged as a viable option or not. So, the only choice I see presented is to use a full-mesh of L1 tunnels between the L12 edge routers.

Anyway, here's a picture

If, in fact, the draft is talking about adding virtual links to IS-IS I think it should say that. There's a lot of experience in OSPF with virtual links and some of the trouble they have caused. There's also important details in making them work right in OSPF that doesn't seem to be in the current draft, so it's not clear what actual level of complexity is going to be required to make this all work, and importantly, if that would then be palatable.

I also do not like the idea of all of these automatic "fowarding-tunnels". They would be disjoint from the advertised "flood reflector" tunnel topology (right?) -- it seems like a management/debugability nightmare to me, and I'm curious which operators are keen to have a bunch of unadvertised tunnels added to their IGP network. :) If the draft really can work in general w/o the tunnels I would appreciate seeing those mechanics described rather than just hinted at.

The ideas are interesting for sure, but the draft doesn't seem fully fleshed out, and I'm worried it'll be overly complex when it is.

Thanks,
Chris.
[as WG member]

---

From: Tony Li <[email-masked]>
Date: Sat, 6 Jun 2020 14:15:01 -0700

Hi Chris,

Thank you for your thoughtful comments.

> A simplified model of this design can be seen as a horseshoe of horseshoes. the Major horseshoe is L2 and the minor horseshoes are L1. Each L1 area has 2 L2 routers for redundancy (I'll consider more though), and all L2 routers are full-mesh connected to support that redundancy.
> 
> Telco
> 
> 

The key point here is not a DC centric view, but one of scale. The above worked just fine back when we were terminating T1’s and E1’s.  Now, PE routers easily have terabit capacity. WIth the 100:1 ratio between the PE and the L1L2 routers (Area Edge routers), the bandwidth requirements drive us to petabit, multi-chassis routers. While these have been great fun to build, operators are not happy with this direction. Forklift upgrades are necessary with every silicon cycle. The complexity of the router has multiplied, the blast radius is off the charts, and the premiums charged for these devices are impressive. As a result, folks are trying to explore a ’scale-out’ approach. Rather than 2 huge area edge routers, they would much rather be able to scale out the area edge to be many routers.

If you pursue this design direction, the first thing you observe is that you can see is that you cannot afford to build a full-mesh of inter-area circuits across all of the area edge routers.

And if your network is a bit more geographically dispersed, you find that it becomes inefficient to have even a full mesh at the area level. This forces some areas to become transit.

Between these two forces, we are compelled to push transit traffic through the internals of an area.

> Now for whatever reason some operators do not want to provision high-bandwidth transit links between their L2 routers in their L1 areas. This is critically important b/c otherwise you would simply…

In this design, you’ve shown ten area edge routers.  Yes, you could provision a full mesh of links between them.  The issue remains one of scalability and uniformity. The number of ports per area edge router has to scale linearly with the size of area edge and the overall number of links used for this purpose is O(n^2).  Combine this with the fact that any non-uniformity in the traffic pattern and your full mesh ends up being congested and under-utilized simultaneously.  

All is not lost, however. Charles Clos to the rescue. By structuring each area as a Clos (or Benes) network (which my employer seems to insist on spelling ‘leaf-spine’), you avoid this. I assume I don’t need to go into details on this.  

> First I'll look at area proxy. This seems a fairly simple idea, basically it's taking the now L1L2 areas and advertising them externally as a single LSP so the impact is very similar to if they wer…

We’ve made some changes in the latest version of the draft.  In the current version, we require that each router in the area be L1L2.  However, only one LSP is advertised externally for each area.  Thus, each router will see 100 L1 LSPs, 100 L2 LSPs and 8 L2 Proxy LSPs.

> The key thing to note here is that if you double the number of areas you only add to the Outside LSP and Proxy count just as it would scale in the Natural Design, so going from 8 to 16 areas here a…

Doubling the number of areas would give you 16 L2 proxy LSPs, so you end up going from 208 LSPs to 216.  The key point here is that the database now scales linearly with the number of areas.

Demo time:

In a lab setup, we have an area of five routers.  We have three L2 routers.  The database on one of the pure L2 routers is just 5 entries (three L2, one proxy LSP, one pseudonode):

rtrmpls8>show isis data

IS-IS Instance: Amun VRF: default
  IS-IS Level 2 Link State Database
    LSPID                 Seq Num   Cksum  Life  IS Flags
    ip6.00-00             5         39442  1077  L2 <>
    ip7.00-00             6         40764  1077  L2 <>
    ip8.00-00             4         17533  1071  L2 <>
    ip8.06-00             1         42260  1071  L2 <>
    Proxy.00-00           1         13553  1087  L2 <>

On an inside node, the L1 data is 7 entries (including 2 pseudonodes), and the L2 database is 12 (including 3 pseudonodes):

rtrmpls4>show isis data

IS-IS Instance: Amun VRF: default
  IS-IS Level 1 Link State Database
    LSPID                 Seq Num   Cksum  Life  IS Flags
    ip1.00-00             6         57570  962   L2 <DefaultAtt>
    ip2.00-00             5         12641  963   L2 <DefaultAtt>
    ip2.06-00             1         26206  963   L2 <>
    ip3.00-00             5         30428  964   L2 <DefaultAtt>
    ip4.00-00             5         58405  962   L2 <DefaultAtt>
    ip4.07-00             1         35124  955   L2 <>
    ip5.00-00             5         55336  962   L2 <DefaultAtt>
  IS-IS Level 2 Link State Database
    LSPID                 Seq Num   Cksum  Life  IS Flags
    ip1.00-00             10        890    966   L2 <>
    ip2.00-00             7         4390   965   L2 <>
    ip2.06-00             1         61120  963   L2 <>
    ip3.00-00             7         43115  965   L2 <>
    ip4.00-00             7         64064  962   L2 <>
    ip4.07-00             1         6798   955   L2 <>
    ip5.00-00             9         34969  975   L2 <>
    ip6.00-00             5         39442  967   L2 <>
    ip7.00-00             6         40764  966   L2 <>
    ip8.00-00             4         17533  961   L2 <>
    ip8.06-00             1         42260  961   L2 <>
    Proxy.00-00           1         13553  975   L2 <>

Regards,
Tony

---

From: Jeff Tantsura <[email-masked]>
Date: Sat, 6 Jun 2020 14:25:50 -0700

Thanks Chris/Tony,

I wish we’d have more of this kind of discussions on the list, discussing pro/cons of the solutions proposed!

Have a great weekend!

Regards,
Jeff

> On Jun 6, 2020, at 14:15, Tony Li <[email-masked]> wrote:
> 
> 
> 
> Hi Chris,
> 
> _______________________________________________
> Lsr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lsr

---

From: Christian Hopps <[email-masked]>
Date: Sat, 6 Jun 2020 20:56:02 -0400

> On Jun 6, 2020, at 5:15 PM, Tony Li <[email-masked]> wrote:
> 
> 

Ah yes, I missed that the Inside Edge Routers were masquerading using the Proxy System ID to the Outside Edge Routers. This does offer even better scaling.

This also does directly address your use case where the Area edge is very large compared to old designs.

Interestingly, if we consider Peter's pair of huge "R2" (L1L2 area edge) in Terastream (Telco Design), in effect you are creating a single Proxy LSP that continues to represent that concept topologically (although with a single "huge" router (Proxy LSP) rather than a pair of "R2"s), while allowing the huge multi-chassis physical routers to now be made up of N-many smaller physical routers.

On the industry direction, it'll be interesting to see if using N-many physical routers with all the external L2/L3 interconnect required and the extra operational/management costs actually works better than the single large multi-chassis with it's internal fabric interconnect that could be managed as a single router. It would seem to be adding a lot of operational cost on to the operator. Terastream was specifically trying to avoid this extra operational cost, as capex wasn't the problem, the opex was. Not having to forklift upgrade these huge routers was critical to that design though, and it sounds like maybe we're giving up on the vendors delivering on this?

Thanks,
Chris.
[as WG member]

P.S. Curiously, I had "fixed" my diagram and text just prior to sending the email as originally I had it as 108 L2 LSPs and then 216 when doubling the network, maybe I had noticed subconsciously. :)

---

From: Christian Hopps <[email-masked]>
Date: Tue, 9 Jun 2020 20:46:47 -0400

Hi,

Given that the flood reflector authors have asked the chairs to do a call for adoption, might someone from that group be able to talk to a couple of the points/questions from my mail? I think it would help me (and maybe others) in making informed responses to any adoption call.

- Is this proposing to add "virtual links" to IS-IS (in addition to the flood reflector part)?

- Is there a general-purpose non-complex non-tunneling solution possible?

Thanks,
Chris.
[as WG member]

> On Jun 6, 2020, at 7:18 AM, Christian Hopps <[email-masked]> wrote:
>

---

From: Tony Przygienda <[email-masked]>
Date: Tue, 9 Jun 2020 19:01:34 -0700

Chris (addressing in WG member context you declared), I reply tersely since
we will put more work into the draft once it's adopted (for which I think
you saw a good amount of support in two threads already).

I deferred from your email since the chain-terrasteam topology you're
showing is simply not what we are dealing in any operational, successful
networks today AFAIK frankly and I saw lots of "assume complexity" and
"dislikes" in your email which I didn't read as technical arguments but
mental attitudes. Likes or dislikes and assumptions are fine but we should
probably focus on existing network/customer technical + operational
arguments & requirements when building solutions and now what you or we
like first.

So trying to extract the technical point you seem to be making inbetween
all that

a) I see how you can try to have a mental model of "virtual links". What we
suggest are not virtual links (I implemented VL @ least once but it's so
long ago I forgot pretty much all the details so had to look stuff up :-)
Virtual links in OSPF were "magic bits on LSAs" that kind of computed "SPF
reachability through the area to change SPF" edge-to-edge and the
asynchronicity of all that flooding-being-tunnels-being-SPF was playing
havoc on us @ this time of 300MHz CPUs + frankly, the complexity of that
was not needed @ this time just as partition healing was never implemented
in ISIS. That's why it never went anywhere much, my take, others may
correct. Saying "virtual links are bad" and "this is virtual links to me so
it's bad" is simply a "strawman fallacy" to me frankly. This draft suggests
(but as I wrote Bruno as answer to his fairly deep email) to run proper
flooding over proper tunnels (we run routing over tunnels all the time in
all kind of scenarios be it BGP proper or SD-WAN or overlays obviously) but
if you choose FRs to be one hop away you can get away without any "L2
tunneled adjacencies", that's deployment choice.
b) Generally we may seem a bit muddled between different types of "tunnels"
and "tunnels are bad" and "lots of tunnels". The draft talks about 2 types
of tunnels and it seemed to be written clearly enough to distinguish that
easily based on feedback I got so far

i) control plane tunnels are proper L2 entities (again, if your FR is one
hop away from leafs then you don't need any tunneling but can run normal L2
adjacencies which I hope are not too scary; whole thing is really
equivalent to BGP RR, do you put it in path, do you want to run multi-hop
and how confident are you in your lower level infra not dropping TCP
"tunnels" under you; every day's business since years really). So, no,
there is no magic and hidden complexity and whatever not, you may or may
not use auto-discovery the draft provides (you can just build something
completely statically configured and in fact several customers told me
they'd prefer it that way just as they don't auto-discover RRs normally) to
build bunch of tunnels towards your 2-3 FRs from your edges and you're in
business after L2 adjacency comes up. Looks like any old ISIS + some
optional TLVs you can ignore that indicate for some smart future folks to
know it's a FR adjacnecy and not "real L2 adjaqcency". No fork-lifting of
whole cluster, no fork-lifting routers outside a cluster, no single point
of failure under some fancy new name, barely any protocol changes (in fact
I didn't see any proposal to run anything more minimal than that except
maybe very simple version of TTZ which however has too many L2 adjacencies
exposed @ any reasonable cluster size to really solve the problem of amount
of L2 information sloshed around)

ii) data plane tunnels. The draft basically explains that for the solution
to work in a simple fashion,full-mesh of data forwarding tunnels can be
established (which are NOT visible in L2) as shortcuts that allow to
utilize all paths through L1 and that will work fine since it doesn't spill
into L2. You want to run L1 adjacencies over those tunnels if you care
whether they are up but you could do just BFD e.g. and use them as
forwarding next-hops in the computation without them being visible in L1
ISIS. The other option is to not use such a data plane mesh and use
reachability instead and we can explain that further in detail after
adoption and we get more people talking through that etc. or you can look @
my preso @ last IETF where I kind of quickly ran through that (and it
seemed relatively obvious to me how it works). In summary, we chose to do
real work rather than polilsh optional points in individual drafts because,
frankly, customers are not interested all that much often whether IETF WG
feels like working on it while they have a pressing problem and need
solutions in a timely manner. And AFAIR the chairs guided the group
multiple times towards "ignore the problem, of no importance" and now with
a certain urgency want to have everything @ the same time.

So, I look forward to this draft being adopted as WG item given the support
seen and urgency of having this in the field and obviously, being running,
working code since a bit ...

-- tony

On Tue, Jun 9, 2020 at 5:46 PM Christian Hopps <[email-masked]> wrote:

> Hi,
>
> Given that the flood reflector authors have asked the chairs to do a call
> for adoption, might someone from that group be able to talk to a couple of
> the points/questions from my mail? I think it would help me (and maybe
> others) in making informed responses to any adoption call.
>
> > I've been thinking a lot about the proposed flooding reduction drafts
> currently vying for adoption by the WG. I've been doing this thinking in
> the context of wearing an operator hat (i.e., trying to leverage my prior
> experience working for DT on Terastream). In the abstract the Terastream
> architecture presented a good way to visualize and compare the benefits of
> using one of these solutions w/o getting too complex. A simplified model of
> this design can be seen as a horseshoe of horseshoes. the Major horseshoe
> is L2 and the minor horseshoes are L1. Each L1 area has 2 L2 routers for
> redundancy (I'll consider more though), and all L2 routers are full-mesh
> connected to support that redundancy.
> >
> > Telco
> >
> >
> > But let's map this to a more DC centric view (I guess?) where each L1
> area now has 10 L2 routers instead of 2 (i.e., 10%, but that could be
> change that later if need be).
> >
> > Natural Design
> >
> >
> >
> > Now for whatever reason some operators do not want to provision
> high-bandwidth transit links between their L2 routers in their L1 areas.
> This is critically important b/c otherwise you would simply use the above
> Natural Design. I'd like to better understand why that isn't just the
> answer here.
> >
> > First I'll look at area proxy. This seems a fairly simple idea,
> basically it's taking the now L1L2 areas and advertising them externally as
> a single LSP so the impact is very similar to if they were L1 only. This
> maps fairly closely to the Telco and Natural Design from above. Each L1
> router in the Telco design would have 100 LSPs The L12 routers would have
> 100 L1 + 16 L2 LSP. In the Natural Design each L1 router has 100 L1 and
> each L12 router would have 100 L1 and 80 L2. With Area Proxy each router
> has 100 L1 and 100 "Inner L2 LSPs" and 80 "Outer LSPs" + 8 "Outer L2 LSPs"
> >
> > The key thing to note here is that if you double the number of areas you
> only add to the Outside LSP and Proxy count just as it would scale in the
> Natural Design, so going from 8 to 16 areas here adds 80 more "Outside
> LSPs" and 8 more L2 Proxy LSPs for a total of 276 L2 LSPs even though
> you've added 800 more routers to your network.
> >
> >
> >
> > Flood Reflectors
> >
> > I'm less sure I understand this draft so please forgive me if I get this
> wrong. After reading this draft a few times I believe it is basically
> providing "L2 virtual links" over an L1 area between the area's L2 edge
> routers and then using a "flood reflector" to reduce the number of required
> "virtual links" by creating a hub-and-spoke topology of them.
> >
> > The draft does a bit of hand-waving at "judicious scoping of
> reachability" being used in place of tunneling. I could guess at what this
> might mean, but I shouldn't have to; the draft should spell it out so it
> can be judged as a viable option or not. So, the only choice I see
> presented is to use a full-mesh of L1 tunnels between the L12 edge routers.
> >
> > Anyway, here's a picture
> >
> >
> > If, in fact, the draft is talking about adding virtual links to IS-IS I
> think it should say that. There's a lot of experience in OSPF with virtual
> links and some of the trouble they have caused. There's also important
> details in making them work right in OSPF that doesn't seem to be in the
> current draft, so it's not clear what actual level of complexity is going
> to be required to make this all work, and importantly, if that would then
> be palatable.
> >
> > I also do not like the idea of all of these automatic
> "fowarding-tunnels". They would be disjoint from the advertised "flood
> reflector" tunnel topology (right?) -- it seems like a
> management/debugability nightmare to me, and I'm curious which operators
> are keen to have a bunch of unadvertised tunnels added to their IGP
> network. :) If the draft really can work in general w/o the tunnels I would
> appreciate seeing those mechanics described rather than just hinted at.
> >
> > The ideas are interesting for sure, but the draft doesn't seem fully
> fleshed out, and I'm worried it'll be overly complex when it is.
> >
> > Thanks,
> > Chris.
> > [as WG member]
> >

---

From: Christian Hopps <[email-masked]>
Date: Wed, 10 Jun 2020 07:27:47 -0400

> On Jun 9, 2020, at 10:01 PM, Tony Przygienda <[email-masked]> wrote:
> 
> Chris (addressing in WG member context you declared), I reply tersely since we will put more work into the draft once it's adopted (for which I think you saw a good amount of support in two threads already). 
> 
> I deferred from your email since the chain-terrasteam topology you're showing is simply not what we are dealing in any operational, successful networks today AFAIK frankly and I saw lots of "assume complexity" and "dislikes" in your email which I didn't read as technical arguments but mental attitudes. Likes or dislikes and assumptions are fine but we should probably focus on existing network/customer technical + operational arguments & requirements when building solutions and now what you or we like first. 

Both Area Proxy and Flood Reflector are proposing to use L1 areas as transit to connect L2, isn't that chaining? It seemed like a decent way to help visualize the proposals along with some numbers, perhaps you have something better...

The Area Proxy draft is making everything L2 and using the L1 areas to redefine the advertised topology information to allow it to scale. Because everything is ultimately L2 nothing changes in the data plane to provide this transit.

The flood reflector draft is keeping the L1-only abstraction so it has to provide for transit some other way.

> So trying to extract the technical point you seem to be making inbetween all that
> 
> a) I see how you can try to have a mental model of "virtual links". What we suggest are not virtual links (I implemented VL @ least once but it's so long ago I forgot pretty much all the details so had to look stuff up :-) Virtual links in OSPF were "magic bits on LSAs" that kind of computed "SPF reachability through the area to change SPF" edge-to-edge and the asynchronicity of all that flooding-being-tunnels-being-SPF was playing havoc on us @ this time of 300MHz CPUs + frankly, the complexity of that was not needed @ this time just as partition healing was never implemented in ISIS.. That's why it never went anywhere much, my take, others may correct. Saying "virtual links are bad" and "this is virtual links to me so it's bad" is simply a "strawman fallacy" to me frankly. This draft suggests (but as I wrote Bruno as answer to his fairly deep email) to run proper flooding over proper tunnels (we run routing over tunnels all the time in all kind of scenarios be it BGP proper or SD-WAN or overlays obviously) but if you choose FRs to be one hop away you can get away without any "L2 tunneled adjacencies", that's deployment choice. 

If you are not using tunnels, but are still trying to provide transit through the L1 area for L2, that is exactly what OSPF virtual links are doing. Part of making those work is the advertisement of reachability into the area from another, changes to router advertisements (to indicate if the area is transit capable), and changes to the SPF calculation to modify route choices based on whether they are based on virtual links or not.

The complexity of OSPF virtual links is there to make them work, right? I'm not trying to make any argument, strawman or otherwise; I'm just trying to understand what is being proposed as a replacement for the full-mesh of tunnels.

It's nice if it reduces to OSPF virtual links b/c we then have an example of how to actually implement it, and years of experience to understand it. If that highlights that getting the non-tunneled choice right isn't easy, well I guess that's important, right?

> b) Generally we may seem a bit muddled between different types of "tunnels" and "tunnels are bad" and "lots of tunnels". The draft talks about 2 types of tunnels and it seemed to be written clearly enough to distinguish that easily based on feedback I got so far
> 
> i) control plane tunnels are proper L2 entities (again, if your FR is one hop away from leafs then you don't need any tunneling but can run normal L2 adjacencies which I hope are not too scary; whole thing is really equivalent to BGP RR, do you put it in path, do you want to run multi-hop and how confident are you in your lower level infra not dropping TCP  "tunnels" under you; every day's business since years really). So, no, there is no magic and hidden complexity and whatever not, you may or may not use auto-discovery the draft provides (you can just build something completely statically configured and in fact several customers told me they'd prefer it that way just as they don't auto-discover RRs normally) to build bunch of tunnels towards your 2-3 FRs from your edges and you're in business after L2 adjacency comes up. Looks like any old ISIS + some optional TLVs you can ignore that indicate for some smart future folks to know it's a FR adjacnecy and not "real L2 adjaqcency". No fork-lifting of whole cluster, no fork-lifting routers outside a cluster, no single point of failure under some fancy new name, barely any protocol changes (in fact I didn't see any proposal to run anything more minimal than that except maybe very simple version of TTZ which however has too many L2 adjacencies exposed @ any reasonable cluster size to really solve the problem of amount of L2 information sloshed around)

I think I understand the flood reflector bit; It's replacing the network of (now data-plane only) tunnels in flooding graph and the advertised topology.

> ii) data plane tunnels. The draft basically explains that for the solution to work in a simple fashion,full-mesh of data forwarding tunnels can be established (which are NOT visible in L2) as shortcuts that allow to utilize all paths through L1 and that will work fine since it doesn't spill into L2. You want to run L1 adjacencies over those tunnels if you care whether they are up but you could do just BFD e.g. and use them as forwarding next-hops in the computation without them being visible in L1 ISIS. The other option is to not use such a data plane mesh and use reachability instead and we can explain that further in detail after adoption and we get more people talking through that etc. or you can look @ my preso @ last IETF where I kind of quickly ran through that (and it seemed relatively obvious to me how it works). In summary, we chose to do real work rather than polilsh optional points in individual drafts because, frankly, customers are not interested all that much often whether IETF WG feels like working on it while they have a pressing problem and need solutions in a timely manner. And AFAIR the chairs guided the group multiple times towards "ignore the problem, of no importance" and now with a certain urgency want to have everything @ the same time.  

So this is creating a full-mesh overlay network of tunnels between the edge routers on the L1 area. I don't think you would want to advertise that overlay network back into the underlay network (L1 area) so you can't just form up L1 adjacencies (unmodified) to determine if they are up or not, some other mechanism would have to be used. OSPF virtual links use the intra-area connectivity to "v-bit" routers to determine if the area is transit capable -- or perhaps BFD as you suggest.

I guess the question is can there be topological control-plane connectivity to the flood reflector, but not data-plane overlay network tunnel connectivity?

The draft is also saying you suppress advertising the overlay network in L2 as well b/c it is represented instead by the advertised topology created by using the flood reflector. So the overlay is network represented, but unadvertised itself. This suppression of advertising the overlay tunnel network seems similar to how the L12 topology inside an area proxy is also suppressed, except ... tunnels. :)

Thanks,
Chris.
[as WG member]

> So, I look forward to this draft being adopted as WG item given the support seen and urgency of having this in the field and obviously, being running, working code since a bit ... 
> 
> -- tony 
> 
>

---

From: Tony Przygienda <[email-masked]>
Date: Wed, 10 Jun 2020 09:40:02 -0700

On Wed, Jun 10, 2020 at 4:27 AM Christian Hopps <[email-masked]> wrote:

> > Chris (addressing in WG member context you declared), I reply tersely
> since we will put more work into the draft once it's adopted (for which I
> think you saw a good amount of support in two threads already).
> >
> Both Area Proxy and Flood Reflector are proposing to use L1 areas as
> transit to connect L2, isn't that chaining? It seemed like a decent way to
> help visualize the proposals along with some numbers, perhaps you have
> something better...
>
> The Area Proxy draft is making everything L2 and using the L1 areas to
> redefine the advertised topology information to allow it to scale. Because
> everything is ultimately L2 nothing changes in the data plane to provide
> this transit.
>
> > a) I see how you can try to have a mental model of "virtual links". What
> we suggest are not virtual links (I implemented VL @ least once but it's so
> in ISIS.. That's why it never went anywhere much, my take, others may
> If you are not using tunnels, but are still trying to provide transit
> through the L1 area for L2, that is exactly what OSPF virtual links are
> doing. Part of making those work is the advertisement of reachability into
> the area from another, changes to router advertisements (to indicate if the
> area is transit capable), and changes to the SPF calculation to modify
> route choices based on whether they are based on virtual links or not.
>
> The complexity of OSPF virtual links is there to make them work, right?
> I'm not trying to make any argument, strawman or otherwise; I'm just trying
> to understand what is being proposed as a replacement for the full-mesh of
> tunnels.
>
> It's nice if it reduces to OSPF virtual links b/c we then have an example
> of how to actually implement it, and years of experience to understand it.
> If that highlights that getting the non-tunneled choice right isn't easy,
> well I guess that's important, right?
>
>
Hmmm, AFAIR the implementation of OSPF virtual links was having no tunnel
at all (and that's how I remember I implemented it then). I cite

"

The InterfaceUp event occurs for
    a virtual link when its corresponding routing table entry becomes
    reachable.  Conversely, the InterfaceDown event occurs when its
    routing table entry becomes unreachable.  In other words, the
    virtual link's viability is determined by the existence of an
    intra-area path, through the Transit area, between the two
    endpoints.
"

 and that was largely the problem, flooding+SPF+routing table was basically
the adjacency keep up and very flappy/circular instead of proper stable
tunnel with hellos. Did you implement and run virtual links in OSPF over
tunnels? which type?

And having run good amount of routing over tunnels I am still to see all
those dragons you are summoning. GRE, incl soft GRE is very widely deployed
and works well as are other tunnel types I worked with.

A valid point of discussion is how many adjacencies you can keep up to the
edge, my experience is hundreds which for all practical purposes is a very
good scale but otherwise we can relax the draft and build a "flood
reflector hierarchy", I'm open to that if it's a real concern. Ultimately
running more than hundreds is largely a question of relaxing timers which
is relatively easily done.

> b) Generally we may seem a bit muddled between different types of
> "tunnels" and "tunnels are bad" and "lots of tunnels". The draft talks
> about 2 types of tunnels and it seemed to be written clearly enough to
> distinguish that easily based on feedback I got so far
> >
> > i) control plane tunnels are proper L2 entities (again, if your FR is
> one hop away from leafs then you don't need any tunneling but can run
> normal L2 adjacencies which I hope are not too scary; whole thing is really
> I think I understand the flood reflector bit; It's replacing the network
> of (now data-plane only) tunnels in flooding graph and the advertised
> topology.
>
> > ii) data plane tunnels. The draft basically explains that for the
> solution to work in a simple fashion,full-mesh of data forwarding tunnels
> can be established (which are NOT visible in L2) as shortcuts that allow to
> So this is creating a full-mesh overlay network of tunnels between the
> edge routers on the L1 area. I don't think you would want to advertise that
> overlay network back into the underlay network (L1 area) so you can't just
> form up L1 adjacencies (unmodified) to determine if they are up or not,
> some other mechanism would have to be used. OSPF virtual links use the
> intra-area connectivity to "v-bit" routers to determine if the area is
> transit capable -- or perhaps BFD as you suggest.
>
> I guess the question is can there be topological control-plane
> connectivity to the flood reflector, but not data-plane overlay network
> tunnel connectivity?
>
> The draft is also saying you suppress advertising the overlay network in
> L2 as well b/c it is represented instead by the advertised topology created
> by using the flood reflector. So the overlay is network represented, but
> unadvertised itself. This suppression of advertising the overlay tunnel
> network seems similar to how the L12 topology inside an area proxy is also
> suppressed, except ... tunnels. :)
>
>
At a scale we look at it was not a big problem to advertise the L1 tunnel
mesh back into the underlay but as you say BFD would work fine etc. Not
really that different from shortcuts we have all over the place in IGPs
already. But again, I explained how you can run it without any L1 tunnels
data mesh in the preso @ IETF and we'll update the draft with details once
it's adopted.

You do seem to be carrying as WG member a hot torch for area-proxy for some
reason, that's fine with me, frankly, I had extensive discussions with
customers when DriveNet was being proposed to them (which AFAIS is
basically area-proxy) and the solution is intriguing but it did not cut
lots of requirements of large customers and there are a lot of unresolved
issues operationally with an approach like that. However, I'm here to make
sure to get flood reflectors adopted so it can be deployed by customers in
ideally multi-vendor, interoperable scenarios and not on crusades so let's
adopt the draft and then we fill the pieces in and maybe extend it to
hierarchy, no-tunnel-one-hop-away bits if people think that is
interesting/relevant and are willing to work on that.  And then customers
can choose whether they run area-proxy or flood reflectors based on what
requirement set is important to them.

thanks

--- tony

---

From: [email-masked]
Date: Wed, 10 Jun 2020 10:23:54 -0700

Tony,

> On Jun 10, 2020, at 9:40 AM, Tony Przygienda <[email-masked]> wrote:
> 
> You do seem to be carrying as WG member a hot torch for area-proxy for some reason, that's fine with me, frankly, I had extensive discussions with customers when DriveNet was being proposed to them (which AFAIS is basically area-proxy) and the solution is intriguing but it did not cut lots of requirements of large customers and there are a lot of unresolved issues operationally with an approach like that. 

Drivenets, as I understand it, is an attempt to physically deaggregate a multi-chassis fabric down to the chip level using a proprietary chip-set-specific cell based interlink protocol. However, it retains a single control plane and as such looks like a single IS-IS system.  It has no relationship whatsoever to area proxy.

Regards,
Tony

---

From: Tony Przygienda <[email-masked]>
Date: Wed, 10 Jun 2020 10:25:36 -0700

ok, let's not drag vendor specific stuff in. I shouldn't have brought it up
I guess, outside the scope of IETF threads ...

thanks

--- tony

On Wed, Jun 10, 2020 at 10:23 AM <[email-masked]> wrote:

> You do seem to be carrying as WG member a hot torch for area-proxy for
> some reason, that's fine with me, frankly, I had extensive discussions with
> issues operationally with an approach like that.
>
>
>
> Drivenets, as I understand it, is an attempt to physically deaggregate a
> multi-chassis fabric down to the chip level using a proprietary
> chip-set-specific cell based interlink protocol. However, it retains a
> single control plane and as such looks like a single IS-IS system.  It has
> no relationship whatsoever to area proxy.
>
> Regards,
> Tony
>
>
>
>

---

From: Christian Hopps <[email-masked]>
Date: Wed, 10 Jun 2020 13:29:37 -0400

> On Jun 10, 2020, at 12:40 PM, Tony Przygienda <[email-masked]> wrote:
> 
> 
> 
> Hmmm, AFAIR the implementation of OSPF virtual links was having no tunnel at all (and that's how I remember I implemented it then). I cite 

Correct, that's why I'm suggesting that any solution without tunnels is going to reduce to OSPF virtual links.

:)

>  and that was largely the problem, flooding+SPF+routing table was basically the adjacency keep up and very flappy/circular instead of proper stable tunnel with hellos. Did you implement and run virtual links in OSPF over tunnels? which type? 

Well I had to implement them (virtual links, not tunnels, since that's just how OSPF virtual links work) back in the day when I added the non-backbone code to the second GateD OSPF version. From that point on it was all IS-IS though. :)

> And having run good amount of routing over tunnels I am still to see all those dragons you are summoning. GRE, incl soft GRE is very widely deployed and works well as are other tunnel types I worked with. 

One solution requires tunnels the other doesn't *shrug*, tunnels aren't free they cost bandwidth and add complexity I think.

> You do seem to be carrying as WG member a hot torch for area-proxy for some reason,

Isn't that the point of me being able to contribute "as a WG member"? If I was saying all this "As a Chair" and/or acting on it "As a chair" by fiat, then it would be seen as unfair. We do want our chairs (who are generally knowledgeable about that for which they chair) to be able to contribute to a technical discussion though, right?

If it seems like I'm "carrying the torch" it's b/c as I understand the drafts currently, the area proxy seems like a much simpler solution (KISS), and I haven't yet been able to figure out what the benefits of using the flood reflector method instead would bring. If those benefits were more obvious I'd probably like it too. :)

Thanks,
Chris.
[as WG member]

> that's fine with me, frankly, I had extensive discussions with customers when DriveNet was being proposed to them (which AFAIS is basically area-proxy) and the solution is intriguing but it did not cut lots of requirements of large customers and there are a lot of unresolved issues operationally with an approach like that. However, I'm here to make sure to get flood reflectors adopted so it can be deployed by customers in ideally multi-vendor, interoperable scenarios and not on crusades so let's adopt the draft and then we fill the pieces in and maybe extend it to hierarchy, no-tunnel-one-hop-away bits if people think that is interesting/relevant and are willing to work on that.  And then customers can choose whether they run area-proxy or flood reflectors based on what requirement set is important to them.

---

From: Tony Przygienda <[email-masked]>
Date: Wed, 10 Jun 2020 10:56:18 -0700

On Wed, Jun 10, 2020 at 10:29 AM Christian Hopps <[email-masked]> wrote:

>
>
>
> >
> >
> > Hmmm, AFAIR the implementation of OSPF virtual links was having no
> tunnel at all (and that's how I remember I implemented it then). I cite
>
Good to hear you implemented virtual w/o tunnels.

But then FR is not a solution "without tunnels" so the whole logic of "it's
not the same hence it's the same for me" escapes me completely now and I
let it rest.

>
>
> If it seems like I'm "carrying the torch" it's b/c as I understand the
> drafts currently, the area proxy seems like a much simpler solution (KISS),
> and I haven't yet been able to figure out what the benefits of using the
> flood reflector method instead would bring. If those benefits were more
> obvious I'd probably like it too. :)
>
>
looking @ the amount of protocol extensions in terms of TLVs, flooding
scoping, configuration etc necessary in either case and involved
fork-lifting, new-constructs necessary in operational tooling I dare say
you hold a somewhat unique "opinion" IMO ...

As to simplicity you see and since your mental model seems fixed around
virtual links concept, I also suggest to look up why in PNNI we ended up
introducing a special "L1 equivalent" computation to the peer group leader
to validate that it was actually reachable for correct operation
(especially hierarchy negotiations) on peer group egresses which in a sense
was like but worse than virtual links operationally IME. I was suggesting
then to keep an SVC from PGL to every border for that reason since a
partition of a PGL led otherwise to the peer groups looking like the same
thing for a while with highly undesirable operational effects (my memory
doesn't reach that far really but we had at least discussions then to
implement proprietary solution in Fore to have such SVCs for more stable
deployments). In more abstract terms, flooding is extremely good to quickly
"route around failures" when e'ones state is completely decentralized but
is simply not a great mechanism if you have to talk to an entity couple
hops away in a stable manner, especially if this entity hold state you need
for correct operation when talking to your peers.

-- tony

---

From: Christian Hopps <[email-masked]>
Date: Wed, 10 Jun 2020 14:04:45 -0400

> On Jun 10, 2020, at 1:56 PM, Tony Przygienda <[email-masked]> wrote:
> 
> 
> 
> But then FR is not a solution "without tunnels" so the whole logic of "it's not the same hence it's the same for me" escapes me completely now and I let it rest. 
>  
> 
> 
> looking @ the amount of protocol extensions in terms of TLVs, flooding scoping, configuration etc necessary in either case and involved fork-lifting, new-constructs necessary in operational tooling I dare say you hold a somewhat unique "opinion" IMO ... 
> 
> As to simplicity you see and since your mental model seems fixed around virtual links concept,

Tony,

I think you have misunderstood me at some point. When asked about the requirement of tunnels you said that they were not required, and that a solution could be done with carefully scoped reachability, this was also mentioned in your second presentation.

*ALL* I've been talking about WRT virtual links is that when you hint at a non-tunnel solution in your draft and presentation involving reachability, that your going to end up with OSPF virtual links. That is all.

Thanks,
Chris.
[as WG member]

> I also suggest to look up why in PNNI we ended up introducing a special "L1 equivalent" computation to the peer group leader to validate that it was actually reachable for correct operation (especially hierarchy negotiations) on peer group egresses which in a sense was like but worse than virtual links operationally IME. I was suggesting then to keep an SVC from PGL to every border for that reason since a partition of a PGL led otherwise to the peer groups looking like the same thing for a while with highly undesirable operational effects (my memory doesn't reach that far really but we had at least discussions then to implement proprietary solution in Fore to have such SVCs for more stable deployments). In more abstract terms, flooding is extremely good to quickly "route around failures" when e'ones state is completely decentralized but is simply not a great mechanism if you have to talk to an entity couple hops away in a stable manner, especially if this entity hold state you need for correct operation when talking to your peers. 
> 
> -- tony 
>

---

From: Tony Przygienda <[email-masked]>
Date: Wed, 10 Jun 2020 11:10:49 -0700

On Wed, Jun 10, 2020 at 11:04 AM Christian Hopps <[email-masked]> wrote:

>
> ...
>
>
> > I also suggest to look up why in PNNI we ended up introducing a special
> "L1 equivalent" computation to the peer group leader to validate that it
> was actually reachable for correct operation (especially hierarchy
> negotiations) on peer group egresses which in a sense was like but worse
> than virtual links operationally IME. I was suggesting then to keep an SVC
> from PGL to every border for that reason since a partition of a PGL led
> otherwise to the peer groups looking like the same thing for a while with
> highly undesirable operational effects (my memory doesn't reach that far
> really but we had at least discussions then to implement proprietary
> solution in Fore to have such SVCs for more stable deployments). In more
> abstract terms, flooding is extremely good to quickly "route around
> failures" when e'ones state is completely decentralized but is simply not a
> great mechanism if you have to talk to an entity couple hops away in a
> stable manner, especially if this entity hold state you need for correct
> operation when talking to your peers.
>
>
So we understand L2 tunnels from leaf to FR are "real" tunnels and have
nothing to do with virtual links.

As to the data plane, nope, you don't end up with any "virtual links" I can
recognize unless you choose to call redistribution "virtual links" as well,
maybe look @ the presentation again.

-- tony

---

From: Christian Hopps <[email-masked]>
Date: Wed, 10 Jun 2020 14:58:21 -0400

> On Jun 10, 2020, at 2:10 PM, Tony Przygienda <[email-masked]> wrote:
> 
> 
> 
> ....
> So we understand L2 tunnels from leaf to FR are "real" tunnels and have nothing to do with virtual links. 
> 
> As to the data plane, nope, you don't end up with any "virtual links" I can recognize unless you choose to call redistribution "virtual links" as well, maybe look @ the presentation again. 

That's exactly it! Excellent. :)

Safely using redistributed reachability to provide transit w/o tunnels is exactly what OSPF virtual links are, so that's why I'm saying

"A solution without tunnels is also possible by judicious scoping of reachability information between the levels."

will reduce to the same thing, if you want it to actually work generically. Perhaps this will become more obvious when that part of the draft is fleshed out.

Thanks,
Chris.