Subject: Re: [Anima] ANIMA when there is a system-wide issue

From: Toerless Eckert <[email-masked]>
Date: Thu, 11 Feb 2021 21:19:10 +0100

Michael:

The re-posted draft is a good reminder:

To me, the draft touches on a very important and hopefully lucrative topic, but i think
the text makes it very hard to figure out what's going on.

Let me try to summarize the situation from where i sit:

draft-ACP section 7. explains how to implement ACP on any low-end L2-only as well as
L2/L3 devices so that the device can operate ACP as if it was just an L3 device, while
in parallel operating unchanged on some or all ports as an L2 switch with STB:
ACP packets are software routed and software IPsec en/decrypted.

The magic to this is also explained, its simply to per-port-punt DULL GRASP packets
and later encapsulated ACP packets to CPU as it is done today for STP, CDP, LLDP,
EAPOL packets. The rest is then standard ACP implementation as on any L3 ACP device. 

The one possible gap for low-end hardware is the function:

  punt-to-cpu ACP packets from ports that are in STP blocked state.

In my past implementation experience, the punt option that always works is
one where you punt a specific ethertype. This is so because all the L2 switch
hardware was designed based on supporting 802.1x, and that is where LLDP and EAPOL
came into play and the minimum HW selector to decide between punt/forward/block
was simply ethertype.

Aka: The most simple extension we would need to drastically improve the ability for
ACP to get into lucrativee markets is to define an ACP encap using its own ethertype.
That is primarily an issue with finding i think 2000 USD and arguing with ethertype
registry owner about the long term value of that ethertype. We should simply make
sure that we build selectors into the encap to make it equal or better extensible
than EAPOL, so that we effectively have an IETF standards controlled equivalent
to EAPOL. There are IMHO a lot of good reasons not to use EAPOL:

- symmetric (ACP) vs. asymmeric (EAPOL use cases)
- control: IETF vs. IEEE
- risk/overhead: demux of such disjoined use-cases deeper into the packet such as different
   EAPOL sub-typing.

Forget using multicast MAC destinations. Maybe i can find the time trying to remember
all the horrible things that could go wrong with it.

The other option missing that may benefit discussion is to use a different unicast MAC
for ACP if the HW can punt-to-CPU a separate unicast-MAC. But it foregoes the big
benefit of the new ethertype option for us to be able to optimize the encap: aka:
remove the unnecessary overhead of an outer IPv6 encapsulation but simply run
native IPv6/IPsec packets across a new ethertype with maybe some extensibility
selector (as in EAPOL) before the IPv6/IPsec packet.

Cheers
    toerless

---

From: Michael Richardson <[email-masked]>
Date: Thu, 11 Feb 2021 17:40:25 -0500

Toerless Eckert <[email-masked]> wrote:
    > draft-ACP section 7. explains how to implement ACP on any low-end
    > L2-only as well as L2/L3 devices so that the device can operate ACP as
    > if it was just an L3 device, while in parallel operating unchanged on
    > some or all ports as an L2 switch with STB: ACP packets are software
    > routed and software IPsec en/decrypted.

}   Predictable scaling requirements for ACP neighbors can most easily be
}   achieved if in topologies such as these, ACP capable L2 switches can
}   ensure that discovery messages terminate on them so that neighboring
}   ACP routers and switches will only find the physically connected ACP
}   L2 switches as their candidate ACP neighbors.

So, your solution is to have L2 switches not forward based upon L3 multicast
addresses.  While this is deterministically mapped to a unique L2 multicast MAC,
and maybe this can work...  but, as you say below, that's not primarily how
the punt has worked in the past.

    > The magic to this is also explained, its simply to per-port-punt DULL
    > GRASP packets and later encapsulated ACP packets to CPU as it is done
    > today for STP, CDP, LLDP, EAPOL packets. The rest is then standard ACP
    > implementation as on any L3 ACP device.

okay.
My only request is that we avoid adding a new thing to the list of per-port
things, because they take up ASIC / NPU space.

    > In my past implementation experience, the punt option that always works
    > is one where you punt a specific ethertype.

Great, so one of the options I proposed is to have a new ethertype for
IPv6-DULL messages, as you also say below.

    > This is so because all the
    > L2 switch hardware was designed based on supporting 802.1x, and that is
    > where LLDP and EAPOL came into play and the minimum HW selector to
    > decide between punt/forward/block was simply ethertype.

    > Aka: The most simple extension we would need to drastically improve the
    > ability for ACP to get into lucrativee markets is to define an ACP
    > encap using its own ethertype.  That is primarily an issue with finding
    > i think 2000 USD and arguing with ethertype registry owner about the
    > long term value of that ethertype.

I think that the IEEE/IETF liason can get us an ethertype, given an
appropriate STD track draft.
Tell me if that you think this is within the ANIMA WG's charter.

    > Forget using multicast MAC destinations. Maybe i can find the time
    > trying to remember all the horrible things that could go wrong with it.

okay.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 12 Feb 2021 17:04:40 +1300

>     > trying to remember all the horrible things that could go wrong with it.

As long as you remember that we will always be emulating multicast for discovery and flooding. There really isn't any mathematically possible way round that, even if the only solution is replicast.

Regards
   Brian

---

From: Michael Richardson <[email-masked]>
Date: Fri, 12 Feb 2021 14:02:17 -0500

Brian E Carpenter <[email-masked]> wrote:
    >> > Forget using multicast MAC destinations. Maybe i can find the time >

    > As long as you remember that we will always be emulating multicast for
    > discovery and flooding. There really isn't any mathematically possible
    > way round that, even if the only solution is replicast.

That's very true.

What we really want is to never accidentally get native multicast involved.
We always want to involve the device that would be providing that native
multicast in the ACP...

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 16 Feb 2021 18:43:15 +0100

On Thu, Feb 11, 2021 at 05:40:25PM -0500, Michael Richardson wrote:
> So, your solution is to have L2 switches not forward based upon L3 multicast addresses.

I can not parse this. Can you rephrase ?

> While this is deterministically mapped to a unique L2 multicast MAC,

What is "this" ? Sorry.... can not parse again.

> and maybe this can work...

> but, as you say below, that's not primarily how  the punt has worked in the past.

> things, because they take up ASIC / NPU space.

Right.

> IPv6-DULL messages, as you also say below.

My point was that its not only about the discovery packets. We also want to make
sure that the ACP secure channel packets could also go across blocked ports.

> appropriate STD track draft. Tell me if that you think this is within the ANIMA WG's charter.

Different encap to make ANI more resilient. Sounds to me like perfectly in scope of
BRSKI/ACP extensions charter point.

I just think we shouldn't rush this but think it through well:

First of all, relying on a separate MAC address for ACP is the most eay way out.
But its not clear to me if thi works on every hardware. On the other hand, i can
see how i would always have a separate MAC address for ACP if for example i have classic
PC hardware and implement ACP on the BMC (e.g.: as extension to OpenBMC). But
many switches/routers also have a MAC address pool they can use.

Might make sense to just document that option (can't remember if i mentioned this
in ACPdraft). But nothing else to do.

When it comes to encapsulating across new ethertype, we need to make it extensible,
thats IEEE requirement for such a scarce resource. So we might even want to
ask around IETF for similar use-cases to have candidate other values for a
selector field for example.

Need to read through EAP over Ethernet to check what we could share. I forgot all about it.
But ultimatly, its going to be a small "selector-header" on top of the new ethertype
that we need to define.

I can see at least two selectors: ACP-secure-channel and Discovery (DULL-GRASP).

I have not thought about use-cases for e.g.: just BRSKI without ACP.

If you read up on the public slides i did eternities ago for my prior employer about that vendors
(somewhat proprietary) ACP implementation in the context of service provide ethernet services,
where a customer L2-ethernet service is multiplexed across a service provider L2VPN service, then those
type of service-edge nodes have all type of pain for discovery protocols such as CDP/LLDP,
e.g.: for their own instance and for the customer instance. So LLDP has explicit provisions
for this, and i have to find time to swap in the context of that stuff again. Much easier
if we had in-person IETFs and someone in the know like Norm Finn could remind me ;-))

As in: lets make sure that our selector field header could support the same flexibility
for those use-cases. At least to the extent that we know for a rev0 spec how many
"reserved/ignore" bits we need and then fill them in later.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 16 Feb 2021 18:48:41 +0100

> As long as you remember that we will always be emulating multicast for discovery and flooding. There really isn't any mathematically possible way round that, even if the only solution is replicast.

Link-local multicast is fine for DULL grasp, i just wouldn't want to overload
the semantic of the multicast address for the punt="break L2 domain" function of th
ACP L2 switch.

Full mesh of ACP secure channels to emulate L2 multicast should be well enough defined
in ACPdraft. If we get ACP capable L2 switches, then that full-mesh problem wouldn't
even arise anymore.

More interestingly though is to make sure that whatever we do to make the solution 
more resilient (by being able to operate independent of STB), we should also make sure
that it does not prohibit HW forwarding if/when nodes implement this. I am not very
confident though that campus/Metro equipment will get HW IPsec any time soon, so
that would leave the open question of how to utilize MacSec. 

The trick is not trying to take too many steps at once but ensure that we don't close doors on them.

Cheers
    Toerless

---

From: Michael Richardson <[email-masked]>
Date: Fri, 19 Feb 2021 12:45:35 -0500

EXECSUM: we need IEEE802.1X political clue.

The WG should adopt my l2-friendly-acp document as a problem statement
document, and then we should propose a solution which can be discussed
outside of the IETF.

Toerless Eckert <[email-masked]> wrote:
    >> Toerless Eckert <[email-masked]> wrote: > draft-ACP section 7. explains
    >> how to implement ACP on any low-end > L2-only as well as L2/L3 devices
    >> so that the device can operate ACP as > if it was just an L3 device,
    >> while in parallel operating unchanged on > some or all ports as an L2
    >> switch with STB: ACP packets are software > routed and software IPsec
    >> en/decrypted.

} Predictable scaling requirements for ACP neighbors can most easily
} achieved if in topologies such as these, ACP capable L2 switches
} can ensure that discovery messages terminate on them so that
} neighboring ACP routers and switches will only find the physically
} connected ACP L2 switches as their candidate ACP neighbors.

    >> So, your solution is to have L2 switches not forward based upon L3

    tte> I can not parse this. Can you rephrase ?

You are asking L2 switches, which don't speak any L3 protocols in their
forwarding engine, to do filtering on L3 things, namely L3 destination
multicast addresses.

    >> While this is deterministically mapped to a unique L2 multicast MAC,

    tte> What is "this" ? Sorry.... can not parse again.

The L3 multicast address is mapped on ethernet to a deterministic L2
multicast address.  So, an L2 switch could filter on that, but unless it then
examines the L3 address, it could catch future L3 multicasts that happen to
map to the same L2 multicast.  When we allocated the ACP l3 multicast
destination, we did not make any attempt to make it a unique L2 target.
Assuming that there are no existing conflicts, to avoid the above, we would
need to allocate all the other L3 multicast targets that map to the same L2
multicast target.

    tte> In my past implementation experience, the punt option that always
    tee> works is one where you punt a specific ethertype.
    >> IPv6-DULL messages, as you also say below.

    tte> My point was that its not only about the discovery packets. We also
    tte> want to make sure that the ACP secure channel packets could also go
    tte> across blocked ports.

Yes, that's a different problem, but I agree it is related.

For the L2 SDN that does not use STP because it does not want blocked ports,
but rather wants to use all the bandwidth, the problem is keep the ACP DULL
multicast from causing loops.

    mcr> I think that the IEEE/IETF liason can get us an ethertype, given an
    mcr> appropriate STD track draft. Tell me if that you think this is within
    mcr> the ANIMA WG's charter.

    tte> Different encap to make ANI more resilient. Sounds to me like
    tte> perfectly in scope of BRSKI/ACP extensions charter point.

Good, we agree.

    tte> I just think we shouldn't rush this but think it through well:

    tte> First of all, relying on a separate MAC address for ACP is the most
    tte> eay way out.  But its not clear to me if thi works on every
    tte> hardware. On the other hand, i can see how i would always have a
    tte> separate MAC address for ACP if for example i have classic PC
    tte> hardware and implement ACP on the BMC (e.g.: as extension to
    tte> OpenBMC). But many switches/routers also have a MAC address pool
    tte> they can use.

Having the ACP on the (Open)BMC is definitely a killer use, and I hope to get there soon.
One might still want the ACP running even if the context of a BMC user who
shoots eirself in the foot.
The Linux kernel gives one the "macvlan" which is effectively a kind of
bridge (actually mutually exclusive with being in the a bridge).
The macvlan gets a kernel allocated randomized mac address, and can be moved
into a network namespace and effectively hidden, however, there does not seem
to be a way to keep the physical interface from being marked down.

    tte> When it comes to encapsulating across new ethertype, we need to make
    tte> it extensible, thats IEEE requirement for such a scarce resource. So
    tte> we might even want to ask around IETF for similar use-cases to have
    tte> candidate other values for a selector field for example.

    tte> Need to read through EAP over Ethernet to check what we could
    tte> share. I forgot all about it.  But ultimatly, its going to be a
    tte> small "selector-header" on top of the new ethertype that we need to
    tte> define.

You want chapter 11 of 802.1X-2020.
Table 11-3 lists the 9 EAPOL types used.
No equivalent to IANA Consideratons exist, so I think that it would require a
revision by the IEEE to allocate a code.  That would really be enough.

    tte> I can see at least two selectors: ACP-secure-channel and Discovery
    tte> (DULL-GRASP).

Both are IPv6 with LL src/dst, different protocols, so as long as we can
carry IPv6, then we can do both.

    tte> I have not thought about use-cases for e.g.: just BRSKI without ACP.

BRSKI can use the DULL GRASP channel to get access to a Join Proxy.
If used without ACP, then the certificate enrollment could well lead to use
of EAP to connect.  That's an important connection.

    tte> If you read up on the public slides i did eternities ago for my
    tte> prior employer about that vendors (somewhat proprietary) ACP
    tte> implementation in the context of service provide ethernet services,
    tte> where a customer L2-ethernet service is multiplexed across a service
    tte> provider L2VPN service, then those type of service-edge nodes have
    tte> all type of pain for discovery protocols such as CDP/LLDP, e.g.: for
    tte> their own instance and for the customer instance. So LLDP has
    tte> explicit provisions for this, and i have to find time to swap in the
    tte> context of that stuff again. Much easier if we had in-person IETFs
    tte> and someone in the know like Norm Finn could remind me ;-))

Yes, I recognize that.  There is a table 11-1 in the 802.1X document that
deals with this.  It seems broken that an end system has to know whether it's
in a some kind of L2-service to do it's announcements, etc.

    tte> As in: lets make sure that our selector field header could support
    tte> the same flexibility for those use-cases. At least to the extent
    tte> that we know for a rev0 spec how many "reserved/ignore" bits we need
    tte> and then fill them in later.

My reading is that they accomplish this via table 11-1, and if we were to
build upon EAPOL, that it would all come for free.

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 23 Feb 2021 12:04:28 +0100

> document, and then we should propose a solution which can be discussed

Personally i feel i would like to see a good amount of the explanations and
clarifications that we exchanged in this thread to first go into this or
any other draft before i personally would feel its appropriate for WG
adoption. Let me see that i can find some time to help.

> forwarding engine, to do filtering on L3 things, namely L3 destination

Actually, for such a switch to become useful, we do expect more, and maybe
it wold help to write down those different "more" options that we think to
understand. Aka: in the most simple form, we expect that the switch is
able to do ACP processing in software - and then the packet filtering
is the only "HW" requirement.

I am still fundamentally struggling with the operational model of the
ACP when it can have significantly different forwarder speeds. But thats
because i saw perfect deployment opportunities dwindle because of implementations
that where only in software.

For example using the ACP for the RFC8368 use case and then going through
the calculation about the maximum performance required by a controller
to update/change configs across a wide range of devices while having all
this traffic go through a switch in the NOC (close to the controller)
that only implements ACP in CPU software forwarding.

So i am wondering if we would even have  good plan how to solve this
issue. FOr example: Could i define the ACP such that during enrollment,
the registrar could determine that the switch sucks (too slow) for the
network and asks it to turn into an ACP host instead of an ACP router
(and data-plane switch) ? That to me looks like one possible option
to let operators decide. And its a great option really only available
with L2 switches (not routers).

I think this would be easy to define except for this missing
linkage between BRSKI and ACP: If i wanted to push this info to
such a device today (ACP router vs. ACP host), i'd have to come
up with a new flag in the AcpNodeName. Which may be ok, and maybe
even helpfull security-wise (consider any RPL info from such a system
except for its own route to be an error and ignore it...  ?)

But having some parameters that can be given at the tailend of
BRSKI to the node would be lovely too. I think we discussed this
in the past.

> need to allocate all the other L3 multicast targets that map to the same L2

Right. Any implementation of what i described in ACP would need to
software forward any L3 multicast address traffic that is aliasing to
the DULL GRASP L3 multicast address. Upon quick browsing i think its
not even mentioned in RFC4541 (problem for every IGMP snooping implemention).

> multicast from causing loops.

Uhmm.. not clar. Forget ACP DULL... You have an ethernet without
STP but with rdundant paths. How do you avoid loops ? Do you use one of
the IEEE SPF alternatives to STP ?

> Having the ACP on the (Open)BMC is definitely a killer use, and I hope to get there soon.

Last time i checked OpenBMC git was very confusing, seemed like mostly
facebook internal adoption, but couldn't figure out any option i could
easily buy individually as an experimentation platform.

> to be a way to keep the physical interface from being marked down.

Right. Ideally you would have SrIOV to create a PCI-bus level disjoint
ethernet interface for the ACP. Alas, today, like MacSEC this is an
option only on high-end Ethernet PCI controllers. Or else you have to much
around in he linux kernel to create protectiona against unintended shutdowns.

> revision by the IEEE to allocate a code.  That would really be enough.

Right. I didn't mean to use EAPOL. I meant to document all the arguments why NOT
to use  it, but then also reuse all the reuseable ideas that we need.

> build upon EAPOL, that it would all come for free.

Yepp... more to digest for me here.

Cheers
    toerless

-- 
---
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Tue, 23 Feb 2021 17:53:54 -0500

Toerless Eckert <[email-masked]> wrote:
    >> For the L2 SDN that does not use STP because it does not want blocked
    >> ports, but rather wants to use all the bandwidth, the problem is keep
    >> the ACP DULL multicast from causing loops.

    > Uhmm.. not clar. Forget ACP DULL... You have an ethernet without STP
    > but with rdundant paths. How do you avoid loops ? Do you use one of the
    > IEEE SPF alternatives to STP ?

Yes, maybe one of the alternatives, maybe because some SDN controller deals
with updating the right forwarding tables.

    > easily buy individually as an experimentation platform.

Well, there is significant progress.  But you are right: it is hard to find
actual hardware you can run it on without working for the hardware company.
But, that's because of TPM mechansim.
I also notice how hard it is to run alternate firmware on Cisco/Junipier/Huawei/etc.

    >> One might still want the ACP running even if the context of a BMC user
    >> who shoots eirself in the foot.  The Linux kernel gives one the
    >> "macvlan" which is effectively a kind of bridge (actually mutually
    >> exclusive with being in the a bridge).  The macvlan gets a kernel
    >> allocated randomized mac address, and can be moved into a network
    >> namespace and effectively hidden, however, there does not seem to be a
    >> way to keep the physical interface from being marked down.

    > option only on high-end Ethernet PCI controllers. Or else you have to
    > much around in he linux kernel to create protectiona against unintended
    > shutdowns.

Yes, so there are issues.

    >>
    >> You want chapter 11 of 802.1X-2020.  Table 11-3 lists the 9 EAPOL
    >> types used.  No equivalent to IANA Consideratons exist, so I think
    >> that it would require a revision by the IEEE to allocate a code.  That
    >> would really be enough.

    > Right. I didn't mean to use EAPOL. I meant to document all the
    > arguments why NOT to use it, but then also reuse all the reuseable
    > ideas that we need.

ah.