Subject: Re: [Idr] BGP autoconfiguration - draft-ymbk-idr-l3nd - Problem scope

From: "Susan Hares" <[email-masked]>
Date: Wed, 2 Mar 2022 18:05:03 -0500

Jeff:
[author-hat-on] 

This is a second message response to your review discussing problem scope
and transport
This portion has high-level comments plus in-line comments. 

DT document 
Problem-scope says:  "The current target environment is BGP used for the
underlay protocol in data center networks."
Simplicity: "The auto-discovery mechanism is designed to be simple. ... This
BGP  
State Requirements:  "enough information to a BGP Speaker to initiate a
connection in BGP protocols"
Discovered session state: 
 - IP address, Transport security parameters, GTSM, and BGP session Protocol
State Version number. 
BGP Protocol state 
- AS numbers, BGP ID, Supported AFI/SAFIs 

Here the draft-ietf-idr-bgp-autconf-considerations-02.txt is unclear. 
Does the BGP-Auto-discovery pass the AFI/SAFIs?  - 
Does pass: LLDP  (Your draft) 
Does not pass:   L3DL, L3DN, draft-minto-idr-bgp-autodiscovery 
The ones who do not pass the AFI/SAFIs could easily add a TLV to do this. 

L3ND 
Scope:  Massive Data Center (MDC) 
Simplicity:  Simplicity with
Discovered state:  IP addresses, transport security, 
Encapsulations + Encapsulation Interfaces, 
L3NDversion, (see Open), DB-version (see serial number),  GTSM
BGP protocol State: AS number, BGP ID, AFI/SAFI

Two exceptions: 
1)  BGP Protocol State Version number - If this means v4 for BGPv4, please
revise the DT auto-configuration. 
2) AFI/SAFI - see question above. 

IMHO design choices for Transport are best: 
reliable transport (lower timers) + ability to plug in good security. 

I'll keep this to section 1 (points 1-5) 

I'll resume commenting on your input. 

Thank you for taking time to write down your review. 

Sue 

-----Original Message-----
From: Idr [mailto:[email-masked]] On Behalf Of Jeffrey Haas
Sent: Tuesday, March 1, 2022 4:22 PM
To: [email-masked]
Subject: [Idr] BGP autoconfiguration - draft-ymbk-idr-l3nd

Working Group,

One of the topics for our last IDR interim was an attempt to conclude
discussions on BGP autoconfiguration proposals prior to sending adoption
requests to the Working Group.  Just prior to that call going out, we were
requested to add one additional proposal for consideration that hadn't yet
been published: l3nd[1] and l3nd-ulpc[2].

The authors have requested a presentation slot at the upcoming IETF 113 for
this proposal.  Since this is a very late entrant, I'd like to try to get
discussion of the properties of this proposal started on the mailing list
using the same discussion points we've had from prior interim meetings.

The Working Group adopted requirements and some of the prior proposal
discussions are part of the design team document.[3]

Below, find my observations on the draft with questions and commentary
intermixed.  I finish this response with my high level analysis and personal
opinion on what these properties accomplish.

-----

Scoping discussion:

[point 1]
The l3nd proposal appears to be designed to start from similar premises as
l3dl.  l3dl has been previously discussed as a candidate for a layer 2 BGP
autoconfiguration proposal and is covered in [3]A.1.2.

:   This document (set) provides a similar solution at Layer-3,
:   attempting to be as similar as reasonable to L3DL.
[Sue: Thank you for noticing the L2 and L3 share BGP]

[Point-2]
The intended scoping is point to point links:
:   In this document, the use case for L3ND is for point to point links
:   in a datacenter Clos ([Clos])

This is somewhat unusual language since the typical deployment interface is
Ethernet.  While such Ethernet connections are typically deployed in a point
to point fashion, Ethernet itself is not specifically point to point.
[Sue: text can be improved. Thank you for noticing.] 
Revised text: 
+    L3DN is intended for use within single IP subnets (IP over
+    Ethernet or other point-to-point or multi-point IP link) in order to
+    exchange the data needed to bootstrap BGP-based peering, EVPN, etc.;
+    often in a datacenter Clos <xref target="Clos"/> environment.

[Point-3] 
There is also a pointer that the proposal may be intended for more general
purposes than BGP Autoconfiguration:

:   L3ND might be found to be widely applicable to a range of routing and
:   similar protocols which need Layer-3 neighbor discovery.

The proposal appears to be intended to be able to expanded to non-data
center cases.
[Sue - yes it can expand]

:   While L3ND is designed for the MDC, there are no inherent reasons it
:   could not run on a WAN.  The authentication and authorization needed
:   to run safely on a WAN need to be considered, and the appropriate
:   level of security options chosen.

Such wider scoping was permitted by the autoconfiguration analysis but such
work was left as out of scope for the DC case.
[Sue - Not exactly.  See the security section 
Section 3.6 "Different deployment models will have different and
requirements"
This section indicate this proposal could support an authentication model
with more security. 
It could also just run TCP]. 

[Point-4] 
l3nd seems to be targeting different levels of scale than prior proposals,
and that appears to motivate a different property:

:   The number of addresses of one Encapsulation type on an interface
:   link may be quite large given a TOR switch with tens of servers, each
:   server having a few hundred micro-services, resulting in an
:   inordinate number of addresses.  And highly automated micro-service
:   migration can cause serious address prefix disaggregation, resulting
:   in interfaces with thousands of disaggregated prefixes.
:
:   To meet such scaling needs, the L3ND protocol is session oriented and
:   uses incremental announcement and withdrawal with session restart, a
:   la BGP ([RFC4271]).

Given that the desired scoping context is point to point (typically a small
number of addresses on a given link), it'd be helpful if the authors discuss
scenarios wherein a large number of BGP sessions are expected to be
discovered on such a link.
[Sue:  The primary target is the massive Data Center (MDC) environment as
the first sentence of the introduction indicates.  The revised text above
shows correction from authors.  ]
[Sue: The introduction provides you truth in advertising that it scales. ]

-----
[Point-5] 
Transport layer considerations:
l3nd leverages a multicast hello message and requires a TCP (possibly
protected by TLS) session to do its work.

In comparison to other proposals previously discussed:
- draft-xu-idr-neighbor-autodiscovery was session based, with comparability
  to the LDP hello mechanism.  It uses TCP after multicast discovery.
- l3dl itself is session based and implements its own transport and session
  layers. 
- draft-raszuk-idr-bgp-autodiscovery, determined to be not a fit for the
  core data center case, used "BGP over BGP".
[Sue = Core data center case is massive data center, or very large data
center. 
Core data center case is not Mini-Data Centers spread out in 5G centers] 

- draft-acee-idr-lldp-peer-discovery operates over layer 2 embedded in the
  LLDP protocol.  It is currently sessionless.  Discussion has suggested
  that if, in the future, security or large packet considerations drive
  requirements to need larger packets that LLDPv2 that is being currently
  stanardized in IEEE can serve those purposes.  LLDPv2 provides a session
  layer.
- draft-minto-idr-bgp-autodiscovery uses layer 3 IP multicast exclusively
  and does not have a session layer. 

[Sue: Jeff - based on the discussions, the next round of text L3DN will 
Hellos with LDP style resolution (lowest IP = Client, Highest client=Server)

Design choice: 
IMHO - a reliable transport and the security reduce errors and downtime. ] 

-----
[Point 6 - will start here in the next message]

Protocol state and FSM considerations:

- Similar to BGP, parallel session are a possibility.  I'm unclear how
  connection collisions are handled.
  + The procedures seem to try to cover this by saying "drop hello when
    there is an established session".  However, timing could result in two
    sessions happening in parallel prior to drop.
  + There isn't a clear idea of the FSM having an "established" state, or
    the related "OpenSent/OpenConfirm" states similar ot  BGP that could
    prevent this?

Open message:
- If Attributes are single-octet values of local significance, is there any
  significance to receiving the same Attribute twice?  If not, consider
  adding text about their uniqueness.
  + Also, why so small?  Payload length appears to be 4 octets.  A larger
    value permits more flexibility for local semantics; e.g. bit vectors.
  + Is there significance to resuming a session using the documented
    procedures when the Attributes may not be identical to the ones
previously
    received in the interrupted session?
- For clarity, consider labeling Serial Number as "last received serial
  number".

Ack message:
- The nybble next to Etype needs some flavor the usual text to deal with its
  contents.  "Must be zero on transmit, should ignored on receipt"
- The Etype conditions are a bit lax for a useful state machine for what we
  have documentation for:
  + If Open can't be acked successfully, should the session be permitted to
    continue?
  + Is "restart is hopeless" a shutdown?  An admindown?  What should the
    implementation do with its PDUs?  Should it tear down the session?
- 3 seconds seems a bit low for a response timeout.  While the protocol
  seems to be structured toward easy restart, why disconnect so
  aggressively?
- The message exchange protocol is "one outstanding ack" which means there
  is no pipelining of the various messages.  Given this, the lack of a
  serial number being acked in the message is understandable, but seems like
  it'd be good debug info.
  + That lack of pipelining means that full exchange of state will be gated
    on slowest sender on the session; the transmitter or the acknowledger..

Encapsulations:
- The text for addressing conflicts needs to specify the Etype along with
  the Error Code.
  + It can't be 0, because that says No Error.
  + It's probably 1 as a warning?
  + In general, normative text needs to specify Etype + Error Code together.
- 10.1, the procedure for expecting an ACK seems conflicting with the
  retransmission text in 9.1.  In that, it says if you're expecting an ack
  and don't get one, the session should be closed.  However, it's implied
  (but unclearly so) that you should have one pending ack at a time.  If
  retransmits of an encapsulation pdu are possible, this means multiple
  outstanding acks may be needed.  In such a case, serial numbers being
  acknowledged are more desirable.
  + In general, the text here for retransmit vs. ack seems to be a mismatch
    between the expectations of working on a reliable stream protocol like
    TCP.
  + Similarly, the "link is broken below layer 3" comment seems a mismatch
    since TCP shouldn't pay direct attention to that.
- 24 bits is a peculiar length for a counter.  Why this size?

Encapsulation Flags:
- 10.2 typo on Encapsulation
- Primary is interesting operational state.  What should an implementation
  do if it gets conflicting primary entries?

Encapsulations (varying):
- You can pack more than one prefix in a PDU for a given serial number.  In
  the event that a single prefix is an error (see question about about Etype
  for such cases), are all prefixes in the bundle intended to be rejected? 
  + How would the ack receiver know which prefix is the bad one if the whole
    bundle isn't rejected?

MPLS Label List/Encapsulation:
- After our "fun" dealing with label issues covering RFC 3107 and others,
  please make sure to tell people the obvious: the label stack needs to be
  well formed and bottom of stack bit should only be set on last entry.

Hello discussion:
- The English of the first paragraph is in need of significant work for
  clarity.
- I think this is trying to say that in the presence of a LAG that
  transmission over more than one LAG member might be okay.
- I also think that it's trying to say that once you have a session up you
  can stop hellos on the individual member links.
- I think the intent of the second paragraph is that if a link is on a
  multi-access network and non-p2p behavior is desired, keep sending hellos.

ULPC draft:

3.1:
:   A peer receiving BGP ULPC PDUs has only one active BGP ULPC PDU for
:   an particular address family on a specific link at any point in time;
:   receipt of a new BGP ULPC PDU for a particular address family
:   replaces any previous one.

Okay, implicit replace.  That's fine.

:   If there are one or more open BGP sessions, receipt of a new BGP ULPC
:   PDU does not affect these sessions 

Do not disturb existing sessions.  That's fine.

:   and the PDU SHOULD be discarded.

Probably not fine.  If the session bounces, you likely intend for it to
connect to the newly advertised entry.

:   If a peer wishes to replace an open BGP session, they MUST first
:   close the running BGP session and then send a new BGP ULPC PDU.

This seems to assume a tight coupling between the implementations for BGP
and for the discovery mechanism.  A peer bounce without having first
advertised the new address can mean a race condition where the bounced peer
continues trying (and possibly failing) to establish the connection with the
old peer.

It'd be cleaner to remove the prior binding first.

However, there's no withdraw procedure currently specified for these TLVs to
remove peer addresses, only to advertise them.

:   For each BGP peering on a link here MUST be one agreed encapsulation,
:   and the addresses used MUST be in the corresponding L3DP IPv4/IPv6
:   Announcement PDUs.  If the choice is ambiguous, an Attribute may be
:   used to signal preferences.

How?  Among other things, the attributes are bound to a session and not any
individual address.

3.1.2/3.1.3 Prefix-Len is probably not appropriate for the addresses.  If
there's an intent to permit peering from more than one address in the
subnet, how should the implementation pick?  From state learned in l3nd
encapsulation bindings for an overlapping subnet?

3.1.4 Authentication: You're referring to multiple possible ways security
could work.  This likely means multiple authentication types and should be
its own code point.

3.1.5 "No flags are currently defined"... and then defines GTSM.

:   As the ULPC PDU may contain keying material, see Section 3.1.4, it
:   SHOULD BE signed.

What does signed mean in this instance?  Do you mean that the transport
session should be given secrecy properties by TLS?

:   Any keying material in the PDU SHOULD BE salted and hashed.

I'm very confused here.

The material that a BGP peer needs for bringing up a session is the key to
pass to the relevant algorithm like TCP-MD5 or TCP-AO.  Typically when we
discuss salt and hashing, we're discussing the ciphertext from those
operations.  

Are you suggesting you send the ciphertext with the intent that an
implementation can consult its own tables to find what input password for
that salt should be used?  If so, this is a flavor of pre-shared key.  It'd
also require coordination of salt values between the implementations.

If this is an obvious technique that I'm not familiar with, a pointer to the
procedure would be useful.

-----

Misc:
- The embedded attempts at humor decrease the clarity of the document for
  readers that are less familiar with English and should be excised.
- In the PDU diagrams, explicitly document what the field lengths are.
  Don't make people interpret ASCII diagrams.
- The HELLO packet is itself without security.  In circumstances in which a
  man-in-the-middle attack is possible and raw TCP is offered as an option,
  active interception and replacement of the HELLO is sufficient to cause
  the less secure mechanism to be used.
- Section 7, the text about GTSM as a SHOULD on SYN should simply be GTSM on
  the TCP session.  GTSM applies to all of the packets in a TCP session.
- Section numbering under encapsulations could be more consistent.  For
  example, each section for a PDU type.  MPLS Label List probably shouldn't
  be a section at the same level as the mpls encapsulation pdus that use it.
- The session is long-lived but not protected similarly to BGP; i.e. tcp-md5
  or tcp-ao.  Given that the intent of the mechanism is to setup BGP which
  may use those properties, but uses TLS for protection, this isn't
  surprising.  But it means that the sessions are vulnerable to reset
attacks
  those TCP mechanisms are intended to protect against.

-----

Jeff's read of the proposal:

1. The proposal is session based.  While the prior discussion was primarily
in the context of draft-xu, sessions were seen as potentially problematic
for a number of reasons.  They also had some possibly useful properties.
1a. Session based makes large amounts of state easy to deal with.  That
said, the requirements show we need very little state for a given session.
That's true even in the ULPC portion of this proposal.
1b. These sessions are long-lived.  You'll have one for each interface you
want to run auto-discovery on.  This minimally doubles the number of active
sockets your stack will need because for each BGP session you get, you'll
have a lingering discovery session.  Perhaps they don't need to be
long-lived if the only thing you're using this for is BGP
auto-configuration?
1c. If you're going to use some flavor of TLS (including DTLS) as we
discussed during the requirements phase as a possible mechanism, you need a
session.
1d. If you're going to use something that rides on top of TCP, you have all
of the TCP vulnerabilities that TCP-MD5 and TCP-AO are intended to mitigate.
Using those mechanisms to mitigate the mechanism means that you're already
solving the keying issue... and perhaps don't need TLS at all.

2. The hello is completely unprotected.  This means there's an edge case
man-in-the-middle condition.  Protecting it means you've already solved the
keying discussions and wouldn't need TLS.

3. TLS crypto infrastructure is operationally weighty if you're covering the
usual things:
- What certificate authority are you using?
- How is revocation handled, if at all?
- If you want to use certificates for authentication, you now have to get
  those distributed to all of your participating routers.
- Certificates have lifetimes - although potentially very long lived ones.
  You have to now worry not only about usual key rollover considerations
  that all of the likley mechanisms need, but also making sure you don't
lose
  your auto-discovered sessions on a bounce because a certificate expired.

I'm loosely aware of the ACME work in IETF and perhaps the intent is that
these things are no longer quite as operationally cumbersome.  However,
that's an entire ecosystem that needs to be integrated into your router
operations if that's the case.

4. TLS provides integrity to the transport session and its contents, even if
it can't protect against attacks on TCP itself.  Privacy is optional,
although given the somewhat odd ULPC security text it seems that a scenario
under considertion is some form of distribution the pre-shared keys or
something similar.  However, the majority of the state for ULPC and l3nd
isn't exactly secret; it's stuff you'll see in ARP and ND if you're on link.

I don't think keychain information is considered something from most models
to need privacy. 

5. The proposal likely shouldn't call itself BGP-like.  The explicit ack
model minimally makes that not the case.  The state machine isn't terribly
comparable.

In particular, the Serial Number restart infrastructure seems to be very
heavy-weight for something running over a reliable transport like TCP.  But
this perception is tied to the idea that the BGP piece of this is small.
And even if there's a modestly large number of interfaces on the link, the
protocol should be able to send the entire bulk of them in a small number of
TCP frames.  (Compare vs. BGP send rates in even low end implementations.)
So, why complicate the mechanism with a highly stateful restart mechanism?
I suspect this makes more sense for the layer 2 proposal.

6. The sheer weight of this thing would make more sense if it's a more
general mechanism that is already deployed and the small BGP piece is along
for the ride.  I understand the motivation for a feature like this at layer
2, especially in the lsvr context.

I don't understand the use case for something like this in its current form
at layer 3.  What's that use case?

7. The crypto itself makes the autoconfiguration mechanism a point of attack
on the system rather than something to mitigate the attack.  The overhead to
deal with an on-link attack on TCP-MD5/-AO (symmetrical ciphers) is lower
than the overhead of TLS negotiation.  Rate limiting the creation of
incoming l3nd sessions might help things to some extent, but then you're
worrying similarly about TCP session exhaustion as well.

-- Jeff

[1] https://datatracker.ietf.org/doc/html/draft-ymbk-idr-l3nd-00
[2] https://www.ietf.org/rfcdiff?url2=draft-ymbk-idr-l3nd-ulpc-02.txt
[3]
https://datatracker.ietf.org/doc/html/draft-ietf-idr-bgp-autoconf-considerat
ions-02

_______________________________________________
Idr mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idr

---

From: Randy Bush <[email-masked]>
Date: Wed, 02 Mar 2022 15:20:33 -0800

> The ones who do not pass the AFI/SAFIs could easily add a TLV to do this. 

the question is not what could be added, but what does not need to be
there or should not be added.

duplicating bgp parameters between discovery and bgp open
  o gains/speeds nothing, you will still do bgp open
  o opens the window for disagreement and thus algs to resolve
  o but you do not discover disagreement until bgp open, way late
  o creates employment for very senior engineers as debuggers

randy, who had planned to stay out of this poolpah and will regret posting

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 7 Mar 2022 22:05:43 -0500

Sue,

> Simplicity: "The auto-discovery mechanism is designed to be simple. ... This

I'll give this one a light pass.  The TLS 1.1 spec is 80 pages of its own
text with multiple call-outs to other dependent specs for implementation.
That said, it's fairly easy to get implementations for embedded use in
applications these days.  

However, the operational and debugging complexity for TLS shouldn't be
discounted.  It's a tradeoff vs. security requirements.

> The ones who do not pass the AFI/SAFIs could easily add a TLV to do this. 

I believe Randy has covered this point.  They are intentionally withheld
from scope from the auto configuration protocol as being a source of
conflicting state.

Contrarily, AS number is part of the state discovered from BGP, yet
L3DL/L3ND have it. :-)

> Encapsulations + Encapsulation Interfaces, 

The L3ND authors have yet to make it clear why encapsulations are relevant
to BGP auto configuration.

Is the expectation that for a given interface L3ND will be advertising for
dozens, hundreds, thousands of BGP peering sessions?  The ULPC PDUs and
procedure seem to argue that for a given L3ND session that you may have one
of each AFI.

As Randy has himself emphasized multiple times in these efforts, 
RFC 1925 §2.(12) applies.

> revise the DT auto-configuration. 

https://datatracker.ietf.org/doc/html/draft-ietf-idr-bgp-autoconf-considerations-02
Section 2.3.1:

:   *  A version number representing when the BGP Session Protocol State
:      has last changed.  This can be used as a hint by an auto-discovery
:      client to determine when the state has been updated from a prior
:      version.  This can reduce repeated connections from an auto-
:      discovery client to the discovered BGP Speaker when information
:      has not changed.

> reliable transport (lower timers) + ability to plug in good security. 

Transport considerations partly depend on the amount of state you're passing
around.  And, as noted elsewhere in these reviews, timers for carrying data
payload vary depending on network loss and framing issues.

A transport layer itself also has security considerations.

"good security" is a subjective point, as we're discussing.  It's perhaps
more precise to state that if you want the properties TLS gives you, you
need a session based transport.

> [Sue - yes it can expand]

Given the intended scope of the work we've chartered is for BGP
autoconfiguration, it's reasonable that since the bulk of the procedure for
L3ND isn't for BGP sessions that it should be justified.

In the case of L3DL, BGP is along for the ride for applications LSVR
requires, so that makes sense.

> It could also just run TCP]. 

The comment was targeted toward "WAN".

> [Point-4] 
[...]
> [Sue: The introduction provides you truth in advertising that it scales. ]

To repeat a question asked elsewhere: How many BGP peering sessions are you
expecting to do on a given link?

You can be as massive as you like.  But for the most part, the data center
discussions throughout this process have largely been in reference to
fabrics.  For most common fabric topologies, the individual per router radix
is not very large.  This puts us either into an unusually large number of
peering sessions per-interface, per-leaf or something that's not a fabric.

Understanding the desired scope of why "because MDC" applies to high scale
per-interface BGP peering is the key detail.

> IMHO - a reliable transport and the security reduce errors and downtime. ] 

I believe these are non sequitir.

Please feel free to expand on why those properties have those impacts.

-- Jeff