Subject: Re: [nvo3] Antw: Re: Antw: Re: Antw: What's a TSI? It depends...

From: "Pat Thaler" <[email-masked]>
Date: Tue, 7 May 2013 18:18:25 +0000

Actually, the number of MAC/VID pairs is more limited than that because the limiting factor is the maximum TLV information field length of 512 bytes (limited by a 9 bit length for that field). Therefore, the maximum number of MAC VID pair entries is 61.

The expectation was that a device using more addresses such as something offering some type of gateway service, would use filter info of just  VID or GroupID/VID.

Pat

From: [email-masked] [mailto:[email-masked]] On Behalf Of Florian Mahr
Sent: Saturday, April 27, 2013 5:05 AM
To: Pat Thaler; [email-masked]; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: [nvo3] Antw: Re: Antw: Re: Antw: What's a TSI? It depends...

AFAIK, the number of MAC/VID pairs in one VDP association TLV is specified by the Number of Filter Info entries field, and this field has a length of 16 bits.
Nevertheless, in regard to the case of a TSI de-associate with 100 addresses associated with that TSI, I think it is not quite efficient to signal them as 100 indivual addresses to the NVE. In such a situation I would prefer to signal only a single TSI Instance ID (TSIID) which represents all the addresses associated with that TSI. However, in order to achieve that, the NVE needs to know the exact relationship between the addresses and the TSI. In VDP this is quite easy to achieve, as an 128-bit VSIID is always part of an VDP association TLV. I personally do not see any disadvantage when each TSI would be assigned a 128-bit UUID at its creation time, which is then signaled to the NVE as part of an associate and de-associate request.

Thanks, Florian

>>> "Larry Kreeger (kreeger)" <[email-masked]<mailto:[email-masked]>> 27.04.13 1.11 Uhr >>>
Hi Pat,

How do we decide what the right number of addresses to support on a TSI is?  If the TS is bridging, it could have a lot.  I have no idea if 100 is the right number either, but it doesn't seem far fetched.  How many is the "small number of MAC/VID pairs" supported by VDP?

Thanks, Larry

From: Pat Thaler <[email-masked]<mailto:[email-masked]>>
Date: Friday, April 26, 2013 3:29 PM
To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>, "Reith, Lothar" <[email-masked]<mailto:[email-masked]>>, David Black <[email-masked]<mailto:[email-masked]>>, Florian Mahr <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: RE: [nvo3] Antw: Re: Antw: What's a TSI? It depends ...

We found similar use cases when we defined VDP in IEEE 802.1Qbg.
To allow a switch to classify traffic as belonging to a VSI (Virtual Service Interface – similar to a TSI), we defined several “Filter Info Formats” – information in the frame that would identify it as belonging to that end point.

To support a system that was providing a service across many addresses (e.g. a firewall or load balancer), the filter info can be a VID (VLAN ID) or a list of VIDs. That allows for a device that receives and sends on any MAC addresses within the VID(s). Another filter info format carries one or more MAC/VID pairs to allow for a VSI that is associated with a small number of MAC/VID pairs.

For the case where the network was managed on the basis of a longer tenant identifier, e.g. a 24-bit I-SID, there are formats that add a 32-bit Group ID to the VID or MAC/VID. We created this format because of the use case where the data center is using PBB, but the concept could also apply to tenant identifiers for NVO.  When this format is used, the profile for the VSI provides the Group ID. A locally-scoped VID is assigned to map to the Group ID.

I agree with Larry that one should allow for cases where there are multiple addresses associated with a VSI without having all addresses going to that VSI though I’m not convinced that the number for that should be as high as 100.

Regards,
Pat

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Larry Kreeger (kreeger)
Sent: Thursday, April 25, 2013 2:55 PM
To: Reith, Lothar; Black, David; Florian Mahr
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Antw: Re: Antw: What's a TSI? It depends ...

Lothar,

While I agree that a NIC would be programmed to accept frames with any MAC address, the reason for me to discuss addresses is more related to how the addresses are relayed to the NVE for inner-to-outer address mapping for delivering frames/packets across the overlay VN to the correct NVE.  It seems like a bad idea for any station that can receive more than one address to request the NVE to deliver all frames/packets to it.  If there are 100 stations on the other side of a VM performing bridging, the overlay should only deliver  those 100 addresses to the NVE the VM is connected to, not any address.

This discussion does make me wonder if we should have a way for an NVE to deliver "unknown" addresses to a TS.  There are applications that (such as one mode of Microsoft Network Load Balancing) that desire unknown frames to be delivered to them.  A VM acting as a bridge may be one of these applications because the bridge may not know exactly what station addresses are available its interfaces if the stations are quiet until spoken to.

- Larry

From: <Reith>, Lothar <[email-masked]<mailto:[email-masked]>>
Date: Thursday, April 25, 2013 1:20 PM
To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>, David Black <[email-masked]<mailto:[email-masked]>>, Florian Mahr <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: AW: [nvo3] Antw: Re: Antw: What's a TSI? It depends ...

Larry,

thinking about it the following may be a solution how we could model this without violating the critical one to one relation between station and MAC-address.

In case where the station needs to be receiver for more than a single unicast MAC address (e.g. if the station is a bridge port), then we need to model it as if the wildcard MAC-address (or broadcast MAC address with the value 0xFFFFFF is assigned to the station.

This would model the behavior of a transparent bridge correctly, I hope at least.

I believe in all the cases you mention, the station is configured with the Station-ID 0xFFFFFF, which is a valid MAC-Address, but not a L2-endpoint address.

The functions you mention such as bridging, IDS, firewall may be considered filtering of relayed frames  while bridging them from the ingress station to a predetermined egress station of the bridge.

L3-Forwarding using a routing table in a router  may be considered bridging from an ingress station to  dynamically determined egress station of the bridge, where said dynamic determining of said egress bridge port happens  on the basis of looking at L3 address information (IP-address) encapsulated in that bridged layer 2 frame.

Loadbalancing may be considered hunting an available egress station of the bridge, while perhaps at the same time also hunting one of multiple end stations present on the virtual or real yellow cable attached to the dynamically selected egress bridge port. Said hunting may for example be using round robin loadbalancing among a group of stations referred to as “hunt group”. There may be DNS lookup involved when terminating different services on different stations (identified each by a MAC-address within the local Layer2 network and also identified by a globally reachable IP address, that not only identifies the end point, but also the virtual yellow cable by the CIDR that identifies the subnet associated with the virtual yellow cable on which the destination station is present.

Lothar

Von: Larry Kreeger (kreeger) [mailto:[email-masked]]
Gesendet: Donnerstag, 25. April 2013 21:20
An: Reith, Lothar; Black, David; Florian Mahr
Cc: [email-masked]<mailto:[email-masked]>
Betreff: Re: [nvo3] Antw: Re: Antw: What's a TSI? It depends ...

Hi Lothar,

In the simplest case of a VM acting as a host with a vNIC, the station has one MAC address and one IP address using that MAC.  I assume this is the most common case, but I think we must also support the cases where the VM is:

 (a) acting as a bridge (perhaps while performing a service such as IDS or firewall, or acting as a L2 VPN gateway),
 (b) acting as a router (perhaps while performing a service such as IDS or firewall, or acting as an L3 VPN gateway),
 (c) acting as load balancer terminating different services on different IP addresses

In case (a), the VM's vNIC may have a MAC address that it uses for an L3 interface used for management, but there will also be frames entering and leaving the vNIC with MAC addresses that belong to other stations for which the VM is performing the bridging function.  The VM could be bridging these frames between two vNICs, or between two VNs represented as VIDs on the same vNIC.

In case (b), the VM always sends/receives frames using the vNIC MAC address, but these frames may be carrying packets with the IP addresses of stations for which the VM is performing the routing function, in addition to the VM's own IP address used for management or control plane traffic.

In case (c), the VM always sends/receives frames using the vNIC MAC address, but these frames may be carrying multiple IP addresses which are terminated on the VM.

I don't think these cases are all covered in the model you describe below.  If they are, can you help describe the relationships you are thinking of?

In the thread below, David is asserting that the NVE does not need to know what TSI any of the addresses are associated with (only the VAP).  I agree that the TSI is not strictly needed, but I think it could be useful for the following reasons.   In the cases above where there are multiple addresses associated with a TSI.  If we look at case (a), the NVE will need to forward frames towards the vNIC for all the MAC addresses for which the VM is performing the bridging function (let's say 100 addresses).  When that VM migrates to another hypervisor, the NVE will need to disassociate the 100 addresses from the NVE port leading towards the vNIC.  This could be signaled from the hypervisor to the NVE as 100 individual addresses, or it could be signaled as one TSI disassociate (if the relationship of the addresses to the TSI are known).

Thanks, Larry

From: <Reith>, Lothar <[email-masked]<mailto:[email-masked]>>
Date: Thursday, April 25, 2013 11:16 AM
To: David Black <[email-masked]<mailto:[email-masked]>>, Florian Mahr <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] Antw: Re: Antw: What's a TSI? It depends ...

David,

I fully agree with your statement: “Because we’re getting into object-oriented design where the nature of relationships is crucial”

Therefore I would like to ask: Are we 100% sure, that an association of multiple MAC-addresses directly with the vNIC is a good idea?

Something is disturbing me with that idea. Looking at Bob Metcalfe’s 40 year old design, it becomes obvious, that the object “station” may be missing.

Requirement1: The object “station” needs a station-name, that is not the MAC address.
Reason: To support use cases where an associated MAC-address or an associated MEP-ID has to be changed, e.g. due to a hardware failure of a NIC, but without having to tell all communication partners, which may even be in other administrative domains.

Requirement2: There needs to be a maximum of one MAC-address assigned per station, or none during times when a station does not have a MAC address assigned.
Reason: To be able to leverage existing standards for endpoint identification and authentication and to ease the IT support for that. I am thinking of DHCP and of DHCP intermediate agent (where the station-id is the identifier that identifies the object that tries to get the IP address assigned to it – I conclude therefore that the object is the station) or the PPPoE intermediate agent (where the station-ID is the identifier that identifies the object that tries to get an IP address dynamically assigned – obviously I conclude that this object is the station).

To relate that to the below: The MAC-address is strongly associated with the station, but the MAC-address is not the station.

And when an IP address gets associated to a MAC-address with DHCP, that does not mean that the IP-address identifies the MAC-address, it does also not mean that the IP address identifies the host or VM, rather one could argue the IP-address always identifies a station on a physical or virtual yellow cable, that is located at the network facing side of the NIC (or vNIC in case of VM).  On the associated compute facing side the corresponding compute facing interface object may be called “port” (at least OpenStack Quantum API2.0 calls this compute-side-facing interface “port”, and assigns a MAC-address to the port, which in reality does not get assigned to the port, but to the station on the “other side” of the interface, i.e. on the network facing side.

As a side note: One could argue, that a loopback address in a router could be considered as a degenerate case of a collapsed virtual yellow cable that is not bound to a physical NIC, but has a single station as one endpoint of a virtual yellow cable, a single interface to a gateway as another endpoint, and the length of the virtual yellow cable being zero – such that the loopback network and the loopback station and the loopback interface are collapsed into one singularity requiring only a single IP address to identify all 3 objects.

Now, what does all have to do with the title of this thread “what is a TSI?”. When looking with a compute side paradigm at this, the answer may be: a port (or in Quantum API: a Quantum network port).

When looking at this with a network side paradigm, the answer may be: a station

Lothar

Von:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]] Im Auftrag von Black, David
Gesendet: Donnerstag, 25. April 2013 15:24
An: Florian Mahr
Cc: [email-masked]<mailto:[email-masked]>
Betreff: Re: [nvo3] Antw: Re: Antw: What's a TSI? It depends ...

> FM> Could you please give me more details why you think it might be better not
> to treat the L2/L3 addresses as attributes of a TSI object and signal them as
> such in a hypervisor-to-NVE control protocol?

Sure, but let me start with a caveat that some of this may seem like hair-splitting,
because we’re getting into object-oriented design where the nature of relationships
is crucial.  From that perspective “attribute of” is a “part of” relationship, whereas
I view addresses as “associated with” TSI objects.  To make this distinction clear by
example:

      - My car is strongly associated with me.
      - Nonetheless, my car is not part of me. :-) :-) :-)

Regarding signaling, I believe that the addresses should be signaled, not TSIs, and
I’m a co-author of draft-kompella, which spells out some of the details of how that
could work.  That draft just expired (mea culpa - my day job’s been limiting the
time that I can put into IETF), I expect to submit a new version either today or
tomorrow (it’s written, just needs some final checks).

OTOH, I do not believe that the TSIs need to be signaled as TSI objects.  To make
this a bit more concrete, I don’t think the following two cases need to be
distinguished at the VAP or NVE for a single VAP and VNI:

- 6 VMs, one untagged vNIC each, one MAC/vNIC, total of 6 TSIs
- 3 VMs, one untagged vNIC each, two MACs/vNIC, total of 3 TSIs

VM events may implicitly distinguish these cases (e.g., one MAC vs. two MACs
affected by VM migration or VM power on/off events), but (IMHO) the signaling
protocol doesn’t have to care - the NVE and VAP just have to know that those
6 MACs on VLAN 72 are for VNI 348.

Thanks,
--David

From:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Florian Mahr
Sent: Thursday, April 25, 2013 4:19 AM
To: [email-masked]<mailto:[email-masked]>; Black, David
Cc: [email-masked]<mailto:[email-masked]>
Subject: [nvo3] Antw: Re: Antw: What's a TSI? It depends ...

Hi David,

Please see my comment below marked as "FM>".

Thanks,

Florian
>>> "Black, David" <[email-masked]<mailto:[email-masked]>> 24.04.13 20.13 Uhr >>>
Lothar,

> Something is disturbing me with your definitions of the term NIC and vNIC.

Ok ...

[... snip ...]

IMHO, it’s important to distinguish definitions of concepts/objects from
the assembly/construction of solutions:

> I understand that many on the list would prefer not having to consider
> NIC-teaming with LAG, but the market realities may not allow such a
> simplistic solutions that unfortunately lacks any resilience.

Not exactly ;-).  To be precise, my position is that I would prefer
“not having to consider NIC-teaming with LAG” *to be part of the TSI*.

I agree that LAG will need to be part of a lot of solutions (there’s a
lot of it deployed out there and its functionality is important), but
I expect that LAG functionality to be part of a solution component other
than the NIC, vNIC, pNIC or TSI.  Possible locations for the LAG
functionality include the hypervisor softswitch and in an operating
system (e.g.,  device driver) running on bare metal or in a VM.

> Pre-emptively, I would further argue, that there may be a couple of
> more objects, such as:

> A L3-interface (network side facing) of the NIC (as well as of the VNIC)
> A L2-interface (network side facing) of the NIC (as well as of the VNIC)

I’m inclined to agree in principle, in part because I’ve taken a strong
position that L2 and L3 addresses should not be part of the TSI object,
so some other objects appear to be needed.

FM> Could you please give me more details why you think it might be better not to treat the L2/L3
addresses as attributes of a TSI object and signal them as such in a hypervisor-to-NVE control protocol?

At the level of abstraction
of this discussion, the objects you’ve proposed seem plausible.  To go
further, I’d suggest taking a look at the IF-MIB (RFC 2863) for some
relevant concepts, although nvo3 surely does not need everything that’s
in that MIB.

Returning to LAG, for LAG implemented in an OS, it would make sense for
LAG to be below the L2 interface but above the TSI.

Thanks,
--David

From: Reith, Lothar [mailto:[email-masked]]
Sent: Wednesday, April 24, 2013 1:12 PM
To: Black, David; Florian Mahr; David Allan I
Cc: [email-masked]<mailto:[email-masked]>
Subject: AW: [nvo3] Antw: What's a TSI? It depends ...

David,

Something is disturbing me with your definitions of the term NIC and vNIC.  So please forgive me that I go back to this definition and make a proposal how some of my perceived ambiguity could be eliminated.

- NIC: Physical network interface (e.g., NIC card), presented as network-side-facing interface by a server to a physical cable.

- vNIC: Virtualized version of a NIC.  Presented by a hypervisor as network-side-facing interface to a virtual yellow cable. Said virtual yellow cable (aka virtual link that may be a multipoint link)  being emulated by a V-Switch that is under the same administrative control as said hypervisor. The virtual yellow cables emulated by the V-Switch connect one vNIC to one NIC, or one vNIC to multiple NICs (NIC teaming/LAG).  In addition, said hypervisor presents a virtual hardware interface as an associated compute-side-facing interface to a Virtual Machine (VM) which is also under the administrative control of said hypervisor.

I understand that many on the list would prefer not having to consider NIC-teaming with LAG, but the market realities may not allow such a simplistic solutions that unfortunately lacks any resilience.

Pre-emptively, I would further argue, that there may be a couple of more objects, such as:

A L3-interface (network side facing) of the NIC (as well as of the VNIC)

-          As client of an underlying server layer network service: Interfacing to a service access point of an underlying layer 2 server layer network service (such as a virtual station Interface on a virtual yellow cable)

A L2-interface (network side facing) of the NIC (as well as of the VNIC)

-          As client of an underlying server layer network service: Interfacing to a service access point of an underlying layer1 server layer network service

The nice part of this definition is, that it fits both cases, when the VAP and NIC are not in the same device.e. with the VAP in the ToRswitch , and when the VAP and NIC are collocated back to back  in the hypervisor (in which case the yellow cable may be considered a collapsed yellow cable).

It is also extensible in multiple dimensions, i.e. in space when a MEF Ethernet EVC may be located in  between the VAP and the NIC, because a MEF EVC is a virtual yellow cable.

It is also extensible from point to point to multipoint, just by adding a station to the yellow cable, even if is a  collapsed virtual yellow cable inside the hypervisor.

Lothar

________________________________
From:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Black, David
Sent: Tuesday, April 23, 2013 11:50 AM
To: Florian Mahr
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] Antw: What's a TSI? It depends ...
Hi Florian,

> Is there already a common understanding of the terms NIC, vNIC, TSI
> and how these terms relate to each other?  I think it would be good
> to summarize the statements received so far in regard to this topic.

IMHO, yes and no ☺.  I think the terms NIC, pNIC and vNIC are relatively
well understood:

- pNIC: Physical network interface (e.g., NIC card).
- vNIC: Virtualized version of a pNIC.  My usage of this term
      always refers to a virtual hardware interface presented by
      a hypervisor to virtual machines (VMs), not anything higher in
a VM’s OS stack (or lower in the hypervisor).
- NIC: General term encompassing both pNICs and vNICs.

OTOH, TSI is more difficult, IMHO, as indicated by the title of this
thread ;-).  Here’s my take on that, which I don’t promise to be much
more than IMHO ...

I believe the goal is that TSI:VAP cardinality be 1:1, so Dave Allen’s
observation on NIC cardinality is relevant (Dave is implicitly assuming
that the vNIC is VLAN-tagged):

> the cardinalities of vNIC to VAP does not line up. It is potentially a
> 1:N relationship.

That’s complemented by Larry Kreeger’s observation on how the mux/demux
works on the VAP side (based on VLAN tags, aka VIDs, no surprise there):

> LK> We are assuming that the physical link connecting a hypervisor
> vSwitch to an external NVE is plain old 12 bit VLAN tags, so when a
> frame arrives at the external NVE, it doesn't necessarily know what
> TSI it originated from

Beyond that, there are two complicating factors that I strongly prefer to
leave out of the definition of a TSI, based on the amount of confusion
they have caused in discussion (much heat, not much light):

- Addresses, both MAC and IP
- Link Aggregation, whether above or below the NICs (below NIC
      case covers vNIC usage of aggregated pNICs).

Hence my proposal for the definition of a TSI starts from:

> When VLAN tagging is visible at the Tenant End system (i.e., NIC or
> vNIC is VLAN-tagged), but is *not* part of the L2 virtual network
> service, each VLAN is attached to a different virtual network instance
> at the NVE, so the TSI would be a <NIC, VID [VLAN ID]> pair.  This is
> an important case that will occur in practice.

Restating that proposal:

- If the NIC is not VLAN-tagged, the NIC is a TSI.
- If the NIC is VLAN-tagged, the TSI is the NIC plus a VID.

I’d prefer to stop there, as that suffices when the L2 network service
provided by the nvo3 overlay is not VLAN-tagged.

If the L2 network service provided by the overlay is VLAN-tagged, the
VLAN-tagged NIC case above needs generalization.  For clarity “L2 network
service provided by the overlay is VLAN-tagged” means that the encapsulated
inner L2 frames exchanged between NVEs are VLAN-tagged.  At an NVE, a VAP
for such a network sends and receives frames to/from a corresponding NIC
on multiple VLANs, not just one. The upshot is that the generalization
of TSI for this case is that the TSI is a NIC plus a set of VIDs, namely
those VIDs that the VAP is using.

That generalization complicates explaining what’s going on - my first
choice is to leave it out of the general drafts, but allow solution drafts
to explain it.  Alternatively, if it needs to be explained in the general
drafts, I think it’s important to cover the “L2 network service provided
by the nvo3 overlay is not VLAN-tagged” case first (TSI is the NIC plus
a VID) before generalizing.

Does that make sense?

Thanks,
--David

p.s.  Of course, once this is settled, the next question will be how it
applies to L3 network services provided by an nvo3 overlay, if at all.
For my part, I hope the L3VPN people are watching this with mild bemusement
based on  the principle that IP addresses plus routing information sort
this all out at the NVE for L3VPNs ... I hope ;-).

From: Florian Mahr [mailto:[email-masked]]
Sent: Tuesday, April 23, 2013 5:34 AM
To: Black, David
Cc: [email-masked]<mailto:[email-masked]>
Subject: Antw: [nvo3] What's a TSI? It depends ...

Hi David,

Is there already a common understanding of the terms NIC, vNIC, TSI and how these terms relate to each other?
I think, it would be good to summarize the statements received so far in regard to this topic.

Thanks,

Florian
>>> "Black, David" <[email-masked]<mailto:[email-masked]>> 19.04.13 15.56 Uhr >>>
I would put the TSI at point B (NIC + VN), but with one important change - IMHO, we should think in terms of VLANs at that interface, not virtual networks.  The TSI concept that we need attaches to one virtual network instance (end system analog to the VAP).  Here’s an attempt to walk through this.

The starting point is that a physical server with an untagged NIC attaches to exactly one virtual network instance at the NVE, therefore the TSI is the NIC.  Ditto for a virtual machine with an untagged vNIC - that untagged vNIC is the TSI (and I like Dave Allen’s approach of using NIC as the general term w/vNIC as a specialization).

Let’s call that vNIC-VAP communication path a “virtual link”; the next topic is what happens when the hypervisor uses VLANs or some other mechanism to trunk multiple untagged virtual links on a single physical link.  The TSI is still the far end of the virtual link and the VAP is still the other end, but there is now mux/demux functionality in both the NVE and End Device to handle this trunking.

Making VLAN tagging visible to the Tenant End System and providing it as part of the L2 virtual network service complicates things, and these need to be looked at independently.

When VLAN tagging is visible at the Tenant End system (i.e., NIC or vNIC is VLAN-tagged), but is *not* part of the L2 virtual network service, each VLAN is attached to a different virtual network instance at the NVE, so the TSI would be a <NIC, VID [VLAN ID]> pair.  This is an important case that will occur in practice.

Adding VLAN tagging to the L2 virtual network service results in at least a couple of possibilities.  The simplest one is that the VLAN tags from the NIC are end-to-end within the encapsulation, and we’re back to the TSI being the NIC which is associated with one virtual network instance via one VAP.
We might want to stop here, at least for the problem statement because things start to get complex beyond here.

The next step is to generalize the mux/demux of virtual links onto the End Device-NVE physical link to in effect allow VLAN tagging of those virtual links.  The result is that the VLANs used by the NIC in the Tenant End System get grouped by virtual network that they’re attached to (i.e., which VAP is used on the NVE), and the TSI would be a NIC plus a set of VIDs.  That’s the fully general definition with the explicit note that the set of VIDs is often empty or contains one VID.

For example, at a vNIC, VLANs 1-6 might attach to VNI A, VLAN 7 to VNI B, VLAN 8 to VNI C and VLANs 9-10 to VNI D.

The next wrinkle is that it’s quite possible for multiple virtual links that are trunked onto a common physical link to want to use the same VIDs, which will lead to having to remap VIDs as part of the mux/demux for trunking (at both ends in full generality).  I’d really prefer not to go there in the problem statement and framework drafts ;-).

Finally, I think the use of IP addresses or MAC addresses *to identify virtual network instances* is a mistake, and hence points C and D are bad ideas because the desired TSI-VAP virtual link is scoped to a virtual network instance.

Thanks,
--David

From: Qin Wu [mailto:[email-masked]]
Sent: Friday, April 19, 2013 3:17 AM
To: Larry Kreeger (kreeger); Anoop Ghanwani
Cc: Black, David; [email-masked]<mailto:[email-masked]>; Reith, Lothar
Subject: RE: [nvo3] 答复: 答复: 答复: 答复: NVO3 Terminology changes

+1,
Point B is a perfect reference point. But I am not sure we only use VNID to distinct one TSI from another, we may use combination of VNID, MAC address and IP address.
e.g., is there any case we use two TSIs belong to the same vNIC connect to the same VN, In this case, both TSIs use the same VNID.

Regards!
-Qin
From: Larry Kreeger (kreeger) [mailto:[email-masked]]
Sent: Friday, April 19, 2013 4:41 AM
To: Anoop Ghanwani
Cc: Qin Wu; Black, David; [email-masked]<mailto:[email-masked]>; Reith, Lothar
Subject: Re: [nvo3] 答复: 答复: 答复: 答复: NVO3 Terminology changes

Hi Anoop,

What benefit do you see in defining the TSI as point C or D?  Do you agree that the TSI should directly correspond to the VAP of the NVE?  If so, it seems like point B corresponds to it.

Thanks, Larry

From: Anoop Ghanwani <[email-masked]<mailto:[email-masked]>>
Date: Wednesday, April 17, 2013 8:01 PM
To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>
Cc: Qin Wu <[email-masked]<mailto:[email-masked]>>, David Black <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "Reith, Lothar" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] 答复: 答复: 答复: 答复: NVO3 Terminology changes

Larry,

In reference to the figure below, I think all of them could be valid TSIs depending on the application.

Anoop

On Wed, Apr 17, 2013 at 9:55 AM, Larry Kreeger (kreeger) <[email-masked]<mailto:[email-masked]>> wrote:
Hi Qin,

A VM is one possible implementation of a Tenant System. I have not seen anyone suggesting that a Tenant System get more granular than that.

Regarding what defines a TSI, someone could choose any branch in the tree below to define as the TSI.  I was assuming that the TSI would be at point A below in the tree.  However, point B would correspond the the VAP.  Pat had mentioned having the TSI at point D.  I think if we were to define the TSI to be either point C or point D, then it would depend on the service being given (L2 or L3).  I would prefer it to be at point A or B.

         VNIC-+-VN-+-MAC-+-IP
           ^  |    |     +-IP ...
           |  |    |
           |  |    +-MAC-+-IP
           |  |          +-IP ...
           |  |
           |  +-VN-+-MAC-+-IP
           |     ^ |     +-IP ...
           |     | |
           |     | +-MAC-+-IP
           |     |    ^  +-IP ...
           |     |    |     ^
           |     |    |     |
           A     B    C     D

 - Larry

From: Qin Wu <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, April 16, 2013 7:14 PM
To: Larry Kreeger <[email-masked]<mailto:[email-masked]>>, Anoop Ghanwani <[email-masked]<mailto:[email-masked]>>, David Black <[email-masked]<mailto:[email-masked]>>

Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "Reith, Lothar" <[email-masked]<mailto:[email-masked]>>
Subject: [nvo3] 答复: 答复: 答复: 答复: NVO3 Terminology changes

Hi, Larry:
Is relation between vNIC and TSI one to one or one to more?
Let me interpret what you said below,
(a)If a VM has only one vNIC, then vNIC that belongs to the VM will be dealt with as one tenant system. Each vNIC corresponds to one TSI.
(b) If a VM has 5 vNICs and each vNIC is assigned with only one IP address, each vNIC that belongs to the same VM will be treated as one Tenant System. Each vNIC corresponds to one TSI. Then one VM have 5 Tenant system. Each Tenant System has one TSI.
(c) If a VM has 5 vNICs and each vNIC is assigned with 2 IP addresses, each vNIC that belongs to the same VM will be treated as one Tenant system, each vNIC with each IP address corresponds to one TSI. Then one VM has 5 tenant system. Each Tenant system has 2 TSIs.
That is to say if the relation between vNIC and TSi is one to more, then we can easy to guarantee that each TSI is associated with only one NVE.

However if one VM is simply regarded as one tenant system and each vNIC with multiple IP addresses belonging to one VM is treated as only one TSI in (b)(c), take (c) as an example. when one vNIC is assigned with multiple IP addresses and connect to multiple VNs through multiple NVE, then one TSI has multiple IP addresses and will be associated with multiple NVEs.
Here is two figure2, figure 1 shows one to one relation between vNIC and TSI, figure 2 shows one to more relation between vNIC and TSI.

                +---------------------+
                |                     |
                |  VM (Tenant System) |
                |                     |
                +-+--------+--------+-+
                  |        |        |
                  |        |        |
                  |        |        |
                  +        + ...... +
            vNIC1(TSI) vNIC2(TSI)  vNICx(TSI)
                  |
                  |
         |--------+------+
    (IP add1) (IP add2)  (IP addr3)
         |        |      |
        VN1      VN2    VN3

   Legend: TSI -- Tenant System Interface
           VN  -- Virtual Network
           IP add1
               -- IP address1
       Figure 1 One to One relation between vNIC and TSI

               +---------------------+
               |                     |
               |  VM (Tenant Systems)|
               |                     |
               +-+--------+--------+-+
                 |        |        |
                 |        |        |
                 |        |        |
                 |                 |
          vNIC1(TS1)   vNIC2(TS2) vNIC3(TS3)
                 |
                 |
         +-------+------+
         |       |      |
         |       |      |
        TSI1    TSI2   TSI3

   (IP add1) (IP add2) (IP addr3)

    Legend: TSI1 -- Tenant System Interface1
            TS1_ Tenat System1

     Figure 2 One to more relation between vNIC and TSI

I think figure 2 well match what you said below. please confirm whether my understanding is correct.

Regards!
-Qin
发件人:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]] 代表 Larry Kreeger (kreeger)
发送时间: 2013年4月17日 7:58
收件人: Anoop Ghanwani; Black, David
抄送:[email-masked]<mailto:[email-masked]>; Reith, Lothar
主题: Re: [nvo3] 答复: 答复: 答复: NVO3 Terminology changes

Hi Anoop,

I definitely see a TSI as belonging to a single tenant, since it is an interface on a single tenant's system (Tenant System).  My assumption is that each TSI is associated with only one NVE, so that there is a one to one correspondence between a TSI and a VAP on the NVE.  It is what is connected to the bottom lines in figure 4 of http://tools.ietf.org/html/draft-ietf-nvo3-framework-02

 - Larry

From: Anoop Ghanwani <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, April 16, 2013 4:47 PM
To: David Black <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "Reith, Lothar" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [nvo3] 答复: 答复: 答复: NVO3 Terminology changes

When I think of TSI, I think of something different than a vNIC (or, or in the case of non-virtualized systems, a NIC).  Of course, it's possible to create an implementation where a TSI is always a vNIC (or NIC), but I don't think it is always limited to that.

It might help to clarify the following properties of a TSI:
- Can a TSI be associated with multiple NVEs?
- Does a TSI carry traffic from only one tenant or multiple tenants?

Anoop

On Tue, Apr 16, 2013 at 3:32 PM, Black, David <[email-masked]<mailto:[email-masked]>> wrote:
I think this may reflect some confusion around possible meanings of “interface”.  I believe that the notion of “interface” that is desired here corresponds to a vNIC, not a bonded set of vNICs (and likewise to a NIC, not a bonded set of NICs).  If that clarification is clear, I think TSI would be a reasonable term.

Thanks,
--David

From:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Reith, Lothar
Sent: Monday, April 15, 2013 7:18 AM
To: Qin Wu; Larry Kreeger (kreeger); [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] 答复: 答复: 答复: NVO3 Terminology changes

I believe that it is not possible to replace the term vNIC with “TSI”, because multiple vNICs may form a single interface.

 AFAIK it is common practice to use vNIC bonding (LAG) for high availability, where two vNICs of one VM are associated with different physical NICs of the virtualization host running the VMs. Creating a LAG of these 2 vNICs ( NIC-Bonding) creates a single Interface of the VM which survives the outage of one of the both physical NICs.

Lothar

Von:[email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] Im Auftrag von Qin Wu
Gesendet: Montag, 15. April 2013 09:17
An: Larry Kreeger (kreeger); [email-masked]<mailto:[email-masked]>
Betreff: [nvo3] 答复: 答复: 答复: NVO3 Terminology changes

>>I believe one tenant system can host multiple VMs, each VM may have multiple vNIC adapters that it uses to communicate with both the virtual and physical networks.

>LK> A VM is one example of a tenant system…so it would not host VMs.  You may be thinking of "End Device".

>[Qin]: Not sure about that, the definition of “Tenant system” in Framework said:
“
>       Tenant System: A physical or virtual system that can play the role
>       of a host, or a forwarding element such as a router, switch,
>       firewall, etc. It belongs to a single tenant and connects to one or
>       more VNs of that tenant.
”
>So tenant system can be a host and host one or multiple VMs on it. What am I missing?

>LK2> I think that you are assuming that "host" is synonymous with "Hypervisor".  In the definition above, I believe the term host relates to the more traditional definition of an internet host such as in RFC 1122.

[Qin]: So “Host” in the definition of tenant system seems misleading since we two have different interpretation to it. I agree Hypersor or Server or Server blade can host multiple VMs, however in the framework document, it also said, a host can be server or server blade in the definition of End device.

LK3> Only servers or server blades that are running hypervisors can host multiple VMs.  When we mean hypervisor we must say hypervisor (what form factor of server it is running on is irrelevant).  All hypervisors run on servers,

[Qin]: Yes.

>but all servers are not running hypervisor software.

[Qin]: Are you saying that not all server are running hypervisor software? If yes, I agree. If not, would you like to clarify a little bit?

Suppose one tenant A have 2 VMs resided in the server1 or hypervisor1. Tenant B have 3 VMs resided in the server 1.

Can we say each VM belonging to the same tenant is a tenant system

LK3> Yes

or multiple VMs of each tenant sharing the same Server belong to the same single tenant system?  i.e., Tenant System A corresponding to Tenant A contains 2 VMs. Tenant System B corresponding to Tenant B contains 3 VMs.

LK3> No, that would be a really bad idea IMO.  The fact that two VMs happen to be running on the same server hardware is an ephemeral thing.  From one moment to the next a VM may migrate to a different server where it becomes separated from the other VMs.  Furthermore, I see no advantage in defining a Tenant System to be a grouping of VMs from a networking perspective since the thing that connects multiple VMs together is the network.

[Qin]: Good point, I fully agree with you. Here is the update to http://www.ietf.org/internet-drafts/draft-wu-nvo3-nve2nve-03.txt, (NVO3 control plane requirement mostly for NVE to NV Authority/Controller interface) which clears terminology confusing based on our discussion.

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: "Reith, Lothar" <[email-masked]>
Date: Fri, 10 May 2013 02:05:30 +0000

Hi Pat,

could you clarify what „limiting factor“ means?  Limiting factor for a delta or limiting factor for a total?

If I understand IEEE 802.1Qbg correctly (and transfer it correctly to this discussion), than the TLV describes the delta of additional MAC/VID pairs (and their associated L3-IP address)  that are about to appear in near future (pre-associate) or are now appearing (associate) or are now disappearing (dis-associate) at some NVE (which is refered to in 802.1qbg  as adjacent bridge).

I therefore maintain my view, that there maybe use cases, where all virtual stations (what you call MAC/VID pairs)  belonging to a subnet migrate one by one or groupwise with maximum group size 61  from NVE A to NVE  B (say because of planned power shutdown of the virtualization host  connected at  NVE  A).

A TLV information field length may just limit the amount of stations that can be moved within one atomic operation, i.e using a single Ethernet Frame for signaling said announcement.

But as always – I may be wrong.

Lothar

PS: Thanks for your response regarding clarification of MSAP location.

Von: Pat Thaler [mailto:[email-masked]]
Gesendet: Dienstag, 7. Mai 2013 20:18
An: Florian Mahr; [email-masked]; Reith, Lothar; [email-masked]
Cc: [email-masked]
Betreff: RE: [nvo3] Antw: Re: Antw: Re: Antw: What's a TSI? It depends...

Pat

Thanks, Florian

Hi Pat,

Thanks, Larry

Regards,
Pat

Lothar,

- Larry

Larry,

Lothar

Hi Lothar,

Thanks, Larry

David,

Lothar

Thanks,
--David

Hi David,

Thanks,

Florian
Lothar,

Ok ...

[... snip ...]

Thanks,
--David

David,

Lothar

Restating that proposal:

Does that make sense?

Thanks,
--David

Hi David,

Thanks,

Florian

Thanks,
--David

+1,

Regards!
-Qin

Hi Anoop,

Thanks, Larry

Larry,

Anoop

         VNIC-+-VN-+-MAC-+-IP
           ^  |    |     +-IP ...
           |  |    |
           |  |    +-MAC-+-IP
           |  |          +-IP ...
           |  |
           |  +-VN-+-MAC-+-IP
           |     ^ |     +-IP ...
           |     | |
           |     | +-MAC-+-IP
           |     |    ^  +-IP ...
           |     |    |     ^
           |     |    |     |
           A     B    C     D

 - Larry

Hi, Larry:

                +---------------------+
                |                     |
                |  VM (Tenant System) |
                |                     |
                +-+--------+--------+-+
                  |        |        |
                  |        |        |
                  |        |        |
                  +        + ...... +

               +---------------------+
               |                     |
               |  VM (Tenant Systems)|
               |                     |
               +-+--------+--------+-+
                 |        |        |
                 |        |        |
                 |        |        |
                 |                 |

Regards!
-Qin

Hi Anoop,

 - Larry

Anoop

Thanks,
--David

Lothar

“
”

[Qin]: Yes.

LK3> Yes