Subject: [nvo3] Shepherd's review of draft-ietf-nvo3-vmm-06

From: "Bocci, Matthew (Nokia - GB)" <[email-masked]>
Date: Wed, 19 Feb 2020 18:00:52 +0000

Authors

Here is my document shepherd’s review. Please treat these comments as you would any other last call comment.

Thanks

Matthew

The document is missing the definite article (‘the’, ‘a’ etc) in numerous places. Please go through the document carefully and correct these.

Please expand the term ‘DC’ on first use.

Section 3, Requirements
VM mobility should not require changing their IP addresses after the
   move.

MB> Who is ‘their’? Do you mean ‘a VM’s’ ?

   There exist "Hot Migration" where transport service continuity is
   maintained, and "Cold Migration" where the transport service needs
   to be restarted, i.e., execution of the tasks   is stopped on the
   "Old" NVE, moved to the "New" NVE and the task is restarted.

MB> What is the requirement here? Can you rephrase this paragraph?

4.1. VM Migration in Layer-2 Network
[...]
Therefore, this scheme is highly desirable for utilization in
     large scale multi-tenant DCs.

MB> Which scheme? Do you mean dynamic VM mobility? Please can you clarify this sentence.

[...]

Such a change enables all NVEs
     to encapsulate the outgoing MAC frames with the current target NVE
     IP address. It may take some time to refresh the ARP/ND cache when
     a VM has moved to a New NVE.  During this period, a tunnel is
     needed for that Old NVE to forward packets destined to the VM
     under the New NVE.

MB> How does the old VM know which tunnel to use?

[...]

Reverse ARP (RARP) which enables the host to discover its IPv4
     address when it boots from a local server [RFC0903], is not used
     by VMs because the VM already knows its IPv4 address. Next, we
     describe a case where RARP is used.

MB> Please clarify. First you say RARP is not used, but then you say it is. Perhaps it is just the was this paragraph is phrased.

4.2. Task Migration in Layer-3 Network

     Layer-2 based DC networks become quickly prohibitive because
     ARP/neighbor caches don't scale.

MB> That is a very strong statement for a BCP. Perhaps you mean to say that ARP/neighbour cache scalability considerations can limit the size of Layer-2 based DC networks?

[...]

Cold task migration, which is a common practice in many data
     centers, involves the following steps:

     - Stop running the task.
     - Package the runtime state of the job.
     - Send the runtime state of the task to the new NVE where the
        task is to run.
     - Instantiate the task's state on the new machine.
     - Start the tasks   continuing it from the point at which it was
        stopped.

     Address migration and connection migration in moving tasks or VMs
     are addressed next.

MB> This last sentence seems redundant. I suggest removing it.

[...]

5. Handling Packets in Flight

     The Old NVE may receive packets from the VM's ongoing
     communications. These packets should not be lost; they should be
     sent to the New NVE to be delivered to the VM.  The steps involved
     in handling packets in flight are as follows:

     Preparation Step:  It takes some time, possibly a few seconds for
     a VM to move from its Old NVE to a New NVE. During this period, a
     tunnel needs to be established so that the Old NVE can forward
     packets to the New NVE. Old NVE gets New NVE address from NVA in
     the request to move the VM. The Old NVE can store the New NVE
     address for the VM with a timer. When the timer expired, the entry
     for the New NVE for the VM can be deleted.

MB> How does the old NVE know this? I assume the NVA pushes this to the old NVE, but it is not at all clear from the text. Please clarify.

[...]

6. Moving Local State of VM

MB> This whole section seems to be out of scope of the DC VPN network. Therefore, I would think it is out of scope of this draft and should be removed.

[...]
There is also a Hot Standby option in addition to the Hot
     Mobility, where there are VMs in both primary and secondary NVEs.

MB> This section title says Hot Mobility, but only talks about Hot Standby.

     They have identical information and can provide services
     simultaneously as in load-share mode of operation.  If the VM in
     the primary NVE fails, there is no need to actively move the VM to
     the secondary NVE because the VM in the secondary NVE already
     contains identical information.  The Hot Standby option is the
     costliest mechanism, and hence this option is utilized only for
     mission-critical applications and services.  In Hot Standby
     option, regarding TCP connections, one option is to start with and
     maintain TCP connections to two different VMs at the same time.
     The least loaded VM responds first and starts providing service
     while the sender (origin) still continues to receive Ack from the
     heavily loaded (secondary) VM and chooses not to use the service
     of the secondary responding VM.  If the situation (loading
     condition of the primary responding VM) changes the secondary VM
     may start providing service to the sender (origin).

[...]

8. VM Operation
     Once a VM moves to a new NVE, the VM's IP address does not change
     and the VM should be able to continue to receive packets to its
     address(es).

MB> How does that work for the hot standby case? Do you swap the old/new VM IP addresses?

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 21 Feb 2020 00:41:05 +0000

Matthew,

Thank you very much for the detailed comments. They are very helpful.
Inserted below are the detailed changes per your comments.

Can you let us know if they are acceptable?

Thank you.

Linda

From: Bocci, Matthew (Nokia - GB) <[email-masked]>
Sent: Wednesday, February 19, 2020 12:01 PM
To: [email-masked]
Cc: NVO3 <[email-masked]>
Subject: Shepherd's review of draft-ietf-nvo3-vmm-06

Authors

Thanks

Matthew

The document is missing the definite article (‘the’, ‘a’ etc) in numerous places. Please go through the document carefully and correct these.
[Linda] thank you. We are going through to add them.

[Linda] added.

Section 3, Requirements

MB> Who is ‘their’? Do you mean ‘a VM’s’ ?
[Linda]. Yes. Changed the sentence to “Virtual machine (VM) mobility should not require changing VMs’ IP addresses after the move.”

[Linda] Add the following sentences to the paragraph:
“Not all DCs support “Hot Migration. DCs that only support Cold Migration should make their customers aware of the potential service interruption during the Cold migration.”

MB> Which scheme? Do you mean dynamic VM mobility? Please can you clarify this sentence.
[Linda] Yes. Revised the sentence to the following:
“Therefore, dynamic VM Mobility is highly desirable for large scale multi-tenant DCs.”

[...]

MB> How does the old VM know which tunnel to use?
[Linda] the procedure is explained in the paragraph right following this sentence:
“In IPv4, the VM immediately after the move should send a gratuitous ARP request message containing its IPv4 and Layer 2 MAC address in its new NVE.  This message's destination address is the broadcast address.  Upon receiving this message, both Old and New NVEs should update the VM's ARP entry in the central directory at the NVA, to update its mappings to record the IPv4 address & MAC address of the moving VM along with the new NVE IPv4 address.  An NVE-to-NVA protocol is used for this purpose [RFC8014].”

[...]

MB> Please clarify. First you say RARP is not used, but then you say it is. Perhaps it is just the was this paragraph is phrased.
[Linda] Changed the sentence to the following:
“Reverse ARP (RARP) which enables the host to discover its IPv4 address when it boots from a local server [RFC0903], is not used by VMs if the VM already knows its IPv4 address (most common scenario). Next, we describe a case where RARP is used.

MB> That is a very strong statement for a BCP. Perhaps you mean to say that ARP/neighbour cache scalability considerations can limit the size of Layer-2 based DC networks?
[Linda] That is correct. Changed to your suggested wording.

[...]

     - Stop running the task.

[Linda] removed. Thanks.

[...]

MB> How does the old NVE know this? I assume the NVA pushes this to the old NVE, but it is not at all clear from the text. Please clarify.
[Linda] “Old NVE gets New NVE address from NVA in the request to move the VM”. Not clear?

[...]

MB> This whole section seems to be out of scope of the DC VPN network. Therefore, I would think it is out of scope of this draft and should be removed.
[Linda] How about we remove the description for the detailed State Transferring. And states that it is out of the scope of this document?
6. Moving Local State of VM
In addition to the VM mobility related signaling (VM Mobility Registration Request/Reply), the VM state needs to be transferred to the New NVE.  The state includes its memory and file system if the VM cannot access the memory and the file system after moving to the New NVE.
The mechanism of transferring VM States and file system is out of the scope of this document.

[...]

MB> This section title says Hot Mobility, but only talks about Hot Standby.
[Linda] Added a new subtitle “Other VM Mobility Options”

[...]

8. VM Operation

MB> How does that work for the hot standby case? Do you swap the old/new VM IP addresses?
[Linda] moved the description of the behaviour on Hot VM Mobility to be under the Section 7 (Handling Hot, Warm and Cold VM Mobility). Changed the section title to “VM Lifecycle Management” (which is out of the scope of the document).

---

From: "Bocci, Matthew (Nokia - GB)" <[email-masked]>
Date: Fri, 21 Feb 2020 11:24:23 +0000

Hi Linda,

Thanks for your responses. I am fine with most of these.

I have just a couple of comments below.

Matthew

From: Linda Dunbar <[email-masked]>
Date: Friday, 21 February 2020 at 00:41
To: "Bocci, Matthew (Nokia - GB)" <[email-masked]>, "[email-masked]" <[email-masked]>
Cc: NVO3 <[email-masked]>
Subject: RE: Shepherd's review of draft-ietf-nvo3-vmm-06

Matthew,

Thank you.

Linda

Authors

Thanks

Matthew

Section 3, Requirements

[...]

[...]

[...]

     - Stop running the task.

[...]

MB2> The directionality of the ‘request to move the NVE’ is not clear to me. I assume what happens is that when some entity decides to move the VM from one server to another then it tells the NVA to update all the NVEs in the DC VPN accordingly.
So the NVA pushes the information about how to reach the VM at the new NVE from the old NVE. I don’t think the NVA actually moves the VM, since the NVA is just managing the network, not the spin up/down/migration of the VMs themselves. Likewise, I don’t think the NVE asks the NVA to move a VM as it is not much more than a tunnel endpoint.  It would be helpful if you can clarify the sequence of steps and the directionality of any messaging here.

[...]

[...]

[...]

8. VM Operation

---

From: "Bocci, Matthew (Nokia - GB)" <[email-masked]>
Date: Fri, 21 Feb 2020 16:44:58 +0000

Hi Linda

Ok let's go forward with that revised wording.

Thanks
Matthew

Get Outlook for Android<https://aka.ms/ghei36>

________________________________
From: Linda Dunbar <[email-masked]>
Sent: Friday, February 21, 2020 4:36:21 PM
To: Bocci, Matthew (Nokia - GB) <[email-masked]>; [email-masked] <[email-masked]>
Cc: NVO3 <[email-masked]>
Subject: RE: Shepherd's review of draft-ietf-nvo3-vmm-06

Matthew,

How about the following change to address your comment?

Current: “Old NVE gets New NVE address from NVA in the request to move the VM.”

Revised: “Old NVE gets New NVE address from its NVA, assuming that the NVA gets the notification when a VM is moved from one NVE to another. It is out of the scope of this document on which entity manages the VM move and how NVA gets notified of the move”.

Thank you.

Linda

From: Bocci, Matthew (Nokia - GB) <[email-masked]>
Sent: Friday, February 21, 2020 5:24 AM
To: Linda Dunbar <[email-masked]>; [email-masked]
Cc: NVO3 <[email-masked]>
Subject: Re: Shepherd's review of draft-ietf-nvo3-vmm-06

Hi Linda,

Matthew

From: Linda Dunbar <[email-masked]<mailto:[email-masked]>>
Date: Friday, 21 February 2020 at 00:41
To: "Bocci, Matthew (Nokia - GB)" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: NVO3 <[email-masked]<mailto:[email-masked]>>
Subject: RE: Shepherd's review of draft-ietf-nvo3-vmm-06

Matthew,

Thank you.

Linda

From: Bocci, Matthew (Nokia - GB) <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, February 19, 2020 12:01 PM
To: [email-masked]<mailto:[email-masked]>
Cc: NVO3 <[email-masked]<mailto:[email-masked]>>
Subject: Shepherd's review of draft-ietf-nvo3-vmm-06

Authors

Thanks

Matthew

[Linda] added.

Section 3, Requirements

   move.

[...]

[...]

     under the New NVE.

[...]

[...]

     - Stop running the task.

        task is to run.

        stopped.

     are addressed next.

[Linda] removed. Thanks.

[...]

[...]

[...]

[...]

8. VM Operation

     address(es).

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 21 Feb 2020 16:36:21 +0000

Matthew,

Thank you.

Linda

Hi Linda,

Matthew

Matthew,

Thank you.

Linda

Authors

Thanks

Matthew

Section 3, Requirements

[...]

[...]

[...]

     - Stop running the task.

[...]

[...]

[...]

[...]

8. VM Operation

---

From: Linda Dunbar <[email-masked]>
Date: Fri, 21 Feb 2020 17:20:26 +0000

Matthew,

Thank you very much for the careful and detailed review. The revised draft has been posted: https://datatracker.ietf.org/doc/draft-ietf-nvo3-vmm/

Hope we can get closure of this draft soon.

Linda

From: Bocci, Matthew (Nokia - GB) <[email-masked]>
Sent: Friday, February 21, 2020 10:45 AM
To: [email-masked]; Linda Dunbar <[email-masked]>
Cc: NVO3 <[email-masked]>
Subject: Re: Shepherd's review of draft-ietf-nvo3-vmm-06

Hi Linda
Ok let's go forward with that revised wording.
Thanks
Matthew
Get Outlook for Android<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fghei36&data=02%7C01%7Clinda.dunbar%40futurewei.com%7C0291e2781a9a44fb4cb708d7b6ed64e9%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637179003030433270&sdata=RS7sZU%2FoXJkM%2B%2BkMooifmIqV6Qs1vr7MOjdIPgKKIrI%3D&reserved=0>

To: Bocci, Matthew (Nokia - GB) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>

Matthew,

Current: "Old NVE gets New NVE address from NVA in the request to move the VM."

Revised: "Old NVE gets New NVE address from its NVA, assuming that the NVA gets the notification when a VM is moved from one NVE to another. It is out of the scope of this document on which entity manages the VM move and how NVA gets notified of the move".

Thank you.

Linda

[Linda] "Old NVE gets New NVE address from NVA in the request to move the VM". Not clear?

MB2> The directionality of the 'request to move the NVE' is not clear to me. I assume what happens is that when some entity decides to move the VM from one server to another then it tells the NVA to update all the NVEs in the DC VPN accordingly.

So the NVA pushes the information about how to reach the VM at the new NVE from the old NVE. I don't think the NVA actually moves the VM, since the NVA is just managing the network, not the spin up/down/migration of the VMs themselves. Likewise, I don't think the NVE asks the NVA to move a VM as it is not much more than a tunnel endpoint.  It would be helpful if you can clarify the sequence of steps and the directionality of any messaging here.

To: Linda Dunbar <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>

Hi Linda,

Matthew

Matthew,

Thank you.

Linda

Authors

Here is my document shepherd's review. Please treat these comments as you would any other last call comment.

Thanks

Matthew

The document is missing the definite article ('the', 'a' etc) in numerous places. Please go through the document carefully and correct these.

Please expand the term 'DC' on first use.

[Linda] added.

Section 3, Requirements

   move.

MB> Who is 'their'? Do you mean 'a VM's' ?

[Linda]. Yes. Changed the sentence to "Virtual machine (VM) mobility should not require changing VMs' IP addresses after the move."

"Not all DCs support "Hot Migration. DCs that only support Cold Migration should make their customers aware of the potential service interruption during the Cold migration."

[...]

"Therefore, dynamic VM Mobility is highly desirable for large scale multi-tenant DCs."

[...]

     under the New NVE.

"In IPv4, the VM immediately after the move should send a gratuitous ARP request message containing its IPv4 and Layer 2 MAC address in its new NVE.  This message's destination address is the broadcast address.  Upon receiving this message, both Old and New NVEs should update the VM's ARP entry in the central directory at the NVA, to update its mappings to record the IPv4 address & MAC address of the moving VM along with the new NVE IPv4 address.  An NVE-to-NVA protocol is used for this purpose [RFC8014]."

[...]

"Reverse ARP (RARP) which enables the host to discover its IPv4 address when it boots from a local server [RFC0903], is not used by VMs if the VM already knows its IPv4 address (most common scenario). Next, we describe a case where RARP is used.

[...]

     - Stop running the task.

        task is to run.

        stopped.

     are addressed next.

[Linda] removed. Thanks.

[...]

[Linda] "Old NVE gets New NVE address from NVA in the request to move the VM". Not clear?

MB2> The directionality of the 'request to move the NVE' is not clear to me. I assume what happens is that when some entity decides to move the VM from one server to another then it tells the NVA to update all the NVEs in the DC VPN accordingly.

So the NVA pushes the information about how to reach the VM at the new NVE from the old NVE. I don't think the NVA actually moves the VM, since the NVA is just managing the network, not the spin up/down/migration of the VMs themselves. Likewise, I don't think the NVE asks the NVA to move a VM as it is not much more than a tunnel endpoint.  It would be helpful if you can clarify the sequence of steps and the directionality of any messaging here.

[...]

[...]

[Linda] Added a new subtitle "Other VM Mobility Options"

[...]

8. VM Operation

     address(es).

[Linda] moved the description of the behaviour on Hot VM Mobility to be under the Section 7 (Handling Hot, Warm and Cold VM Mobility). Changed the section title to "VM Lifecycle Management" (which is out of the scope of the document).