Subject: [OPSAWG] Comments from niw.com.au operator on draft on

From: Michael MacFaden <[email-masked]>
Date: Wed, 17 Sep 2014 17:45:18 +0000

Hi authors,

Regarding: https://datatracker.ietf.org/doc/draft-ietf-opsawg-vmm-mib/

Ian as allowed me to forward you his comments on the draft.

Cheers,
Mike MacFaden

From: Ian West <[email-masked]>
Subject: RE: Mail regarding draft-ietf-opsawg-vmm-mib
Date: September 16, 2014 at 3:18:08 AM PDT
To: 'Michael MacFaden' <[email-masked]>

Overall I think it looks great.
 
It would be really nice to have IO service times as well, ie something like accumulated milliseconds of wait time for io servicing,
this might be a harder to instrument. A counter64 of cumulative ms (us?) awaiting delivery is a really handy read/write latency metric.
 
The cputime and memory figures look good.
 
I am thinking there would be a storage object for the swap space which would give most of the paging stats if actual byte counts were added ?
 
Links back to the ifindex should also tie into vswitch I think ? My current walks of vmware are nesting vswitch ports nicely so that’s
looking good. Network stats seem good already, its primarily the per vhost cpu usage and io stats that are weak from a real time
perspective, this MIB looks like it should cover it really well.
 
Is this intended to be a vsphere (central management) resource, or a per physical host (or both, which would be great) ?
 
If you were to query this from a vsphere central management host, I am thinking you might want to add to the vmtable an identifier
for the physical host that the vm is currently running on. This would make the table useful either standalone or  across a bunch of
physical hosts.
 
Ill think on it some more, this seems like a great step forward over whats there now already in VMware’s implementation.

The vmStorageEntry statistics while having read operation and write operation counters don't allow for measuring the volume of read and write traffic. It would be useful to include 64 bit counters for read and write octets.

---

From: Keiichi SHIMA <[email-masked]>
Date: Sat, 20 Sep 2014 17:32:51 +0900

Hello,

> this might be a harder to instrument. A counter64 of cumulative ms (us?) awaiting delivery is a really handy read/write latency metric.

Let me clarify the above sentence.  Is the service time, you are talking about, the time between an I/O request arrives to the storage subsystem (in a hypervisor) and the time its response is returned to the calling VM?  And you want to add an entry to keep an accumulated value of those I/O request delay?

> The vmStorageEntry statistics while having read operation and write operation counters don't allow for measuring the volume of read and write traffic. It would be useful to include 64 bit counters…

I don’t have any particular opinion on this topic.  If people in this WG support adding the above entries, then maybe it is possible to add them.

Regards,
---
Keiichi SHIMA (島 慶一)
Research Laboratory, IIJ Innovation Institute, Inc <[email-masked]>
WIDE project <[email-masked]>

2014/09/18 2:45、Michael MacFaden <[email-masked]> のメール：

> _______________________________________________
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg

---

From: "Ian West" <[email-masked]>
Date: Sun, 21 Sep 2014 11:57:22 +0930

Hello,

Thankyou for your clarification and feedback.

Your description of the accumulated milliseconds (or potentially microseconds to allow for things getting a lot faster) is exactly
what I had in mind. The delta of this figure divided by the delta iops, should give an approximate measure of the load the io
subsystem is under. 

Regards,
Ian West.

-----Original Message-----
From: Keiichi SHIMA [mailto:[email-masked]] 
Sent: Saturday, 20 September 2014 18:03
To: [email-masked]
Cc: SHIMA Keiichi; ASAI Hirochika; Jürgen Schönwälder; SEKIYA Yuji; Tina Tsou; Cathy Zhou; ESAKI Hiroshi; Ian West; Michael MacFaden
Subject: Re: [OPSAWG] Comments from niw.com.au operator on draft on

Hello,

Regards,
---
Keiichi SHIMA (島 慶一)

---

From: Keiichi SHIMA <[email-masked]>
Date: Sun, 21 Sep 2014 21:36:55 +0900

Hello Ian,

> what I had in mind. The delta of this figure divided by the delta iops, should give an approximate measure of the load the io

Thank you for the clarification.  I got better understanding of your proposal.  I tend to agree that the above usage may be useful.

I again don’t have any strong support/objection for adding the above new entry to the VMM-MIB.  I’d like to hear if other authors and people in this WG support this.

Regards,
---
Keiichi SHIMA (島 慶一)

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 22 Sep 2014 07:55:22 +0200

Hi,

it would help if people would propose concrete text of what the
description of the counters would be. What is most important is that
we define counters that are implementable in a meaningful way on a
large set of different hypervisors.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "Ian West" <[email-masked]>
Date: Wed, 24 Sep 2014 10:29:06 +0930

My apologies for the delay in response, and thankyou for this feedback.

I will endeavour to put my suggestions into the form you suggest over the weekend and submit
them back to the list.

Regards,
Ian West

-----Original Message-----
From: Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: Monday, 22 September 2014 15:25
To: Keiichi SHIMA
Cc: Ian West; [email-masked]; ASAI Hirochika; SEKIYA Yuji; Tina Tsou; Cathy Zhou; ESAKI Hiroshi; Michael MacFaden
Subject: Re: [OPSAWG] Comments from niw.com.au operator on draft on

Hi,

/js

---

From: Keiichi SHIMA <[email-masked]>
Date: Mon, 29 Sep 2014 16:27:13 +0900

Hello Ian,

Thank you for contributing the text.  I personally think the text is fine.  Let's wait for a while if we receive any comments about the text from other authors and WG people.

I believe this modification doesn't affect the current consensus we have about the draft.  If the text is accepted by the WG, then we'll integrate it to the draft and would like to move forward.

Regards,
---
Keiichi SHIMA (島 慶一)

On 2014/09/29, at 15:33, Ian West <[email-masked]> wrote:

> Attached is a diff against the text form of the draft, I believe I have addressed the changes that
> might be required.
> 
> Also for convenience the actual text of the proposed additional counters is as follows.
> 
> vmStorageReadOctets OBJECT-TYPE
>       SYNTAX       Counter64
>       MAX-ACCESS   read-only
>       STATUS       current
>       DESCRIPTION
>               "The total number of bytes read from this device.
>               Discontinuities in the value of this counter can occur
>               at re-initialization of the hypervisor, and
>               administrative state (vmAdminState) changes of the
>               virtual machine."
>       ::= { vmStorageEntry 15 }
> 
> vmStorageWriteOctets OBJECT-TYPE
>       SYNTAX       Counter64
>       MAX-ACCESS   read-only
>       STATUS       current
>       DESCRIPTION
>               "The total number of bytes written to this device.
>               Discontinuities in the value of this counter can occur
>               at re-initialization of the hypervisor, and
>               administrative state (vmAdminState) changes of the
>               virtual machine."
>       ::= { vmStorageEntry 16 }
> 
> vmStorageReadLatency OBJECT-TYPE
>       SYNTAX       Counter64
>       MAX-ACCESS   read-only
>       STATUS       current
>       DESCRIPTION
>               "The total number of microseconds read requests have been
>               queued for this device.
>               This would typically be implemented by storing the high precision
>               system time stamp of when the request is received from the Virtual
>               Machine with the request, the difference between this initial time
>               stamp and the time at which the requested operation has completed
>               should be converted to microseconds and accumulated.
>               Discontinuities in the value of this counter can occur
>               at re-initialization of the hypervisor, and
>               administrative state (vmAdminState) changes of the
>               virtual machine."
>       ::= { vmStorageEntry 17 }
> 
> vmStorageWriteLatency OBJECT-TYPE
>       SYNTAX       Counter64
>       MAX-ACCESS   read-only
>       STATUS       current
>       DESCRIPTION
>               "The total number of microseconds write requests have been
>               queued for this device.
>               This would typically be implemented by storing the high precision
>               system time stamp of when the request is received from the Virtual
>               Machine with the request, the difference between this initial time
>               stamp and the time at which the requested operation has completed
>               should be converted to microseconds and accumulated.
>               Discontinuities in the value of this counter can occur
>               at re-initialization of the hypervisor, and
>               administrative state (vmAdminState) changes of the
>               virtual machine."
>       ::= { vmStorageEntry 18 }
> 
> Thankyou again for your consideration and feedback.
> 
> Regards,
> Ian West.
> 
> <draft-ietf-opsawg-vmm-mib-01.diff>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 29 Sep 2014 10:07:29 +0200

Hi,

for me, it is most important to finish up. This project is ongoing for
quite some time meanwhile and I think it is time to deliver. MIB
moduels can be extended after publication as Proposed Standard. I am
OK to accept these modifications if we at the same time close the door
for new features and instead wrap up this MIB module.

/js

---

From: Keiichi SHIMA <[email-masked]>
Date: Mon, 29 Sep 2014 20:38:58 +0900

>  I am

I support this approach.

Regards,
---
Keiichi SHIMA (島 慶一)

---

From: "Ian West" <[email-masked]>
Date: Mon, 29 Sep 2014 16:03:48 +0930

Regards,
Ian West.

-----Original Message-----
From: Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: Monday, 22 September 2014 15:25
To: Keiichi SHIMA
Cc: Ian West; [email-masked]; ASAI Hirochika; SEKIYA Yuji; Tina Tsou; Cathy Zhou; ESAKI Hiroshi; Michael MacFaden
Subject: Re: [OPSAWG] Comments from niw.com.au operator on draft on

Hi,

/js

---

From: Hirochika Asai <[email-masked]>
Date: Wed, 1 Oct 2014 13:06:23 +0900

Hi,

Thank you for your contribution, and sorry for my delayed response.

I think they are useful information for hypervisor operators,
so I personally support to add these objects to the MIB definition.

However, I would like to clarify two objects related to access latency,
vmStorageReadLatency and vmStorageWriteLatency, from the
viewpoint of the implementation.

If I understand the common behavior of VM’s disk access correctly,
it consists of the following steps.

1. A VM sets up a virtual AHCI command structure for disk access.
2. The VM issues hypervisor call (and the VM suspends).
3. The hypervisor schedules a virtual disk handling process/thread or something equivalent.
4. The process/thread processes the queued request and does something to physical storage system.
5. The process/thread receives a reply from the physical storage system.
6. The process/thread constructs a virtual AHCI reply message structure for the VM.
7. The hypervisor schedules the VM to be resumed.

According to my understanding of lan’s suggestion about latency is to measure the
time from Step 4 to 6, and I think it could be implemented (although not yet implemented
in Xen or KVM as far as I know).  However, if you mean to measure the latency from
Step 1 to 7, I don’t think it measurable from hypervisors.

Is my understanding correct?

Thank you.
Hirochika

-- 
Hirochika Asai <[email-masked]>, The University of Tokyo