Subject: [Nmlrg] links to each slide of presentations//RE: slides of NMLRG #3, June 27th, Athens

From: Sheng Jiang <[email-masked]>
Date: Tue, 28 Jun 2016 14:03:29 +0000

Oops... The proceeding page for interim meeting seems not be as intelligent as the proceeding pages of IETF meetings. Our proceeding page does not autonomically show the slides. I have sent an email to IETF secretary to ask them to fix it. Meanwhile, in this email, here is the links for each presentations:

Slide Filename Edit Replace Delete 
Chair Slides 
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-0.pdf

Introduction to Network Machine Learning & NMLRG
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-1.pdf

Data Collection and Analysis At High Security Lab
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-2.pdf

Use Cases of Applying Machine Learning Mechanism with Network Traffic 
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-3.pdf

Mobile network state characterization and prediction
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-4.pdf

Learning how to route 
https://www.ietf.org/proceedings/interim-2016-nmlrg-01/slides/slides-interim-2016-nmlrg-01-5.pdf

Regards,

Sheng
________________________________________
From: nmlrg [[email-masked]] on behalf of Sheng Jiang [[email-masked]]
Sent: 28 June 2016 21:03
To: [email-masked]
Subject: [Nmlrg] slides of NMLRG #3, June 27th, Athens

Hi, nmlrg,

All slides that have been presented in our NMLRG #3 meeting, June 27th, 2016, Athens, Greece, with EUCNC2016, have been uploaded. They can be accessed through below link

https://www.ietf.org/proceedings/interim/2016/06/27/nmlrg/proceedings.html

Best regards,

Sheng
_______________________________________________
nmlrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/nmlrg

---

From: David Meyer <[email-masked]>
Date: Tue, 28 Jun 2016 08:30:38 -0700

Sheng,

Thanks for the pointers. One thing I notice is that we don't have much on
what the characteristics of network data might be, and as such, what kinds
of existing learning algorithms might be suitable and where additional
development might be required. For example, is the collected data IID? Is
it time series? Is the underlying distribution stationary? If not, how does
that constrain algorithms we might use or develop? For example, how does a
given algorithm deal with concept drift/internal covariate shift (in the
case of DNNs; see e.g., batch normalization [0]). There are many other such
questions, such as is the data categorical (e.g., ports, IP addresses) or
is it continuous/discrete (e.g., counters). And if the data in question is
categorical, what is the cardinality of the categories (this will inform
how such data can be encoded); in the case of IP addresses we can't really
one-hot encode addresses because their cardinality is too large (2^32 or
2^128); this has implications for how we build classifiers (in particular,
for softmax layers in DNNs of various kinds).

Related to the above is the question of features? What are good features
for networking? Where do they come from? Are they domain specific? Can we
learn features the way a DNNs does in the network space? Can we use
autoencoders to discover such features? Or can we use GANs to train DNNs
for network classification tasks in an unsupervised manner? Are there
other, non-ad-hoc (well founded) methods we can use, or is every use case a
one-off (one would hope not).

We can carry the same kinds of analyzes to the algorithms applied. For
example, while something k-means is an effective way to get a feeling for
how continuous/discrete hangs together, if our data is categorical
statistical clustering approaches such as LDA might provide a more
well-founded approach (of course, as with most Bayesian techniques is the
question of approximate inference arises since in most interesting cases
the integral that we need to solve, namely the marginal probability data
isn't tractable so we need to resort to MCMC or more likely variational
inference). And what about the use of SGD/batch normalization etc with
DNNs, and perhaps more importantly, can we use network data to train DNN
policy networks for reinforcement learning like we saw in deep Q-learning
and AlphaGo?

These comments are all by way of saying that we don't have a solid
theoretical understanding (yet) of how techniques that have been so
successful in other domains (e.g., DNNs for perceptual tasks) generalize to
networking use cases. We will need this understanding if our goal is to
provide accurate, repeatable, and explainable results.

In order to accomplish all of this we need, as I have been saying , not
only a good understanding of how these algorithms work but also
standardized data sets and associated benchmarks so we can tell if we are
making progress (or even if our techniques work). Analogies here include
MNIST and ImageNet and their associated benchmarks, among others. As
mentioned  standardized data sets are key to making progress in the ML for
networking space (otherwise how do you know your technique works and/or
improves on another techniques?). One might assume that these data sets
would need to be labeled (as supervised learning is where most of the
progress is being made these days), but not necessarily; Generative
Adversarial Networks (GANs) have emerged as a new way to train DNNs in an
unsupervised manner (this is moving very rapidly; see e.g.,
https://openai.com/blog/generative-models/).

The summary here is that the "distance" between theory and practice in ML
is effectively zero right now due to the incredible rate of progress in the
field; this means we need  to understand both sides of the theory/practice
coin in order to be effective. None of the slide decks provide much
background on what the proposed algorithms are, how they work, or why they
should be expected to work on network data.

Finally, if you are interested in LDA or other algorithms there are a few
short explanatory pieces I have written for my team on
http://www.1-4-5.net/~dmm/ml (works in progress).

Thanks,

Dave

[0] https://arxiv.org/pdf/1502.03167.pdf

On Tue, Jun 28, 2016 at 7:03 AM, Sheng Jiang <[email-masked]> wrote:

> Oops... The proceeding page for interim meeting seems not be as
> intelligent as the proceeding pages of IETF meetings. Our proceeding page
> does not autonomically show the slides. I have sent an email to IETF
> secretary to ask them to fix it. Meanwhile, in this email, here is the
> links for each presentations:
>
> All slides that have been presented in our NMLRG #3 meeting, June 27th,
> 2016, Athens, Greece, with EUCNC2016, have been uploaded. They can be
> accessed through below link
>

---

From: Sheng Jiang <[email-masked]>
Date: Wed, 29 Jun 2016 08:10:24 +0000

Hi, David,

Thanks so much for your email. I am sharing the same concern you have. Actually, this Monday in our NMLRG #3, Athens, we did discussed this issue regarding to the potential standardized dataset. And we are planning to have an open discussion to regarding to the requirements and validation qualification of the potential common dataset in the coming NMLRG #4, Berlin, IETF96.

However, upto now, I have serveral concerns. Firstly, before we could discuss such dataset, we need to narrow our target scenarios to a few very specific use cases, giving that there are so many various scenarios in the network area and they are so different from each other and require different dataset. Secondly, as we did talk with operators, the real data is the important properties for them. So, it would be almost impossible for them to share them. It leaves two possibilities on how to get dataset: a) generating dataset using simulation; b) try to get some partial dataset from operators that have removed their sensenble data. Both way are problemstic, in the risk that a lot of efforts may waste in wrong areas. Thirdly, the availability of data may very different in different network environments. It depends on the measurement/congnitive functions that have been deployed or implemented.

Although having the above concerns, I  think we should: a) research on certain dataset. It would be very useful to prove the machine learning could perform better or provide more flexibility and adaptibility; b) continue our discuss on various network use cases that could benefit from applying machine learning, in order to serve wider network communities.

Another important point I would like to discuss is that you mentioned "our goal is to provide accurate, repeatable, and explainable results." Ideally, we do want this. It is our traditional way for logic-based programming. However, I am not sure that is a doable target with machine learning. Many of machine learning algorithms are based on statistics theory, which may not be that much accurate. Some of machine learning algorithms are black box or have very complexicated internal computing process. So, the resulf of these algorithms may be hard to be explained. Also the process of machine learning may be hard to be controlled or intervened. Repeatable, yes, we have to have this feature to be widely adopted. However, repeatable may have to come with some level of toleration for the inaccurate.

Best regards,

Sheng

________________________________

From: David Meyer [[email-masked]]
Sent: 28 June 2016 23:30
To: Sheng Jiang
Cc: [email-masked]
Subject: Re: [Nmlrg] links to each slide of presentations//RE: slides of NMLRG #3, June 27th, Athens

Sheng,

Thanks for the pointers. One thing I notice is that we don't have much on what the characteristics of network data might be, and as such, what kinds of existing learning algorithms might be suitable and where additional development might be required. For example, is the collected data IID? Is it time series? Is the underlying distribution stationary? If not, how does that constrain algorithms we might use or develop? For example, how does a given algorithm deal with concept drift/internal covariate shift (in the case of DNNs; see e.g., batch normalization [0]). There are many other such questions, such as is the data categorical (e.g., ports, IP addresses) or is it continuous/discrete (e.g., counters). And if the data in question is categorical, what is the cardinality of the categories (this will inform how such data can be encoded); in the case of IP addresses we can't really one-hot encode addresses because their cardinality is too large (2^32 or 2^128); this has implications for how we build classifiers (in particular, for softmax layers in DNNs of various kinds).

Related to the above is the question of features? What are good features for networking? Where do they come from? Are they domain specific? Can we learn features the way a DNNs does in the network space? Can we use autoencoders to discover such features? Or can we use GANs to train DNNs for network classification tasks in an unsupervised manner? Are there other, non-ad-hoc (well founded) methods we can use, or is every use case a one-off (one would hope not).

We can carry the same kinds of analyzes to the algorithms applied. For example, while something k-means is an effective way to get a feeling for how continuous/discrete hangs together, if our data is categorical statistical clustering approaches such as LDA might provide a more well-founded approach (of course, as with most Bayesian techniques is the question of approximate inference arises since in most interesting cases the integral that we need to solve, namely the marginal probability data isn't tractable so we need to resort to MCMC or more likely variational inference). And what about the use of SGD/batch normalization etc with DNNs, and perhaps more importantly, can we use network data to train DNN policy networks for reinforcement learning like we saw in deep Q-learning and AlphaGo?

These comments are all by way of saying that we don't have a solid theoretical understanding (yet) of how techniques that have been so successful in other domains (e.g., DNNs for perceptual tasks) generalize to networking use cases. We will need this understanding if our goal is to provide accurate, repeatable, and explainable results.

In order to accomplish all of this we need, as I have been saying , not only a good understanding of how these algorithms work but also standardized data sets and associated benchmarks so we can tell if we are making progress (or even if our techniques work). Analogies here include MNIST and ImageNet and their associated benchmarks, among others. As mentioned  standardized data sets are key to making progress in the ML for networking space (otherwise how do you know your technique works and/or improves on another techniques?). One might assume that these data sets would need to be labeled (as supervised learning is where most of the progress is being made these days), but not necessarily; Generative Adversarial Networks (GANs) have emerged as a new way to train DNNs in an unsupervised manner (this is moving very rapidly; see e.g., https://openai.com/blog/generative-models/).

The summary here is that the "distance" between theory and practice in ML is effectively zero right now due to the incredible rate of progress in the field; this means we need  to understand both sides of the theory/practice coin in order to be effective. None of the slide decks provide much background on what the proposed algorithms are, how they work, or why they should be expected to work on network data.

Finally, if you are interested in LDA or other algorithms there are a few short explanatory pieces I have written for my team on http://www.1-4-5.net/~dmm/ml (works in progress).

Thanks,

Dave

On Tue, Jun 28, 2016 at 7:03 AM, Sheng Jiang <[email-masked]<mailto:[email-masked]>> wrote:
Oops... The proceeding page for interim meeting seems not be as intelligent as the proceeding pages of IETF meetings. Our proceeding page does not autonomically show the slides. I have sent an email to IETF secretary to ask them to fix it. Meanwhile, in this email, here is the links for each presentations:

Learning how to route

Regards,

Sheng
________________________________________
From: nmlrg [[email-masked]<mailto:[email-masked]>] on behalf of Sheng Jiang [[email-masked]<mailto:[email-masked]>]
Sent: 28 June 2016 21:03
To: [email-masked]<mailto:[email-masked]>
Subject: [Nmlrg] slides of NMLRG #3, June 27th, Athens

Hi, nmlrg,

Best regards,

Sheng
_______________________________________________
nmlrg mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/nmlrg
_______________________________________________
nmlrg mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/nmlrg

---

From: David Meyer <[email-masked]>
Date: Wed, 29 Jun 2016 08:35:37 -0700

On Wed, Jun 29, 2016 at 1:10 AM, Sheng Jiang <[email-masked]> wrote:

> Hi, David,
>
>
>
> Thanks so much for your email. I am sharing the same concern you have.
> Actually, this Monday in our NMLRG #3, Athens, we did discussed this issue
> regarding to the potential standardized dataset. And we are planning to
> have an open discussion to regarding to the requirements and validation
> qualification of the potential common dataset in the coming NMLRG #4,
> Berlin, IETF96.
>

Looking forward to it.

>
>
> However, upto now, I have serveral concerns. Firstly, before we could
> discuss such dataset, we need to narrow our target scenarios to a few
> very specific use cases, giving that there are so many various scenarios in
> the network area and they are so different from each other and require
> different dataset.
>

I'm hoping that not every network is not a one off. Rather, what we seek is
to learn what are the fundamental properties of networks that can be the
basis of reasoning about networks do. For example, we should be able to
learn something from work done on transfer learning [0] and representation
learning [1]. In the image recognition space there are even startups that
provide pretrained AlexNet, GooLeNet, etc trained on ImageNet (and others)
that you can further train/customize for your particular application. In
any event, if every network requires something totally different then we
haven't yet understood what is fundamental to the structure of our
networks.

> Secondly, as we did talk with operators, the real data is the important
> properties for them. So, it would be almost impossible for them to share
> them. It leaves two possibilities on how to get dataset: a) generating
> dataset using simulation;
>

This likely won't work. For example, if you do this then try to train a DNN
on that data set the NN will learn how the simulator works. This is
unlikely to be the outcome that we'd like (we already know how the
simulator works).

 b) try to get some partial dataset from operators that have removed their
sensenble data. Both way are problemstic, in the risk that a lot of efforts
may waste in wrong areas.

Agree that this is problematic.

Thirdly, the availability of data may very different in different network
environments. It depends on the measurement/congnitive functions that have
been deployed or implemented.

What is a "cognitive function"?

In any event, you have outlined just a few of the problems with creating
"standardized" data sets for our use. I have quoted standardized because
I'm beginning to think its the wrong word (invokes "standards" e.g., IETF,
<X>..., and that is not what is meant here).  Not sure what the right term
is but  "standardized" seems to be the wrong word here.

>
> Although having the above concerns, I  think we should: a) research on
> certain dataset. It would be very useful to prove the machine learning
> could perform better or provide more flexibility and adaptibility;
>

Not sure what you mean here. By "certain dataset" you mean what?

> b) continue our discuss on various network use cases that could benefit
> from applying machine learning, in order to serve wider network communities.
>

Use cases are good, however machine learning isn't something we can just
apply to a use case because we've defined the use case. There are really at
least three parts of this that need to be considered as part of a solution:
(i). use case (as you mention), (ii). algorithm(s) that are going to
provide the regression/classification/ desired in (i), and (iii). data
sets.

(ii). requires understanding of the algorithms that can be applied to
produce the inferences (regressions, ...)  we are interested in. What
algorithms we can use depend on the data we can get (iii).

That said, one of the things we've learned over the past decade or so is
that learning itself is a very general process. We see this in how powerful
something as simple as SGD/backprop is (see e.g., [2]). We can also use
this to our advantage and re-purpose algorithms originally developed for
other tasks. A simple example of this is LDA (see [3,4]). Here the original
idea was to model topics in a corpus of documents, where the topics
themselves (along with the per-document topic proportions) were the
latent/hidden variables. One way to re-purpose LDA for applications like
ours (essentially classification/anomaly detection tasks) is to view each
to/from pair of IP addresses as a "document" (in the language of LDA) and
then words are constructed from features of the to/from traffic (e.g., to
ports, etc). See [5] for a decent example of how you might think about this
kind of "statistical clustering" using the LDA Probabilistic Graphical
Model approach. So there is hope, but there is a lot of work left to be
done on both the theory and practice sides of this (which as I mentioned
earlier are, at least currently, essentially the same thing).

>
>
> Another important point I would like to discuss is that you mentioned "our
> goal is to provide accurate, repeatable, and explainable results." Ideally,
> we do want this. It is our traditional way for logic-based programming.
>
What is "logic-based programming"?  Do you mean logic programming, e.g.,
Prolog? If so I understand logic programming (I did a lot of work on logic
programming back in the late 80s/early 90s, see
http://dl.acm.org/citation.cfm?id=87991). But even so, I don't understand
the connection, so can you be more explicit?

> However, I am not sure that is a doable target with machine learning. Many
> of machine learning algorithms are based on statistics theory, which may
> not be that much accurate.
>
Sorry, I don't understand this. Things like Bayesian inference (e.g., LDA)
are inherently about quantifying uncertainty. In any event can you be more
specific? A learning algorithm can "accurately" approximate something like
a posterior distribution, its just that the output is just that, a
posterior distribution. So there is a difference between "accuracy" (which
isn't really defined here) and the stochastic nature of the results we get
from ML.

OTOH, if you mean accuracy of our estimated distributions then there are
obviously ways this can go wrong. For example, even an algorithm as simple
as k-means can go wrong since the optimization objective (distortion
function) is non-convex, so it is sensitive to initial conditions (in
particular, where you start; if you are unlucky you can get stuck in a
local minimum  due to the non-convex nature of the objective). See e.g.,
[8]. BTW,  k-means was called out in one of the decks below but as I
mentioned, not much discussion of the important topics we are discussing
here there. There is also approximate inference (for example, MCMC or
variational approaches), but these approaches also have
provably quantitative error bounds (http://www.1-4-5.net/~dmm/ml/ps.pdf
talks a bit about how variational approaches, essential trading inference
for optimization where the inference (integral) is intractable, find a
tight lower-bound using Jensen's inequality.

> Some of machine learning algorithms are black box or have very
> complexicated internal computing process. So, the resulf of these
> algorithms may be hard to be explained.
>
This is definitely true to some extent today with DNNs, but that is
changing rapidly. Results can be repeatable but explaining exactly what
units (artificial neurons) far from the input are doing can be difficult.
But this too is an area of active research and development (see [6] from
last year's ICML, or the work I pointed to on GANs in my last note as
progress in this area). Other approaches such probabilistic graphical
models or signal processing approaches are perhaps more easily explained,
but I don't think this is the level of explanation that is going to be
useful to people deploying and operating networks, so we'll have to find
different ways of explaining what things like DNNs to in a way that is
consumable by operators, etc.

All this by way of saying that we are gaining better understanding how DNNs
work every day (literally) and as a result  I'm optimistic that we can
build technology that will be accurate (again, stochastic results),
repeatable and explainable.

I'll just note here that if you have a model with literally billions of
parameters being trained by SGD its not that surprising that humans have a
hard time understanding what the computation is. As a concrete example,
DNNs can solve the so-called "selectivity-invariance" problem [7], but we
humans don't really  know how to do, so this is another example.

> Also the process of machine learning may be hard to be controlled or
> intervened.
>

Again, not sure what you mean here. Please give a bit more detail.

> Repeatable, yes, we have to have this feature to be widely adopted.
> However, repeatable may have to come with some level of toleration for the
> inaccurate.
>

All of this is inherently probabilistic. A DNN, for example, can output the
posterior distribution over some hidden variables of interest accurately.
LDA finds the latent "topics" and word distributions, both of which are
multinomals so again we're essentially getting a posterior over the hidden
variables.  Even k-means is  just the EM (Expectation Maximization)
algorithm applied to a particular Naive Bayes model. The same can be said
of almost every learning algorithm you might mention. So "inaccurate" isn't
the correct word here (unless you mean whatever you trained didn't predict
whatever you wanted to predict, in which case something else is wrong). So
the DNN gives you as stochastic result, which may be different than what
many network people are used to. But its not inaccurate, its just
stochastic.

Thx,

--dmm

[0] http://people.csail.mit.edu/mcollins/papers/CVPR_2008.pdf
[1]  https://arxiv.org/abs/1206.5538
[2] http://deliprao.com/archives/153
[3] http://www.1-4-5.net/~dmm/ml/lda_intro.pdf
[4] http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf
[5] http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=5949386
[6]
http://yosinski.com/media/papers/Yosinski__2015__ICML_DL__Understanding_Neural_Networks_Through_Deep_Visualization__.pdf
[7]
https://cbmm.mit.edu/sites/default/files/publications/Anselmi_Invariance_CBMM_memo_29.pdf
[8] http://cs229.stanford.edu/notes/cs229-notes7a.pdf

> Best regards,
>
>
>
> Sheng
>
>
>
> ------------------------------
>
> *From:* David Meyer [[email-masked]]
> *Sent:* 28 June 2016 23:30http://deliprao.com/archives/153
> *To:* Sheng Jiang
> *Cc:* [email-masked]
> *Subject:* Re: [Nmlrg] links to each slide of presentations//RE: slides
> of NMLRG #3, June 27th, Athens
>
> Sheng,
>

---

From: David Meyer <[email-masked]>
Date: Wed, 29 Jun 2016 11:09:30 -0700

This question is on this use case:

Isn't it the case that you will know if something is registered to the MBB,
MMC, or MCC slice at device authentication time? If not what is the
scenario in which you don't know?

Thanks,

--dmm

---

From: Sheng Jiang <[email-masked]>
Date: Thu, 30 Jun 2016 17:44:37 +0000

Hi, David,

I am not sure your question is for me or for the speaker. Copied this email to the speaker, Panagiotis. If the question is on me, I don't really understand the point of your question. Could you please provide more context or explanation?

Thanks,

Sheng

Sent: 30 June 2016 2:09

Isn't it the case that you will know if something is registered to the MBB, MMC, or MCC slice at device authentication time? If not what is the scenario in which you don't know?

Thanks,

--dmm

Learning how to route

Regards,

Sheng

Hi, nmlrg,

Best regards,

Sheng

---

From: David Meyer <[email-masked]>
Date: Thu, 30 Jun 2016 10:58:32 -0700

Thanks Sheng, I meant it for the authors, but didn't see email addresses in
the deck...

--dmm

On Thu, Jun 30, 2016 at 10:44 AM, Sheng Jiang <[email-masked]> wrote:

> Hi, David,
>
>
>
> I am not sure your question is for me or for the speaker. Copied this
> email to the speaker, Panagiotis. If the question is on me, I don't really
> understand the point of your question. Could you please provide more
> context or explanation?
>
>
>
> Thanks,
>
>
>
> Sheng
>
>
> *Sent:* 30 June 2016 2:09
> *To:* Sheng Jiang
> *Cc:* [email-masked]
> Isn't it the case that you will know if something is registered to the
> MBB, MMC, or MCC slice at device authentication time? If not what is the

---

From: Panagiotis Demestichas <[email-masked]>
Date: Thu, 30 Jun 2016 21:28:38 +0300

Dear all

just saw the stream of emails.

The goal of the work is to learn traffic patterns, and other associated
aspects, e.g., in time and space. The goal is not to instantaneously
identify the class of the stream; such streams are used for fine tuning the
learning.

Voula Vassaki (did the presentation) will complement.

Best regards
Panagiotis

On Thu, Jun 30, 2016 at 8:58 PM, David Meyer <[email-masked]> wrote:

> Thanks Sheng, I meant it for the authors, but didn't see email addresses
> in the deck...
>
> --dmm
>
>

---

From: David Meyer <[email-masked]>
Date: Thu, 30 Jun 2016 11:39:42 -0700

Thanks. Do you folks have a paper or other additional description of your
data and methods?

Thx,

--dmm

---

From: "Liubing (Leo)" <[email-masked]>
Date: Thu, 30 Jun 2016 19:13:09 +0000

Hi David,

Pardon for jump in the thread. A couple of comments inline.

发件人: nmlrg [mailto:[email-masked]] 代表 David Meyer
发送时间: 2016年6月29日 18:36
收件人: Sheng Jiang
抄送: [email-masked]
主题: Re: [Nmlrg] links to each slide of presentations//RE: slides of NMLRG #3, June 27th, Athens

On Wed, Jun 29, 2016 at 1:10 AM, Sheng Jiang <[email-masked]<mailto:[email-masked]>> wrote:

Hi, David,

Looking forward to it.

However, upto now, I have serveral concerns. Firstly, before we could discuss such dataset, we need to narrow our target scenarios to a few very specific use cases, giving that there are so many various scenarios in the network area and they are so different from each other and require different dataset.

I'm hoping that not every network is not a one off. Rather, what we seek is to learn what are the fundamental properties of networks that can be the basis of reasoning about networks do. For example, we should be able to learn something from work done on transfer learning [0] and representation learning [1]. In the image recognition space there are even startups that provide pretrained AlexNet, GooLeNet, etc trained on ImageNet (and others) that you can further train/customize for your particular application. In any event, if every network requires something totally different then we haven't yet understood what is fundamental to the structure of our networks.

[Bing] In technical perspective, I think you’re raising an inspiring idea, and probably the future direction of this research field. OTOH, I also feel this might be too far away, where should we start?

Secondly, as we did talk with operators, the real data is the important properties for them. So, it would be almost impossible for them to share them. It leaves two possibilities on how to get dataset: a) generating dataset using simulation;

This likely won't work. For example, if you do this then try to train a DNN on that data set the NN will learn how the simulator works. This is unlikely to be the outcome that we'd like (we already know how the simulator works).

[Bing] I think ML could do two things: 1) mining some unknown knowledge; 2) do something which is already known, but very hard to be solved by programming rules.
I think the simulation data could fit into 2).

 b) try to get some partial dataset from operators that have removed their sensenble data. Both way are problemstic, in the risk that a lot of efforts may waste in wrong areas.

Thirdly, the availability of data may very different in different network environments. It depends on the measurement/congnitive functions that have been deployed or implemented.

In any event, you have outlined just a few of the problems with creating "standardized" data sets for our use. I have quoted standardized because I'm beginning to think its the wrong word (invokes "standards" e.g., IETF, <X>..., and that is not what is meant here).  Not sure what the right term is but  "standardized" seems to be the wrong word here.

Although having the above concerns, I  think we should: a) research on certain dataset. It would be very useful to prove the machine learning could perform better or provide more flexibility and adaptibility;

b) continue our discuss on various network use cases that could benefit from applying machine learning, in order to serve wider network communities.

Use cases are good, however machine learning isn't something we can just apply to a use case because we've defined the use case. There are really at least three parts of this that need to be considered as part of a solution: (i). use case (as you mention), (ii). algorithm(s) that are going to provide the regression/classification/ desired in (i), and (iii). data sets.

(ii). requires understanding of the algorithms that can be applied to produce the inferences (regressions, ...)  we are interested in. What algorithms we can use depend on the data we can get (iii).

That said, one of the things we've learned over the past decade or so is that learning itself is a very general process. We see this in how powerful something as simple as SGD/backprop is (see e.g., [2]). We can also use this to our advantage and re-purpose algorithms originally developed for other tasks. A simple example of this is LDA (see [3,4]). Here the original idea was to model topics in a corpus of documents, where the topics themselves (along with the per-document topic proportions) were the latent/hidden variables. One way to re-purpose LDA for applications like ours (essentially classification/anomaly detection tasks) is to view each to/from pair of IP addresses as a "document" (in the language of LDA) and then words are constructed from features of the to/from traffic (e.g., to ports, etc). See [5] for a decent example of how you might think about this kind of "statistical clustering" using the LDA Probabilistic Graphical Model approach. So there is hope, but there is a lot of work left to be done on both the theory and practice sides of this (which as I mentioned earlier are, at least currently, essentially the same thing).

[Bing]  I agree the algorithms are basically general to different applications. However, besides algorithms, another important part is the “learning path”. E.g. feature selecting, identifying the target problem as classification/regression/clustering etc., and probably multiple learning stages (e.g. do clustering first, and then do classification). Such kind of “learning path” in my view is case-by-case, and seems very difficult to be generalized for current technologies.

Best regards,
Bing

Another important point I would like to discuss is that you mentioned "our goal is to provide accurate, repeatable, and explainable results." Ideally, we do want this. It is our traditional way for logic-based programming.
What is "logic-based programming"?  Do you mean logic programming, e.g., Prolog? If so I understand logic programming (I did a lot of work on logic programming back in the late 80s/early 90s, see http://dl.acm.org/citation.cfm?id=87991). But even so, I don't understand the connection, so can you be more explicit?

However, I am not sure that is a doable target with machine learning. Many of machine learning algorithms are based on statistics theory, which may not be that much accurate.
Sorry, I don't understand this. Things like Bayesian inference (e.g., LDA) are inherently about quantifying uncertainty. In any event can you be more specific? A learning algorithm can "accurately" approximate something like a posterior distribution, its just that the output is just that, a posterior distribution. So there is a difference between "accuracy" (which isn't really defined here) and the stochastic nature of the results we get from ML.

OTOH, if you mean accuracy of our estimated distributions then there are obviously ways this can go wrong. For example, even an algorithm as simple as k-means can go wrong since the optimization objective (distortion function) is non-convex, so it is sensitive to initial conditions (in particular, where you start; if you are unlucky you can get stuck in a local minimum  due to the non-convex nature of the objective). See e.g., [8]. BTW,  k-means was called out in one of the decks below but as I mentioned, not much discussion of the important topics we are discussing here there. There is also approximate inference (for example, MCMC or variational approaches), but these approaches also have provably quantitative error bounds (http://www.1-4-5.net/~dmm/ml/ps.pdf talks a bit about how variational approaches, essential trading inference for optimization where the inference (integral) is intractable, find a tight lower-bound using Jensen's inequality.

Some of machine learning algorithms are black box or have very complexicated internal computing process. So, the resulf of these algorithms may be hard to be explained.
This is definitely true to some extent today with DNNs, but that is changing rapidly. Results can be repeatable but explaining exactly what units (artificial neurons) far from the input are doing can be difficult. But this too is an area of active research and development (see [6] from last year's ICML, or the work I pointed to on GANs in my last note as progress in this area). Other approaches such probabilistic graphical models or signal processing approaches are perhaps more easily explained, but I don't think this is the level of explanation that is going to be useful to people deploying and operating networks, so we'll have to find different ways of explaining what things like DNNs to in a way that is consumable by operators, etc.

All this by way of saying that we are gaining better understanding how DNNs work every day (literally) and as a result  I'm optimistic that we can build technology that will be accurate (again, stochastic results), repeatable and explainable.

I'll just note here that if you have a model with literally billions of parameters being trained by SGD its not that surprising that humans have a hard time understanding what the computation is. As a concrete example, DNNs can solve the so-called "selectivity-invariance" problem [7], but we humans don't really  know how to do, so this is another example.

Also the process of machine learning may be hard to be controlled or intervened.

Repeatable, yes, we have to have this feature to be widely adopted. However, repeatable may have to come with some level of toleration for the inaccurate.

All of this is inherently probabilistic. A DNN, for example, can output the posterior distribution over some hidden variables of interest accurately. LDA finds the latent "topics" and word distributions, both of which are multinomals so again we're essentially getting a posterior over the hidden variables.  Even k-means is  just the EM (Expectation Maximization) algorithm applied to a particular Naive Bayes model. The same can be said of almost every learning algorithm you might mention. So "inaccurate" isn't the correct word here (unless you mean whatever you trained didn't predict whatever you wanted to predict, in which case something else is wrong). So the DNN gives you as stochastic result, which may be different than what many network people are used to. But its not inaccurate, its just stochastic.

Thx,

--dmm

[0] http://people.csail.mit.edu/mcollins/papers/CVPR_2008.pdf
[1]  https://arxiv.org/abs/1206.5538
[2] http://deliprao.com/archives/153
[3] http://www.1-4-5.net/~dmm/ml/lda_intro.pdf
[4] http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf
[5] http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=5949386
[6] http://yosinski.com/media/papers/Yosinski__2015__ICML_DL__Understanding_Neural_Networks_Through_Deep_Visualization__.pdf
[7] https://cbmm.mit.edu/sites/default/files/publications/Anselmi_Invariance_CBMM_memo_29.pdf
[8] http://cs229.stanford.edu/notes/cs229-notes7a.pdf

Best regards,

Sheng

________________________________
From: David Meyer [[email-masked]<mailto:[email-masked]>]
Sent: 28 June 2016 23:30http://deliprao.com/archives/153
To: Sheng Jiang
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [Nmlrg] links to each slide of presentations//RE: slides of NMLRG #3, June 27th, Athens
Sheng,

Thanks,

Dave

Learning how to route

Regards,

Sheng

Hi, nmlrg,

Best regards,

Sheng

---

From: Sheng Jiang <[email-masked]>
Date: Thu, 30 Jun 2016 20:16:42 +0000

Hi, David,

Please see my replies with <Sheng/> in the lines. Sorry for being slow response due to my traveling.

<Sheng>If I understand you correctly, you meant we should do some generic learning in order to better understand the network fundamentally, and the result should be helpful for solving these specific tasks in the networks. It is different to what we have done - working on applying ML to solve specific network tasks. I would agree this proposed research direction are important and  I am very interesting in such direction although  it looks ambitions and no guarantee for success so far. </Sheng>

<Sheng>Here, we seems reach the agreement on that we do NOT have a good way to obtain the useful dataset. It is not good at all. Does that mean we have to wait for operators to work on themselves alone? Any suggestion on how we could get quality real dataset?</Sheng>
Thirdly, the availability of data may very different in different network environments. It depends on the measurement/congnitive functions that have been deployed or implemented.

<Sheng>Here, I use the "cognitive function" to represent that device or network function that collects the existing status or configuration data, against measurement.</Sheng>

<Sheng>We did discuss a potential standardized dataset, didn't we?</Sheng>

<Sheng>What we have done up to now is, in a specific use case, to design an algorithm or a combination of algorithm on chosen dataset (this may be dynamic data from the network environment) to complete a specific task.</Sheng>

<Sheng>Agree on that learning and learning algorithm is general. But what we did was only use it in specific way to solve specific tasks/use cases. As I agreed earlier, generlized learning on network could be an important direction and I am interesting in. Let's see how far we could go. On another side, I have to emphasis that the general learning must be helpful on solving specific tasks, in other words, specific tasks should be earlier to solve from the result of general learning. Otherwise, the general learning is not mearningful.</Sheng>

<Sheng>Here, "logic-based programming" I referred to all traditional programming that realize human pre-designed logic into computing programs. It is comparing to ML-based programming.</Sheng>

<Sheng>The reason that statistics-based ML algorithms can work is that they chose to focus on only high probability and igrone corner cases. But when mearningful corner cases happen or we want to target the corner cases, machine learning may not accurately work. Again, "accuracy" itself does not properly defined. So, we may talk about very different things. On other side, I also agree that accurate is not important topic here. ML can reach a high-level accurate, which are suffient for majority cases. But ML may never reach full accurate while traditional program can 100% certain the result on giving circumstance.</Sheng>

<Sheng>Sharing the same opinion here. Every we could really explain how DNNs works in a very complicated way. Our customers would not have enough patient to listen. They would still not trust ML. We, actually the whole AI/ML society, needs to find other way to explain and prove things like DNN are trustable. Only then, they can be use for more serious or autonomic tasks.</Sheng>

<Sheng>I hope I can share you optimistic. Actually, I do believe we could do so one day. But I am not sure how long it will take. </Sheng>

<Sheng> Here, I actually refer to DNN too. We don't know how to intervened DNN in its running time.

<Sheng>Inaccurate or stochastic, they are not what many network people can accept. In their eyes, network is definite, every parameter is definite. However, actually, if you could analyze every network parameter, the value may be definite, but actually they all have toleration for some level of inaccurate as long as their design purposes could meet. It is the fundemental that we could use ML to solve network task.

Thanks so much. Regards,

Sheng </Sheng>

Thx,

--dmm

Best regards,

Sheng

Sheng,

Thanks,

Dave

Learning how to route

Regards,

Sheng

Hi, nmlrg,

Best regards,

Sheng

---

From: David Meyer <[email-masked]>
Date: Thu, 30 Jun 2016 13:52:39 -0700

Hey Sheng, see in-line. Thx, --dmm

On Thu, Jun 30, 2016 at 1:16 PM, Sheng Jiang <[email-masked]> wrote:

> I'm hoping that not every network is not a one off. Rather, what we seek
> is to learn what are the fundamental properties of networks that can be the
> haven't yet understood what is fundamental to the structure of our networks.
>
> <Sheng>If I understand you correctly, you meant we should do some generic
> learning in order to better understand the network fundamentally, and the
> result should be helpful for solving these specific tasks in the networks.
> It is different to what we have done - working on applying ML to solve
> specific network tasks. I would agree this proposed research direction are
> important and  I am very interesting in such direction although  it looks
> ambitions and no guarantee for success so far. </Sheng>
>

No, not really. What I meant is if every network is a one-off then by
definition we haven't learned to generalize. Now, generalization might not
be possible, but I don't see any reason why it wouldn't be. It certainly is
for perceptual tasks, so there's one data point (for example, if you want
to do image recognition you use a Conv Net because there is both theory and
practice behind the framework).

> This likely won't work. For example, if you do this then try to train a
> DNN on that data set the NN will learn how the simulator works. This is
>  b) try to get some partial dataset from operators that have removed
> their sensenble data. Both way are problemstic, in the risk that a lot of
> efforts may waste in wrong areas.
>
> <Sheng>Here, we seems reach the agreement on that we do NOT have a good
> way to obtain the useful dataset. It is not good at all. Does that mean we
> have to wait for operators to work on themselves alone? Any suggestion on
> how we could get quality real dataset?</Sheng>
>

This is a huge challenge, and one we should spend time talking about.
Without something like ImageNet (or MNIST), we want to able to measure
progress (or even if our techniques work).

> <Sheng>Here, I use the "cognitive function" to represent that device or
> network function that collects the existing status or configuration data,
> against measurement.</Sheng>
>

So is a network element, say a router, a cognitive function?

> <Sheng>We did discuss a potential standardized dataset, didn't we?</Sheng>

Sorry,  I didn't understand the language.

> <Sheng>What we have done up to now is, in a specific use case, to design
> an algorithm or a combination of algorithm on chosen dataset (this may be
> dynamic data from the network environment) to complete a specific
> task.</Sheng>
>
> <Sheng>Agree on that learning and learning algorithm is general. But what
> we did was only use it in specific way to solve specific tasks/use cases.
>

Understand, but I'm questioning whether this is a legitimate topic for a
research group. Building one-off solutions is obviously possible (look at
all of the startups, products etc in this space), but none of these
approachs give us insight into how to learn from network data, what its
properties are, or how to generalize (really the holy grail).

As I agreed earlier, generlized learning on network could be an important
> direction and I am interesting in. Let's see how far we could go. On
> another side, I have to emphasis that the general learning must be helpful
> on solving specific tasks, in other words, specific tasks should be earlier
> to solve from the result of general learning. Otherwise, the general
> learning is not mearningful.</Sheng>
>
>
>> Another important point I would like to discuss is that you mentioned
>> "our goal is to provide accurate, repeatable, and explainable results."
>> Ideally, we do want this. It is our traditional way for logic-based
>> programming.
>>
> <Sheng>Here, "logic-based programming" I referred to all traditional
> programming that realize human pre-designed logic into computing programs.
> It is comparing to ML-based programming.</Sheng>
>

None of those terms are well defined, but I get your sense.

>
>
>> However, I am not sure that is a doable target with machine learning.
>> Many of machine learning algorithms are based on statistics theory, which
>> may not be that much accurate.
>>
> <Sheng>The reason that statistics-based ML algorithms can work is that
> they chose to focus on only high probability and igrone corner cases.
>

I don't believe this is correct, but perhaps I don't understand your point.
Can you give a concrete example of (a). what you mean by statistics-based
ML algorithm and (b). one that ignores corner cases and how/why?

> But when mearningful corner cases happen or we want to target the corner
> cases, machine learning may not accurately work. Again, "accuracy" itself
> does not properly defined. So, we may talk about very different things. On
> other side, I also agree that accurate is not important topic here. ML can
> reach a high-level accurate, which are suffient for majority cases. But ML
> may never reach full accurate while traditional program can 100% certain
> the result on giving circumstance.</Sheng>
>
>
> <Sheng>Sharing the same opinion here. Every we could really explain how
> DNNs works in a very complicated way. Our customers would not have enough
> patient to listen. They would still not trust ML. We, actually the whole
> AI/ML society, needs to find other way to explain and prove things like DNN
> are trustable. Only then, they can be use for more serious or
> autonomic tasks.</Sheng>
>

I'm optimistic that we will be able to understand what the higher layers in
DNNs are doing; much progress has already been and is being made. That
said, this problem is really more of an issue for consumers of ML
technology, and not so much for those building it (although it would be
useful to better understand the computation a DNN is making; as I mentioned
much progress has been made here). As a loose analogy, how many consumers
of the Linux Kernel really understand what it is doing in any detailed way?
Of course explainability is important and I don't mean to diminish that
importance in any way.

BTW, people are going to deploy ML based systems not because they are
interesting or cool (which they are), but rather because their competitors
are/will. We already see this in the hyper-scales, automotive industry,
process automation industry, and even our industry (check out Chris'
description of ML as "automating your automation"
https://www.youtube.com/watch?v=PZAAOVbiK9A).

> All this by way of saying that we are gaining better understanding how
> DNNs work every day (literally) and as a result  I'm optimistic that we can
> <Sheng>I hope I can share you optimistic. Actually, I do believe we could
> do so one day. But I am not sure how long it will take. </Sheng>
>

Do you mean that once you've trained a DNN you can only use it for
classification (or whatever) and can't further train it, or do you mean
that we can't learn from additional data, or something else?

> All of this is inherently probabilistic. A DNN, for example, can output
> the posterior distribution over some hidden variables of interest
> accurately. LDA finds the latent "topics" and word distributions, both of
> which are multinomals so again we're essentially getting a posterior over
> the hidden variables.  Even k-means is  just the EM (Expectation
> Maximization) algorithm applied to a particular Naive Bayes model. The same
> can be said of almost every learning algorithm you might mention. So
> "inaccurate" isn't the correct word here (unless you mean whatever you
> trained didn't predict whatever you wanted to predict, in which case
> something else is wrong). So the DNN gives you as stochastic result, which
> may be different than what many network people are used to. But its not
> inaccurate, its just stochastic.
>
> <Sheng>Inaccurate or stochastic, they are not what many network people can
> accept. In their eyes, network is definite, every parameter is definite.
> However, actually, if you could analyze every network parameter, the value
> may be definite, but actually they all have toleration for some level of
> inaccurate as long as their design purposes could meet. It is the
> fundemental that we could use ML to solve network task.
>

I didn't understand what you mean here.

Thanks,

--dmm

---

From: Voula Vassaki <[email-masked]>
Date: Thu, 30 Jun 2016 22:53:30 +0300

Dear  David,

as Prof. Demestichas explained the main reason of the classification is 
to be able to learn the characteristics of the traffic patterns in order 
to be able to allocate in an efficient manner the network resources. In 
case you need any further explanation regarding the presentation, do not 
hesitate to contact me.

Best regards,
Voula

>
> ---------- Forwarded message ----------
> From: *Panagiotis Demestichas* <[email-masked] 
> <mailto:[email-masked]>>
> Date: Thu, Jun 30, 2016 at 9:28 PM
> To: David Meyer <[email-masked] <mailto:[email-masked]>>
> Cc: Sheng Jiang <[email-masked] 
> <mailto:[email-masked]>>, "[email-masked] 
> <mailto:[email-masked]>" <[email-masked] 
> <mailto:[email-masked]>>, "[email-masked] <mailto:[email-masked]>" 
> <[email-masked] <mailto:[email-masked]>>
>
>
> Dear all
>
> The goal of the work is to learn traffic patterns, and other 
> associated aspects, e.g., in time and space. The goal is not to 
> instantaneously identify the class of the stream; such streams are 
> used for fine tuning the learning.
>
> On Thu, Jun 30, 2016 at 8:58 PM, David Meyer <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Thanks Sheng, I meant it for the authors, but didn't see email
>     addresses in the deck...
>
>     --dmm
>
>
>     On Thu, Jun 30, 2016 at 10:44 AM, Sheng Jiang
>     <[email-masked] <mailto:[email-masked]>> wrote:
>
>         Hi, David,
>
>         I am not sure your question is for me or for the speaker.
>         Copied this email to the speaker, Panagiotis. If the question
>         is on me, I don't really understand the point of your
>         question. Could you please provide more context or explanation?
>
>         Thanks,
>
>         Sheng
>
>         ------------------------------------------------------------------------
>         *From:* David Meyer [[email-masked] <mailto:[email-masked]>]
>         *Cc:* [email-masked] <mailto:[email-masked]>
>         *Subject:* Re: [Nmlrg] links to each slide of
>         presentations//RE: slides of NMLRG #3, June 27th, Athens
>
>         Isn't it the case that you will know if something is
>         registered to the MBB, MMC, or MCC slice at device
>         authentication time? If not what is the scenario in which you
>         don't know?
>
>         Thanks,
>
>         --dmm
>
>
>
>         On Tue, Jun 28, 2016 at 7:03 AM, Sheng Jiang
>         <[email-masked] <mailto:[email-masked]>> wrote:
>
>             Oops... The proceeding page for interim meeting seems not
>             be as intelligent as the proceeding pages of IETF
>             meetings. Our proceeding page does not autonomically show
>             the slides. I have sent an email to IETF secretary to ask
>             them to fix it. Meanwhile, in this email, here is the
>             From: nmlrg [[email-masked]
>             <mailto:[email-masked]>] on behalf of Sheng Jiang
>             [[email-masked] <mailto:[email-masked]>]
>             Sent: 28 June 2016 21:03
>             To: [email-masked] <mailto:[email-masked]>
>             All slides that have been presented in our NMLRG #3
>             meeting, June 27th, 2016, Athens, Greece, with EUCNC2016,
>             have been uploaded. They can be accessed through below link
>
>             [email-masked] <mailto:[email-masked]>
>             [email-masked] <mailto:[email-masked]>

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus