Subject: Re: [Nmlrg] Using Machine Learning for Network Device Initial Configurations-//RE: Machine Learning in network - solicitation for use cases

From: "Liubing (Leo)" <[email-masked]>
Date: Tue, 1 Sep 2015 11:46:38 +0000

Hi Sheng,

> >We're working on a use case of applying Machine Learning technologies
> >to network device initial configuration.
> >The basic scenario is to learn knowledge/patterns of how to configure a
> >device from historical data (device configuration data of a number of
> >networks ) and then apply them to the new networks.
> 
> Hi, Bing,
> 
> Thanks for sharing. It would be helpful if you could introduce a little bit more
> on how to leverage the machine learning in your use case, such as the
> learning objectives, what a result of the learning, how it apply to a network,
> etc. 

[Bing] The program processes the database which is a collection of a number of existing/historical network information/configurations as the training data, to learn the correlation pattern between a specific parameter configuration and the network information. The network information includes network planning manifest, topology (mostly local partial topology around the device to configure), and every device's information including device manufacture, type, capacity etc.

The learning results are be kind of rules or functions that used for predicting configurations for a given device in a network to be configured. 
For example, the result could be a classification model. Because many of the parameter values could be considered as tags, even for some numeric value, it could also be divided into several value segments that are identical as tags.
When applying it to the new network, just input the device and network information, then we can get the output classification result ( the parameter value). 
Comparing to current device configuration approach, the input is towards minimum requirements for human operations. And it is very flexible, because the configurations are generated dynamically based on the input. 

There are various ways to perform machine learning in this use cases. The design space relies on the data pattern and algorithms chosen.

> It would also useful to discuss the precondition of your use case and the
> constrain of the machine learning mechanism, either in general way or
> specific in your use case.

[Bing] The most important precondition is the training data. The training data should contain a decent number of network samples. The minimum number depends on the quality of the data. And each network sample should contain complete topology information, network planning manifest and every device's information and complete configuration lines.

For constrains, in general there are two:
1) As you see, this is actually a Data-Mining process, which strongly consumes computing resources and time, especially when the training data is huge.
2) The results are only prediction, they are NOT guaranteed to be valid, and currently we lack a mechanism to evaluate them instantly.

B.R.
Bing

> Best regards,
> 
> Sheng
> 
> >This is to leverage the automation on network initial configuration and
> >only require a minimal set of input such as high-level network
> >planning, which includes architecture design, address pool/block
> >assignment, protocol selection etc.
> >
> >The advantages of using Machine Learning on network initial
> >configuration could possibly be:
> >- Saving human cost
> >- Flexibility
> >  - dynamically generating configurations on site
> >  - adaptive to different types of networks
> >- The ability to continuously optimize the parameters, if the program
> >also learning the running performance
> >- etc.
> >
> >Comments are welcomed.
> >
> >Best regards,
> >Bing
> >
> >
> >> -----Original Message-----
> >> From: nmlrg [mailto:[email-masked]] On Behalf Of Sheng Jiang
> >> Sent: Monday, August 31, 2015 11:16 AM
> >> To: [email-masked]
> >> Subject: [Nmlrg] Machine Learning in network - solicitation for use
> >> cases
> >>
> >> Hi, all,
> >>
> >> Thanks for subscribe to NMLRG (Network Machine Learning) mail list.
> >> As we know, there are already many ongoing researches for Machine
> >> Learning in network, in many areas. But up to now, there are few
> >> matured applications yet. So it is the time for a Research Group to
> >> work on this future-oriented technology.
> >>
> >> The first step would be to collect possible use cases: where the
> >> machine learning mechanism could be used in networks. The use cases
> >> does not need to be mature, but should have potential.
> >>
> >> Note that this topic is rapidly moving from academic research into
> >> practical application. Therefore, use cases from university
> >> environments, industrial research and development organizations are all
> welcome.
> >>
> >> Best regards,
> >>
> >> Sheng
> >> _______________________________________________
> >> nmlrg mailing list
> >> [email-masked]
> >> https://www.irtf.org/mailman/listinfo/nmlrg
> >
> >_______________________________________________
> >nmlrg mailing list
> >[email-masked]
> >https://www.irtf.org/mailman/listinfo/nmlrg

---

From: Sheng Jiang <[email-masked]>
Date: Wed, 2 Sep 2015 05:48:04 +0000

>[Bing] The most important precondition is the training data. The training data
>should contain a decent number of network samples. The minimum number
>depends on the quality of the data. 

What does you mean the "quality" of the data here? Is there any general standards to qualify or measure the "quality" of the data?

Regards,

Sheng

>And each network sample should contain
>complete topology information, network planning manifest and every
>device's information and complete configuration lines.
>
>1) As you see, this is actually a Data-Mining process, which strongly consumes
>computing resources and time, especially when the training data is huge.
>2) The results are only prediction, they are NOT guaranteed to be valid, and
>currently we lack a mechanism to evaluate them instantly.
>
>B.R.
>Bing
>
>
>> Best regards,
>>
>> Sheng
>>

---

From: "Liubing (Leo)" <[email-masked]>
Date: Wed, 2 Sep 2015 07:55:52 +0000

> >[Bing] The most important precondition is the training data. The
> >training data should contain a decent number of network samples. The
> >minimum number depends on the quality of the data.
> 
> What does you mean the "quality" of the data here? Is there any general
> standards to qualify or measure the "quality" of the data?

[Bing] There are some dimensions (afaik) to judge the quality of the data:

A. The data should contain the finable knowledge. 
Incomplete or chaotic data is useless for the program. (This is obvious.)

B. The content of the training data should be consistent with the target scenarios. 
One specific database cannot cover all scenarios. So the quality is highly target scenario related. 

C. The data should represent the most usual/typical cases rather than corner cases. 
However, this is not saying the corner case data is NOT needed. It is valuable, but it should NOT be the main objects of the database.

D. Diversity of the data.
For example, for a specific type of network (e.g. DC networks, IPRAN, fixed access, core etc.) , if the data covers different scalability, different designs of that specific type, it is good quality.

E. Additionally, it might be better that the data are from real operational networks.

If the data fits all of the dimensions, maybe a small number of samples could be sufficient for the program to learn the expected patterns.

B.R.
Bing

> >1) As you see, this is actually a Data-Mining process, which strongly
> >consumes computing resources and time, especially when the training data
> is huge.
> >2) The results are only prediction, they are NOT guaranteed to be
> >valid, and currently we lack a mechanism to evaluate them instantly.
> >
> >B.R.
> >Bing
> >
> >
> >> Best regards,
> >>
> >> Sheng
> >>
> >> >This is to leverage the automation on network initial configuration
> >> >and only require a minimal set of input such as high-level network
> >> >- The ability to continuously optimize the parameters, if the
> >> >program also learning the running performance
> >> >- etc.
> >> >
> >> >Comments are welcomed.
> >> >
> >> >Best regards,
> >> >Bing
> >> >
> >> >
> >> >> Subject: [Nmlrg] Machine Learning in network - solicitation for
> >> >> use cases
> >> >>
> >> >> Hi, all,
> >> >>
> >> >> matured applications yet. So it is the time for a Research Group
> >> >> to work on this future-oriented technology.
> >> >>
> >> >> machine learning mechanism could be used in networks. The use
> >> >> cases does not need to be mature, but should have potential.
> >> >>

---

From: Jérôme François <[email-masked]>
Date: Wed, 16 Sep 2015 18:17:37 +0200

I have experienced using ML for device fingerprinting meaning that by
observing traffic pattern (message sequence and timing information) it
is possible to automatically retrieve the precise types of device (name,
version, series).
It is particularily interesting to make network inventory as most of
cases there are some unknwon devices on the network (user or old ones)
and finally potentially identifying vulnerable devices from a security
point of view.

jerome

Le 31/08/2015 05:15, Sheng Jiang a écrit :
> Hi, all,
>
> Thanks for subscribe to NMLRG (Network Machine Learning) mail list. As we know, there are already many ongoing researches for Machine Learning in network, in many areas. But up to now, there are few matured applications yet. So it is the time for a Research Group to work on this future-oriented technology.
>
> The first step would be to collect possible use cases: where the machine learning mechanism could be used in networks. The use cases does not need to be mature, but should have potential.
>
> Note that this topic is rapidly moving from academic research into practical application. Therefore, use cases from university environments, industrial research and development organizations are all welcome.
>
> Best regards,
>
> Sheng

---

From: Sheng Jiang <[email-masked]>
Date: Thu, 17 Sep 2015 06:36:50 +0000

Hi, Jerome,

Thanks for sharing. It would be helpful if you could introduce a little bit more on how to leverage the machine learning in your use case, such as the learning objects and objectives, etc. It would also useful to discuss the working principle in your use case.

Many thanks and best regards,

Sheng

>From: nmlrg [mailto:[email-masked]] On Behalf Of Jér?me Fran?ois
>Sent: Thursday, September 17, 2015 12:18 AM
>To: [email-masked]
>Subject: Re: [Nmlrg] Machine Learning in network - solicitation for use cases
>
>> Thanks for subscribe to NMLRG (Network Machine Learning) mail list. As we
>know, there are already many ongoing researches for Machine Learning in
>network, in many areas. But up to now, there are few matured applications
>yet. So it is the time for a Research Group to work on this future-oriented
>technology.
>>
>> The first step would be to collect possible use cases: where the machine
>learning mechanism could be used in networks. The use cases does not need
>to be mature, but should have potential.
>>
>> Note that this topic is rapidly moving from academic research into practical
>application. Therefore, use cases from university environments, industrial
>research and development organizations are all welcome.
>>
>> Best regards,
>>
>> Sheng
>

---

From: Sebastian Abt <[email-masked]>
Date: Thu, 17 Sep 2015 21:46:54 +0200

> Am 16.09.2015 um 18:17 schrieb Jérôme François <[email-masked]>:
> 

Just to add to this fingerprinting: we also did some research on fingerprinting/recognising individuals in network traffic, which worked very well.  I also know that there’s work on ML-based web browser fingerprinting. 

In general, I find fingerprinting an interesting topic from both, a privacy-protection as well as a forensics (can we find evidence that a suspect was using a specific computer network?) perspective.

sebastian

---

From: Jérôme François <[email-masked]>
Date: Thu, 17 Sep 2015 22:53:36 +0200

> Thanks for sharing. It would be helpful if you could introduce a little bit more on how to leverage the machine learning in your use case, such as the learning objects and objectives, etc. It would…
Assuming a protocol, the goal of this work was to automatically
identifiy every device type using this protocol. By identification, I
mean the name and version for software and the manufacturer and series
for hardware.

As usual, the feature definition was the core of the problem.:
- case 1 relies on the syntactic trees extracted from messages using the
protocol grammar
- case 2 relies on a temporal  behavioral tree which represents observed
sequence of message with inter-arrival time information

We then applied multi-class support vector machine with a learning
stage. Hence, specific kernel functions had to been defined. We also
tested other approaches like unsupervised graph-based clustering with
tree-based based on isomorphic trees. In our work, we specifically
targeted SIP and so applied the identification for one or a set of a
single session. Then, different use cases can benefit from this
approach: network inventory, detection of vulnerable/forbidden devices
or automatically generate rules for stateful firewall (in that case, the
classes which are learnt are not the device types but the anomalies). Of
course, in a perfect world, device type can be easily identified by some
configuration information or, for example, a user agent header can be
included in message. However, this can be easily faked.

Feature engineering was so very important. Messages were preprocessed to
create these trees and metrics (distances, kernels) had to be defined. I
have some other experiences with flow-based anomaly detection or
phishing detection and I observed that fine-tuning the feature is a key
step (more than ML algorithm parameter tuning).

jerome