Subject: Re: [Nmlrg] Machine Learning in network - solicitation for use cases

From: "Dacheng Zhang" <[email-masked]>
Date: Sun, 06 Sep 2015 11:46:19 +0800

>Some more detailed introduction.
>
>DDoS and APT are very active research topics. Application layer DDoS
>attacks are more difficult to detect than layer 4 DDoS attacks. In many
>cases, the application layer DDoS does not introduce large amount
>traffics.However, by using big data and data mining tech, it is possible
>to find out the clues of such attacks.
>
>There were some related discussions in Dots. If you are interested, I
>could find them out later.
>
>
>>>>>
>>>>>>-----Original Message-----
>>>>>>From: Dacheng Zhang [mailto:[email-masked]]
>>>>>>Sent: Monday, August 31, 2015 3:39 PM
>>>>>>To: Sheng Jiang; [email-masked]
>>>>>>Subject: Re: [Nmlrg] Machine Learning in network - solicitation for
>>>>>>use
>>>>>>cases
>>>>>>
>>>>>>I think ML on security would be a good use case. For instance, Huawei
>>>>>>is
>>>>>>proposing an approaching using big data for more accurate DDoS
>>>>>>detection.
>>>>>>
>>>>>>Cheers
>>>>>>
>>>>>>Dacheng
>>>>>>
>>>>>>
>>>>>>
>>>>>>在 15-8-31 上午11:15， "nmlrg on behalf of Sheng Jiang"
>>>>>><[email-masked] on behalf of [email-masked]> 写入:
>>>>>>
>>>>>>>Hi, all,
>>>>>>>
>>>>>>>Thanks for subscribe to NMLRG (Network Machine Learning) mail list.
>>>>>>>As
>>>>>we
>>>>>>>know, there are already many ongoing researches for Machine Learning
>>>in
>>>>>>>network, in many areas. But up to now, there are few matured
>>>>>>>applications
>>>>>>>yet. So it is the time for a Research Group to work on this
>>>>>>>future-oriented technology.
>>>>>>>
>>>>>>>The first step would be to collect possible use cases: where the
>>>>>>>machine
>>>>>>>learning mechanism could be used in networks. The use cases does not
>>>>>need
>>>>>>>to be mature, but should have potential.
>>>>>>>
>>>>>>>Note that this topic is rapidly moving from academic research into
>>>>>>>practical application. Therefore, use cases from university
>>>>>>>environments,
>>>>>>>industrial research and development organizations are all welcome.
>>>>>>>
>>>>>>>Best regards,
>>>>>>>
>>>>>>>Sheng
>>>>>>>_______________________________________________
>>>>>>>nmlrg mailing list
>>>>>>>[email-masked]
>>>>>>>https://www.irtf.org/mailman/listinfo/nmlrg
>>>>>>
>>>>
>>>
>>
>

---

From: Sheng Jiang <[email-masked]>
Date: Sun, 6 Sep 2015 09:01:44 +0000

>traffics. However, by using big data and data mining tech, it is possible

Hi, Dacheng,

Applying machine learning in DDoS protection is an interest use case. For my understanding, the machine would learn the potential attack behaviors, am I right?

If yes, I have two questions: a) does the machine learning has the possibility to learn/identify new attack behaviors, which was not recognized before? If yes, what is the working principles? b) is it possible for autonomic reaction from the network operational perspective after detect such DDoS attack? Give the machine learning may not be accurate, my guess is human intervention is needed.

Best regards,

Sheng

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 7 Sep 2015 07:52:39 +1200

> If yes, I have two questions: a) does the machine learning has the possibility to learn/identify new attack behaviors, which was not recognized before? If yes, what is the working principles? b) is…

I suppose this is in some ways similar to spam processing in email. For example,
a Bayesian spam filter is surprisingly good, but sometimes makes mistakes, so human
training to correct false positives and false negatives is essential. How can
human training be achieved for a real-time case like DDOS?

   Brian

---

From: "Dacheng Zhang" <[email-masked]>
Date: Mon, 07 Sep 2015 09:46:18 +0800

在 15-9-6 下午5:01， "Sheng Jiang" <[email-masked]> 写入:

>Applying machine learning in DDoS protection is an interest use case. For
>my understanding, the machine would learn the potential attack behaviors,
>am I right?
>
>If yes, I have two questions: a) does the machine learning has the
>possibility to learn/identify new attack behaviors, which was not
>recognized before? If yes, what is the working principles? b) is it
>possible for autonomic reaction from the network operational perspective
>after detect such DDoS attack? Give the machine learning may not be
>accurate, my guess is human intervention is needed.
>
>Best regards,
>
>Sheng
>

---

From: "Dacheng Zhang" <[email-masked]>
Date: Mon, 07 Sep 2015 09:52:38 +0800

>my understanding, the machine would learn the potential attack behaviors,

Yes， you are right.
>recognized before? If yes, what is the working principles?

Normally we need to generate a normal behavior model and some “abnormal
behavior models”, the machine will detect whether certain behavior of a
client will be located in an ‘abnormal’ area.
I need to check with my colleagues to see whether we could disclose more
detailed information for the moment.

> b) is it possible for autonomic reaction from the network operational
>perspective after detect such DDoS attack? Give the machine learning may
>not be accurate, my guess is human intervention is needed.

In the current practice, machine learning procedure is normally offline.
1) machine learning may not very that accurate. 2) big data processing
needs time and computing resources.  Human involvement is required.

Looking for future discussion on this topic.

Cheers

Dacheng

---

From: Sheng Jiang <[email-masked]>
Date: Mon, 7 Sep 2015 02:26:10 +0000

>> Applying machine learning in DDoS protection is an interest use case. For my
>understanding, the machine would learn the potential attack behaviors, am I
>right?
>>
>> If yes, I have two questions: a) does the machine learning has the possibility
>to learn/identify new attack behaviors, which was not recognized before? If
>yes, what is the working principles? b) is it possible for autonomic reaction
>from the network operational perspective after detect such DDoS attack?
>Give the machine learning may not be accurate, my guess is human
>intervention is needed.
>
>a Bayesian spam filter is surprisingly good, but sometimes makes mistakes, so
>human training to correct false positives and false negatives is essential.

Hi, Brian,

I believe there is feedback-style training in spam processing. But what do you mean by "human" training? Do you mean the feedback is decided and feed by human administrators? I believe this could be done by machine learning mechanisms.

Actually, spam filtering was one of the earliest network-relevant area that starts to use machine learning. It would worth to study the machine learning applications in spam filtering. Or we could invite some expert in this area to join nmlrg discussion.

Best regards,

Sheng

>How can

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 7 Sep 2015 15:52:13 +1200

> I believe there is feedback-style training in spam processing. But what do you mean by "human" training? Do you mean the feedback is decided and feed by human administrators? I believe this could b…

In the spam case it is definitely a real live human who must detect an error
by the classifier. In my case I estimate that maybe 0.5% of the messages
in my Gmail spam folder are not spam, and maybe 1% in my Gmail Inbox are actually
spam.

I think that for real-time DDOS protection, the solution has to include real-time
input from an operator for both cases: "This is not an attack" and "You missed
an attack". I think that means that the machine-learning system will always
run in training mode, even if training is only needed in 1% of cases.

> Actually, spam filtering was one of the earliest network-relevant area that starts to use machine learning. It would worth to study the machine learning applications in spam filtering. Or we could…

Agreed
  Brian

---

From: Sheng Jiang <[email-masked]>
Date: Tue, 8 Sep 2015 03:11:59 +0000

>my
>possibility
>> I believe there is feedback-style training in spam processing. But what do
>you mean by "human" training? Do you mean the feedback is decided and
>feed by human administrators? I believe this could be done by machine
>learning mechanisms.
>

The automatic in DDoS scenarios may be more critic than the SPAM scenarios. In the worst case, the time of waiting human input/feedback may already be enough for attackers to make significant damage. However, this may not be directly relative to machine learning. The programs that react according to machine learning results may be if-else style. It may also potentially be another machine learning mechanism.

Sheng

>> Actually, spam filtering was one of the earliest network-relevant area that
>starts to use machine learning. It would worth to study the machine learning
>applications in spam filtering. Or we could invite some expert in this area to
>join nmlrg discussion.
>
>Agreed
>  Brian
>
>>
>> Best regards,
>>
>> Sheng
>>
>>> How can

---

From: Sheng Jiang <[email-masked]>
Date: Tue, 8 Sep 2015 03:39:48 +0000

>detailed information for the moment.

This would be an interesting use case. Looking forward to see more information. For now, I guess we do not need much details in algorithms or specific implementation. It would be interested to learn, in general, the objects of learning, and the expected output of the learning in the DDoS protection use case.

>needs time and computing resources.  Human involvement is required.

What may influence the accuracy of the mechanism learning result? In another word, how to improve the accuracy in machine learning mechanism? This question may not be DDoS protection specific.

Best regards,

Sheng

---

From: "Liubing (Leo)" <[email-masked]>
Date: Tue, 8 Sep 2015 04:00:29 +0000

> >In the spam case it is definitely a real live human who must detect an
> >error by the classifier. In my case I estimate that maybe 0.5% of the
> >messages in my Gmail spam folder are not spam, and maybe 1% in my
> Gmail
> >Inbox are actually spam.
> >
> >I think that for real-time DDOS protection, the solution has to include
> >real-time input from an operator for both cases: "This is not an
> >attack" and "You missed an attack". I think that means that the
> >machine-learning system will always run in training mode, even if
> >training is only needed in 1% of cases.
> 
> The automatic in DDoS scenarios may be more critic than the SPAM
> scenarios. In the worst case, the time of waiting human input/feedback may
> already be enough for attackers to make significant damage. 

[Bing] For SPAM machine learning case, I guess the human feedback is only the source of labeled training data. All the Gmail users (I read there were 900M users) are together training the SPAM filter. So, the filter might not directly response to one single feedback.
If DDoS is the similar approach, then I guess the human feedback won't make the learning program react either? 

But I'm curious about what is the item that could be labeled as "This is not an attack " or " You missed an attack ". E.g., the item is an packet, a stream, or any other kind of N-tuple things.

Best regards,
Bing

> However, this may not be directly relative to machine learning. The programs that react
> according to machine learning results may be if-else style. It may also
> potentially be another machine learning mechanism.
> 
> Sheng
> 
> >starts to use machine learning. It would worth to study the machine
> >learning applications in spam filtering. Or we could invite some expert
> >in this area to join nmlrg discussion.
> >
> >Agreed
> >  Brian
> >
> >>
> >> Best regards,
> >>
> >> Sheng
> >>
> >>> How can
> >>>>> There were some related discussions in Dots. If you are
> >>>>> interested, I could find them out later.
> >>>>>

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 8 Sep 2015 16:38:32 +1200

On 08/09/2015 16:00, Liubing (Leo) wrote:

...
> But I'm curious about what is the item that could be labeled as "This is not an attack " or " You missed an attack ". E.g., the item is an packet, a stream, or any other kind of N-tuple things.

The two cases are rather different.

1. The system signals "attack in progress" to the NOC. The operators have a look
and decide that there is no attack, it is just some unusual traffic.
(Example: you are live-streaming the Olympic Games. Two seconds after the
end of the 100 metres final, there is an enormous burst of traffic.
The machine learning system signals an attack, because it was not trained
on the data set from the previous Olympic Games.)

In this case the NOC operators urgently tell the algorithm it is wrong.
It needs to learn that the signature of a sudden burst just after the
end of an event is less likely to be an attack than a sudden burst
at another time.

2. Someone invents a new kind of DDoS attack, which is therefore not
in the historical training data. The system doesn't identify it.
In this case, the NOC operators tell the algorithm "Attack started
at <time>." This automatically becomes high quality training data
for the algorithm: the signature of the new traffic at that time
is 100% certain to be an attack.

I think the hard part is extracting useful signatures from the
traffic stream in real time; the learning/training part is fairly
standard.

    Brian

---

From: "Liubing (Leo)" <[email-masked]>
Date: Tue, 8 Sep 2015 08:13:56 +0000

Hi Brian,

Thanks for your elaborating explanation.
Please see inline.

> From: Brian E Carpenter [mailto:[email-masked]]
> Sent: Tuesday, September 08, 2015 12:39 PM
> To: Liubing (Leo); Sheng Jiang; Dacheng Zhang; [email-masked]
> Subject: Re: [Nmlrg] Machine Learning in network - solicitation for use cases
> 
> > But I'm curious about what is the item that could be labeled as "This is not
> an attack " or " You missed an attack ". E.g., the item is an packet, a stream,
> or any other kind of N-tuple things.
> 
> 1. The system signals "attack in progress" to the NOC. The operators have a
> look and decide that there is no attack, it is just some unusual traffic.
> The machine learning system signals an attack, because it was not trained on
> the data set from the previous Olympic Games.)
> 
> It needs to learn that the signature of a sudden burst just after the end of an
> event is less likely to be an attack than a sudden burst at another time.
> 
> 2. Someone invents a new kind of DDoS attack, which is therefore not in the
> historical training data. The system doesn't identify it.
> In this case, the NOC operators tell the algorithm "Attack started at <time>."

[Bing] It feels like the learning objects are mostly traffic burst events? When traffic burst happens, the machine judges whether it's a DDoS or not.
Then the training data might be a bunch of traffic burst evens marked as normal or abnormal. And the challenge should be how to pick a set of features out of a burst event for the machine learning program to discover the pattern of normal/abnormal classification.

This is just my hypothetical case, could be all wrong.

> This automatically becomes high quality training data for the algorithm: the
> signature of the new traffic at that time is 100% certain to be an attack.

> I think the hard part is extracting useful signatures from the traffic stream in
> real time; the learning/training part is fairly standard.
[Bing] When you said the signature of "100% certain", my perception is that it is something like the typical virus detection approach, which doing exact match of a particular piece of code that could identify a virus.
If my perception was correct, I think the signatures are some special combinations of the features (as mentioned above) where the classification pattern has a 100% confidence. Then I think it doesn't need to extract the signatures in real time, because the features are all pre-defined. 

However, this is only from my view, I might have totally misunderstood you.

Best regards,
Bing

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 9 Sep 2015 11:05:23 +1200

Bing,

We are at the limits of my knowledge here. However, as I said, I think
that automatically extracting event signatures from a real time
packet stream is the hard part.

I think a typical approach would be to run a variety of statistical
algorithms over a sliding window of incoming packet headers.
This paper talks about one technique (but not used in real time):
http://www.cs.auckland.ac.nz/CDMTCS//researchreports/266eimann.pdf
and the corresponding PhD thesis:
https://researchspace.auckland.ac.nz/bitstream/handle/2292/3427/02whole.pdf

Regards
   Brian

---

From: "Liubing (Leo)" <[email-masked]>
Date: Wed, 9 Sep 2015 02:13:02 +0000

Hi Brian,

Thanks much for the information. 
The papers are very good stuff to get a sense of how modern technologies detect the DDoS event, rather than my arbitrary imagination:)

Best regards,
Bing

> Sent: Wednesday, September 09, 2015 7:05 AM
> We are at the limits of my knowledge here. However, as I said, I think that
> automatically extracting event signatures from a real time packet stream is
> the hard part.
> 
> I think a typical approach would be to run a variety of statistical algorithms
> over a sliding window of incoming packet headers.
> >> use cases
> >> an attack " or " You missed an attack ". E.g., the item is an packet,
> >> a stream, or any other kind of N-tuple things.
> >>
> >> 1. The system signals "attack in progress" to the NOC. The operators
> >> have a look and decide that there is no attack, it is just some unusual
> traffic.
> >> (Example: you are live-streaming the Olympic Games. Two seconds after
> >> the end of the 100 metres final, there is an enormous burst of traffic.
> >> The machine learning system signals an attack, because it was not
> >> trained on the data set from the previous Olympic Games.)
> >>
> >> end of an event is less likely to be an attack than a sudden burst at
> another time.
> >>
> > [Bing] It feels like the learning objects are mostly traffic burst events?
> When traffic burst happens, the machine judges whether it's a DDoS or not.
> > Then the training data might be a bunch of traffic burst evens marked as
> normal or abnormal. And the challenge should be how to pick a set of
> features out of a burst event for the machine learning program to discover
> the pattern of normal/abnormal classification.
> >
> >> This automatically becomes high quality training data for the
> >> algorithm: the signature of the new traffic at that time is 100% certain to
> be an attack.
> >
> >> traffic stream in real time; the learning/training part is fairly standard.
> > [Bing] When you said the signature of "100% certain", my perception is
> that it is something like the typical virus detection approach, which doing
> exact match of a particular piece of code that could identify a virus.
> > If my perception was correct, I think the signatures are some special
> combinations of the features (as mentioned above) where the classification
> pattern has a 100% confidence. Then I think it doesn't need to extract the
> signatures in real time, because the features are all pre-defined.
> >

---

From: Jérôme François <[email-masked]>
Date: Wed, 16 Sep 2015 18:13:16 +0200

Hi ,

> traffic stream in real time; the learning/training part is fairly
Extracting the signatures should be the output of the learning part but
ML algorithms will not necessarily produce standards signatures like
patterns, for example this can consist in a set of classifier functions.

Also, I think most of efforts have been done on the automated detection
in the past making the human relatively not active during the detection,
usually limited to tell yes or no (case 1), more interactive real-time
ML would be helpful for complex attacks like APT to support the human
decisions.

jerome

---

From: Sebastian Abt <[email-masked]>
Date: Thu, 17 Sep 2015 21:01:11 +0200

Hi all.

I’m a bit late in the threat, but I’d like to share some insights/experience I gained when researching ML-based systems and also when deploying them in operational networks...

> Am 07.09.2015 um 03:52 schrieb Dacheng Zhang <[email-masked]>:
> 
> 
> 

This depends on your anomaly detection system.  In general, you can apply two-class or one-class systems (multi-class systems are typically deployed as many two-class comparisons).  So, for the DDoS case you could either:

1. Build a model that learns a normal class (benign traffic) and an attack class (DDoS traffic).  For this, you typically need labelled traffic of both classes which are equally balanced in size.  If one class is more dominant than the other, models tend to generalise towards the more dominant class, just because guessing-probability in that case is higher and, hence, classification error is lower.  Using such a model, an ML-based ADS would tell you to which class a specific input sample belongs.

2. Build a model that learns just the normal class.  In that case your system typically needs to be able to map input samples to a point in an euclidean space.  For anomaly detection, you then typically compare distance between new data points and the center/representative of the previously learned normal class.  If distance is too large, than anomaly is assumed.  For learning the normal class you „simply“ need a clean sample of normal traffic.

From my experience and in my opinion, I would only call the second case a true anomaly detection system as the system would not make an assumption of what might be an attack.  Instead, it simply makes an assumption (i.e., the learnt model) of what normality looks like.  The drawback of such systems is that normality might change, e.g. you deploy a new application or - even more difficult - one of your customers deploys a new application or uses a new protocol. In that case you raise false alarms.

> needs time and computing resources.  Human involvement is required.

There are also online learning/continuous learning systems which update their models according to learning decisions.  This may also be enhanced by operator feedback (i.e., manually accepting or declining a predicted class after inspection).  This is a very interesting approach, but unfortunately has some fallbacks: attackers can trick systems to displace the decision boundary in a way that attacks get unnoticed.

Regards,
sebastian

---

From: Sebastian Abt <[email-masked]>
Date: Thu, 17 Sep 2015 21:17:34 +0200

> Am 08.09.2015 um 05:39 schrieb Sheng Jiang <[email-masked]>:
> 

I think there are many different factors that affect accuracy of a ML system.  Most crucial in my opinion are the following two:

1. You need to find an appropriate description of the class(es) you try to learn.  In ML, this processing of finding/generating an appropriate description is commonly called feature extraction.  For network security this means that you need to find a way to transform your given representation of traffic (e.g., packets, flow records, SNMP counters, …) such that only the bits relevant to describe normality/to distinguish between two classes A and B are reflected and everything else is dismissed, effectively reducing entropy. The resulting feature vectors should have high intra-class and low inter-class similarity - for whatever notion of similarity you choose.

2. Especially for one-class systems that only learn models of normality, it is important to be able to track a change of normality. Otherwise, these systems render themselves useless over time / generate too much false alarms.  As operator, you can only rely on the results if there are no (significant) baseline changes.  However, detecting this is probably not trivial and as far as I know this is not heavily researched by the network security community.  Some years ago, I read a paper that claimed that such baseline confidence checks are successfully employed in voice recognition systems and crucial for those system’s reliability.  Unfortunately, I don’t have this paper at hand.

sebastian

---

From: Sebastian Abt <[email-masked]>
Date: Thu, 17 Sep 2015 21:40:11 +0200

> Am 08.09.2015 um 10:13 schrieb Liubing (Leo) <[email-masked]>:
> 
> Hi Brian,
> 

As you say, the point really is the representation you choose, i.e. the features your system uses.  If you encode traffic as bps and pps, then bursts won’t tell you anything.  Both in the case of a flash crowd and during a volumetric DDoS attack these features would increase.  You can predict a little bit more if you also incorporate bps/pps, i.e. bits-per-packet, as in reality DDoS attacks typically cause a shift there, but this is all pretty coarse.  And what might be a DDoS attack for a customer might not be considered an attack for the carrier - it may even go unnoticed by the carrier as it is too low-volume when compared to its usual backbone traffic levels.   What this example should tell: for reliable attack detection choosing features that are independent of traffic volume is important.  Looking at volume/bursts can only be indicative. 

> If my perception was correct, I think the signatures are some special combinations of the features (as mentioned above) where the classification pattern has a 100% confidence. Then I think it doesn…

In both examples the difficulty is backwards-mapping from features to packets/flows/etc.  Typically, your features are derived from a set of packets packets/flows/etc. collected over a specific period of time.  So, you have to bin the packets/flows/etc. in a sensible way (i.e., not only time) before actually extracting feature vectors in order to be able to draw conclusions on very specific packets/flows - which is especially required for automatic response.

BTW: the same applies to the „high quality training data“ as mentioned by Brian.  While I agree that manually labelled training data is the gold standard, in the case sketched here an operator would only see the prediction your ADS makes on the feature vectors, which may be based on a bunch of packets/flows/… which might contain both, legitimate and illegitimate activity.  So, being able to map labels down to packets or flows will be a time consuming manual process.

sebastian

---

From: Sebastian Abt <[email-masked]>
Date: Thu, 17 Sep 2015 21:41:30 +0200

> Am 17.09.2015 um 21:01 schrieb Sebastian Abt <[email-masked]>:
> 
> I’m a bit late in the threat

.. of course I meant thread.  Too much security-thinking :)

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 18 Sep 2015 08:02:37 +1200

Sebastian,

On 18/09/2015 07:17, Sebastian Abt wrote:
> 1. You need to find an appropriate description of the class(es) you try to learn.  In ML, this processing of finding/generating an appropriate description is commonly called feature extraction.  For network security this means that you need to find a way to transform your given representation of traffic (e.g., packets, flow records, SNMP counters, …) such that only the bits relevant to describe normality/to distinguish between two classes A and B are reflected and everything else is dismissed, effectively reducing entropy. 

For a real-time system we need a transformation of the packet stream that can run
at line speed and produce a fairly small number of parameters per second, right? Would you
see that as meaning a statistical algorithm, e.g. one producing entropy measures?

> The resulting feature vectors should have high intra-class and low inter-class similarity - for whatever notion of similarity
you choose.
> 2. Especially for one-class systems that only learn models of normality, it is important to be able to track a change of normality. Otherwise, these systems render themselves useless over time / ge…

Strangely enough my first published paper a million years ago was on human
factors in real-time speech recognition. It turned out that emotional changes
such as user frustration caused large changes in gross parameters of the speech
signal, as large as the differences between speakers. So I think your point
is very important: how to distinguish between normal anomalies (like a surge
in valid traffic) from abnormal anomalies (like a surge in DOS packets).
Feature extraction that makes this possible sounds like a hard problem.

   Brian

---

From: "Liubing (Leo)" <[email-masked]>
Date: Fri, 18 Sep 2015 03:48:45 +0000

Hi Sebastian,

Thanks for your further explanation. 
Please see inline.

> the
> on
> As you say, the point really is the representation you choose, i.e. the
> features your system uses.  If you encode traffic as bps and pps, then bursts
> won’t tell you anything.  Both in the case of a flash crowd and during a
> volumetric DDoS attack these features would increase.  You can predict a
> little bit more if you also incorporate bps/pps, i.e. bits-per-packet, as in
> reality DDoS attacks typically cause a shift there, but this is all pretty coarse.

[Bing] Indeed. The trick/art is in feature selection.
However, feature selection is basically made by man who understands both the application and the machine learning well. We were always wondering, is there any possibility that machine can select features by itself dynamically according to some general/universal methods.

> And what might be a DDoS attack for a customer might not be considered an
> attack for the carrier - it may even go unnoticed by the carrier as it is too
> low-volume when compared to its usual backbone traffic levels.   What
> this example should tell: for reliable attack detection choosing features that
> are independent of traffic volume is important.  Looking at volume/bursts
> can only be indicative.

[Bing] Yes, it should be one-sided to consider DDoS as "burst" events, especially in a carrier aspect.

> In both examples the difficulty is backwards-mapping from features to
> packets/flows/etc.  Typically, your features are derived from a set of
> packets packets/flows/etc. collected over a specific period of time. 

[Bing] A clarification question: when you said "your features are derived from a set of packets packets/flows/etc", for the "features", you were referring features that selected manually at the system design stage; not dynamically/automatically selected by the machine at the running stage, right?
 
> So, you have to bin the packets/flows/etc. in a sensible way (i.e., not only time)
> before actually extracting feature vectors in order to be able to draw
> conclusions on very specific packets/flows - which is especially required for
> automatic response.

> BTW: the same applies to the „high quality training data“ as mentioned by
> Brian.  While I agree that manually labelled training data is the gold
> standard, in the case sketched here an operator would only see the
> prediction your ADS makes on the feature vectors, which may be based on a
> bunch of packets/flows/… which might contain both, legitimate and
> illegitimate activity.  So, being able to map labels down to packets or flows
> will be a time consuming manual process.

[Bing] If I understand you correctly, you basically meant the DDoS detection needs to be done at a integral view level (e.g. a bunch of flows/packets in a time period); but the action needs to be performed on a partial objects (e.g. some specific flows). This really is a puzzle.

However, if the goal of the machine is just alerting the DDoS event, then human only need to label the packet/flow "bin" you mentioned, and save a large number of examples of the labeled "bin", then it might be much easier for labeling. But of course, such kind of a system that can only "alert" DDoS makes much less sense than a system that can action on specific packets/flows.

Best regards,
Bing

---

From: Jérôme François <[email-masked]>
Date: Fri, 18 Sep 2015 14:50:22 +0200

Hi Bing,

> However, feature selection is basically made by man who understands both the application and the machine learning well. We were always wondering, is there any possibility that machine can select fe…
Automated selection of features could work on very generic features I
guess like byte values, bytelength... but avoiding the expert with
knowledge to design targeted features is still very challenging. But I
think most of the time, there are some desciption of the data format we
are relying which is already a good basis to decompose in
attributes/features. Then, feature selection algortihms can be applied.
jerome

---

From: Sheng Jiang <[email-masked]>
Date: Sat, 19 Sep 2015 02:11:41 +0000

>[Bing] Indeed. The trick/art is in feature selection.

Not only the feature selection, the learning direction or path design are also important. It needs the implementors/designers to apply the specific prior knowledge to indicate/guide the mechanism learning process. The good design with valuable prior knowledge would enhance the efficiency and accuracy of the machine learning application. However, the more prior knowledge applied, the less generality it would be.

Sheng

>However, feature selection is basically made by man who understands both
>the application and the machine learning well. We were always wondering, is
>there any possibility that machine can select features by itself dynamically
>according to some general/universal methods.
>

---

From: Sheng Jiang <[email-masked]>
Date: Sat, 19 Sep 2015 02:34:47 +0000

>Especially for one-class systems that only learn models of normality, it is
>important to be able to track a change of normality. Otherwise, these systems
>render themselves useless over time / generate too much false alarms.  

The feedback-style learning could enhance the efficient and accuracy, I believe. The analysis results, either good or bad, after confirmed by human operators, could be fed back as new data. In these way, the quality of the data would be improved over time. However, this is general for machine learning. Security may be slightly different, and need to be continuously learn the new attacks, which may be invited by the attackers over the time.

Sheng

>As
>operator, you can only rely on the results if there are no (significant) baseline
>changes.  However, detecting this is probably not trivial and as far as I know
>this is not heavily researched by the network security community.  Some
>years ago, I read a paper that claimed that such baseline confidence checks
>are successfully employed in voice recognition systems and crucial for those
>system’s reliability.  Unfortunately, I don’t have this paper at hand.
>
>sebastian

---

From: "Liubing (Leo)" <[email-masked]>
Date: Sat, 19 Sep 2015 05:59:30 +0000

Hi Sheng,

> From: Sheng Jiang
> Sent: Saturday, September 19, 2015 10:12 AM
> To: Liubing (Leo); Sebastian Abt
> Cc: Brian E Carpenter; Dacheng Zhang; [email-masked]
> Subject: RE: [Nmlrg] Machine Learning in network - solicitation for use cases
> 
> Not only the feature selection, the learning direction or path design are also
> important. It needs the implementors/designers to apply the specific prior
> knowledge to indicate/guide the mechanism learning process. The good
> design with valuable prior knowledge would enhance the efficiency and
> accuracy of the machine learning application. However, the more prior
> knowledge applied, the less generality it would be.

[Bing] For " learning direction or path design ", did you mean this: one application could be divided into multiple parts or stages, each part/stage might involve different learning models/algorithms (or maybe the same learning models/algorithm but for different features at each stage). Then the "learning direction or path" is about how to separate the stages and choose what models/algorithms. It indeed needs more human wise involved. And I guess maybe it is more practical in real application?

B.R.
Bing

---

From: Sheng Jiang <[email-masked]>
Date: Tue, 22 Sep 2015 06:04:32 +0000

>From: Liubing (Leo)
>Sent: Saturday, September 19, 2015 2:00 PM
>To: Sheng Jiang; Sebastian Abt
>[Bing] For " learning direction or path design ", did you mean this: one
>application could be divided into multiple parts or stages, each part/stage
>might involve different learning models/algorithms (or maybe the same
>learning models/algorithm but for different features at each stage). Then the
>"learning direction or path" is about how to separate the stages and choose
>what models/algorithms. It indeed needs more human wise involved.

Your understanding is partially what I meant. I actually meant all the designed processes of the analysis mechanism that utilities machine learning mechanism. It includes selecting and designing the intermediate pattern/feature/learning objects, selecting and combining multiple learning models/algorithms together, and maybe combining machine learning mechanism with some tradition if-else programming, or even the interacting between machine learning and human, etc.

Regards,

Sheng

>And I
>guess maybe it is more practical in real application?
>
>B.R.
>Bing
>
>> Sheng

---

From: Sheng Jiang <[email-masked]>
Date: Tue, 22 Sep 2015 06:04:32 +0000

Regards,

Sheng