Subject: [Sipping] Spit Score Semantics

From: "Jan Seedorf" <[email-masked]>
Date: Thu, 22 Nov 2007 12:26:07 +0100

Dear all,

We (Saverio Niccolini and Jan Seedorf) believe there is a need to signal SPIT scores (as calculated with various methods at single signalling hops) along the signalling path to protect against SPIT. In order to make such a signalling of SPIT scores compatible among heterogeneous system, it must be standardised in some way. The draft-wing-sipping-spam-score addresses this issue by proposing additional headers/values for SPIT scores that could be passed along the downstream (and possibly upstream) signalling path. One question is whether it makes sense to have a SPIT score ranging from 0-100. The concern raised by Henning Schulzrinne on the SPITSTOP mailing list was that a scale from 0-100 has no clear semantics defined for values other than 0 ("I am sure this is not SPIT") and 100 ("I am sure this is SPIT"). We thought about how to define semantics for SPIT scores and would like to receive feedback on our thoughts.

First, we believe there are SPIT detection methods resulting in a binary score (0="not SPIT",1="SPIT"). For such methods with a binary result range it is quite easy to come up with well-defined semantics. For instance, for methods like "whitelist" or "blacklist", it could be defined that "0" means "not on list" and "1" means "on list". Similar, for a CAPTCHA-test you can image "0" for passed (i.e., not SPIT according to this test) and "1" for failed (i.e., SPIT according to this test).

Second, we believe it is also possible to come up with semantics for statistical methods (which generally do not produce 0/1 results), resulting in a score between 0 and 100. For anomaly-based statistical methods, the semantics of a score x could be that x percent of previous messages where below/above (depending on the method) the result for this method compared to the current message. For instance, think of a call-rate detection method that checks how many calls are initiated from a particular SIP-URI (or IP-address) within a time-frame. A result of 85 for such a test would mean that the call-rate for the SIP-URI in the From-header of the SIP message in the time-frame is higher than 84% of all calls monitored in the past. Note, as Henning pointed out, this implies nothing directly on the SPIT probability for that message. It is simply a way to precisely define the semantics for the score of anomaly-based statistical modules.

For statistical modules the score could also be based on user-feedback. Then the semantics of score x would be that in the past x percent of messages with the same method-result were marked as SPIT by users (using some user-feedback mechanism). This has probably more relation to a SPIT probability. However, this is not important. The point is that the score is well-defined.

In general, we think that for EACH method it needs to be (publicly) well-defined what the corresponding score means. This score does NOT (necessarily) imply a SPIT-probability. Instead, it has a per-method semantic. Then, for each method it is clearly defined what the signalling of corresponding SPIT-scores means and any SPIT-prevention device can make a local policy decision on these scores at its own discretion. In particular, whether the SPIT-prevention device using scores from previous hops makes arithmetic calculations on these scores or how it combines them to a final 0/1 decision is out of the scope of standardisation. In addition, any hop can express its accumulated result for the single values in a total score. This total is based on this hops calibrated algorithm on SPIT detection and is either 0="not SPIT" or 1="SPIT".

To give an example (based on the current draft-wing-sipping-spam-score, whether the Via header is the correct place for SPIT-score signalling or not is a different issue and not part of this discussion):

       Via: SIP/2.0/UDP biloxi.com;branch=z9hG4bKnashds8
         ;received=192.0.2.1
  -->    ;spam-detail="callrate-anomaly=30;iprange-userfeedback=45;whitelist=0;blacklist=1"
  -->    ;spam=1

This would imply that proxy biloxi.com calculated a callrate value of 30 for this message, based on anomaly detection, meaning 30 percent of all previous messages had a lower or equal call-rate, meaning nothing more and nothing less. Also, the iprange method for this message resulted in a score of 45, based on user-feedback. This means that 45% of all times the result of the iprange method for this message was marked as SPIT by users. The message was not on the whitelist of the proxy but on its blacklist. With its own (proprietary and secret) algorithm, the proxy would aggregate these scores to a "this is SPIT" decision. 

We are looking forward to receiving comments from others how they view the problem of assigning semantics to SPIT scores and on our thoughts towards a solution. We are thinking of writing these thoughts into a separate draft or inside the draft-wing-sipping-spam-score to have the semantic of SPIT scores defined.

 - Jan & Saverio

============================================================
Jan Seedorf
Research Scientist
NEC Europe Ltd., NEC Laboratories Europe, Network Division	
Kurfuerstenanlage 36, D-69115 Heidelberg
Tel.     +49 (0)6221 4342-221
Fax:     +49 (0)6221 4342-155
e-mail:  [email-masked]
============================================================
NEC Europe Limited Registered Office: NEC House,
1 Victoria Road, London W3 6BL Registered in England 2832014 

_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: Eric Rescorla <[email-masked]>
Date: Sat, 24 Nov 2007 13:16:22 -0800

$Id: niccolini-spit-rev.txt,v 1.3 2007/11/24 21:11:22 ekr Exp $

Saverio,

Thanks for sending me some of your research papers as background to 
the SPIT discussion [QNT07], [RNT06], [FGN07]. 

Briefly, the contents of these papers are as follows:

- [RNT06] describes a multistage filtering framework comprising
  two types of filters, non-intrusive and intrusive. The nonintrusive
  filters use things like source IP address. The intrusive filters
  involve intercepting the call and doing something like a reverse
  turing test. This paper seems mostly to be about how to collect
  the data and redirect the calls as appopriate.

- [FGN07] describes an evaluation of the system described in [RNT06]
  based on synthetic data.

- [QNT07] describes a particular method for analyzing voice
  patterns on a call to try to determine whether the call is
  SPIT.

As I indicated in our discussion in Prague, my primary concern
with having a BOF or WG intended to carry indicators of SPIT or
non-SPITness, as described in draft-niccolini-sipping-feedback-spit,
is that it's not clear that any set of filters we currently 
understand will do an acceptable job of determining whether
a message is SPIT or not. Reviewing these papers has reinforced
that impression. In particular, there's no evidence that the
nonintrusive indicia will actually be useful for detecting SPIT
and the intrusive methods are both extremely intrusive and seem
easy to spoof.

Let's start with the nonintrusive indicia. [FGN07] lists the 
following modules:

- Call rate.
- Identity checking.
- "Statistical"

All of these modules seem to be based on unwarranted assumptions about
attacker behavior. In particular, the first and last seem to be based
on the assumption that the attacker will just bomb the recipient with
as many calls as possible. But that's just one attacker strategy, and
it's trivial for attackers to rate limit, since they after all are
likely to attack a large number of recipients, so they simply need to
space out the calls some. In addition, in order for suppression to be
useful, the recipient needs to be able to continue to receive
legitimate phone calls even during high traffic periods, which is
problematic if identity detection does not work properly.

This brings us to the identity checking module, which seems to work on
usernames and IP addresses, all of which are trivially easy to
forge. I would note at this point that email has a fairly similar
traffic model to VoIP and all the same information is available to
anti-SPAM filters, and yet this sort of identity binding has not, as
far as I know, been a major success in suppressing spam, hence more
sophisticated strategies such as RMX/SPF and DKIM. I note that SIP
already has an analog to DKIM in the form of RFC 4474 Identity.

Moving on to intrusive methods, [QNT07] describes a system that
attempts to determine whether a call is SPIT by examining its
conformance to "human communication patterns." The algorithm
description here is a little vague, but seems to be based on (S 3.1)
whether the caller is silent while waiting for the callee's greeting,
(S 3.2) checking the "answer length of the caller and (S 3.3)
"challenges.

First, it's worth noting that either you've put the caller in jail
while you do these tests, which is highly annoying for the caller, or
you are monitoring while the caller is connected to the callee, at
which point surely the caller can do better SPIT detection than these
algorithms can.

Second, these algorithms appear to me to be both susceptible to false
positives and easy to attack. I don't really understand the silence
detection in S 3.1, but based on S 4.1 and S 4.4.,, it sounds like if
I call you and just say "Hello, Hello" when the phone stops ringing or
I hear a click, I have a real risk of being regarded as SPIT. That
seems bad. As far as easy to circumvent, detecting silence is an easy
problem, so why can't the SPIT caller just wait for the callee to say
"Hello".  My understanding is that this is a standard feature of call
center software. Have you tried collecting a real corpus of telemarketing
calls and testing your algorithms against it.

Your basic assumption (see S 3.3) seems to be that
attackers won't put smarts in their software, but as the number of
SPIT calls you are going to generate is far smaller than the number of
SPAM messages, so much more CPU power is available, this assumption
seems unwarranted.  Moreover, it's not clear to me that these tests
are really expensive to defeat. Have you tried? 

I realize that this is a difficult problem made more difficult by
the lack of real-world data to compare against. That said, after
reading this work, I come away that actually building reasonable
SPIT filters is far more of a research problem than an engineering
problem and that the current level of work doesn't come close to
actually representing a solution.

I understand that what you're asking for is not an IETF effort to
develop SPIT filters, but just for an effort to define protocol data
carry the scores emitted by said filters (and feedback in the other
direction) However, as I said in Prague, I don't see the value in
investing effort in standardizing mechanisms to carry data that is
itself of extremely questionable usefulness.  An additional concern
here is that it's not at all obvious what data one would like to carry
around. If you look at current spam filters, they're multifactorial
and it's not at all obvious how to combine information about each
indicator into a single score. I've heard suggestions that each
agent would just emit its score, but that destroys a lot of
data and may be really suboptimal. Accordingly, I think it's
really unwise to start specifying mechanism to carry filter
information until we know waht the actual filters will look like
and whether they can conform to such an interface.

-Ekr

[QNT07] J. Quittek, S. Niccolini, S. Tarterelli, M. Stiemerling, 
	M. Brunner, T. Ewald, "Detecting SPIT Calls by Checking
	Human Communication Patterns", to appear in proceedings
	of ICC 2007, June 2007.

[RNT06] R. Schlegel, S. Niccolini, S. Tartarelli, M. Brunner,
	"SPam over Internet Telephony (SPIT) Prevention Framework,
	Proceedings of IEEE GLOBECOM 2006, November, 2006.

[FGN07] M. Falomi, R. Garroppo, S. Niccolini, "Simulation and
	Optimization of SPIT Detection Frameworks", 2007.

---

From: Hannes Tschofenig <[email-masked]>
Date: Sun, 25 Nov 2007 14:29:57 +0100

Hi Ekr,

unfortunately I have to agree with you that the papers do not
give enough reason to believe that the statistical techniques
would work very well. (There are other techniques described,
such as turing tests, white lists, etc., that are more useful IMHO.)

Some comments regarding the papers you mentioned:

[RNT06] says

"
One module we developed
stores and checks the recent call rate of a caller by keeping
information about past calls for a limited and configurable
amount of time. Whenever a new call is received the identity
of the caller is looked up to determine his current call rate.
Example parameters could be to allow at most 2 calls within
60 seconds and assign a positive score if this threshold is
passed.
"

This work makes the assumption that the user is
classified into a group regarding their calling behavior.
This might impose a significant operational cost when you
require the user to make this indication. Alternatively,
one could imagine that this "normal" user behavior is
learned dynamically. However, a couple of questions remain.
For example, how do you deal with users where no
such data is available? What happens if the call originates
from a different domain and this information is not "forwarded"?
How does this mechanism interwork with privacy mechanisms?

I have, however, learned that some SBC vendors
have implemented such a feature for SPIT prevention.
I am not sure how widely it is used; I guess it isn't
given that there is almost no SPIT.

[RNT06] also talks about the IP address to
originating domain correlation.

I am always very reluctant when it comes to
the usage of IP address for anything.
One should not that the IP address
of the sender may change due to a number of
things (VPN, all sorts of mobility protocol related
tunneling techniques, TURN, SBCs, mobility in general)

I believe that the turing test is useful but a fairly orthogonal issue.
This also has nothing todo with the SPIT score.

In [QNT07] a technique is used that inspects the audio itself.
I also see this as a separate category since it has nothing todo
with the SPIT score.

[FGN07] talks about statistical techniques.
"
Rate based Intrusion Prevention Systems (RBIPS) work by
monitoring and learning normal network behaviors. Through
real-time traffic monitoring and comparison with stored
statistics, RBIPS can identify abnormal rates for certain types
of traffic (e.g. TCP, UDP or ARP packets, connections per
second, packets per connection, packets to specific ports etc.).
A baseline of traffic patterns is established, usually during a
learning phase in which the device only “listens” without
blocking packets or giving alarms.
A good system will have
default parameters set to reasonable levels, but the “listening”
period is required to learn the specific traffic behavior in the
network. Once a baseline is established, rate-based systems
watch for deviations from the known traffic patterns to detect
anomalies. RBIPS should be set to avoid false positives and
not to misidentify legitimate changes in traffic patterns as
attacks.
"

These techniques have been developed in the past already for the
PSTN. At Siemens we had products mainly to detect fraud but
the techniques are essentially the same. The basic idea is to learn the
calling behavior of a user (mostly using techniques known from the
machine learning community, such as artificial neural networks,
Support Vector Machines, or other statistical classification mechanisms).

These techniques have been deployed and used. I am not sure whether
there are so many literature references available since most
companies probably did not want to disclose what they did in their
products (mainly to avoid giving information to "adversaries").

I personally see some value in these techniques (as long as you
give the user the ability to turn them off) but I am not
sure whether we will see a lot of publications from
manufacturers.

Ciao
Hannes

---

From: "Saverio Niccolini" <[email-masked]>
Date: Sun, 25 Nov 2007 15:55:29 +0100

Dear Eric,

thanks for taking time to read into the papers.

Clearly you are right in one sense:
-- they are research papers and they describe research issues, but this
is what they were written for

We could not make a paper detailing trivial detection methodologies 
(maybe the ones you asked us to provide you) like:
-- does an user has simultaneous active calls? if yes how many, the
more he has the more suspucious he is
-- what is the average duration of calls? are there users with strange numbers?
-- misdialled calls resulting in 4xx messages
-- call trends checking if the number of calls placed by one identity is
increasing or decreasing in adjacent time windows
-- number of calls initiated by one user over number of calls initiated on
average by an user
-- white and black lists
-- etc.
Even if all these methodologies are in principle possible to be supported
by our architecture described in [RNT06]. These are methodologies that for
sure will work, but they would not have granted us a publication at Globecom
or ICC.
 
Moreover (as also Hannes pointed out):
-- they are based on building an history of patterns of users and
understand deviations from these patters, this is common practice
also in PSTN and in Intrusion Detection Systems, this is how the
spam score is going to be computed for SIP (alternative is basing
the score user feedbacks)

There are other methodologies that do not involve scoring that would for sure
be even more effective (even if more intrusive):
-- reverse Turing tests, CAPTCHAs, etc., in our paper maybe you missed
the fact that the human communication patter is checked by an answering machine,
e.g. if the call is detected to be as suspicious (caller unknown), the call
is redirected to an answering machine that plays a message like "Welcome,
wait until you call is being transferred" and meanwhile checks the caller
(a variant could be to pose a challenge...)
--> that is why I am planning to contirbute this to the draft of Hannes on
CAPTCHAs

Also we are currently working on other methodologies based social relationships
(similar to the paper Hannes circulated) or based on the fact that real callers
call each other and not just get called, but the methodologies can not be disclosed
publicily because of obvious IPR issues now (that will not go into standards since they
are belonging to the methodologies and not to the standard issue of spam scoring)

For what it is worth (since I do not think you are really going to change your
opinion no matter what I say...):
-- we are currently in the process of testng with real data (only real calls
that are not SPIT since there are not too many repeorted incidents of SPIT
in the real world) --> I will send you another paper that addresses the issue
of profiling real traffic to detect spam in another email for you to have a look...
-- we are also planning to test the Turing test methodologies against software
that is claimed to defeat Turing tet like ours and understand the success rate

Hope this helps in understanding our direction.

As I said: we want to have a container where spam scores can be exchanged, we have
ideas for methodologies (simple ones, not only complicated ones) and ideas for their
semantics, maybe it is better if we describe them in a draft in a way that everybody
can have a look, what do you think?
Can we have a chat at next IETF about this?

Cheers,
Saverio

============================================================
Dr. Saverio Niccolini
Senior Researcher
NEC Laboratories Europe, Network Research Division	
Kurfuerstenanlage 36, D-69115 Heidelberg
Tel.     +49 (0)6221 4342-118
Fax:     +49 (0)6221 4342-155
e-mail:  [email-masked] <-- !!! NEW ADDRESS !!!
============================================================
NEC Europe Limited Registered Office: NEC House, 1 Victoria
Road, London W3 6BL Registered in England 2832014

> -----Original Message-----
> From: Eric Rescorla [mailto:[email-masked]] 
> Sent: Saturday, November 24, 2007 10:16 PM
> To: Saverio Niccolini
> Cc: [email-masked]
> Subject: Re: [Sipping] Spit Score Semantics
> 
> Thanks for sending me some of your research papers as 
> background to the SPIT discussion [QNT07], [RNT06], [FGN07]. 
> 
> As I indicated in our discussion in Prague, my primary 
> concern with having a BOF or WG intended to carry indicators 
> of SPIT or non-SPITness, as described in 
> draft-niccolini-sipping-feedback-spit,
> understand will do an acceptable job of determining whether a 
> message is SPIT or not. Reviewing these papers has reinforced 
> nonintrusive indicia will actually be useful for detecting 
> SPIT and the intrusive methods are both extremely intrusive 
> and seem easy to spoof.
> 
> All of these modules seem to be based on unwarranted 
> assumptions about attacker behavior. In particular, the first 
> and last seem to be based on the assumption that the attacker 
> will just bomb the recipient with as many calls as possible. 
> But that's just one attacker strategy, and it's trivial for 
> attackers to rate limit, since they after all are likely to 
> attack a large number of recipients, so they simply need to 
> space out the calls some. In addition, in order for 
> suppression to be useful, the recipient needs to be able to 
> continue to receive legitimate phone calls even during high 
> traffic periods, which is problematic if identity detection 
> does not work properly.
> 
> This brings us to the identity checking module, which seems 
> to work on usernames and IP addresses, all of which are 
> trivially easy to forge. I would note at this point that 
> email has a fairly similar traffic model to VoIP and all the 
> same information is available to anti-SPAM filters, and yet 
> this sort of identity binding has not, as far as I know, been 
> a major success in suppressing spam, hence more sophisticated 
> strategies such as RMX/SPF and DKIM. I note that SIP already 
> has an analog to DKIM in the form of RFC 4474 Identity.
> 
> 
> Moving on to intrusive methods, [QNT07] describes a system 
> that attempts to determine whether a call is SPIT by 
> examining its conformance to "human communication patterns." 
> The algorithm description here is a little vague, but seems 
> to be based on (S 3.1) whether the caller is silent while 
> waiting for the callee's greeting, (S 3.2) checking the 
> "answer length of the caller and (S 3.3) "challenges.
> 
> First, it's worth noting that either you've put the caller in 
> jail while you do these tests, which is highly annoying for 
> the caller, or you are monitoring while the caller is 
> connected to the callee, at which point surely the caller can 
> do better SPIT detection than these algorithms can.
> 
> Second, these algorithms appear to me to be both susceptible 
> to false positives and easy to attack. I don't really 
> understand the silence detection in S 3.1, but based on S 4.1 
> and S 4.4.,, it sounds like if I call you and just say 
> "Hello, Hello" when the phone stops ringing or I hear a 
> click, I have a real risk of being regarded as SPIT. That 
> seems bad. As far as easy to circumvent, detecting silence is 
> an easy problem, so why can't the SPIT caller just wait for 
> the callee to say "Hello".  My understanding is that this is 
> a standard feature of call center software. Have you tried 
> collecting a real corpus of telemarketing calls and testing 
> your algorithms against it.
> 
> Your basic assumption (see S 3.3) seems to be that attackers 
> won't put smarts in their software, but as the number of SPIT 
> calls you are going to generate is far smaller than the 
> number of SPAM messages, so much more CPU power is available, 
> this assumption seems unwarranted.  Moreover, it's not clear 
> to me that these tests are really expensive to defeat. Have 
> you tried? 
> 
> I realize that this is a difficult problem made more 
> difficult by the lack of real-world data to compare against. 
> That said, after reading this work, I come away that actually 
> building reasonable SPIT filters is far more of a research 
> problem than an engineering problem and that the current 
> level of work doesn't come close to actually representing a solution.
> 
> I understand that what you're asking for is not an IETF 
> effort to develop SPIT filters, but just for an effort to 
> define protocol data carry the scores emitted by said filters 
> (and feedback in the other
> direction) However, as I said in Prague, I don't see the 
> value in investing effort in standardizing mechanisms to 
> carry data that is itself of extremely questionable 
> usefulness.  An additional concern here is that it's not at 
> all obvious what data one would like to carry around. If you 
> look at current spam filters, they're multifactorial and it's 
> not at all obvious how to combine information about each 
> indicator into a single score. I've heard suggestions that 
> each agent would just emit its score, but that destroys a lot 
> of data and may be really suboptimal. Accordingly, I think 
> it's really unwise to start specifying mechanism to carry 
> filter information until we know waht the actual filters will 
> look like and whether they can conform to such an interface.
> 
> -Ekr
> 
>

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 25 Nov 2007 08:13:08 -0800

At Sun, 25 Nov 2007 15:55:29 +0100,
> sure will work, but they would not have granted us a publication at Globecom

What I was asking for was evidence that *any* set of techniques
such as the ones you describe habove or the ones in your papers
actually worked--and more importantly, could be applied by
someone who isn't at the calling party's proxy. You have 
provided none. And BTW, I regularly serve on conference PCs
and if you had a paper that actually demonstrated useful
SPIT detection with real data sets, that would be quite 
publishable.

> be even more effective (even if more intrusive):

Well, if we don't care about how intrusive things are, I can just cut
the wire leading to my phone. You need to worry about both type I and
type II errors.

> (a variant could be to pose a challenge...)

No, I noticed this. I think it's unacceptable to most callers.

> opinion no matter what I say...):

Well, that's sure a nice friendly basis on which to start a conversation.

I prefer to think that I make my decisions based on the evidence and that
you haven't provided much.

> in the real world)

This sort of data generally isn't very useful, since you often
end up fitting your classifier to features that are easily
changed by the attacker.

> As I said: we want to have a container where spam scores can be
> exchanged, we have ideas for methodologies (simple ones, not only
> complicated ones) and ideas for their semantics, maybe it is better
> if we describe them in a draft in a way that everybody can have a
> look, what do you think?

Absent some evidence that such methodologies do anything useful,
I don't see the point of this.

> Can we have a chat at next IETF about this?

Sure.

-Ekr

---

From: Hadriel Kaplan <[email-masked]>
Date: Sun, 25 Nov 2007 22:04:58 -0500

> From: Hannes Tschofenig [mailto:[email-masked]]
> Sent: Sunday, November 25, 2007 8:30 AM
>

It's used, but it can also be done on the registrar or S-CSCF-type nodes, assuming all inbound calls from subscribers go through them. (which they usually do, afaik)

Stopping SPIT at the originating domain is one of the easier things to do, depending on how you define "SPIT".  The really hard part is stopping it at the terminating domain, because by then all discriminating information is probably gone.  None of the technique's I've seen on paper will work in the real world, because the things they key off of don't exist.

> This also has nothing todo with the SPIT score.

Why do you think it's useful?  I don't doubt some people/geeks will be ok with it, just as they are with Email versions of it.  But in general I don't think most consumers and Enterprises would accept this behavior.  For one thing it means the clock starts on billing early; it would probably have to be language-specific; and it's more annoying than the PSTN experience.

-hadriel

---

From: "Saverio Niccolini" <[email-masked]>
Date: Mon, 26 Nov 2007 10:45:23 +0100

Eric,

I was saying you are not going to change your opinion since what you
are asking is impossible:
-- show that solutions are working on blocking real SPIT incidents,
since there are no data of such attacks available it is a dead end (now)

The aim of this standardization is to have some useful standards (in
my opinion) *before* these attacks happen, thus there is a clear
mismatch between your request and what I was trying to push...

But maybe we should just implement it without any standard (we are
doing it anyway).

I do not want to bother folks on this mailing list with this discussion
thus I will take it offline and talk to you in Vancouver.

Thanks for you time you spent commenting on this.

Cheers,
Saverio

> Sent: Sunday, November 25, 2007 5:13 PM
> To: Saverio Niccolini
> Cc: Eric Rescorla; [email-masked]
> > -- they are research papers and they describe research issues, but 
> > this is what they were written for
> > 
> > -- call trends checking if the number of calls placed by 
> one identity 
> > is increasing or decreasing in adjacent time windows
> > -- number of calls initiated by one user over number of calls 
> > initiated on average by an user
> > -- white and black lists
> > -- etc.
> > Even if all these methodologies are in principle possible to be 
> > supported by our architecture described in [RNT06]. These are 
> > methodologies that for sure will work, but they would not 
> have granted 
> > us a publication at Globecom or ICC.
> 
> What I was asking for was evidence that *any* set of 
> techniques such as the ones you describe habove or the ones 
> in your papers actually worked--and more importantly, could 
> be applied by someone who isn't at the calling party's proxy. 
> You have provided none. And BTW, I regularly serve on 
> conference PCs and if you had a paper that actually 
> demonstrated useful SPIT detection with real data sets, that 
> would be quite publishable.
> 
> 
> > There are other methodologies that do not involve scoring 
> that would 
> > for sure be even more effective (even if more intrusive):
> 
> Well, if we don't care about how intrusive things are, I can 
> just cut the wire leading to my phone. You need to worry 
> about both type I and type II errors.
> 
> 
> > the fact that the human communication patter is checked by an 
> > answering machine, e.g. if the call is detected to be as suspicious 
> > (caller unknown), the call is redirected to an answering 
> machine that 
> > plays a message like "Welcome, wait until you call is being 
> > transferred" and meanwhile checks the caller (a variant could be to 
> > pose a challenge...)
> 
> > For what it is worth (since I do not think you are really going to 
> > change your opinion no matter what I say...):
> 
> I prefer to think that I make my decisions based on the 
> evidence and that you haven't provided much.
> 
> 
> > -- we are currently in the process of testng with real data 
> (only real 
> > calls that are not SPIT since there are not too many repeorted 
> > incidents of SPIT in the real world)
> 
> This sort of data generally isn't very useful, since you 
> often end up fitting your classifier to features that are 
> easily changed by the attacker.
> 
> 
> > complicated ones) and ideas for their semantics, maybe it 
> is better if 
> > we describe them in a draft in a way that everybody can 
> have a look, 
> > what do you think?
> 
> Absent some evidence that such methodologies do anything 
> useful, I don't see the point of this.
> 
>

---

From: "Jan Seedorf" <[email-masked]>
Date: Thu, 29 Nov 2007 13:59:44 +0100

Dear Eric,

please allow me some comments on this discussion:

I think we all agree that the SPIT prevention work Saverio and others published so far is not a complete solution but rather provides first steps towards fighting the problem. From a scientific point you are absolutely right: you can only be 100% sure that algorithms work when you have tested them against real SPIT. I used to work in the antivirus test center (http://agn-www.informatik.uni-hamburg.de/vtc/index.htm) where we tested antivirus products against a testbed of ~350.000 samples of malware to evaluate their algorithms' effectiveness, so I understand your concern. The question is: do we want to wait with SPIT prevention until we have 100.000 samples of SPIT calls? So: to really prove that the algorithms work we need to wait for real SPIT. However, we think we need standardisation of SPIT scores before that so that a protection architecture is in place when the problem appears "in the wild" (antivirus people jargon).

>From my point of view: the research discussion is not getting us far with standardisation because - as you pointed out - currently we cannot evaluate our algorithms against real SPIT, we can only estimate the effectiveness of possible solutions. The core question is thus: standardise a container field for SPIT prevention information before there is actually SPIT or not? We think yes, you (and possibly others) do not. I think around this question the discussion in Vancouver (or on the mailing list) should focus. 

One more point: It is well-known that the credit card industry, the intrusion-detection community, and as Hannes pointed out also PSTN equipment vendors use anomaly-based detection mechanisms (which - fully agreed - all suffer from false positives). So even if we cannot test anomaly-based SPIT prevention at the current stage, are you doubting that these mechanisms are feasable for SPIT prevention in general? If the answer is no, then why not have standards ready to enable distributed input to local algorithms before the problem pops up?

 - Jan

> Use [email-masked] for questions on current 
> sip Use [email-masked] for new developments of core SIP
>