Subject: [Sipping] Re: Spit Score Semantics

From: Cullen Jennings <[email-masked]>
Date: Thu, 22 Nov 2007 10:51:03 -0800

This is a questions of architecture and without an architecture it s  
pretty hard to decide what to do. If your architecture requires the  
combination of two scores generated by two different sources - then  
you need them to mean something more than an ordinal value. Negative  
log probabilities are pretty popular in the bayesian world but there  
are many other things such as bounded error rates that could work  
too. If on the other hand the number is generated by one source and  
it the only thing is to execute a threshold on it, then it could just  
be an arbitrary range. Anyways, before we understand what the basic  
architecture is, it seems hard to sort out the semantics we need.

On Nov 22, 2007, at 3:17 AM, Jan Seedorf wrote:

> Dear all,
>
> We (Saverio Niccolini and Jan Seedorf) believe there is a need to  
> signal SPIT scores (as calculated with various methods at single  
> signalling hops) along the signalling path to protect against SPIT.  
> In order to make such a signalling of SPIT scores compatible among  
> heterogeneous system, it must be standardised in some way. The  
> draft-wing-sipping-spam-score addresses this issue by proposing  
> additional headers/values for SPIT scores that could be passed  
> along the downstream (and possibly upstream) signalling path. One  
> question is whether it makes sense to have a SPIT score ranging  
> from 0-100. The concern raised by Henning Schulzrinne on the  
> SPITSTOP mailing list was that a scale from 0-100 has no clear  
> semantics defined for values other than 0 ("I am sure this is not  
> SPIT") and 100 ("I am sure this is SPIT"). We thought about how to  
> define semantics for SPIT scores and would like to receive feedback  
> on our thoughts.
>
> First, we believe there are SPIT detection methods resulting in a  
> binary score (0="not SPIT",1="SPIT"). For such methods with a  
> binary result range it is quite easy to come up with well-defined  
> semantics. For instance, for methods like "whitelist" or  
> "blacklist", it could be defined that "0" means "not on list" and  
> "1" means "on list". Similar, for a CAPTCHA-test you can image "0"  
> for passed (i.e., not SPIT according to this test) and "1" for  
> failed (i.e., SPIT according to this test).
>
> Second, we believe it is also possible to come up with semantics  
> for statistical methods (which generally do not produce 0/1  
> results), resulting in a score between 0 and 100. For anomaly-based  
> statistical methods, the semantics of a score x could be that x  
> percent of previous messages where below/above (depending on the  
> method) the result for this method compared to the current message.  
> For instance, think of a call-rate detection method that checks how  
> many calls are initiated from a particular SIP-URI (or IP-address)  
> within a time-frame. A result of 85 for such a test would mean that  
> the call-rate for the SIP-URI in the From-header of the SIP message  
> in the time-frame is higher than 84% of all calls monitored in the  
> past. Note, as Henning pointed out, this implies nothing directly  
> on the SPIT probability for that message. It is simply a way to  
> precisely define the semantics for the score of anomaly-based  
> statistical modules.
>
> For statistical modules the score could also be based on user- 
> feedback. Then the semantics of score x would be that in the past x  
> percent of messages with the same method-result were marked as SPIT  
> by users (using some user-feedback mechanism). This has probably  
> more relation to a SPIT probability. However, this is not  
> important. The point is that the score is well-defined.
>
> In general, we think that for EACH method it needs to be (publicly)  
> well-defined what the corresponding score means. This score does  
> NOT (necessarily) imply a SPIT-probability. Instead, it has a per- 
> method semantic. Then, for each method it is clearly defined what  
> the signalling of corresponding SPIT-scores means and any SPIT- 
> prevention device can make a local policy decision on these scores  
> at its own discretion. In particular, whether the SPIT-prevention  
> device using scores from previous hops makes arithmetic  
> calculations on these scores or how it combines them to a final 0/1  
> decision is out of the scope of standardisation. In addition, any  
> hop can express its accumulated result for the single values in a  
> total score. This total is based on this hops calibrated algorithm  
> on SPIT detection and is either 0="not SPIT" or 1="SPIT".
>
> To give an example (based on the current draft-wing-sipping-spam- 
> score, whether the Via header is the correct place for SPIT-score  
> signalling or not is a different issue and not part of this  
> discussion):
>
>        Via: SIP/2.0/UDP biloxi.com;branch=z9hG4bKnashds8
>          ;received=192.0.2.1
>   -->    ;spam-detail="callrate-anomaly=30;iprange- 
> userfeedback=45;whitelist=0;blacklist=1"
>   -->    ;spam=1
>
> This would imply that proxy biloxi.com calculated a callrate value  
> of 30 for this message, based on anomaly detection, meaning 30  
> percent of all previous messages had a lower or equal call-rate,  
> meaning nothing more and nothing less. Also, the iprange method for  
> this message resulted in a score of 45, based on user-feedback.  
> This means that 45% of all times the result of the iprange method  
> for this message was marked as SPIT by users. The message was not  
> on the whitelist of the proxy but on its blacklist. With its own  
> (proprietary and secret) algorithm, the proxy would aggregate these  
> scores to a "this is SPIT" decision.
>
> We are looking forward to receiving comments from others how they  
> view the problem of assigning semantics to SPIT scores and on our  
> thoughts towards a solution. We are thinking of writing these  
> thoughts into a separate draft or inside the draft-wing-sipping- 
> spam-score to have the semantic of SPIT scores defined.
>
>  - Jan & Saverio
>
> ============================================================
> Jan Seedorf
> Research Scientist
> NEC Europe Ltd., NEC Laboratories Europe, Network Division	
> Kurfuerstenanlage 36, D-69115 Heidelberg
> Tel.     +49 (0)6221 4342-221
> Fax:     +49 (0)6221 4342-155
> e-mail:  [email-masked]
> ============================================================
> NEC Europe Limited Registered Office: NEC House,
> 1 Victoria Road, London W3 6BL Registered in England 2832014

_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: "Saverio Niccolini" <[email-masked]>
Date: Thu, 22 Nov 2007 20:43:54 +0100

Hi Cullen,

all we were trying to do was to answer the question about semantics
raised by Henning some time ago. He was arguing that without semantics
the spam score has no meaning and that is why we answered to it (after
thinking quite some time...).

As you know some examples of architecture are out already together with a
reference scenario (draft expired):
http://tools.ietf.org/html/draft-tschofenig-sipping-framework-spit-reduction-02
http://tools.ietf.org/id/draft-niccolini-sipping-spitstop-00.txt

Moreover some reference architecture for prototypes/products exist as
well (we demonstrated at last IETF our demo to interested people)
and there are scientific results in the literature that would justify
the semantics we are reporting here.

If an architecture needs to be defined then let's define it, let's
have this work started, it seems there is enough interest around the
topic and people willing to work on it :-)

Looking forward to meeting you in Vancouver.

Cheers,
Saverio

============================================================
Dr. Saverio Niccolini
Senior Researcher
NEC Laboratories Europe, Network Research Division	
Kurfuerstenanlage 36, D-69115 Heidelberg
Tel.     +49 (0)6221 4342-118
Fax:     +49 (0)6221 4342-155
e-mail:  [email-masked] <-- !!! NEW ADDRESS !!!
============================================================
NEC Europe Limited Registered Office: NEC House, 1 Victoria
Road, London W3 6BL Registered in England 2832014

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of 
> Cullen Jennings
> Sent: Thursday, November 22, 2007 7:51 PM
> To: Jan Seedorf
> Cc: Eric Rescorla; [email-masked]; 
> [email-masked]; [email-masked]
> Subject: Re: [spitstop] Spit Score Semantics
> 
> 
> This is a questions of architecture and without an 
> architecture it s pretty hard to decide what to do. If your 
> architecture requires the combination of two scores generated 
> by two different sources - then you need them to mean 
> something more than an ordinal value. Negative log 
> probabilities are pretty popular in the bayesian world but 
> there are many other things such as bounded error rates that 
> could work too. If on the other hand the number is generated 
> by one source and it the only thing is to execute a threshold 
> on it, then it could just be an arbitrary range. Anyways, 
> before we understand what the basic architecture is, it seems 
> hard to sort out the semantics we need.
> 
> 
> > additional headers/values for SPIT scores that could be 
> passed along 
> > the downstream (and possibly upstream) signalling path. One 
> question 
> > is whether it makes sense to have a SPIT score ranging from 
> 0-100. The 
> > concern raised by Henning Schulzrinne on the SPITSTOP 
> mailing list was 
> > that a scale from 0-100 has no clear semantics defined for values 
> > other than 0 ("I am sure this is not
> > define semantics for SPIT scores and would like to receive 
> feedback on 
> > our thoughts.
> >
> > binary score (0="not SPIT",1="SPIT"). For such methods with 
> a binary 
> > result range it is quite easy to come up with well-defined 
> semantics. 
> > For instance, for methods like "whitelist" or "blacklist", 
> it could be 
> > defined that "0" means "not on list" and "1" means "on 
> list". Similar, 
> > for a CAPTCHA-test you can image "0"
> > for passed (i.e., not SPIT according to this test) and "1" 
> for failed 
> > (i.e., SPIT according to this test).
> >
> > Second, we believe it is also possible to come up with 
> semantics for 
> > statistical methods (which generally do not produce 0/1 results), 
> > resulting in a score between 0 and 100. For anomaly-based 
> statistical 
> > methods, the semantics of a score x could be that x percent of 
> > previous messages where below/above (depending on the
> > the call-rate for the SIP-URI in the From-header of the SIP 
> message in 
> > the time-frame is higher than 84% of all calls monitored in 
> the past. 
> > Note, as Henning pointed out, this implies nothing directly on the 
> > SPIT probability for that message. It is simply a way to precisely 
> > define the semantics for the score of anomaly-based statistical 
> > modules.
> >
> > percent of messages with the same method-result were marked 
> as SPIT by 
> > users (using some user-feedback mechanism). This has probably more 
> > relation to a SPIT probability. However, this is not important. The 
> > point is that the score is well-defined.
> >
> > well-defined what the corresponding score means. This score 
> does NOT 
> > (necessarily) imply a SPIT-probability. Instead, it has a 
> per- method 
> > semantic. Then, for each method it is clearly defined what the 
> > signalling of corresponding SPIT-scores means and any SPIT- 
> prevention 
> > device can make a local policy decision on these scores at its own 
> > discretion. In particular, whether the SPIT-prevention device using 
> > scores from previous hops makes arithmetic calculations on these 
> > scores or how it combines them to a final 0/1 decision is 
> out of the 
> > scope of standardisation. In addition, any hop can express its 
> > accumulated result for the single values in a total score. 
> This total 
> > is based on this hops calibrated algorithm on SPIT detection and is 
> > either 0="not SPIT" or 1="SPIT".
> >
> > This would imply that proxy biloxi.com calculated a 
> callrate value of 
> > 30 for this message, based on anomaly detection, meaning 30 
> percent of 
> > all previous messages had a lower or equal call-rate, 
> meaning nothing 
> > more and nothing less. Also, the iprange method for this message 
> > resulted in a score of 45, based on user-feedback.
> > This means that 45% of all times the result of the iprange 
> method for 
> > this message was marked as SPIT by users. The message was 
> not on the 
> > whitelist of the proxy but on its blacklist. With its own 
> (proprietary 
> > and secret) algorithm, the proxy would aggregate these scores to a 
> > "this is SPIT" decision.
> >
> > We are looking forward to receiving comments from others 
> how they view 
> > the problem of assigning semantics to SPIT scores and on 
> our thoughts 
> > towards a solution. We are thinking of writing these 
> thoughts into a 
> > separate draft or inside the draft-wing-sipping- spam-score to have 
> > the semantic of SPIT scores defined.
> >
> >  - Jan & Saverio
> >
> spitstop mailing list
> [email-masked]
> https://listserv.netlab.nec.de/mailman/listinfo/spitstop
>

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 22 Nov 2007 11:41:38 -0800

At Thu, 22 Nov 2007 20:43:54 +0100,
> the semantics we are reporting here.

Saverio,

I'd be very interested in reading these results. Could you please
point me to the appropriate papers?

Thanks,
-Ekr

---

From: "Dan Wing" <[email-masked]>
Date: Mon, 26 Nov 2007 12:22:02 -0800

(dropping CC's.)

I have a simple idea; the presence of a "." could indicate a percentage,
whereas the lack of a period indicates a true/false test.  For example:

  callrate-anomaly=30.0         indicates 30% likelyhood of spam
  iprange-userfeedback=45.0     indicates 45% likelyhood of spam
  whitelistlist=0               indicates false
  blacklist=1                   indicates true

-d

> From: Jan Seedorf [mailto:[email-masked]] 
> Sent: Thursday, November 22, 2007 3:18 AM
> To: [email-masked]
> Cc: [email-masked]; Henning Schulzrinne; 
> [email-masked]; Eric Rescorla; [email-masked]; Dan Wing
> Subject: Spit Score Semantics
> 
> Dear all,
> 
> We (Saverio Niccolini and Jan Seedorf) believe there is a 
> need to signal SPIT scores (as calculated with various 
> methods at single signalling hops) along the signalling path 
> to protect against SPIT. In order to make such a signalling 
> of SPIT scores compatible among heterogeneous system, it must 
> be standardised in some way. The 
> draft-wing-sipping-spam-score addresses this issue by 
> proposing additional headers/values for SPIT scores that 
> could be passed along the downstream (and possibly upstream) 
> signalling path. One question is whether it makes sense to 
> have a SPIT score ranging from 0-100. The concern raised by 
> Henning Schulzrinne on the SPITSTOP mailing list was that a 
> scale from 0-100 has no clear semantics defined for values 
> other than 0 ("I am sure this is not SPIT") and 100 ("I am 
> sure this is SPIT"). We thought about how to define semantics 
> for SPIT scores and would like to receive feedback on our thoughts.
> 
> First, we believe there are SPIT detection methods resulting 
> in a binary score (0="not SPIT",1="SPIT"). For such methods 
> with a binary result range it is quite easy to come up with 
> well-defined semantics. For instance, for methods like 
> "whitelist" or "blacklist", it could be defined that "0" 
> means "not on list" and "1" means "on list". Similar, for a 
> CAPTCHA-test you can image "0" for passed (i.e., not SPIT 
> according to this test) and "1" for failed (i.e., SPIT 
> according to this test).
> 
> semantics for statistical methods (which generally do not 
> produce 0/1 results), resulting in a score between 0 and 100. 
> For anomaly-based statistical methods, the semantics of a 
> score x could be that x percent of previous messages where 
> below/above (depending on the method) the result for this 
> method compared to the current message. For instance, think 
> of a call-rate detection method that checks how many calls 
> are initiated from a particular SIP-URI (or IP-address) 
> within a time-frame. A result of 85 for such a test would 
> mean that the call-rate for the SIP-URI in the From-header of 
> the SIP message in the time-frame is higher than 84% of all 
> calls monitored in the past. Note, as Henning pointed out, 
> this implies nothing directly on the SPIT probability for 
> that message. It is simply a way to precisely define the 
> semantics for the score of anomaly-based statistical modules.
> 
> For statistical modules the score could also be based on 
> user-feedback. Then the semantics of score x would be that in 
> the past x percent of messages with the same method-result 
> were marked as SPIT by users (using some user-feedback 
> mechanism). This has probably more relation to a SPIT 
> probability. However, this is not important. The point is 
> that the score is well-defined.
> 
> In general, we think that for EACH method it needs to be 
> (publicly) well-defined what the corresponding score means. 
> This score does NOT (necessarily) imply a SPIT-probability. 
> Instead, it has a per-method semantic. Then, for each method 
> it is clearly defined what the signalling of corresponding 
> SPIT-scores means and any SPIT-prevention device can make a 
> local policy decision on these scores at its own discretion. 
> In particular, whether the SPIT-prevention device using 
> scores from previous hops makes arithmetic calculations on 
> these scores or how it combines them to a final 0/1 decision 
> is out of the scope of standardisation. In addition, any hop 
> can express its accumulated result for the single values in a 
> total score. This total is based on this hops calibrated 
> algorithm on SPIT detection and is either 0="not SPIT" or 1="SPIT".
> 
> To give an example (based on the current 
> draft-wing-sipping-spam-score, whether the Via header is the 
> correct place for SPIT-score signalling or not is a different 
> issue and not part of this discussion):
> 
>   -->    
> ;spam-detail="callrate-anomaly=30;iprange-userfeedback=45;whit
> elist=0;blacklist=1"
>   -->    ;spam=1
> 
> This would imply that proxy biloxi.com calculated a callrate 
> value of 30 for this message, based on anomaly detection, 
> meaning 30 percent of all previous messages had a lower or 
> equal call-rate, meaning nothing more and nothing less. Also, 
> the iprange method for this message resulted in a score of 
> 45, based on user-feedback. This means that 45% of all times 
> the result of the iprange method for this message was marked 
> as SPIT by users. The message was not on the whitelist of the 
> proxy but on its blacklist. With its own (proprietary and 
> secret) algorithm, the proxy would aggregate these scores to 
> a "this is SPIT" decision. 
> 
> We are looking forward to receiving comments from others how 
> they view the problem of assigning semantics to SPIT scores 
> and on our thoughts towards a solution. We are thinking of 
> writing these thoughts into a separate draft or inside the 
> draft-wing-sipping-spam-score to have the semantic of SPIT 
> scores defined.
> 
>  - Jan & Saverio
>

---

From: "Jan Seedorf" <[email-masked]>
Date: Thu, 29 Nov 2007 19:08:12 +0100

Dear Cullen,

you are right that there are arguments for other ranges than 0 to 100. Actually, the main point of our proposal is to have a per-method-semantic defined. You are suggesting that there will also probably be a per-method-range depending on the semantic. In the proposal below we just gave an example of anomaly-based algorithms where a score between 0 and 100 could be meaningful (as described). There can probably be other ranges depending on the method and corresponding semantic, I agree with you. I also agree with you (and Eric) that at this stage we cannot say which (combination of) algorithms will provide the best protection against SPIT. That is why we envision a field that is flexible to work also with new (upcoming) methods by including the name of the method and additional information about the semantic (e.g., "score is based on anomaly-based techniques and means this" or "score is based on user feedback and means that") in the header.

Regarding architecture I can imagine the following: a combination of scores from different sources and a local algorithm which combines this input from different sources and methods to a local decision. This local algorithm is apart from standardisation and can get calibrated over time, as is done with spam-assassin today, for instance. What would be part of standardisation is a field to get the information from different sources to the local decision engine.

 - Jan