Subject: Re: [sip-clf] The issue is storage space (?)

From: Peter Musgrave <[email-masked]>
Date: Wed, 1 Sep 2010 05:34:10 -0400

HI Hadriel,

I simply meant that "on the wire" the information was in UTF-8.

Of course you're right - these can be processed from e.g. a several
character CSeq number into a binary representation. This applies for
the other fields you mentioned. This provides economy of log data
storage at a modest processing cost (which was likely being done
anyway).

Peter Musgrave

On Tue, Aug 31, 2010 at 10:00 PM, Hadriel Kaplan <[email-masked]> wrote:
>
> On Aug 31, 2010, at 10:54 AM, Peter Musgrave wrote:
>
>> Hi Vijay,
>>
>> <as individual>
>> I agree metrics are useful. Size is an obvious one, as is search
>> speed. Implementation complexity is useful - but less objective.
>>
>> I also agree that intrusion detection systems can be made to run from
>> text input. It also seems to me that since SIP is fundamentally a text
>> based messaging scheme having text based logs seems logical.
>
> Actually, of the 18 or so mandatory fields to record, 11 are not text in nature: timestamp, record size, message type, directionality, CSeq number, dest-ip, dest-port, transport, source-ip, source-port, resp status.  The natural text ones are Req-URI, To, From, Call-ID, Server-txn, Client-txn, and probably method type.
>
>
>> Perhaps
>> this should be the WG output and then the specification of an
>> "adapter" for IPFIX processing (as a stream to e.g. an intrusion
>> detection system) could be done as a new work item (perhaps within
>> sipclf as a charter change - or somewhere else in the IETF universe).
>> I do think the ipfix stream has merit in some applications.
>
> That's just a cop-out to picking one.  We should just pick one.
>
> -hadriel

---

From: Benoit Claise <[email-masked]>
Date: Wed, 01 Sep 2010 12:23:13 +0200

Peter,
> It also seems to me that since SIP is fundamentally a text
Can you expend on this, I don't understand.

Regards, Benoit.

---

From: Peter Musgrave <[email-masked]>
Date: Wed, 1 Sep 2010 06:51:11 -0400

Simply that SIP sends the String INVITE instead of e.g method byte
with 0x01=INVITE 0x02= etc.

---

From: Brian Trammell <[email-masked]>
Date: Thu, 2 Sep 2010 17:50:05 +0200

Hi Peter, all,

Two issues here... one, it sounds like there may be some resistance to encoding the method as an integer in IPFIX. The reason we proposed this is that it's common IPFIX practice: enumerated types get integer codepoints. There may be good reasons for not doing so: the confusion over what to do with garbage methods. We can certainly define the sipMethod IE to be a UTF-8 string with the literal content of the requested method; not a problem at all. This is rather separate from the question of "binary-vs-text".

The separate question depends on what you mean by "text based logs". We've already conceded that if we want to meet the requirements of logging what comes off the wire, text framing as generally understood (newline separated records of whitespace or otherwise delimited values) is inadequate; it's unclear to me if the decision to keep things escaped as literal changes this, and I think not. To me the question isn't text versus binary. It's almost-text with text-based framing (indexed-ascii) versus slightly-less-texty-but-still-almost-text with binary framing (ipfix). The tradeoffs are pretty fine and seem use-case-dependent, and I suspect that the preferences for one or the other represent a bias toward thinking about a particular set of use cases.

Indexed ASCII, as I understand it, is a positional representation with additional text-based framing and text-based indexing (which itself must be parsed to be evaluable) to compensate for the fact that simple space-delimiting misses some edge casing. It's not clear to me how applicable the standard unix textutils are (at the very least, you need to strip out the indexes, or ignore them as a human). In considering processing, the tools I reach for are regex-based: Ruby, PCRE, etc.

IPFIX trades text-based framing for binary framing; this would seem to speed seeking. You gain extensibility (you can add things to templates beyond what's specified, or even interleave records with non-sipclf records) and non-positionality (you can reorder the fields on export because the field order is encoded in the template) at the expense of requiring templates (this slightly increases state requirements on the processing side). Also, certain fields which can naturally be represented with integers or octet arrays (timestamps, IP addresses) are stored (big-endian) binary as opposed to in text. This breaks grep without preprocessing, but eliminates the need to parse integers on read. With IPFIX you also get a transport protocol for free, but this is explicitly outside the charter at this time. When considering processing, the tools I reach for here are of course IPFIX implementations (ripfix, we've already talked about; see also CERT/NetSA libfixbuf or Fraunhofer libipfix on the open-source side); if I want to do something quick and dirty, I'm using stdio and C structures, or pack/unpack in a scripting language.

If you're talking archival storage, you're most probably compressing. I suspect that an IPFIX file and an indexed-ascii file would compress down to about the same size for the same data. In any case, the size of a record in either representation is in the common case going to be dominated by the size of the fields which are necessarily strings.

Regards,

Brian

On Sep 1, 2010, at 12:51 PM, Peter Musgrave wrote:

> _______________________________________________
> sip-clf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sip-clf

---

From: Peter Musgrave <[email-masked]>
Date: Thu, 2 Sep 2010 12:18:55 -0400

Hi Brian,

Pls see inline.

Peter (as individual)

On Thu, Sep 2, 2010 at 11:50 AM, Brian Trammell <[email-masked]> wrote:
> Hi Peter, all,
>
> Two issues here... one, it sounds like there may be some resistance to encoding the method as an integer in IPFIX. The reason we proposed this is that it's common IPFIX >practice: enumerated types get integer codepoints. There may be good reasons for not doing so: the confusion over what to do with garbage methods. We can certainly define the >sipMethod IE to be a UTF-8 string with the literal content of the requested method; not a problem at all. This is rather separate from the question of "binary-vs-text".

I agree it's separate - and I think if we're doing binary then by all
means let's encode Method as well as CSeq, IP addresses etc. into
binary forms.

>
> The separate question depends on what you mean by "text based logs". We've already conceded that if we want to meet the requirements of logging what comes off the wire, text >framing as generally understood (newline separated records of whitespace or otherwise delimited values) is inadequate; it's unclear to me if the decision to keep things escaped as >literal changes this, and I think not. To me the question isn't text versus binary. It's almost-text with text-based framing (indexed-ascii) versus slightly-less-texty-but-still-almost-text >with binary framing (ipfix). The tradeoffs are pretty fine and seem use-case-dependent, and I suspect that the preferences for one or the other represent a bias toward thinking >about a particular set of use cases.

The list seemed to have no one who was defending the logging of
bodies. IIRC this was the issue which made text logging inadequate.
There was one opinion about optional logging of bodies - so this would
tip us back into the binary world.

<rest removed>

---

From: Hadriel Kaplan <[email-masked]>
Date: Thu, 2 Sep 2010 14:53:13 -0400

Yeah the method thing is my fault - we (the authors of the ipfix draft) talked about that field and weren't sure whether to make it an enumeration or a string, and I said we should just keep it an enumeration for now and we can always change it if the WG wants it to be a string.  We can even do both: have an enumeration and a string, whose length is zero unless the enumeration is of "unknown" type in which case the string field's length is not 0. (though that's kinda hokey)
But anyway, all the fields are up for a decision by the WG regardless.

-hadriel

On Sep 2, 2010, at 11:50 AM, Brian Trammell wrote:

> IPFIX trades text-based framing for binary framing; this would seem to speed seeking. You gain extensibility (you can add things to templates beyond what's specified, or even interleave records with non-sipclf records) and non-positionality (you can reorder the fields on export because the field order is encoded in the template) at the expense of requiring templates (this slightly increases state requirements on the processing side). Also, certain fields which can naturally be represented with integers or octet arrays (timestamps, IP addresses) are stored (big-endian) binary as opposed to in text. This breaks grep without preprocessing, but eliminates the need to parse integers on read. With IPFIX you also get a transport protocol for free, but this is explicitly outside the charter at this time. When considering processing, the tools I reach for here are of course IPFIX implementations (ripfix, we've already talked about; see also CERT/NetSA libfixbuf or Fraunhofer libipfix 
> on the open-source side); if I want to do something quick and dirty, I'm using stdio and C structures, or pack/unpack in a scripting language.
>

---

From: Saverio Niccolini <[email-masked]>
Date: Wed, 15 Sep 2010 22:34:33 +0000

Dear Dan,

one issue is storage space for offline processing as you seem
to hint

another thing is the online monitoring case where your (distributed)
probes starts recording these logs (as records), export them, 
merge them in collectors and giving them to centralized IDS systems,
etc.

--> this clearly calls for a IPFIX-like approach that has all
I mentioned above (export, mediations, etc.)

(a text base log can hardly handle this)

I know this is out of the scope of the charter now but it would 
be really silly to choose text-based only now and then realize
you need to start again for just a slightly modified use case/requirement

Just my two cents,
Saverio

============================================================
Dr. Saverio Niccolini
Manager, Real-Time Communications Group
NEC Laboratories Europe, Network Research Division     
Kurfuerstenanlage 36, D-69115 Heidelberg
Tel.     +49 (0)6221 4342-118
NEC*NET Phone: 800-49-33-118
Fax:     +49 (0)6221 4342-155
e-mail:  [email-masked]
============================================================
NEC Europe Limited Registered Office: NEC House, 1 Victoria
Road, London W3 6BL Registered in England 2832014

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Dan Burnett
> Sent: 30 August 2010 23:24
> To: [email-masked]
> Subject: [sip-clf] The issue is storage space (?)
> 
> One of the criteria presented in Maastricht as a way to compare
> between the two formats -- Indexed ASCII and IPFIX -- is storage space.
> 
> At Voxeo we make heavy use of text formats for all of our logging,
> saving substantial amounts of data every day (multiple terabytes, and
> we are still a fairly small company).  However, all of our saved data
> has been zipped using gzip.  Although a highly-optimized, custom
> binary format could potentially save some space, we have found gzip in
> practice to give approximately the same reduction in space (because of
> the repetitive nature of the data) while being a) easy to use and b)
> easy to convert back to text.
> 
> The issue then is not storage space, but utility.  Flexibility might
> be an even better word.  We have many different logs, from different
> software and hardware packages, many from different vendors.  We
> frequently need to aggregate these logs, especially in a multi-channel
> world where web, gateway, and device logs may need to be interleaved.
> The advent of full-text tools such as Splunk and map-reduce tools make
> such interleaved text and processing even more useful and simpler.  In
> short, these tools have, like programming language compilers,
> dramatically reduced the need for custom binary programming.
> Additionally, it can be expensive to write, test, deploy or even
> purchase new tools where a wide variety of free tools already exist.
> 
> As far as universality goes, it's hard to beat text.  Heck, the IETF
> still uses it for all RFCs . . .
> 
> -- dan
>

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Thu, 16 Sep 2010 08:52:04 -0500

> (a text base log can hardly handle this)

In the end, both text and IPFIX-like approaches can handle
what you describe above.  A few years ago, we built a
distributed IDS system using HTTP CLF and ssh CLF as sources
of events (i.e., event generators).  We mined information directly
from the text files and created specific events, aggregating them, 
determined root cause analysis, etc. [1].

The good thing about systems like HTTP CLF is the rather obvious
requirement that the log record has a fixed format that can be
leveraged by other tools.

[1] Gurbani, V.K., Cook, D.L., Menten, L.E., and Reddington, T.B., "A
Case Study on Constructing a Security Event Manegement System,"
Proceedings of the German Informatics Society's 3rd International
Conference on IT-Incident Management and IT-Forensics," , pp. 83-102,
September 11-13, 2007, Stuttgart, Germany.

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
Web:   http://ect.bell-labs.com/who/vkg/

---

From: Saverio Niccolini <[email-masked]>
Date: Fri, 17 Sep 2010 13:00:44 +0000

Vijay,

> determined root cause analysis, etc. [1].

well, I was saying something slightly different.
Do you mean you aggregate events after having mined locally?
I was thinking of exporting the full SIP CLF file or a processed
version of it (but still keeping the IPFIX nature in order to
follow the IPFIX model of exporting a record) to an intermediate 
element, do the merging there following the mediation model of 
IPFIX and then analyse the aggregated data.

if you want you can have a look at [2]

This you can not do with a text-based model (either you export the
full file, but then there is no improvement in merging it)

anyway this is out of the scope of the charter now, I wanted just
to make people to think about future...

Cheers,
Saverio

[2] Sven Anderson, Saverio Niccolini, Dieter Hogrefe: SIPFIX: A scheme for
distributed SIP monitoring. Integrated Network Management 2009: 382-389