Subject: [sip-clf] The issue is storage space (?)

From: Dan Burnett <[email-masked]>
Date: Mon, 30 Aug 2010 17:24:16 -0400

One of the criteria presented in Maastricht as a way to compare  
between the two formats -- Indexed ASCII and IPFIX -- is storage space.

At Voxeo we make heavy use of text formats for all of our logging,  
saving substantial amounts of data every day (multiple terabytes, and  
we are still a fairly small company).  However, all of our saved data  
has been zipped using gzip.  Although a highly-optimized, custom  
binary format could potentially save some space, we have found gzip in  
practice to give approximately the same reduction in space (because of  
the repetitive nature of the data) while being a) easy to use and b)  
easy to convert back to text.

The issue then is not storage space, but utility.  Flexibility might  
be an even better word.  We have many different logs, from different  
software and hardware packages, many from different vendors.  We  
frequently need to aggregate these logs, especially in a multi-channel  
world where web, gateway, and device logs may need to be interleaved.   
The advent of full-text tools such as Splunk and map-reduce tools make  
such interleaved text and processing even more useful and simpler.  In  
short, these tools have, like programming language compilers,  
dramatically reduced the need for custom binary programming.   
Additionally, it can be expensive to write, test, deploy or even  
purchase new tools where a wide variety of free tools already exist.

As far as universality goes, it's hard to beat text.  Heck, the IETF  
still uses it for all RFCs . . .

-- dan

---

From: Peter Musgrave <[email-masked]>
Date: Tue, 31 Aug 2010 06:37:40 -0400

(as individual)

Hi Dan,

I agree with your points on text. The arguments for IPFIX are related
more towards the alignment with NetFlow and the suitability for
processing by e.g. intrusion detection systems.

I am planning to do a quick and dirty implementation of both logging
formats (in my ample spare time) so that I can then post some typical
examples to the list for some real call flows through a B2BUA. My hope
is that this will help crystallize the relative merits of the two
formats and provide us with the information needed for the discussion
in which one should be adopted (or if there is a way to merge the
two).

It's my hope to get this done in September - and if people think it's
useful to put it on the agenda for Beijing.

Peter Musgrave

On Mon, Aug 30, 2010 at 5:24 PM, Dan Burnett <[email-masked]> wrote:
> One of the criteria presented in Maastricht as a way to compare between the
> two formats -- Indexed ASCII and IPFIX -- is storage space.
>
> At Voxeo we make heavy use of text formats for all of our logging, saving
> substantial amounts of data every day (multiple terabytes, and we are still
> a fairly small company).  However, all of our saved data has been zipped
> using gzip.  Although a highly-optimized, custom binary format could
> potentially save some space, we have found gzip in practice to give
> approximately the same reduction in space (because of the repetitive nature
> of the data) while being a) easy to use and b) easy to convert back to text.
>
> The issue then is not storage space, but utility.  Flexibility might be an
> even better word.  We have many different logs, from different software and
> hardware packages, many from different vendors.  We frequently need to
> aggregate these logs, especially in a multi-channel world where web,
> gateway, and device logs may need to be interleaved.  The advent of
> full-text tools such as Splunk and map-reduce tools make such interleaved
> text and processing even more useful and simpler.  In short, these tools
> have, like programming language compilers, dramatically reduced the need for
> custom binary programming.  Additionally, it can be expensive to write,
> test, deploy or even purchase new tools where a wide variety of free tools
> already exist.
>
> As far as universality goes, it's hard to beat text.  Heck, the IETF still
> uses it for all RFCs . . .
>
> -- dan
>
> _______________________________________________
> sip-clf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sip-clf
>

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Tue, 31 Aug 2010 08:30:24 -0500

> processing by e.g. intrusion detection systems.

A minor point: there is absolutely nothing that prevents IDS
from operating on text data (a la indexed-ASCII).  In fact, there
are research and production IDS systems in existence that have
been build around HTTP CLF and the ssh log format.

> in which one should be adopted (or if there is a way to merge the

I still think that having having two representations, while not
optimal, is certainly preferable to forcing the chosen representation
on the party that did not prefer it.

> useful to put it on the agenda for Beijing.

Sure; your work would be very helpful.  That said, if we go this
route, then we ought to have some canonical metrics that everyone
agrees to and that do not provide an undue advantage to either of
the format.

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
Web:   http://ect.bell-labs.com/who/vkg/

---

From: Peter Musgrave <[email-masked]>
Date: Tue, 31 Aug 2010 10:54:30 -0400

Hi Vijay,

<as individual>
I agree metrics are useful. Size is an obvious one, as is search
speed. Implementation complexity is useful - but less objective.

I also agree that intrusion detection systems can be made to run from
text input. It also seems to me that since SIP is fundamentally a text
based messaging scheme having text based logs seems logical. Perhaps
this should be the WG output and then the specification of an
"adapter" for IPFIX processing (as a stream to e.g. an intrusion
detection system) could be done as a new work item (perhaps within
sipclf as a charter change - or somewhere else in the IETF universe).
I do think the ipfix stream has merit in some applications.

<as chair>
There was a clear consensus from the hum in Maastricht that one
representation was the desired output from the WG.

Regards,

Peter Musgrave

---

From: Hadriel Kaplan <[email-masked]>
Date: Tue, 31 Aug 2010 22:00:38 -0400

On Aug 31, 2010, at 10:54 AM, Peter Musgrave wrote:

> based messaging scheme having text based logs seems logical.

Actually, of the 18 or so mandatory fields to record, 11 are not text in nature: timestamp, record size, message type, directionality, CSeq number, dest-ip, dest-port, transport, source-ip, source-port, resp status.  The natural text ones are Req-URI, To, From, Call-ID, Server-txn, Client-txn, and probably method type.

> Perhaps

That's just a cop-out to picking one.  We should just pick one.

-hadriel