Subject: [sip-clf] Merging IPFIX and Indexed ASCII Logging?

From: Peter Musgrave <[email-masked]>
Date: Mon, 20 Sep 2010 18:19:04 -0400

Hi,

Here is an idea which might be useful - or might be really dumb. As I
implement both formats it's clear how very similar they are in
concept.

Ideally we want an indexed format for fast processing but one which is
also easily manipulated by text based tools like grep and vim.

What if:

1) There is a single apache-like log string containing the log data
(which is tab delimited, \n terminated) and stored as an IPFIX string.
This would need to have a fixed set of fields and a symbol e.g. "-"
when a field is not populated.

2) This string has a unique "magic prefix" (a pseudo-random
UTF-encoded value which is very unlikely to otherwise appear in either
the text or natural IPFIX entries). The string starts with \n.
[tabs in element sub-strings would be suppressed as described in the
indexed ascii draft]

3) To allow indexing there are a number of preceding data elements in
the IPFIX entry which provides pointers into the apache string which
indicate where any given element of interest starts. (i.e. To URL,
>From tag etc.). The length could also be included if desired - would
need to think about time/space tradeoff.

I think this provides:
a)  a file which can be easily sifted by grep-ing the "magic prefix"
and would show log lines on a single line (with a bit of random junk
above and below due to the IPFIX values.)
b)  file which can be used with code that wants to skip through only
the entries it cares about. Yes - this code would need to understand
the IPFIX templates - but it was going to have to do some sipclf
specific stuff anyway. Let's assume some sample code can be open
sourced so the effort to get started is not too great.

Opinions?

Is there an issue I am not seeing?

Peter Musgrave
(as individual)

---

From: Brian Trammell <[email-masked]>
Date: Tue, 21 Sep 2010 10:51:39 +0200

Hi, Peter,

This is an interesting idea, but I'm not sure what it would buy you. Basically it's indexed-ASCII with binary as opposed to ASCII indexes, and the indexes are IPFIX-framed. Right?

This would seem to retain the main drawbacks of the indexed-ascii approach (records which are very difficult to extend in a backward-compatible way, ad-hoc data modeling), the main drawbacks of the IPFIX approach (you need to do some framing you otherwise wouldn't, which complicates implementation and (imperceptibly if compressed) increases file size), and doesn't yield any of the benefits of IPFIX (potential reuse of IPFIX implementations, with interoperability and convergence with the rest of an (IPFIX-based) network management stack, an information model defined by something other than regular expression, and a flexible, extensible record format.)

Best regards,

Brian

On Sep 21, 2010, at 12:19 AM, Peter Musgrave wrote:

> _______________________________________________
> sip-clf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sip-clf