Subject: [sip-clf] New CLF Syntax draft (text with index)

From: Adam Roach <[email-masked]>
Date: Thu, 07 May 2009 13:56:23 -0500

This version defines a text format in which each record is composed of 
two lines in a log file. The first line is primarily pointers into the 
second line. The second line contains the actual logged fields, 
separated by tab characters.

This approach retains the fast-search capabilities that I've been 
advocating, while allowing the use of simple, text-based tools, as Vijay 
has been promoting. This hybrid approach does come with a slight 
increase in log file size; for example, writing out the same 100,000 log 
entries in each of the three formats proposed so far:

   - Text:   25 Mb
   - Binary: 29 Mb
   - Hybrid: 37 Mb

This is about a 20% premium over the binary format, and a 48% premium 
over the text format. Speed of generation should be on the same order of 
speed as the other two versions, and speed of processing should be 
approximately the same as processing binary.

The new version of the document can be found here:

http://tools.ietf.org/html/draft-roach-sipping-clf-syntax-01

/a

---

From: Simon Perreault <[email-masked]>
Date: Thu, 07 May 2009 15:16:55 -0400

> approximately the same as processing binary.

Iteresting! Thanks for writing this.

Why did you include pointers as well as lengths? It seems to me that length is
redundant, since you can deduce it by computing the difference between two
pointers (still an O(1) operation).

Thanks,
Simon
-- 
STUN/TURN server    --> http://numb.viagenie.ca
Interplanetary news --> http://reeves.viagenie.ca
vCard 4.0           --> http://www.vcarddav.org

---

From: Adam Roach <[email-masked]>
Date: Thu, 07 May 2009 14:20:17 -0500

> pointers (still an O(1) operation).

I considered doing it both ways, but decided that I preferred the slight 
space impact of being able to get to any fixed field by looking at only 
two numbers instead of all the preceding fields. I'm open to doing just 
lengths if people are very sensitive to the size of each record, but 
have a personal preference for what's in the draft currently.

/a

---

From: Simon Perreault <[email-masked]>
Date: Thu, 07 May 2009 15:23:59 -0400

> have a personal preference for what's in the draft currently.

I'm sorry, I don't understand. Why would one need to look at all the preceding
fields?

Let's say you want to look at field 3. You look up the third pointer (p3), this
gives you the start of the field. Then you look up the fourth pointer (p4), this
give you the end of the field. If you want the length you compute p4 - p3.

Is it because of a possible overflow in the pointers? They're only 16-bit big,
while each field's length is also 16-bit big...

Thanks,
Simon

---

From: Adam Roach <[email-masked]>
Date: Thu, 07 May 2009 14:29:55 -0500

> give you the end of the field. If you want the length you compute p4 - p3.

Ah, yes! That's a good point. I had considered encoding *lengths* only, 
which would require adding all the preceding fields together (and I had 
mis-read your suggestion to mean lengths instead of pointers). You are, 
of course, correct -- encoding only pointers gets me what I need, while 
cutting the index size in half. Thanks! I'll try to get a revised draft 
out reflecting this change sometime soon.

/a

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Fri, 08 May 2009 08:51:08 -0500

> separated by tab characters.

Adam: Thanks for getting this out.  I will read it in more
depth, but a couple of quick questions as I was looking at
the format.

1) Any special reason why the flags field is separated from
the record length field by a comma?  And the same for the
record length field being separated by a comma from the
server transaction pointer field.  Eliminating them would
save 2-bytes per record.

2) I am not sure I follow the 0x0A in byte 80 -- is it the
length of the Date/Time field (i.e., upto the period in
byte 91)?

3) What about 0x09 in byte 98 and 0x09 in byte 109?  What
are they used for?

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
Web:   http://ect.bell-labs.com/who/vkg/

---

From: Adam Roach <[email-masked]>
Date: Fri, 08 May 2009 10:25:52 -0500

Vijay K. Gurbani wrote:
> server transaction pointer field.

I actually had started out with all the fields on the first line 
separated by commas, but removed the ones between the indices to save 
space. I have a feeling that breaking the total record length out with a 
delimiter will serve us well in the future if we ever decide to extend 
the format, but don't have any concrete examples. I'll ruminate on this 
longer.

> length of the Date/Time field (i.e., upto the period in

No; it's an ASCII line feed.

> 3) What about 0x09 in byte 98 and 0x09 in byte 109?  What

Those are ASCII "tab" characters.

Really, I think the fastest way to get the gestalt of what I'm proposing 
is to uudecode the log message at the bottom of the draft and examine it.

/a

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Fri, 08 May 2009 10:49:30 -0500

> the format, but don't have any concrete examples. I'll ruminate on this 

OK.

> No; it's an ASCII line feed.
[...]
> Those are ASCII "tab" characters.

Ah, OK.

> is to uudecode the log message at the bottom of the draft and examine it.

OK, I just did that and I see your intent.

Thanks,

- vijay