Subject: [sip-clf] Large log elements

From: Peter Musgrave <[email-masked]>
Date: Wed, 4 Aug 2010 07:51:07 -0400

Hi all,

In the Maastrichct session the issue of large headers came up.

The example specifically cited by Robert Sparks was a request URI
which could have a body in it - so even in the case of mandatory
logged elements there can be very large entries.

Do we wish to dictate a maximum element size for the purposes of
logging? Should this be in the problem statement?

Regards,

Peter Musgrave
(as co-chair)

---

From: Brian Trammell <[email-masked]>
Date: Thu, 5 Aug 2010 09:51:20 +0200

Hi, Peter, all, 

If we adopt limits, I would say they should appear in the problem statement.

Personally (speaking as a one-time sysadmin rather than as a draft author), I think a maximum element size is a good thing. It decreases the the complexity of securing the log facility against resource exhaustion attacks, and limit the scope for bounds-related security issues in implementations. 

It would appear, however, that SIP was not designed with such considerations in mind. Okay. I'm not a SIP guy, so I probably missed some draft where this is explained and where it makes perfect sense. But the Request URI case is a puzzling one to me. URIs are identifiers for things, not things in themselves, and embedding a body seems, well, wrong. Most applications using URIs and URLs limit them to some sane identifier length; for the most part they put sub-64k limits on these. Even Apache falls over at 4k. (see http://www.boutell.com/newfaq/misc/urllength.html)

So if it is an _actual_ requirement of this WG to log a _full_ (i.e., non-truncated) request URI that contains an embedded body, well, then, we have to handle that, and we cannot impose limits. I remain unconvinced that this is an essential requirement. And we still have to answer the same question for non-truncated bodies.

More generally, there is one use case where limits on SIP-CLF are a bad idea: debug dumps, instead of logs. There is in my opinion a fundamental difference between a log and a dump. Back to the CLF analogy, you _could_ configure (or hack) Apache to dump the entire HTTP transaction to its log, but I can't imagine what nominal logging_purpose this would serve. Dumping "i'm giving up, here you debug this" type info inline to a log increases the implementation complexity of reading the log in the general case, regardless of the representation used. This is why Apache writes multiple log files by default: one for normal activity, one for errors.  

By another analogy, this time to Snort... Another way to handle logging and dumping of  multiple granularities of detail to have the top-level log in one file, and dumping the full data to per-transaction or other per-dimension logs. Snort by default dumps packet traces per source, but keeps the alert log at the summary level. Admittedly, standardizing a layout like this is probably a bad idea (now you're talking about directory structures and filenames in an Internet-Draft... no, thanks...) But the approach may be of some inspirational use.

Speaking again as a draft author: fortunately both considered representations can handle both limited "log" data and unlimited "dump" data, though both do dumps at the expense of some of their elegance (IPFIX needs continuation records, which we are detailing in the next rev of our draft; indexed ASCII loses a good deal of its "textiness"). 

An alternate approach to dumping would be to dump data raw (much nicer from an implementation standpoint than using either IPFIX or indexed ASCII: when you're panicking, it's nice to be able just to blit a chunk of memory to an already-open file descriptor) then reference that with a (length limited :) ) URI stored in the log.

Regards,

Brian

On Aug 4, 2010, at 1:51 PM, Peter Musgrave wrote:

> _______________________________________________
> sip-clf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sip-clf

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Fri, 13 Aug 2010 08:34:49 -0500

> logging? Should this be in the problem statement?

Peter: There was, IIRC, an understanding in Maastricht that
logging an entire body is perhaps not in the spirit of SIPCLF.
I agree to this, and for folks who want to log bodies to
observe codecs, etc. can always use the extension portion of
a SIP CLF representation format to log parts of the body they
are interested in.

The following is a list of mandatory fields in the SIPCLF
where a R-URI appears: R-URI, From, To.  Of these, the
 From and To generally do not contain great many parameters;
a R-URI can.  Bounding the size of the URI seems reasonable,
especially if we agree that a R-URI parameter conveying a
body is not that interesting from a CLF perspective.

The next question, then, is how large should the bounds be?
Brian noted that the Apache server's limit is 4K.  We
could use that length as a maximum for logging URIs.

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
Web:   http://ect.bell-labs.com/who/vkg/

---

From: Peter Musgrave <[email-masked]>
Date: Fri, 13 Aug 2010 09:40:44 -0400

Hi Vijay,

I agree we can leave the "large body problem" to the representation format.

I also agree that setting a 4K limit for mandatory headers is
pragmatic. If people really need a full logging of e.g. R-URI then
would they need to have "full R-URI" as an extension field in the
representation format?

[Over lunch we did dream up one possible use of bodies in R-URIs.
There may come a day when people include location info in the R-URI
and that could exceed the size of more typical R-URIs. I agree it's
not in the wild today - but in the cause of future proofing the drafts
it might be a case to bear in mind]

Thx

Peter Musgrave
(as individual)

---

From: Hadriel Kaplan <[email-masked]>
Date: Wed, 18 Aug 2010 19:57:55 -0400

I also think the message body should be left out of the SIPCLF log entry.  This isn't a replacement for wireshark - yeah, sipclf is for helping to "troubleshoot" (as well as other things), but not really at a wireshark level - more like help to detect when there is a problem, rather than what the root cause of it was.  Once you know you have a problem and what peer/remote-device you're having the problem with, you will go do wireshark or something similar anyway, no?

Even 4k per field seems excessive to me - I'm not letting someone fill up my files and consume resources by sending long URIs of junk - I only need to record enough to show it's junk and more importantly who's doing it. 

-hadriel

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Peter Musgrave
> Sent: Friday, August 13, 2010 9:41 AM
>

---

From: Peter Musgrave <[email-masked]>
Date: Thu, 19 Aug 2010 06:14:51 -0400

Hi Hadriel,

I see you point about logging bodies and limiting header lengths. How
would 1K feel as a truncation point for long headers?

As for the wireshark comment - there would be cases in which *if* the
body was SDP then having it in the log would remove the need for me to
go wireshark things in many cases.

How do you feel about (optionally) allowing the logging only of SDP
bodies? (including SDP when it is multi-part bodies) ? In this case a
truncation of 10K might be reasonable?

Peter Musgrave
(as individual)

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Thu, 19 Aug 2010 08:27:22 -0500

> truncation of 10K might be reasonable?

The SIPCLF logging procedure should not impose undue
computation on the element generating the record.  Thus,
my personal opinion is that we should stay away from any
special rules that will require a processing entity to
look at certain headers and make decision on whether or
not to log the required information.

Thanks,

- vijay

---

From: Mary Barnes <[email-masked]>
Date: Thu, 19 Aug 2010 08:55:26 -0500

I totally agree with Hadriel.

Mary.

On Wed, Aug 18, 2010 at 6:57 PM, Hadriel Kaplan <[email-masked]>wrote:

>
> I also think the message body should be left out of the SIPCLF log entry.
>  This isn't a replacement for wireshark - yeah, sipclf is for helping to
> "troubleshoot" (as well as other things), but not really at a wireshark
> level - more like help to detect when there is a problem, rather than what
> the root cause of it was.  Once you know you have a problem and what
> peer/remote-device you're having the problem with, you will go do wireshark
> or something similar anyway, no?
>
> Even 4k per field seems excessive to me - I'm not letting someone fill up
> my files and consume resources by sending long URIs of junk - I only need to
> record enough to show it's junk and more importantly who's doing it.
>
> -hadriel
>

---

From: Peter Musgrave <[email-masked]>
Date: Thu, 19 Aug 2010 17:20:55 -0400

[as individual]
Ok - I can see the point about avoiding bodies. I assume that any
bodies in e.g. reqUri header would just be left alone...so header
inspection is not required.

[as chair]
Can I please call for a sense of the list about whether making the
problem statement say body logging is not supported in sipclf is
acceptable?

Thanks,

Peter Musgrave

---

From: Gonzalo Salgueiro <[email-masked]>
Date: Thu, 19 Aug 2010 20:33:52 -0400

Peter -

I don't see much value in logging the message bodies. I believe the individual implementation drafts should clearly state that logging bodies is out of scope. For the sake of completeness, I do think it might be helpful for the different implementations to optionally allow for it (i.e. a flag setting, etc.), but obviously have that function disabled by default.

Regards,

Gonzalo

---

From: Peter Musgrave <[email-masked]>
Date: Thu, 19 Aug 2010 21:22:19 -0400

Hi,

If the consensus is that logging bodies is undesirable then I would
suggest that the problem statement make that clear (so it does not
depend on the implementation draft).

I have not heard any supporters for logging bodies....(except briefly
me before I saw the error of my ways).

Peter Musgrave
(as individual)

---

From: Dan Burnett <[email-masked]>
Date: Mon, 30 Aug 2010 17:23:05 -0400

I agree.  I do particularly like your suggestion that implementations  
can allow for it, meaning that we need to not *prohibit* it.

-- dan

On Aug 19, 2010, at 8:33 PM, Gonzalo Salgueiro wrote:

> Peter -
>
> I don't see much value in logging the message bodies. I believe the  
> individual implementation drafts should clearly state that logging  
> bodies is out of scope. For the sake of completeness, I do think it  
> might be helpful for the different implementations to optionally  
> allow for it (i.e. a flag setting, etc.), but obviously have that  
> function disabled by default.
>
> Regards,
>
> Gonzalo
>
>
>> On Thu, Aug 19, 2010 at 9:55 AM, Mary Barnes <[email-masked] 
>> > wrote:
>>> On Wed, Aug 18, 2010 at 6:57 PM, Hadriel Kaplan <[email-masked] 
>>> >
>>> wrote:
>>>>
>>>>> From: [email-masked] [mailto:[email-masked]]  
>>>>> On
>>>>> Behalf
>>>>> Of Peter Musgrave
>>>>> I agree we can leave the "large body problem" to the  
>>>>> representation
>>>>> format.
>>>>>