Subject: [Sipping] SIP CLF Format

From: "Vijay K. Gurbani" <[email-masked]>
Date: Mon, 02 Feb 2009 13:06:39 -0600

Hi: On behalf of my co-authors, I have just submitted a draft
on SIP Common Log File format, the same CLF that has been
successful in HTTP.

It would be great to get some comments on the draft.  It is a
short draft, though a little intricate as it tries to explain
the CLF format through examples.  It is important to point out
that a SIP CLF does not replace a call detail record, it's use
is orthogonal to billing (and this is explained much further in
the draft.)

The text version is available in the IETF archives at:
http://www.ietf.org/internet-drafts/draft-gurbani-sipping-clf-00.txt

If you prefer the HTML version, you can get it from:
http://ect.bell-labs.com/who/vkg/IETF/I-D/draft-gurbani-sipping-clf-00.html

Any feedback would be welcome.

Thank you for your time.

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
Web:   http://ect.bell-labs.com/who/vkg/
_______________________________________________
Sipping mailing list  https://www.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: "Dale Worley" <[email-masked]>
Date: Mon, 02 Feb 2009 15:00:17 -0500

> Any feedback would be welcome.

I like the concept, but there are a lot of details that aren't
specified.  On the first read, I see:

- What separates fields?  It appears that a single space is intended,
but I don't see that specified.

- date (%d) - should read something like: "Date/time of the request,
specified as the decimal representation of the integral number of
seconds since the Unix epoch."  (This representation is time-zone
independent.)

- authuser (%u) - This specifies the user but not the realm used in
authentication.  Is it safe to assume the realm is always known from the
context?

- method (%m) - Presumably this is as appears in the request and is
*not* necessarily upcased.  But it would be wise to specify the casing
rules.

- from (%f) - Stated as "The From URI, including the tag."  But the tag
is not part of the From *URI* and cannot possibly be part of the From
URI.  Do you mean the From name-addr, the full value of the field?  But
in that case, spaces can appear in the field, messing up the separation
of fields.  Or pragmatically, should it be a synthesized name-addr:
"<[From-URI]>;tag=[from-tag]"?

- contactlist (%c) - This is stated to contain only the Contact URIs,
but not their field-parameters, which could be of great interest e.g. in
regard to REGISTER responses.  But giving the entire name-addr leads to
the same problems as with the %f field.

- association codes (%x and %y) - What is the permitted syntax?  Any
non-empty string of non-space printable characters?

- What exactly are the events that are logged?  E.g., section 4.1.2
refers to a "method event", which is a phrase I've never heard before.
Does it refer to SIP requests?  In which case, does it mean the
*reception* of a SIP request or the *sending* of a SIP request?  Or is
the intention to log the request/response pair as one line (as would
make sense in an HTTP server, but less sense in a SIP proxy that
implements forking)?

- In regard to the remotehost (%h) field, does "upstream" mean (as it
usually does) "the source if the request that initiated this
processing"?  In which case, there seems to be no place to identify the
host/address to which an outgoing request is sent, or the host/address
from which an incoming response is received (since that is a
"downstream" host).

- Some care needs to be taken to describe how CANCEL and ACK are logged.
(E.g., ACK for success and ACK for failure *might* be logged differently
-- what is intended?)  This need is discussed in section 1, but I don't
see clear prescriptions of what is to be done.

- Are the "..." around %c really necessary?  Or is that a legacy of the
Apache format?  (And if so, does the Apache format allow %c to contain
spaces?  How does that affect the identification of fields?)

- What are the extension mechanisms (other than additional fields at the
ends of the two line formats)?  Is there any way to identify which
software version generated a particular log file, so that the reader can
unambiguously determine how to interpret the extension fields?

- Why is the to-URI (%t) present in the response format?  It's redundant
with the same information in the corresponding request format.  And if
we want such redundant information, why not provide %f as well?

- In the examples, a number of spaces between fields seem to have been
omitted.  (Remember, within <allOneLine>, line-break is removed and does
not represent a space, only spaces at the beginning of a line represent
spaces.)  E.g., this:

<allOneLine>
1230756550 192.168.1.2 - REGISTER sip:example.com
sip:[email-masked];tag=iu8u76 sip:[email-masked];tag=8hy
[email-masked] 401 - - -
</allOneLine>

is probably intended to be:

<allOneLine>
1230756550 192.168.1.2 - REGISTER sip:example.com
 sip:[email-masked];tag=iu8u76 sip:[email-masked];tag=8hy
[email-masked] 401 - - -
</allOneLine>

Dale

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Mon, 02 Feb 2009 16:39:16 -0600

Dale Worley wrote:
> I like the concept,

Dale: I am glad; thanks for a quick turnaround.

> but there are a lot of details that aren't specified.

You are, no doubt, right.  Our intent in -00 is to present
the idea to the WG to stimulate discussions.  However,
you do bring some salient points below; please see inline.

>  but I don't see that specified.

Ah, good catch.  It'll be added in S4.1.

> seconds since the Unix epoch."  (This representation is time-zone 

Sure; I think it should be okay to take UTC out.

> authentication.  Is it safe to assume the realm is always known from 
> the context?

Ah, another good catch.  I do not believe that the HTTP CLF
saves the realm in its log file.  However, that does not mean
that we cannot do better.

> *not* necessarily upcased.  But it would be wise to specify the 
> casing rules.

Sure.

> - from (%f) - Stated as "The From URI, including the tag."  But the 
> tag is not part of the From *URI* and cannot possibly be part of the 
> From URI.  Do you mean the From name-addr, the full value of the 
> field?  But in that case, spaces can appear in the field, messing up 
> the separation of fields.  Or pragmatically, should it be a 
> synthesized name-addr: "<[From-URI]>;tag=[from-tag]"?

We could possibly be more judicious here and go with addr-spec.
Furthermore, since the tag is not a URI parameter, the intent
is as you write above.

>  but not their field-parameters, which could be of great interest 
> e.g. in regard to REGISTER responses.  But giving the entire 
> name-addr leads to the same problems as with the %f field.

We could possibly go with addr-spec here as well and enclose the
entire %c in " and ".

>  non-empty string of non-space printable characters?

The syntax is tangentially explained in S4.2.  At this time,
I do not have a ABNF representation of these new fields.

> refers to a "method event", which is a phrase I've never heard 
> before. Does it refer to SIP requests?

Ah, yes; another good catch.  Method event refers to SIP requests.
I will make this more clear in a subsequent revision.

> In which case, does it mean the *reception* of a SIP request or the
> *sending* of a SIP request? Or is the intention to log the
> request/response pair as one line (as would make sense in an HTTP
> server, but less sense in a SIP proxy that implements forking)?

For SIP proxies you cannot log the request/response pair in one
line.  That is where S4.2 comes in.  The directives in S4.2
allow the representation of a method arriving at a proxy as
well as departing from the proxy (or B2BUA).

> processing"?  

Yes.

> In which case, there seems to be no place to identify 
> the host/address to which an outgoing request is sent, or the 
> host/address from which an incoming response is received (since that 
> is a "downstream" host).

That is captured in S4.2.  The entity to which a request is
sent downstream is identified by the R-URI (see top of page
13.)  I agree this is a bit hard to follow until you sit down
and work it out, but that is the nature of the complexity.
Any suggestions to make it less complex would be greatly
welcome.

> - Some care needs to be taken to describe how CANCEL and ACK are 
> logged. (E.g., ACK for success and ACK for failure *might* be logged 
> differently -- what is intended?)  This need is discussed in section 
> 1, but I don't see clear prescriptions of what is to be done.

There is an example in S4.1.4; CANCEL is discussed in detail
there.  Regarding ACK for 2xx and ACK for non-2xx, I believe
that they ought to be logged similarly but the interpretation of
them is different (i.e., the automata reading the CLF file will
trigger different states based on whether or not the ACK is
for a 2xx or non-2xx.)

> - Are the "..." around %c really necessary?  Or is that a legacy of 
> the Apache format?  (And if so, does the Apache format allow %c to 
> contain spaces?  How does that affect the identification of fields?)

Apache, of course, does not have a "%c"; it does, however, have
a "%r", which represents the request line as it comes in from
a browser.  The Apache CLF encloses this in a pair of braces
since the "%r" represents three different tokens in Apache:
the method (GET), the resource (/index.html) and the protocol
version ("HTTP/1.0").

"%c" for SIP CLF was put in braces just to include any LWS (and
I will probably have to look at this in more detail as the work
moved ahead.)

> - What are the extension mechanisms (other than additional fields at 
> the ends of the two line formats)?  Is there any way to identify 
> which software version generated a particular log file, so that the 
> reader can unambiguously determine how to interpret the extension 
> fields?

Excellent question.  There isn't a versioning system for the
SIP CLF itself; however, the extension mechanism is rather
simple: you can include any other field you want besides the
fields that constitute the CLF.  We do not restrict this,
subject to the discussion in S5 (Security Considerations.)
Of course, what we want to standardize are the fields we
have listed and in the order that we have listed them in.
These constitute the canonical definition of a "SIP CLF".
Implementations are free to add other fields as long as they
provide/maintain updated SIP CLF parsers.

> - Why is the to-URI (%t) present in the response format?  It's 
> redundant with the same information in the corresponding request 
> format.  And if we want such redundant information, why not provide 
> %f as well?

There is a discussion of this in S4.1.3.  Should we embellish
the discussion in there some more?

> - In the examples, a number of spaces between fields seem to have 
> been omitted.  (Remember, within <allOneLine>, line-break is removed 
> and does not represent a space, only spaces at the beginning of a 
> line represent spaces.) [...]

Ah, yes, of course.  I will correct this in the next revision.

Thanks a LOT for your comments, Dale.  I am glad you liked the
concept of the work.  The details, of course, need to be fleshed
out.

- vijay

---

From: Hadriel Kaplan <[email-masked]>
Date: Mon, 16 Feb 2009 03:48:49 -0500

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Vijay K. Gurbani
> Sent: Thursday, February 05, 2009 5:56 PM
> >
> > Then what is its purpose?
>
> It's purpose is much the same as the Apache CLF format: it is
> not used as much to debug as it is used for trend analysis and
> anomaly detection.

I really like the idea of having a common log file format for third party tools to rely on, but even if it's just for trend analysis and anomaly detection you'd need a much larger list of fields if this is for anything but registrations.  For example you'd need fields for localhost, p-asserted-id/preferred-id, local-reject-reason, diversion/hist-info, and probably some others I'm forgetting.

And that really gets to the problem with these things - everyone wants something more/different.  For example someone would probably want to see the Via/Record-Route list too.  And that's not even covering the SDP side.

Do you think the particular set you defined is necessary and sufficient?  Or is this extensible somehow?

-hadriel

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Mon, 16 Feb 2009 09:00:09 -0600

Hadriel Kaplan wrote:
> I really like the idea of having a common log file format for third
> party tools to rely on, but even if it's just for trend analysis and

Thanks, Hadriel; much appreciate your thoughts.

> anomaly detection you'd need a much larger list of fields if this is
> for anything but registrations.  For example you'd need fields for
> localhost, p-asserted-id/preferred-id, local-reject-reason,
> diversion/hist-info, and probably some others I'm forgetting.
> 
> And that really gets to the problem with these things - everyone
> wants something more/different.  For example someone would probably
> want to see the Via/Record-Route list too.  And that's not even
> covering the SDP side.
> 
> Do you think the particular set you defined is necessary and
> sufficient?  Or is this extensible somehow?

We have tried to apply Occam's razor to the problem set to derive
what we think are the minimal headers that allow one to
model SIP client/server state transitions and in the process
provide some useful information for trend analysis and such.

We do discuss the extensibility aspect; it is buried in
the last sentence of S4.1.  Dale has argued for a more
structured approach to extensibility such that any
pre-existing CLF parsers will not choke on new fields
added an so on; the next revision will contain some
more discussion on this.

However, the set of fields described in the draft now are
considered immutable for the purpose of the "SIP CLF"
moniker.

Thanks,

- vijay