Subject: Re: [sip-clf] WGLC: SIPCLF Problem Statement(draft-gurbani-sipclf-problem-statement-01)

From: "David Harrington" <[email-masked]>
Date: Tue, 2 Feb 2010 10:47:15 -0500

Hi,

I think the problem statement should be adopted as a WG item.
I agree with Cullen that it is a good starting place for a WG problem
statement doc. 

I definitely do NOT agree this is ready for WGLC.

There are multiple use cases that lead to different requirements. Some
people want to correlate flows of conversation; others only want to
pay attention to the data dumped by one server. I think there are lots
of conflicting requirements.

This problem statement document lacks any analysis of existing
protocols
that might address the different problems people seek to address.
Everything is just munged together into one problem statement, as if
everybody was in agreement about a single problem to be solved, and
the single solution that should solve all those problems.

Different use cases can already be dealt with, to varying degrees,
using existing standards, such as syslog and ipfix. 
I am very concerned about developing a logging standard for one single
protocol. Following this approach could easily lead to standards for a
MPLS CLF and a PCE CLF and a XYZ-protocol CLF. IETF is a standards
organization and we should be striving to reduce the number of
protocols and data modeling langauges that need to be learned by
operators 
in order to run their networks, not developing a totally new standard
for every use case. 

I disagree with the following analysis in the document:
   It can be argued
   that a good part of the success of Apache has been its CLF because
it
   allowed third parties to produce tools that analyzed the data and
   generated traffic reports and trends.  The Apache CLF has been so
   successful that not only did it become the de-facto standard in
   producing logging data for web servers, but also many commercial
web
   servers can be configured to produce logs in this format. 

I think Apache was successful because it was widely available,
strongly supported in the open source community, and free. The Apache
CLF became the defacto standard for web server logging because Apache
became the defacto standard for web servers. If Apache had a high
price tag, was not widely available, and was not supported by the open
source community, do you still think the Apache CLF would have become
the defacto standard for web server logging?

I think claiming Apache was successful because it supported a common
log format would be akin to claiming that UNIX was successful because
it supported a common logging format (syslog).
 
I think the document is claiming the wide deployment of the Apache CLF
as a reason to model a solution on that approach. If wide deployemnt
is the justification for choosing a solution, then we really should
also be considering other "common logging formats" that are already
used widely, such as syslog.

This WG needs to really understand the **multiple** problems that
people are trying to solve, and the different requirements these
different use cases place on the solution space. The abstract talks
about problems such as mining the log files to produce reports and
trends, training anomaly detection systems and feeding events into a
security event management system. But nowhere in this document is
there an analysis of the requirements that must be met to address
producing reports and trends, training anomaly detection systems, or
feeding events into a security event management system, such as
aggregation and correlation. 

I think the section on "Alternative approaches to SIP CLF" is woefully
inadquate. Existing IETF standards, such as syslog, ipfix, and SNMP
are already being used to address some of the problems WG contributors
have stated they want to address. The data from these existing
standards is already being widely used to produce reports and trends,
train anomaly detection systems, and feed events into a security event
management system. These existing standards already support a wide
array of technologies, and have been developed to be extensible by
defining new data models for new targeted technologies. These existing
standards are already often used in combination to correlate events
reported by different mechanisms to help pinpoint the cause of
reported problmes, to better analyze trends, to reduce false positives
in intrusion detection systems, and to recognize potential security
breaches. Yet this document does not even **mention** these existing
IETF standards as possible alternative approaches.

I support adoption of the document as a working group item, as a
starting place for a WG problem statement doc, but I do not think the
WG's work is done yet. This document is NOT ready for WGLC.

David Harrington
[email-masked]
[email-masked]
[email-masked]

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Spencer Dawkins
> Sent: Monday, January 18, 2010 1:44 PM
> To: SIP-CLF Mailing List
> Subject: [sip-clf] WGLC: SIPCLF Problem 
> Statement(draft-gurbani-sipclf-problem-statement-01)
> 
> Just to let people know, Theo and I agree that this draft 
> should be adopted 
> as a working group draft. After a quick once-over, I think 
> it's solid enough 
> to start WGLC now.
> 
> WGLC will end on 2010/02/05, which gives nearly three weeks 
> for people to 
> post and discuss comments on the mailing list.
> 
> Once WGLC is completed, I will ask Vijay to post a 
> draft-ietf-sipclf 00 
> version addressing WGLC comments. I hope to be able to 
> request publication 
> for that version of the draft-ietf-sipclf version, so please 
> don't be shy 
> about sending comments on the individual draft currently posted (at 
>
http://www.ietf.org/id/draft-gurbani-sipclf-problem-statement-01.txt).
> 
> Thanks,
> 
> Spencer, as co-chair
> 
> 
> > Folks: I have updated the SIPCLF problem statement draft
> > pursuant to the discussion we had in Hiroshima.  Sorry for
> > the delay in getting this out.
> >
> > By far, the biggest discussion point in Hiroshima for the
> > problem statement was to provide reasons for what we have
> > today (CDR, Wireshark) and why it does not work.  To that extent,
> > I have done the following:
> >
> > 1) I have added a new Section 5 (Alternative approaches to SIP
> >  CLF);
> > 2) I have added an explicit section that contains the problem
> >  statement (newly added Section 3.)
> > 3) I have taken out the old S7 (Relationship to other
> >  protocols.)  In Hiroshima, we discarded syslog; no one spoke
> >  for or against IDMEF.
> >
> >  The only other protocol that is being considered right now
> >  is IPFIX.
> >
> > Another point of discussion in Hiroshima was that some manner
> > of correlation to trace a call end-to-end is needed and should
> > be supported by the SIP CLF mechanism.  As far as SIP CLF
> > supporting such a mechanism is concerned, it will do so since
> > it is designed to extensible (and in fact, in S6, one of the
> > use cases --- the third one --- is exactly this one.)  As to
> > the problem statement draft or the solutions draft specifying
> > such a correlation header, my understanding is that this is
> > out of scope for the working group.
> >
> > Certainly the solutions exist in form of Hadriel's session
> > correlation header that can be used or some ad-hoc means
> > between two enterprises to correlate calls [1].  But whether
> > or not the problem statement or the solutions draft should spell
> > out explicitly how to standardize this correlation primitive
> > and perform this correlation, I believe that is currently out
> > of scope.
> >
> > The URL for the -01 version is:
> >
http://www.ietf.org/id/draft-gurbani-sipclf-problem-statement-01.txt
> >
> > The IETF tools page should be updated shortly so you can perform
> > a diff against -00 (until then, use the following tiny URL:
> > http://tinyurl.com/00-01-diff).
> >
> > I suspect that at least one more revision will be required, but
> > I promise to get the revision out as soon as I have some
> > sizable review comments in.  Please do review the -01 revision
> > and post your comments.
> >
> > [1] Adam Roach mentioned in Hiroshima at the mic that even
> >  without an explicit correlation header, one could correlate
> >  calls simply by noting that the request left my network with a
> >  Call-ID of X and shows up at your network with a Call-ID of Y,
> >  etc.
> >
> > Thank you,
> >
> > - vijay
> > -- 
> > Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
> > 1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
> > Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
> > Web:   http://ect.bell-labs.com/who/vkg/
> > _______________________________________________
> > sip-clf mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/sip-clf
> > 
> 
> _______________________________________________
> sip-clf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sip-clf
>

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 2 Feb 2010 10:54:25 -0600

Hi, Dave,

Thanks for the comments.

Just a point of clarification - my purpose in starting WGLC was to encourage 
people to see whether this draft is ready for publication, because we 
haven't gotten a lot of post-Hiroshima feedback.

In my opinion, we either work on documents or forward them for publication 
("hatching" documents by sitting on them for extended periods of time isn't 
actually part of the workflow).

I'm reading most of your comments as things we should consider before 
requesting publication of this document. I can agree with that.

Thanks,

Spencer, as co-chair, who's looking for other people to express opinions on 
the draft...

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Tue, 02 Feb 2010 14:58:51 -0600

David Harrington wrote:
> I think the problem statement should be adopted as a WG item. I agree
> with Cullen that it is a good starting place for a WG problem 
> statement doc.

David: Thank you for your comments.

> I definitely do NOT agree this is ready for WGLC.

IMHO, the differences are not that great to hold up WGLC.
More inline -- let's see if we can iron them out.

> There are multiple use cases that lead to different requirements.
> Some people want to correlate flows of conversation; others only want
> to pay attention to the data dumped by one server. I think there are
> lots of conflicting requirements.

At the highest level there are two requirements:

1) Correlate transaction information produced by a single
  SIP entity.

  The correlation here is to figure out which responses
  correspond to which requests, and furthermore enunciate
  the number of branches that an incoming INVITE was
  forked to.

  This correlation will also allow one to recreate SIP dialog
  information (for those in the list not steeped in SIP lore,
  a SIP dialog is an end-to-end relationship between two peers.
  There are certain headers in the SIP field that constitute
  to create a dialog identifier.)

  This sort of correlation is what I had in mind when I initially
  started the work.  This correlation is done entirely on the
  CLF file produced by one SIP entity.

2) Correlate SIP calls across multiple SIP service providers.

  As the work progressed and was socialized, some list
  participants felt that it will be advantageous to trace a
  SIP call end-to-end, i.e., across different SIP domains.

  Because we are concerned with a SIP call across different
  operating domains, this type of correlation requires some
  more work, specifically a unique session ID that survives
  in signaling as the session request leaves on operating
  domain and enters another.

  Furthermore, this type of correlation cannot be done by using
  only the CLF file produced by one entity since that entity
  would only have information about the session in its own domain.
  One would need to correlate the logs of one domain with the
  other.  Clearly, such type of correlation is best treated
  by implicitly supporting it but explicitly leaving the exact
  mechanism on how to do this outside of the draft.

  Thus, coming up with a unique session identifier is out of scope
  for SIPCLF working group.  However, using such an identifier,
  if one exists now or later, is perfectly within scope and can
  be supported by keeping the logging format such that other
  fields can be added in later.

In either of these cases, I do not think it is fair to say
that use cases lead to different requirements.  As far as I
can see, requirements for both use cases are supported.

> protocols that might address the different problems people seek to
> address.

I strongly disagree.  Version -00 had a section entitled
"Relationship to other protocols" (c.f., Section 7,
http://tools.ietf.org/html/draft-gurbani-sipclf-problem-statement-00#section-7).

In that we discuss syslog, IDMEF, IPFIX and PCAP.  In
Hiroshima during the meeting we decided that syslog was
not a good fit (simply because of the number of messages
that would need to be logged.)  I believe that syslog is
a good solution if one wants to enunciate messages based
on severity, etc.  In SIP CLF, the intent is to log the
summary of *every* SIP message.  A SIP server can still
use syslog to send out critical messages for the NOC, but
using it as a substitute for CLF is probably overkill.

No one spoke up for IDMEF, and the proponent of PCAP
decided not to pursue it.  That left IPFIX, which has a
dedicated block of people looking at it.  Thus in version
-01, I did not put in a section analyzing other protocols
since it appeared to me that we had reached a decision on
what to do with syslog, IDMEF and IPFIX.

> the single solution that should solve all those problems.

My attempt at a problem statement is given in Section 3 of
the sipclf draft 
(http://tools.ietf.org/html/draft-gurbani-sipclf-problem-statement-01).
If this gives you the feeling that everything is munged together,
I will be happy to work with you on any text you can contribute
that makes this more transparent.

> I am very concerned about developing a logging standard for one
> single protocol. Following this approach could easily lead to
> standards for a MPLS CLF and a PCE CLF and a XYZ-protocol CLF.

MPLS and PCE are lower-level routing protocols.  SIP is an
application layer protocol.  I am unsure whether we can use the
same hammer across the entire IP stack.  The logging behavior
and requirements for MPLS and PCE are probably much different
than SIP.

> I disagree with the following analysis in the document: It can be
> argued that a good part of the success of Apache has been its CLF
> because it allowed third parties to produce tools that analyzed the
> data and generated traffic reports and trends.  The Apache CLF has
> been so successful that not only did it become the de-facto standard
> in producing logging data for web servers, but also many commercial 
> web servers can be configured to produce logs in this format.
[...]
> it supported a common logging format (syslog).

Fair enough.  Please allow me to refine my statement: I do
not claim that Apache was successful because of CLF, but rather
that the CLF model espoused by Apache has been a successful one --
one that should be emulated not shunned.

> different use cases place on the solution space.

I believe that Section 3 of the SIPCLF draft lays out the
problem statement, and I don't think there are multiple
problems being enunciated there.  Again, I am happy to refine
it based on your input.  However, I would request that you
sent me specific text that we can argue and wordsmith.

> The abstract talks about problems such as mining the log files to
> produce reports and trends, training anomaly detection systems and
> feeding events into a security event management system.

But those are not problems.  They are the benefits of having a
CLF.

> I think the section on "Alternative approaches to SIP CLF" is
> woefully inadquate.

During the Hiroshima IETF, we spent a lot of time talking about
CDR and how we should distinguish this work from CDR.  Specifically,
why existing solutions like CDR and Wireshark are inadequate.
That is exactly what the section now contains.  If there are
other reasons why you feel this section is inadequate, please
let me know.

> Existing IETF standards, such as syslog, ipfix, and SNMP are already
> being used to address some of the problems WG contributors have
> stated they want to address. [...] Yet this document does not even
> **mention** these existing IETF standards as possible alternative
> approaches.

I respectfully disagree.  We ruled out syslog and IDMEF after
version -00 and during the Hiroshima meeting.  An IPFIX-based
study is under way.  The current draft (-01) contains the
two remaining (quasi)-standards we talked about in Hiroshima:
CDRs and Wireshark format.  I think we have looked at both IETF
and non-IETF protocols.

Thanks,

- vijay

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 2 Feb 2010 15:33:26 -0600

Speaking as co-chair, this time...

>> everybody was in agreement about a single problem to be solved, and the 
>> single solution that should solve all those problems.
>

Where Spencer and Theo think we are, is that we're adopting the next version 
of this draft as a working group draft. We haven't seen anyone disagreeing 
with that.

At that point - which we might as well declare to be "now", because Vijay 
can send in "the next version" as draft-ietf-sipclf at any time, and Theo 
and I will approve it - the working group becomes responsible for the text 
in the working group draft.

That means that it's not up to Vijay and Dave to work this out, with 
everyone else cheering them on from the sidelines - please feel free to send 
text that the editor can incorporate, as we identify that we have consensus 
on it!

Thanks,

Spencer

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 3 Feb 2010 11:20:05 +0100

On this specific point, it does not look to me that the WG reached a
consensus on this decision, at least in what concerns syslog. IPFIX is
still analyzed you say. In any case I think that it would be good to
keep the analysis of the existing protocols session in the document even
after a decision is reached - describing shortly what in the respective
protocols matched and what did not match the requirements. 

Dan

> [mailto:[email-masked]] On Behalf Of Vijay K. Gurbani
> 

In that we discuss syslog, IDMEF, IPFIX and PCAP.  In Hiroshima during
the meeting we decided that syslog was not a good fit (simply because of
the number of messages that would need to be logged.)  I believe that
syslog is a good solution if one wants to enunciate messages based on
severity, etc.  In SIP CLF, the intent is to log the summary of *every*
SIP message.  A SIP server can still use syslog to send out critical
messages for the NOC, but using it as a substitute for CLF is probably
overkill.

No one spoke up for IDMEF, and the proponent of PCAP decided not to
pursue it.  That left IPFIX, which has a dedicated block of people
looking at it.  Thus in version -01, I did not put in a section
analyzing other protocols since it appeared to me that we had reached a
decision on what to do with syslog, IDMEF and IPFIX.

---

From: "David Harrington" <[email-masked]>
Date: Wed, 3 Feb 2010 11:15:11 -0500

> From: Spencer Dawkins [mailto:[email-masked]] 
[...] 

> text that the editor can incorporate, as we identify that we 

Let me be clear about my role here.
I am syslog wg chair, and know the syslog specs fairly well, but I am
not an operator, and I do not personally deploy syslog, and I do not
know the SIP specs intimately or deploy SIP. If I did, I would be an
active technical contributor to this group rather than just a
technical advisor.

My tendency is going to be to recommend using technologies I am aware
of that I think might fit the requirements. If you have a hammer,
every problem is a nail. As I read the problem-statement-01, my
impression is syslog and/or ipfix might fit well to meet certain
goals, however, it is highly possible that syslog and ipfix are WRONG
to solve the sipclf problem. 

As Technical Advisor - not as technical contributor - I recommend that
the WG consider syslog and ipfix seriously. My advice as Technical
Advisor is to do a real analysis about the applicability of existing
standards like syslog and ipfix to the various sip clf use cases. This
should not be just a cursory analysis; we spent probably less than one
minute talking about syslog at the last IETF meeting. The typical
response to ipfix has been "well, we don't really know much about
ipfix, so we don't think it's a good choice." Those are NOT real
analyses.

I am concerned about the rush attitude of this WG. Let's not rush to
produce a new standard, when existing standards could be applied
reasonably well. 

David Harrington
[email-masked]
[email-masked]
[email-masked]

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 3 Feb 2010 10:45:57 -0600

Hi, Dave,

<as-co-chair>

Thanks for clarifying your role - it's helpful.

</as-co-chair>

<as-participant>

On one specific point:
>
> As I read the problem-statement-01, my
> minute talking about syslog at the last IETF meeting.

I think the discussion lasted longer than a minute, but I suspect the longer 
time I'm remembering included people walking to the mike :D

So, my understanding of the pushback on SYSLOG in Hiroshima - and I think 
this is what Vijay was saying in his followup - was that SYSLOG entries 
become really important when Bad Things Happen(TM), and people weren't sure 
that producing a lot of SIPCLF SYSLOG entries, which might even be 
message-by-message for a large SIP proxy, that you needed to prune through 
in order to find more "interesting" SYSLOG entries like "file system full", 
was the right thing to do.

Since you are a smart guy about SYSLOG - is this a realistic concern for us 
to have? and, if the answer is "yes", does modern SYSLOG with structured 
data elements provide a way to either (1) quickly separate the SIPCLF SYSLOG 
entries from other "oh, no" SYSLOG entries, or (2) send SIPCLF log entries 
using the same protocol to two different destinations, one potentially 
containing massive amounts of SIPCLF log entries, and the other containing 
"everything else"?

I would really like to use one of the existing management frameworks (still 
speaking as a participant here), IFF it's the right thing to do.

</as-participant>

Thanks,

Spencer

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 3 Feb 2010 18:09:26 +0100

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Spencer Dawkins
> Sent: Wednesday, February 03, 2010 5:46 PM
> To: David Harrington; 'Vijay K. Gurbani'; [email-masked]
> Subject: Re: [sip-clf] WGLC: SIPCLFProblemStatement(draft-gurbani-
> sipclf-problem-statement-01)
> 
...
> containing massive amounts of SIPCLF log entries, and the other

This is actually done with ancient syslog in practice. Syslog provides a
so-called facility, which can be used as a very rough indication of the
message type. If you dig into the details, there is much that one can argue
about (and the syslog WG has done in length), but properly deployed this is a
very fast and efficient mechanism to separate different types of log sources.
Also, you can send log entries to as many destinations as you like. I myself
are doing this for over 15 years now, and others do it much longer.

A big problem was that traditional syslog mainly used UDP transport and thus
was not well-suited (to phrase it lightly) for important data (but see [1]
below). Nonetheless even financial institutions used (and continue to use)
traditional syslog with great success for audit data.

Modern syslog solves or at least considerably improves on traditional syslog,
by proving much better reliability, and enhanced filtering capabilities. It
should be noted that each of the large syslog implementations (syslog-ng,
rsyslog, WinSyslog, ...) provide very complex filtering rules, and do so for
year, as it is typical that a stream of syslog messages must be dispersed
into separate streams, for example, to log everything to an audit log, feed
specific events to a security console and feed some other events to a billing
system (in the service provider case). 

In my *personal* view, there are three main points that lead to the
overwhelmingly large deployments of syslog:

* ease of implementation (aka support for 
  lazy implementations ;))
* capability to create very flexible 
  filtering and message routing scenarios
* wide availability of components, both servers
  and clients (related to point 1 above ;)))

Please note that syslog has been implemented outside of the pure system
management space. It is being used at least for

* billing purposes (ISPs, other service providers)
* capturing audit data
* storing connection records for 
  legal purposes (at least in Germany)
* IHE (Healthcare) [1]

I am aware of at least one other larger body that expressed interest in using
syslog for application-level data that has *absolutely* nothing to do with IT
management (but cannot tell more about it due to NDA - just let me say that I
was really *surprised* that this was a use case for syslog).

So I think it would be useful to list the requirements, and see if syslog can
provide a solution or not.

Rainer

[1]
http://wiki.ihe.net/index.php?title=ATNA_Profile_FAQ#Why_does_ATNA_use_UDP_SY
SLOG.3F

---

From: "David Harrington" <[email-masked]>
Date: Wed, 3 Feb 2010 14:50:07 -0500

Hi,

For those unacquainted with syslog, let me try to summarize how it
works. I will focus on the forwarding and filtering and security
mechanisms, since I think those will be important considerations for
SIP logging.

Rainer is the guy I go to when I have a question about syslog. He has
answered Vijay's concern in considerable detail in another email. I
don't personally deploy syslog, so Rainer can feel free to correct
anything I get wrong. ;-)

I will recommend looking at the wikipedia entry, which has a summary
of its history and usage, but some factual details about the protocol
are about the limitations of BSD syslog, rather than the IETF standard
syslog. As technical advisor, I recommend the IETF standard syslog
because it overcomes many of the limitations and interoperability
issues with BSD syslog. The wikipedia entry mentions some work that is
being done, work that relates to anomaly detection and security
management systems.

The IETF has worked at standardizing syslog across operating systems,
and focused mainly on standardizing aspects of syslog that impacted
security and filtering and correlation (such as standardizing the
header, and more detailed info about the source and time of the
messages).

---- The forwarding mechanism ----

Syslog allows log information for both kernel processes and user
processes to be forwarded to a file, a set of users, or another node
in a network. 

IETF syslog provides support for UDP, TLS, and (soon) DTLS. It is
designed to allow additional transports to be defined, so could also
be used in non-IP environments, I suppose. 

This flexibility in the forwarding rules allows syslog to concentrate
on collecting data and putting it where the operator wants it so other
applications can focus on sorting, filtering, and alerting on the
contents. Forwarding to a collector elsewhere in the network can help
prevent attackers from modifying the local logs to cover their tracks,
and allows a centralized location for operators and tools to do log
analysis rather than having to go to each device to look at the logs.
But you can also store it locally if you want.

---- The filtering mechanism ----

syslog messages start with a <priority> byte that is calculated from
two factors - the facility that generated the message and the severity
of the message. This is the primary mechanism for sorting messages
that are logged.

Facilities define the source of a syslog message. There are 23
different facilities defined for use in the <priority field>, ranging
from "kernel" to system daemons to mail to security to local
applications. A SIP application would use one of the "local"
facilities to log messages. If there are multiple SIP applications,
they might use different "local" facilities to generate separate
logging streams for forwarding purposes, but whether this is done
depends on the implementations and operator configurations. Additional
header fields and the SDE feature of IETF syslog allow much finer
filtering during log analysis.

There are 7 levels of severity ranging from emergency to critical to
informational to debug messages. The IETF standard syslog severities
can be aligned with the ITU-T values of perceived severity (M.2100,
X.733, X.736), and with the SNMP values defined in the ALARM-MIB. This
helps operators correlate events reported by different applications or
by different mechanisms.

syslog.conf is the normal mechanism for configuring what to log and
how to distribute it. The filtering/forwarding instructions look
something like this:

mail.info	/var/log/maillog	-- send all informational
mail-related messages
                                 to the file /var/log/maillog
cron.*      /var/log/cron     -- send all messages from cron,
regardless of
                                 severity to the file /var/log/cron
security.*  @loghost.netlab.lander.edu
                              -- send all security related messages,
regardless
                                 of severity to a remote system
(loghost.netlab.lander.edu)
*.notice     root             -- send all notices, regardless of
source, to 
                                 the root user, if root is logged on.

and so on.

These are all configured administratively. Operators are usually very
acquainted with syslog as a tool. 

---- Post logging analysis ----

Other header values can also be used for analyzing logs, including
timestamp, hostname, application name, process ID, etc. Filtering
based on header values has been in wide use for a long time, although
the header format for different syslog implementations vary widely.
IETF syslog standardizes the order and format of header fields, but
still keeps roughly the same historic header fields. There are many
tools available for syslog log analysis.

The Structured Data Elements provide even more capability. This is
only available in the IETF standard syslog. My earlier message pointed
out some of the already-standardized SDEs defined by the WG that can
be used to filter log entries.

Hope this helps,
dbh

> Sent: Wednesday, February 03, 2010 11:46 AM
> Subject: Re: [sip-clf] WGLC: SIPCLF 
> ProblemStatement(draft-gurbani-sipclf-problem-statement-01)
> 
> Hi, Dave,
> 
> <as-co-chair>
>

---

From: "Rainer Gerhards" <[email-masked]>
Date: Wed, 3 Feb 2010 21:07:05 +0100

> Behalf Of David Harrington
> Sent: Wednesday, February 03, 2010 8:50 PM
> To: 'Spencer Dawkins'; 'Vijay K. Gurbani'; [email-masked]
> Subject: [sip-clf] A primer on syslog.
> 
> Hi,
> 

Thanks David, that is an excellent primer. Just one note... 

> processes to be forwarded to a file, a set of users, or another node

While this is beyond the scope of the IETF, current mainstream implementation
supports many more options. Popular options are writing to databases and
passing messages to another application. Rsyslog, for example, has an open
plug-in architecture and every now and then I hear about plugins written for
custom projects that I would never have envisioned.

> be used in non-IP environments, I suppose.

It definitely is. The most prominent non-IP "transport" is Unix local domain
sockets. But real non-IP transports are used quite seldom, now that almost
every device runs IP. In former days, we had log sources (PBXs mostly) that
sent messages via RS232 to a syslog relay which forwarded the messages via IP
to the final destination.

Rainer

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 3 Feb 2010 14:40:40 -0600

Hi, David,

Thanks for your work in familiarizing us with SYSLOG.

To David and Rainer,

Most of my SYSLOG exposure was with the BSD variant, not with the newer IETF 
variant, and most of what I knew about the newer IETF variant was about its 
transports (including TLS). Perhaps others in the working group have had 
similar exposures.

Spencer

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 3 Feb 2010 17:25:36 -0700

On Feb 3, 2010, at 12:50 PM, David Harrington wrote:

> There are 23
> facilities to log messages. 

Just out of curiosity, would it ever make sense for the IETF to designate an facilities value for SIP?

For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 3 Feb 2010 21:20:45 -0700

Some random comments - all sent in my individual contributor role.

I like this draft, think it is fine work, but it needs a few more things before it is done. These are all things that I think can help us speed up the actual clf work if we can agree on them.

Section 8 should be renamed to "Information Model"

remotehost does not seem like enough info when talking about multi interface machines like SBC with insides and outsides. I would prefer to replace this a source and destination address that contains the IP and port of both the source and destination of whatever was in the layer 3 transport protocol. 

I'd update the method and status to make it clear  that there must be exactly one of a request method name, or a response code 

I would separate the to and to-tag to two separate elements. You want to be able to match the to search the to without including the to-tag in a rapid way. I don' feel strongly about this and can live with them mixed together in one field if that what folks want. 

I'd remove the contact list. It's just a generic header like others. This is debatable so worth discussion but I'm in the "less not more" camp

server and client transaction both need to be optional. For example, a stateless proxy will not have this.

Explicitly state that there all other headers go in an order list of pairs of header field names and header field values

A field to hold one optional body

I don't think this information model need to be extensible. In fact the one thing I am going to continue to strongly argue is that it should not be extensible. Note this does not mean we can not have new sip headers or new body types, we can, they just go in the list of sip headers or body type. 

So to summarize, I think the following need to be the elements of the Information Model,

Layer 3 source IP/port
Layer 3 dest IP/port
Timestamp
server transaction
client transaction
From header field value
To header field value minus the to-tag
to-tag
callID header field value
ordered list of header field name value pairs
status or request line 
message body 

I could be convinced there was value in:

method
sip status code
contact header field value for certain types of messages 
syslog style severity level 

I'm against the needless complexity or arbitrary extensibility and would want to see a good argument made for adding things.  

I would add a data model consideration section where we talk about desirable properties of the data model. We don't pick the data model in this draft, that is the job of  actual spec draft, but we can put requirements or other points about the data model here. 

Specifically in this part I want to make sure we are compatible with the important parts of syslog such that if someone is writing a program to read one of these SIPCLF log files and transmit it over syslog, it is easy and there are no semantic level mapping problems. To do that that I would add to this section "Compatible with the syslog definitions of severity." 

I was considering also saying compatibility with syslog definition of timestamp but my recollection of that (which may be very wrong - been awhile since I looked at it) left me more than a bit confused about syslog tmestamps. It seems that every syslog packet I have ever seen in the wild uses the BSD Syslog timestamp format in RFC 3164 but clearly that is more or less borked for any real correlation across time zones so it seems like 5424 version of timestamp would be better. The 5424 allowed the local offsets which resulted in no real canonical format for time which mean some people implemented the full things and some ignored the local offsets results in interoperability problems. Then there is the "-00:00" offset being something much different than "+00:00". And then there is the leap seconds stuff which is nearly untested as far as I can tell in most products. I really don't understand how one write a 5424 compliant syslog message that occurs during the leap second but I'm probably just confused. What exactly does one do if one wants to write something to a log file when an extra leap second is being added? SIP is using the email style time formats and mandating GMT. That seems pretty trivially translatable into whatever one believes is the right way to do it in syslog so I'd be tempted to just allow SIP implementation use the timestamps they are using for SIP and define that as the data representation for the timestamps. I went to look at stealing the time format from IPFIX but gave up at the point where the 5102 said leap seconds where excluded from the dateTimeMicroseconds but in RFC 5101 it looked like dateTimeMicroseconds included the leap seconds. I like how 5101 uses the NTP stuff - that is very clear to everyone but does not map well to these human readable formats people want to use in the log files. All in all, the email style fixed to GMT in a single canonical format is looking pretty appealing.

---

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 4 Feb 2010 12:20:04 +0100

> I was considering also saying compatibility with syslog definition of
> timestamp but my recollection of that (which may be very wrong - been
> awhile since I looked at it) left me more than a bit confused about
> syslog tmestamps. It seems that every syslog packet I have ever seen in
> the wild uses the BSD Syslog timestamp format in RFC 3164 but clearly
> that is more or less borked for any real correlation across time zones
> so it seems like 5424 version of timestamp would be better. The 5424
> allowed the local offsets which resulted in no real canonical format
> for time which mean some people implemented the full things and some
> ignored the local offsets results in interoperability problems.

Well, I don't know any such broken implementations, but they are clearly
broken. That is because time zone information MUST be part of the timestamp.
Basically, syslog reuses a RFC3339 timestamp (see section 5.6 for it's ABNF)
but places some further restrictions on it. We do not force implementers to
use a single canonical on-the-wire format, but it is always easy to obtain a
canonical representation for correlation. As far as I remember, it was
considered useful to have some information inside the timestamp of where it
was generated.

> Then
> there is the "-00:00" offset being something much different than
> "+00:00".

When parsing the timestamp, the sign s is read and "+" interpreted as 1, "-"
as -1. Then, the offset o is parsed. The final offset calculation is o * s,
and as 0*1 = 0*-1 there is no difference between the two. A agree that a
slightly more complex logic is needed to adjust the time to UTC, but the same
is true if you have UTC timestamps only and need to adjust them to local
time, what often is desired by operators. So I fail to see why this is a
problem or not "canonical enough".

> And then there is the leap seconds stuff which is nearly
> untested as far as I can tell in most products.

I have to admit I don't understand this comment. What is untested here?
RFC5424 specifically forbids leap seconds, as during our discussions we found
many cases where leap seconds caused grief. I also think the the TAI is
considering aborting the use of leap seconds for this reason as well. To the
best of my knowledge, GSP also does not use leap seconds. The ultimate reason
to abandon UTC leap seconds in syslog was the we failed to identify an
operating system that would expose leap seconds to a user process. So a
syslogd or any other syslog sender would not even be able to see that one was
introduced. From the syslog perspective, a leap second is just like any other
second, but time flows "somewhat slower". I guess we are in the same boat as
many operating systems with this perspective.

> I really don't
> understand how one write a 5424 compliant syslog message that occurs
> during the leap second but I'
>  m probably just confused. What exactly does one do if one wants to
> write something to a log file when an extra leap second is being added?

I have to admit we didn't explicitely state that case because we thought it
could actually never happen. But I would say that "Leap seconds MUST NOT be
used" to me means that it should be expressed as the 59th second of said
minute. But even if you bump the minute and use the 0 second, I cannot see
how this should be problematic. On a single system, time should still evolve
serially. For correlating events form multiple systems, the timestamp alone
is insufficient in any case. You cannot closely enough synchronize the
different real time clocks. So you need a different meachanism (like Lamport
clocks) for this in any case.

> SIP is using the email style time formats and mandating GMT. That seems
> pretty trivially translatable 

indeed, this is trivial ;) If the sip timestamp is "2010-02-02
23:15:10.1234", you just need to replace the space by "T" and add an extra
"Z", so it becomes the valid syslog timestamp
"2010-02-02T23:15:10.1234Z". It is equally trivial to convert back ;)

Rainer

---

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 4 Feb 2010 12:51:36 +0100

> Behalf Of Cullen Jennings
> Sent: Thursday, February 04, 2010 1:26 AM
> To: David Harrington
> Cc: SIP-CLF Mailing List
> Subject: Re: [sip-clf] A primer on syslog.
> 
> 

Actually, the syslog WG was (very unfortunately) forced to stick with this
too-small set of facilities in order to keep compatible with existing
implementations. Other than facility 0, which seems to be universally used by
the OS kernel, there is no exact semantic for the facility codes. Think about
how many important subsystems we have and how small this facility set is...
The rest of the header provides ample ways to express this is a sip packet,
but an operator will still assign a facility and filter on it simply because
that is the fastest way to do it. This is another thing where some operator
configuration is required, at least in a reasonable busy envrionment.

Rainer

---

From: Benoit Claise <[email-masked]>
Date: Fri, 05 Feb 2010 12:47:08 +0100

Dear all,
> in order to run their networks, not developing a totally new standard
Exactly.
My main concern is that this WG is thinking about yet another data 
modelling language, which implies correlation tools and proxy in the future
because we can see it coming, one day this WG will require flow related 
information elements such as the ones defined in 
http://www.iana.org/assignments/ipfix/ipfix.xhtml. And one day, SIP 
related information elements will be required in IPFIX. And that day, we 
will left with two different data models.

My concern is that, the way the charter is written, any common log 
format is certainly fine.
But then what is the next step? Information transfer, correlation with 
other data model such as flow related information, etc...

Note: I just give my feedback to Saverio in helping to write a draft 
called " Advantages of using an IPFIX File Format for SIPCLF". Hopefully 
this draft will be out soon. It will contain more detailed information.
Note2: It's not because I've been involved in IPFIX that I try to push 
IPFIX at all costs... Numerous performance management partners have been 
stressing the cost/pain of correlation tools and proxies.

So my message is: let's avoid to have yet another data model. And if we 
think about the problems we want to solve from a little bit long term 
point of view, I believe the conclusion will be obvious.

Regards, Benoit.

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Fri, 05 Feb 2010 13:07:16 -0600

Cullen Jennings wrote:
> I like this draft, think it is fine work, but it needs a few more
> things before it is done. These are all things that I think can help
> us speed up the actual clf work if we can agree on them.
> 
[...]

Cullen: Thanks for your comments.

The need for an "Information Model" seems to fairly high
right now --  Rainer refers to it as well in one of his emails
as being the part that we need to get right.  Others have
as well in the past, so I suspect doing this right is worth
the time.

So, this is what I will do:  I will go ahead and respin
draft-gurbani-sipclf-problem-statement as
draft-ietf-sipclf-problem-statement (i.e., no changes except
file name).  Then I will work on updating the draft with
the information model, which we can discuss in Anaheim.

Specific comments you had on what fields should be in the
information model and what should be out of it can be
discussed in more detail once we have a section that lays
out this model.

Thanks,

- vijay