Subject: [sip-clf] A syslog approach to sip logging

From: "David B Harrington" <[email-masked]>
Date: Tue, 2 Feb 2010 13:53:00 -0500

Hi,

Operators often use syslog to carry Apache CLF log data.  Syslog, in
practice, is primarily used for tunneling Apache CLF format. This
seems to be attractive for operators, because of already existing
infrastructure and syslog knowledge.
I suggest it makes sense to develop a sipclf format that can be
carried within an IETF syslog message. In general, that means
information in an ascii format dumped into a local file that can be
parsed with tools like grep, and that can later be secured, filtered,
transported, aggregated, and correlated using existing infrastructure.

Multiple use cases raised by contributors in this WG lead to different
requirements. Some want to pay attention to the data dumped by one
server; some want to follow traffic flows through the network; some
want to filter on standardized fields; some want to aggregate and
correlate log information. 

It is not enough to figure out how to dump data on a single system;
that data will need to be compatible with infrastructure used to
provide secure transport, filtering, correlation, etc. Operators
already have existing infrastructures designed for long-term archiving
of (potentially enormous) logging information, and the goal to
correlate log records, including data-mining.

Syslog is already widely deployed, is well understood by operators,
and the IETF syslog WG has standardized many aspects of security and
transport, such as (D)TLS-secured transport, support for large
messages, optional digitally signed logging for law enforcement and
for message stream integrity checking, etc. 

IETF syslog standardizes a number of parameters useful for
correlation, such as
facility (specific applications), severity classification, timestamp,
hostname, the name of the application sending the message (often
syslogd), process ID, and message ID that are in the syslog header in
ascii format. These were designed to be compatible with ITU logging
standards, and the ALARM-MIB, to provide easier correlation of events
across different event reporting mechanisms. Also to improve
correlation, work has been done to translate syslog messages into SNMP
traps, and SNMP traps into syslog messages.

The IETF syslog standard also provides structured data elements. SDEs
are
designed to supplement the human-readable text with
application-parseable data fields (also encoded in 7-bit ascii), which
makes it easier for applications, such as security management systems,
to extract and correlate the data across vendor implementations, and
across nodes in a network. 

The IETF syslog standard already defines some SDEs that would likely
be useful for the problems sip clf is trying to resolve, like
precisiely tracking sequence inside a networked system: a
high-precision timestamp, the quality of the time source for a given
system, time zone accuracy, whether a node is synched with a network
time source, the origin of a log entry (useful after aggregation and
relay), the ip address at time of logging, an enterprise identifier,
the software that generated the message (i.e., the application that
asked syslogd to send the message), the software version, a sequence
number to provide sequence as seen by a single node, the sysUpTime of
a co-resident SNMP system, and the language used within the
human-readable MSG. The IETF syslog standard also contains some
recommendations on intelligently dropping messages if the log volume
becomes overwhelming.

The syslog WG deliberately did not standardize the content of the
human-readable message field. The WG standardized the header, and has
provided SDEs to standardize certain aspects of the information where
consensus can be reached. Having both (potentially non-standardized)
human-readable data, and standardized human-and-machine-readable
structured data in the same message addresses a wide range of use
cases, and gives the human more information to work with to interpret
an event. 

I propose that the WG reach consensus on specific fields of data that
would be good to standardize, such as those defined in the problem
statement doc, and define them as syslog SDEs (which, remember, are
text fields so they would be greppable and printable and diff-able and
human-readable). Structured data elements would better support
application-parsing of the data, such as for training IDS/IPS anomaly
engines.

There are only a few restrictions placed on the content of a MSG field
in a syslog message. According to the problem-statement document,
there already exist a number of proprietary sip clf formats. Well, if
those are in a format that can fit within the MSG field within an IETF
standard syslog message, then that proprietary data can also be
carried in the syslog message. Any vendor-specific log-parsing tools
would continue to work with the extracted MSG field, and they could be
supplemented by tools that can parse the standardized SDE information.
The IETF syslog standard also supports vendor-specific SDEs for
extensibility of structured data.

In a similar manner to the dual stack approach for IPv4/IPv6
transition, implementers could choose to drop specific fields from
their proprietary formats as consensus on useful SDEs is reached, and
their tools are adapted to use the standardized header and SDE
information.

This approach would work with the WG goal to constrain its focus to
the "useful  information" and not need to reinvent solutions such as a
data modeling language, character sets, delimiters, secure transport,
log integrity checking, log filtering, log aggregation and correlation
issues, and so on. 

I do not see much benefit from designing a whole new ascii file format
that no existing tools support (except generic text-handling tools
like grep), and operators would need to learn in addition to the
semantics in the information model. 

I recommend the WG focus on specific and actually existing problem
cases, and build the semantic "information model" incrementally. Then
use the existing standard syslog format to provide an example data
model, which inherits the benefits of an existing widely-deployed
infrastructure for logging.

David Harrington
[email-masked]
[email-masked]
[email-masked]

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 2 Feb 2010 15:24:32 -0600

Speaking as individual,

Just as a heads-up, David and I had a private conversation late last week, 
where David told me that he did some investigation after our Hiroshima 
meeting, and discovered (to his surprise) that it's fairly common to use 
SYSLOG for Apache CLF.

I'm very interested in hearing reactions to this note, because it makes me 
much more comfortable with our charter - most of our chartered work is to 
identify fields that need to be logged and figure out a story for 
correlation; once we have identified the fields that should be logged, IFF 
we agree that SYSLOG makes sense, defining SYSLOG structured data elements 
for those fields should be pretty straightforward.

Thanks,

Spencer

> _______________________________________________
> sip-clf mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sip-clf

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Tue, 02 Feb 2010 15:49:23 -0600

Spencer Dawkins wrote:
> Speaking as individual,
> 
> Just as a heads-up, David and I had a private conversation late last 
> week, where David told me that he did some investigation after our 
> Hiroshima meeting, and discovered (to his surprise) that it's fairly 
> common to use SYSLOG for Apache CLF.

As far as I know, only the error messages in Apache CLF go to
syslog [1] (someone can correct me if I am wrong.)  The CLF
logs are still saved on disk.

> I'm very interested in hearing reactions to this note, because it makes 
> me much more comfortable with our charter - most of our chartered work 
> is to identify fields that need to be logged and figure out a story for 
> correlation; once we have identified the fields that should be logged, 
> IFF we agree that SYSLOG makes sense, defining SYSLOG structured data 
> elements for those fields should be pretty straightforward.

Note that syslog may not work when a web server (or SIP server) is
running on a small footprint device that is not able to contact
the syslog daemon.  In that case, the syslog records are probably
written to local cache

However, insofar as there is a default mode to producing Apache
CLF, it is produced as a disk file.  Also, syslog records are
typically small; SIP CLF records can get pretty big depending on
what all is being logged.

[1] Ben Laurie, "Apache: The Definitive Guide," 2e.

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
Web:   http://ect.bell-labs.com/who/vkg/

---

From: "Spencer Dawkins" <[email-masked]>
Date: Tue, 2 Feb 2010 16:16:58 -0600

Hi, Vijay,

>> elements for those fields should be pretty straightforward.

IFF, of course... :-)

Spencer

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 3 Feb 2010 16:39:25 -0700

One requirement that I suspect you will find fairly universal about transporting around SIP log like informations is that 

1) it is reliable

2) we can include complete SIP messages. These can get very large. (Magnus posted a 40k SDP to the mmusic awhile back, don't even ask how large MESSAGE messages get in the wild)

Can you say a bit more about how to use syslog to achieve these. 

> infrastructure for logging.

For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: Hadriel Kaplan <[email-masked]>
Date: Wed, 3 Feb 2010 23:16:45 -0500

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Cullen Jennings
> Sent: Wednesday, February 03, 2010 6:39 PM
> 
> One requirement that I suspect you will find fairly universal about
> transporting around SIP log like informations is that
> 
> 1) it is reliable
> 
> 2) we can include complete SIP messages. These can get very large. (Magnus
> posted a 40k SDP to the mmusic awhile back, don't even ask how large
> MESSAGE messages get in the wild)

Also, if it is required for it to support including complete SIP messages, then it has to support binary content, since those can be in SIP message bodies (e.g., ISUP, QSIG).

-hadriel

---

From: Adam Roach <[email-masked]>
Date: Wed, 03 Feb 2010 23:13:37 -0600

> Also, if it is required for it to support including complete SIP messages, then it has to support binary content, since those can be in SIP message bodies (e.g., ISUP, QSIG).

And even if it isn't required to transport complete messages, the 
various fields that are clearly within the scope of even the most basic 
logging can contain UTF-8 characters, which means *almost* any octets 
can potentially appear in these fields anyway.

/a

---

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 4 Feb 2010 12:41:28 +0100

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Cullen Jennings
> Sent: Thursday, February 04, 2010 12:39 AM
> To: David B Harrington
> Cc: 'SIP-CLF Mailing List'
> Subject: Re: [sip-clf] A syslog approach to sip logging
> 
> 

I need to first ask what level of reliability do you need? There are several
flavors of reliability and usually loss of message is acceptable at some
level. 

For example, let's assume the process writes out log messages to a text file.
Under (allmost?) all modern operating systems and by default, this means the
OS accepts the information to write, acks it, does NOT persist it to storage
and lets the application continue. The actual data block is usually written a
short while later. Obviously, this is not reliable: you can lose log data if
an unrecoverable i/o error happens or something else goes fatally wrong.

This can be solved by instructing the operating system to actually persist
the information to durable store before returning back from the API. You have
to pay a big performance toll for that. This is also a frequent question for
syslog data, and many operators do NOT sync and accept a small message loss
risk to save themselves from requiring a factor of 10 servers of what they
now need.

But even if writes are synchronous, how does the application react? For
example: what shall the application do if log data cannot be written? If you
really need reliable logging, the only choice is to shutdown the application
when it can no longer log. I know of very few systems that actually do that,
even though "reliability" is highly demanded. Here, the cost of shutting down
the application may be so high (or even fatal), that the limited risk of log
data loss is accepted.

There are a myriad of things when thinking about reliability. So I think it
is important to define the level of reliability that is required by the
solution and do that in detail. To the best of my knowledge, this is also
important for operators who are required by law to do "reliable" logging. If
they have a risk matrix, they can define where it is "impossible" (for
technical or financial reasons) to achieve full reliability and as of my
understanding this is information auditors are looking for.

Having said that, current RFC5425 provided good, but not perfect reliability.
A big problem is that RFC5425 does not provide an application-layer
backchannel and relies on TCP to achieve reliability. That means that when a
connection breaks, there is a window of uncertainty (sender's tcp send
buffers + current TCP window) for which messages may or may not be lost. This
is comparable to the unsynced file access paradigm.

RFC3195 provides a "fully reliable" protocol, but was not accepted by the
syslog community because it was a too-large departure from traditional
syslog. This is the main reason RFC5424/5425 do not specify backchannels. The
syslog WG will hopefully create a "RFC3195bis", which will be a bit simpler
than 3195 was and supports the new header formats. But this is "hopeware" and
not something you can put on a system today or some time this year).

> 
> 2) we can include complete SIP messages. These can get very large.
> (Magnus posted a 40k SDP to the mmusic awhile back, don't even ask how
> large MESSAGE messages get in the wild)

Size was a large problem with traditional syslog and the syslog community
still prefers small sizes. Syslog recommend "maximum message sizes", which
place an upper bound for the message on a given transport and implementation.
The minumum (maximum message size), so the max size that must be supported by
every implementation, is just 2k. The recommended (maximum message size) is
8k. Both are obviously too low. Thankfully, we managed to set no upper limit
per se on the log records. So a syslog receiver may be configured to use any
maximum message size an operator requires - but implementations are not
required to support that capability. IHE, for example, uses 32k for their
messages. Our Windows Event-Log to syslog conversion tool sometimes need to
emit messages in excess of 128K. 

So syslog can support that requirement, but some syslog implementations may
not support it. Also, an operator must specifically configure syslog
components for non-default sizes to make everything work. There is definitely
a failure potential in this process, and it is not trivial in a larger
network with different syslog users.

Rainer

---

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 4 Feb 2010 12:55:35 +0100

> Behalf Of Hadriel Kaplan
> Sent: Thursday, February 04, 2010 5:17 AM
> To: Cullen Jennings; David B Harrington
> Behalf
> (Magnus
> Also, if it is required for it to support including complete SIP
> messages, then it has to support binary content, since those can be in
> SIP message bodies (e.g., ISUP, QSIG).

This is problematic, but could be done with some transformation rules. syslog
is *not* designed as a transfer protocol for binary data, even though it may
occasionally support it. If sip clf has a large binary content part, syslog
is probably wrong. Apache clf is a text format, so I did not yet think this
may be a problem.

How do you so far intended to write them out? My understanding so far was
that a sip clf log entry would be a set of values, terminated by a LF. But
binary data may contain LF. So don't you need to encode it in any case?

Rainer

---

From: "Rainer Gerhards" <[email-masked]>
Date: Thu, 4 Feb 2010 12:58:09 +0100

> Behalf Of Adam Roach
> Sent: Thursday, February 04, 2010 6:14 AM
> To: Hadriel Kaplan
> Cc: David B Harrington; 'SIP-CLF Mailing List'

UTF-8 is well defined, and not any sequence of octets is a valid UTF-8
sequence. syslog supports UTF-8 (but only the minority of currently existing
implementations do support it well!), but syslog demands proper UTF-8
sequences for security reasons.

Rainer