Subject: RE: Charset mandatory in unix/linux
From: Philip Hazel <[email-masked]>
To: Misha Wolf <[email-masked]>
Cc: [email-masked], ietf-822 <[email-masked]>, [email-masked], John Cowan <[email-masked]>
Date: Mon, 27 Mar 2006 13:55:26 +0100 (BST)
List-Id: <ietf-822.imc.org>

> > Or whether that's the "Unicode" that has among its design principles
> > a uniform code width of 16 bits and an encoding strictly of text
> > (specifically excluding musical notation), or the "Unicode" that has a
> > much wider code width and includes non-textual cruft such as (yes, you
> > guessed it) musical notation.

Though the encoding of the full printed representation of music is 
beyond the capabilities of mere "characters", it is common to want to
print musical symbols (notes, sharps, flats, clefs, etc) within ordinary
text when writing about music. I don't think they are any more "cruft"
than, say, the characters in what is commonly called the "dingbats"
group, which includes things like scissors for "cut here", telephone
symbols, and so on. I think what I'm saying is that it is hard to define 
"strictly of text".

-- 
Philip Hazel, University of Cambridge Computing Service.

Date: Mon, 27 Mar 2006 12:01:43 +0100
From: Misha Wolf <[email-masked]>
Subject: RE: Charset mandatory in unix/linux
To: [email-masked]
Cc: ietf-822 <[email-masked]>, [email-masked], John Cowan  <[email-masked]>
Content-class: urn:content-classes:message
Sender: [email-masked]

[John Cowan has writtten this reply to Bruce's mail and 
has given me permission to forward it.  Misha]

> These typically (and specifically for tar and zip) do not include
> media type information or charset or other type parameter information.
> The information in the tar format, for example, carries time stamps,
> permissions, and file type (where "type" means plain file vs. directory
> vs. device, etc.).

ZIP format, however, has the ability to store extended attributes,
though this is normally only used on VMS and OS/2 systems.  See
http://www.info-zip.org/pub/infozip/doc/appnote-iz-latest.zip for the
details.

> The holy grail of a single unified character set that will supposedly
> solve the problem sounds nice until one looks at the details.

I think this line, and the rant which follows, is a serious exaggeration
of the facts.

> "Unicode" is itself a "vast array" (ever-increasing in number) of
> character code sets.  Saying "Unicode" doesn't tell me if that's
> pre-"Korean mess" (see RFC 2279) "Unicode" or post-"Korean mess"
> "Unicode".

The pre-Korean-mess versions (1.0 and 1.1) do in fact have their own
charsets: Unicode-1-1 and Unicode-1-1-UTF-8.  However, no one has ever
come forward with actual text encoded in anger that contains pre-mess
Korean syllables, so it's almost entirely academic.

> Or whether that's the "Unicode" that has among its design principles
> a uniform code width of 16 bits and an encoding strictly of text
> (specifically excluding musical notation), or the "Unicode" that has a
> much wider code width and includes non-textual cruft such as (yes, you
> guessed it) musical notation.  Or whether it's one of the "Unicode"s
> that has an attempt at encoding language information (versions 3.1
> and 3.2), or one of the "Unicode"s (earlier and later) that do not.
> And so on.

All this is about one thing and one thing only:  what characters
your implementation can handle.  If you have an old 8-bit or 16-bit
implementation trying to process text involving new characters, it
won't know what to make of them, but it won't be seriously confused.
New implementations can of course handle old text fine using their more
advanced models, because everything is backward compatible.

As for language tagging in plain text, it was demanded by an IETF WG,
was in effect born deprecated, and is formally deprecated as of 4.0 but
of course remains present and will forever.

> as far as I know, it's not even possible to have multiple versions of
> Unicode and to transcode between them on the same machine.

It's not possible because (other than the Korean mess) it's not necessary
either.

> [S]uppose Jacob has received a text file in Korean and the issue of
> labeling the charset and language is solved.  If it is labeled as
> "ISO-2022-KR", he can proceed to make sense of the file; conversely
> if it is labeled as "utf-7" he cannot because he lacks information
> to determine whether the result of the transformation to "Unicode"
> should be interpreted as groups of 16 bits or some other code width,
> as well as which code points represent various hangul characters.

The former point is not relevant: you can interpret it as 16-bit or
21-bit codes without any difference in effect.  The latter point I have
already addressed.

-- 
John Cowan  [email-masked]  www.ccil.org/~cowan  www.ap.org
The competent programmer is fully aware of the strictly limited size of his own
skull; therefore he approaches the programming task in full humility, and among
other things he avoids clever tricks like the plague.  --Edsger Dijkstra

To find out more about Reuters visit www.about.reuters.com

Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Reuters Ltd.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: [email-masked]
Subject: Re: Charset mandatory in unix/linux
Date: Sun, 26 Mar 2006 18:50:45 -0500
Cc: ietf-822 <[email-masked]>, [email-masked], Jacob Palme <[email-masked]>
Sender: [email-masked]

On Sun March 12 2006 10:10, [email-masked] wrote:
> 
> (cc'ing the ietf-types list since this doesn't seem like an appropriate topic
> for ietf-822)
[this response to types, cc to 822, Reply-To set to types]

[Jacob Palme wrote, regarding charset] 
> > However, such a parameter is not mandatory in
> > Unix or Linux.
> 
> I could say the same thing about media types. File extensions or type codes are
> commonly used to determine the media type. This is a huge problem that has led
> to serious security glitches as well as poor user experiences.

Agreed.
 
> > This is causing more and more problems, when
> > people have a mixture of files with different charsets,
> > which you easily get when you download files from the
> > Internet or receive them via e-mail.
> 
> The reality is it is causing less and less problems as things gradually shift
> towards Unicode-based charsets and away from the vast array of less capable
> charsets. The security issues caused by non-use or misuse of media type labels
> are a far bigger problem, and worse, one that doesn't appear to be going away.

Agreed about the security issues w.r.t. non-use/misuse of type labels.
However, I have a different perspective regarding Unicode (see below).
> 
> > Would it be possible to get the people responsible for the
> > file systems in Unix and Linux to add a mandatory charset
> > attribute to all text files?
> 
> Knowing the charset buys you very little without also knowing the media type.
> You seem to be focused on plain text here and hence you're ignoring the larger
> media type issue. Lots of media types have parameters and even when the media
> type can be determined - it frequently cannot be done reliably - it is often
> done in a way that doesn't allow additional parameters to be attached.

I note that Unix and Unix-like systems don't have the notion of a "text
file"; unlike some other systems, there is no distinction between "text"
and "binary" files.  Moreover, it is one of the characteristics of Unix
that file system semantics apply not only to files per se, but also
apply to devices (disk drives, communications ports, etc., and in recent
implementations as interfaces to system information for processes).
 
> > Best is probably to add a
> > generalized property list to files, so that also other
> > properties than charset can be added in the future.
> 
> The ability to attach metadata to files is indeed a very useful feature, one
> that has been around for decades on some platforms at least. (I'm not going to
> bother with the history here.) And it is already available on Linux - at a
> minimum the ext2, ext3, and XFS file systems support it. (There are probably
> others but I'm too lazy to go look them up.)

ReiserFS is one of the more important Linux file systems...
 
> So in the sense of getting the filesystem to support this sort of tagging, your
> problem is already solved in many cases. But this is the easy part. You now
> have to get applications to agree on a specific use of metadata tags for
> charsets or media types or whatever. Good luck on getting that to happen.

Specifically regarding Unix-like systems, there is a long history of
representing metadata as character strings containing attribute/value
pairs; that is how environment variables are passed, how command-line
options with parameters are passed to programs, etc.  IIRC, Tom Duff
had a paper in the "papers" volume of one of the recent editions of
the Unix manuals about the use of pairs within graphics data files
for conveying such information.  So if the filesystem metadata can be
represented as character strings containing attribute/value pairs,
that's a very good fit (with one important caveat) to media type
parameters as that is precisely what those parameters are.

> > The advantage would be that programs which transport files
> > across the Internet, such as e-mail, ftp and http, would
> > more often use the correct charset and not munge the files
> > by giving then an incorrect charset. The commonly occuring
> > problem with incorrect charset would be reduced. Also local
> > problems such as text editors would benefit from knowing
> > the charset of a file.
> 
> First of all, email and http do not "transfer files" per se. They transfer data
> objects and each protocol defines the metadata it considers approproate to
> attach to data objects.

It's a bit more complicated than that specifically for text; email
is fairly consistent regarding line endings -- the message format
is quite specific on that point, as are the MIME specifications
(2046 in particular).  HTTP however does not specify line endings
for text, and that is a source of various inconsistencies and
problems.  There is no way to specify line ending with HTTP; the
protocol specification expressly permits implementation
discrepancies.  A potent source of trouble when transferring
unlabeled or mislabeled binary content containing 0x0D octets via
HTTP.

[...]
> This situation means that in situations where retention of file metadata is
> important some sort of additional container has to be used. A vast number of
> such container formats have been defined - tar files, zip files,
> AppleSingle/AppleDouble, etc.

These typically (and specifically for tar and zip) do not include
media type information or charset or other type parameter
information.  The information in the tar format, for example,
carries time stamps, permissions, and file type (where "type"
means plain file vs. directory vs. device, etc.).

Media types can be used to label such containers (IIRC there is
a defined media type for the "AppleDouble" stuff), and media type
parameters can be used to convey additional information;
alternatively, media types could be defined to label an octet
stream as a particular type, with metadata carried via parameters.

[...]
> So what's the bottom line? The bottom line is that you appear to be focusing on
> the wrong problem along several dimensions. First, charset information
> specifically isn't as interesting or essential as you claim, and the degree to
> which is it interesting is dropping for a variety of reasons. Second, you
> appear to have missed the larger and much more important problem of not having
> correct parameterized type information available. (And we haven't even
> discussed the many other sorts of metadata, like say language information, that
> is also useful to have.) Third, your focus on getting metadata support into
> filesystems is mostly misplaced - this is a solved problem in a lot of cases.
> And fifth, you don't seem to appreciate the difficulty of getting everyone to
> agree to actually use filesystem metadata to solve any of these problems. This
> last is a complete showstopper and I dispair of there ever being significant
> progress in this area because of it.

In reverse order:

Getting agreement to use metadata has several issues: aside from the
slow evolutionary process of having protocols convey the metadata and
having applications store and retrieve that metadata, there is the
issue of some sort of standard(s) for APIs for the metadata
storage/retrieval.  That's not really in IETF's bailiwick; perhaps
it's something that might spark some interest in ECMA or a similar
SDO.  So far as Unix-like systems are concerned, the use of
character-stream representation of attribute/value pairs would seem
to be a good fit as noted above, with one caveat; in MIME 
Content-Type fields, one knows how to interpret the "character
strings" because they are specified to be comprised of characters
from a limited repertoire -- that is, one does not need to know the
"charset" of the character strings themselves; they are composed of
a small, well-defined set of characters which fit in octets (in 7
bits, in fact).  So long as one can count on being able to interpret
the character strings as a stream of octets, I won't despair.
Conversely, if the very problem that Jacob has described, viz. the
inability to determine what sort of "character strings" one is
looking at, extends to the metadata storage, I will abandon all hope
of a solution.

As far as additional metadata (language, etc.) is concerned, the
attribute/value pair paradigm still works, but at a higher level.
In email and HTTP, for example, there are header fields -- comprised
of attribute/value pairs (specifically header field name and field
body) -- which convey not only MIME media type and parameters, but
other MIME fields (e.g. Content-Language) as well as non-MIME data.

I do think Jacob has a valid concern; having unlabeled text files in
various charsets is a big problem, and from my perspective it is
getting worse, not better.  The holy grail of a single unified
character set that will supposedly solve the problem sounds nice
until one looks at the details.  Fortunately, the notion of a
"charset" being somewhat more complex than the notion of a "character
set" helps a little; at least knowing the charset, one can
distinguish among utf-7, utf-8, utf-32be, utf-32le, utf-16be, and
utf-16le, all of which have "Unicode" as the underlying character
code.  But that doesn't help much, precisely because "Unicode" is
itself a "vast array" (ever-increasing in number) of character code
sets.  Saying "Unicode" doesn't tell me if that's pre-"Korean mess"
(see RFC 2279) "Unicode" or post-"Korean mess" "Unicode".  Or whether
that's the "Unicode" that has among its design principles a uniform
code width of 16 bits and an encoding strictly of text (specifically
excluding musical notation), or the "Unicode" that has a much wider
code width and includes non-textual cruft such as (yes, you guessed
it) musical notation.  Or whether it's one of the "Unicode"s that
has an attempt at encoding language information (versions 3.1 and
3.2), or one of the "Unicode"s (earlier and later) that do not.  And
so on.  Ned, you're quite right that simply adding attributes isn't
a panacea; specifying "Unicode" version doesn't help.  Obviously an
implementation using "Unicode" version 2.1 won't be able to make
sense of the "features" which crept into "Unicode" version 3.2 -- and
not everybody is able to "upgrade".  Aside from the much greater
resources required to support an increased code width (think of small,
battery-powered hand-held devices like cell phones), and the costs
(monetary and otherwise) that an "upgrade" might entail, some hardware
and/or software might be incapable of supporting a version change
(not merely insufficient hardware resources, but perhaps a vendor has
dropped support or has gone out of business). And there are related
costs (monetary and otherwise) associated with changing entire suites
of software, font support, etc.  And as I see it, the problem is much
worse than when I had to deal with ANSI X3.4 and the ISO-8859 variants;
at least I COULD have software that dealt with the various encodings,
fonts that support the glyphs, locales to switch among, etc. -- as far
as I know, it's not even possible to have multiple versions of Unicode
and to transcode between them on the same machine.  Even if that is
theoretically possible, the fundamental version problem remains;
"utf-8" still doesn't tell me (or my software) anything about the
underlying "Unicode" version.  Whereas "ANSI X3.4" has a specific
meaning -- it's code width didn't suddenly double one day.  To put
the issue in the perspective of Jacob's problem, suppose Jacob has
received a text file in Korean and the issue of labeling the charset
and language is solved.  If it is labeled as "ISO-2022-KR", he can
proceed to make sense of the file; conversely if it is labeled as
"utf-7" he cannot because he lacks information to determine whether
the result of the transformation to "Unicode" should be interpreted as
groups of 16 bits or some other code width, as well as which code points
represent various hangul characters.

From: Bruce Lilly <[email-masked]>
Reply-To: [email-masked]
Organization: Bruce Lilly
To: ietf-822 <[email-masked]>
Subject: Re: Charset mandatory in unix/linux
Date: Sun, 26 Mar 2006 15:15:17 -0500
Cc: [email-masked], Jacob Palme <[email-masked]>, ietf-822 <[email-masked]>
Sender: [email-masked]

On Sun March 12 2006 10:10, [email-masked] wrote:
> 
> (cc'ing the ietf-types list since this doesn't seem like an appropriate topic
> for ietf-822)
[this response to types, cc to 822, Reply-To set to types]

[Jacob Palme wrote] 
> > The charset parameter is mandatory in the MIME content-type
> > attribute.
> 
> Actually, I don't know of a single case where this is true. All media type
> pararameters are either type or subtype specific, so there is no general rule
> that applies to all charset parameters. Nevertheless, the charset parameters
> that attach to the text top-level type are optional, as is the charset
> parameter on application/xml.

What about text/directory and application/shf+xml (RFCs 2425 and
4194 respectively)?

RFC 2425:

5.3.  Required parameters

   Required parameters: charset

RFC 4194:

9.  IANA Considerations

   This section contains the registration information for the MIME type
   to SHF.  The media type has been chosen to comply with the guidelines
   in [4].

   o  Registration: application/shf+xml
   o  MIME media type name: application
   o  MIME subtype name: shf+xml
   o  Required parameters: charset

   Required parameters: charset

   This parameter must exist and must be set to "UTF-8".  No other
   character sets are allowed for transporting SHF data.  The character
   set designator MUST be uppercase.

[so in the latter case, the parameter is mandatory but conveys no
useful information as the charset is fixed, i.e. known a priori for
each instance of the media type, rather than variable...
4194 also conflicts with established usage of charset parameter
values as specified in RFC 2046: "Unlike some other parameter values,
the values of the charset parameter are NOT case sensitive" (That's
for text, but having case-sensitivity of the values FOR A GIVEN
PARAMETER ATTRIBUTE vary with type/subtype greatly complicates
parsing).  Sigh...]

From: [email-masked]
Cc: Ned Freed <[email-masked]>, Martin Duerst <[email-masked]>, [email-masked], IETF mailing list on MIME and e-mail <[email-masked]>, Lars Enderin <[email-masked]>, Jacob Palme <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 13 Mar 2006 08:26:18 -0800 (PST)
Subject: Re: Charset mandatory in unix/linux
Sender: [email-masked]

> Ned Freed writes:
> > Since it does mean that in email, I fail to see why such a change
> > would be (a) Correct or (b) A good idea.

> If «content-type: text/plain; charset=us-ascii", «content-type;
> text/plain» and a missing content-type field are equivalent, I would
> say that encoding the character set is mandatory in email, just as
> Jakob Palme wrote in the first message. A message sender can't avoid
> specifying the character set, even by leaving out the charset
> parameter.

It is quite true that a user of text/plain in email has no choice but to
specify the charset, either implicitly by omitting the parameter, in which case
it defaults to US-ASCII, or explicitly by including it. But that's not what
Jacob said. He said:

> The charset parameter is mandatory in the MIME content-type
> attribute.

This is quite clearly incorrect. The _parameter_ is not mandatory,
_specification of the charset_ by one means or another is. (More precisely, it
is unavoidable due to the way the defaults are set up. Referring to mechanisms
involving defaults as mandatory is not good use of terminology IMO.)

Now, perhaps Jacob meant to say that the charset of plain text has to be
specified in some way. I'm not a mind reader and cannot divine intent. All I
can do is respond to what he did say, which was incorrect.

				Ned

Date: Mon, 13 Mar 2006 15:03:21 +0000
From: Tony Finch <[email-masked]>
To: Jacob Palme <[email-masked]>
cc: IETF mailing list on MIME and e-mail <[email-masked]>, Lars Enderin <[email-masked]>
Subject: Re: Charset mandatory in unix/linux
Sender: [email-masked]

On Sun, 12 Mar 2006, Jacob Palme wrote:
>
> The charset parameter is mandatory in the MIME content-type
> attribute. However, such a parameter is not mandatory in
> Unix or Linux. This is causing more and more problems, when
> people have a mixture of files with different charsets,
> which you easily get when you download files from the
> Internet or receive them via e-mail.

Unix doesn't really support fine-grained switching between locales, and
therefore it doesn't have good support for switching between charsets
either. Files are untyped, and the way they are treated depends on
context. The context is principally defined by the program the file is fed
to, but the interpretation can also be changed by the locale settings in
the environment. Switching locales within a program is not well supported.
The problem is bigger than charsets: for example, the Unix locale API also
doesn't have good support for manipulating dates in multiple timezones.

So if you are going to solve your problem, you'll have to re-do the locale
API as well as defining how to use extended attributes to store charset
information. This is a problem for POSIX not the IETF.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
MULL OF KINTYRE TO ARDNAMURCHAN POINT: SOUTH OR SOUTHEAST 6 TO GALE 8,
OCCASIONALLY SEVERE GALE 9 NEAR EXPOSED HEADLANDS, DECREASING 5 OR 6
OVERNIGHT.

Date: Mon, 13 Mar 2006 15:45:38 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: Ned Freed <[email-masked]>
Subject: Re: Charset mandatory in unix/linux
Cc: Martin Duerst <[email-masked]>, [email-masked], IETF mailing list on MIME and e-mail <[email-masked]>, Lars Enderin <[email-masked]>, Jacob Palme <[email-masked]>
Sender: [email-masked]

Ned Freed writes:
> Since it does mean that in email, I fail to see why such a change 
> would be (a) Correct or (b) A good idea.

If «content-type: text/plain; charset=us-ascii", «content-type; 
text/plain» and a missing content-type field are equivalent, I would 
say that encoding the character set is mandatory in email, just as 
Jakob Palme wrote in the first message. A message sender can't avoid 
specifying the character set, even by leaving out the charset 
parameter.

Arnt

From: [email-masked]
Cc: Martin Duerst <[email-masked]>, Ned Freed <[email-masked]>, [email-masked], IETF mailing list on MIME and e-mail <[email-masked]>, Lars Enderin <[email-masked]>, Jacob Palme <[email-masked]>
To: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 13 Mar 2006 06:17:47 -0800 (PST)
Subject: Re: Charset mandatory in unix/linux
Sender: [email-masked]

> Martin Duerst writes:
> > At 00:10 06/03/13, Ned Freed wrote:
> > >(cc'ing the ietf-types list since this doesn't seem like an
> > appropriate topic
> > >for ietf-822)
> > >
> > >> The charset parameter is mandatory in the MIME content-type attribute.
> > >
> > >Actually, I don't know of a single case where this is true.
> >
> > I don't, either.

> Huh.

> So I looked closely, and I find that you're right. How about stressing
> this a little more in the successor to RFC 2045?

RFC 2045 is about message structure and only deals with with the aspects
of media types that interact with message structure. RFC 2046 is where
media type behavior is defined, and section 4.1.2 of that document is
quite clear about the behavior of the charset parameter.

> Ned? Section 5.2 could
> state clearly that «Content-Type: text/plain» does NOT mean the same as
> a missing Content-Type header.

Since it does mean that in email, I fail to see why such a change would be (a)
Correct or (b) A good idea.

				Ned

Date: Mon, 13 Mar 2006 14:04:11 +0100
From: Arnt Gulbrandsen <[email-masked]>
To: Martin Duerst <[email-masked]>
Subject: Re: Charset mandatory in unix/linux
Cc: Ned Freed <[email-masked]>, [email-masked], IETF mailing list on MIME and e-mail <[email-masked]>, Lars Enderin <[email-masked]>, Jacob Palme <[email-masked]>
Sender: [email-masked]

Martin Duerst writes:
> At 00:10 06/03/13, Ned Freed wrote:
> >(cc'ing the ietf-types list since this doesn't seem like an 
> appropriate topic
> >for ietf-822)
> >
> >> The charset parameter is mandatory in the MIME content-type attribute.
> >
> >Actually, I don't know of a single case where this is true.
>
> I don't, either.

Huh.

So I looked closely, and I find that you're right. How about stressing 
this a little more in the successor to RFC 2045? Ned? Section 5.2 could 
state clearly that «Content-Type: text/plain» does NOT mean the same as 
a missing Content-Type header.

Arnt

Date: Mon, 13 Mar 2006 15:31:04 +0900
To: Ned Freed <[email-masked]>, Jacob Palme <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: Charset mandatory in unix/linux
Cc: IETF mailing list on MIME and e-mail <[email-masked]>, Lars Enderin <[email-masked]>, [email-masked]
Sender: [email-masked]

At 00:10 06/03/13, Ned Freed wrote:
 >(cc'ing the ietf-types list since this doesn't seem like an appropriate topic
 >for ietf-822)
 >
 >> The charset parameter is mandatory in the MIME content-type
 >> attribute.
 >
 >Actually, I don't know of a single case where this is true.

I don't, either.

 >All media type
 >pararameters are either type or subtype specific, so there is no general rule
 >that applies to all charset parameters. Nevertheless, the charset parameters
 >that attach to the text top-level type are optional, as is the charset
 >parameter on application/xml. And making the parameter optional doesn't even
 >imply that there's a default. For exmaple, In the case of XML the allowed
 >charsets for unlabelled material are intentionally limited so they can be
 >determined by inspection.

Almost correct, but wrong: In the case of application/xml, if there
is no 'charset' parameter on the mime type, information inside the
XML document is used to determine the character encoding according
to a clearly defined bootstrap algorithm. If you start a document
with
     <?xml version='1.0' encoding='foobar'?>
then it's in the "foobar" encoding. That doesn't mean that your
parser will be able to understand the "foobar" encoding, XML
parsers are only required to understand UTF-8 and UTF-16.

 >And fifth, you don't seem to appreciate the difficulty of getting everyone to
 >agree to actually use filesystem metadata to solve any of these problems. This
 >last is a complete showstopper and I dispair of there ever being significant
 >progress in this area because of it.

I fully agree with this at the present stage. But there is still
the hope that this problem will solve itself in 5 or 10 years,
like other, similar problem have been solved or are being solved
(ASCII for very basic text exchange, Unicode for worldwide text
exchange, XML for structured data exchange, and so on).

Regards,     Martin. 

From: [email-masked]
Cc: IETF mailing list on MIME and e-mail <[email-masked]>, Lars Enderin <[email-masked]>, [email-masked]
To: Jacob Palme <[email-masked]>
Date: Sun, 12 Mar 2006 07:10:55 -0800 (PST)
Subject: Re: Charset mandatory in unix/linux
Sender: [email-masked]

(cc'ing the ietf-types list since this doesn't seem like an appropriate topic
for ietf-822)

> The charset parameter is mandatory in the MIME content-type
> attribute.

Actually, I don't know of a single case where this is true. All media type
pararameters are either type or subtype specific, so there is no general rule
that applies to all charset parameters. Nevertheless, the charset parameters
that attach to the text top-level type are optional, as is the charset
parameter on application/xml. And making the parameter optional doesn't even
imply that there's a default. For exmaple, In the case of XML the allowed
charsets for unlabelled material are intentionally limited so they can be
determined by inspection.

> However, such a parameter is not mandatory in
> Unix or Linux.

I could say the same thing about media types. File extensions or type codes are
commonly used to determine the media type. This is a huge problem that has led
to serious security glitches as well as poor user experiences.

> This is causing more and more problems, when
> people have a mixture of files with different charsets,
> which you easily get when you download files from the
> Internet or receive them via e-mail.

The reality is it is causing less and less problems as things gradually shift
towards Unicode-based charsets and away from the vast array of less capable
charsets. The security issues caused by non-use or misuse of media type labels
are a far bigger problem, and worse, one that doesn't appear to be going away.

> Would it be possible to get the people responsible for the
> file systems in Unix and Linux to add a mandatory charset
> attribute to all text files?

Knowing the charset buys you very little without also knowing the media type.
You seem to be focused on plain text here and hence you're ignoring the larger
media type issue. Lots of media types have parameters and even when the media
type can be determined - it frequently cannot be done reliably - it is often
done in a way that doesn't allow additional parameters to be attached.

Additionally, usage is shifting away from plain text and towards other, more
sophisticated, formats - so much so that support for plain text has become a
bit problematic in some places. This is one of the reasons given for moving
away from plain text RFCs to some other format.

> Best is probably to add a
> generalized property list to files, so that also other
> properties than charset can be added in the future.

The ability to attach metadata to files is indeed a very useful feature, one
that has been around for decades on some platforms at least. (I'm not going to
bother with the history here.) And it is already available on Linux - at a
minimum the ext2, ext3, and XFS file systems support it. (There are probably
others but I'm too lazy to go look them up.)

So in the sense of getting the filesystem to support this sort of tagging, your
problem is already solved in many cases. But this is the easy part. You now
have to get applications to agree on a specific use of metadata tags for
charsets or media types or whatever. Good luck on getting that to happen.

> The advantage would be that programs which transport files
> across the Internet, such as e-mail, ftp and http, would
> more often use the correct charset and not munge the files
> by giving then an incorrect charset. The commonly occuring
> problem with incorrect charset would be reduced. Also local
> problems such as text editors would benefit from knowing
> the charset of a file.

First of all, email and http do not "transfer files" per se. They transfer data
objects and each protocol defines the metadata it considers approproate to
attach to data objects. The matchup of this metadata to file system metadata is
imperfect at best, and there are plenty of issues in this space. (It is far
from clear that adding additional type information will solve these problems,
however.)

Ftp does transfer files, but ftp is kinda old and doesn't understand file
metadata too well if at all.

This situation means that in situations where retention of file metadata is
important some sort of additional container has to be used. A vast number of
such container formats have been defined - tar files, zip files,
AppleSingle/AppleDouble, etc.

> (Mac OS earlier had a very good feature, you could add to
> every file a property list called the "resource fork").
> This still works in Mac OS X, but is less and less often
> used, since Unix, on which Mac OS X is based, does not have
> this facility. In Mac OS X the resource fork is stored in a
> separate file whose file name starts with ".", in the same
> directory as the file described).

Sigh. There are so many things wrong here it is difficult to know where to
begin. I guess I'll start with resource forks.

The HFS+ and HFSX file systems used by Mac OS X support resource forks
natively. They are accessible through the Posix APIs not by prepending a dot to
the file name (given how often adjunct files with this naming convention are
created by UNIX facilites such a convention would cause all sorts of havoc) but
rather by appending "/rsrc" - it's a sort of file-in-a-pseudo-subdirectory. And
while it used to be true that most utilities on Mac OS X were unaware of
resource forks, this has changed in recent versions - most of the UNIX-derived
OS utilieis now handle resource forks correctly.

However, resource forks are used to store file data, not metadata, so none of
this is especially relevant to the metadata issue. Now, Apple apparently had
some notion of extending the fork model to allow arbitrary named forks. Such a
facility could in theory be used to store metadata, although using such a
heavyweight mechanism to store metadata never seemed like a very good idea to
me. (A fork is in effect a separate file as far as the on-disk structures are
concerned, so even if you hide this from applications the overhead is still
there.) Regardless, Apple has backed away from actually completing the
implementation of this facility. My understanding is if you try to create forks
with other names with the provided APIs you get an error.

What Apple has done instead is implement full support for file metadata
separate from forks. This IMO is the right approach. Apparently there's a
separate B-tree for this somewhere in HFS+ - you don't even need to be
running HFSX for it to work. It took Apple a while to get this done, but API
routines to access this stuff now exist, and even better, they are apparently
the same as the ones used on Linux.

This support also works on other filesystems that lack built in metadata
storage - the metadata is instead stored in a separate file with "._" prepended
to the name. (I know there are ext2 and ext3 drivers for Mac OS X but I don't
know if or how they handle the metadata issue - hopefully they store it in a
way that's compatible with usage on Linux.)

There's lots more to know about metadata on Mac OS X. This article
describes the landscape pretty well:

    http://arstechnica.com/reviews/os/macosx-10.4.ars/6

So what's the bottom line? The bottom line is that you appear to be focusing on
the wrong problem along several dimensions. First, charset information
specifically isn't as interesting or essential as you claim, and the degree to
which is it interesting is dropping for a variety of reasons. Second, you
appear to have missed the larger and much more important problem of not having
correct parameterized type information available. (And we haven't even
discussed the many other sorts of metadata, like say language information, that
is also useful to have.) Third, your focus on getting metadata support into
filesystems is mostly misplaced - this is a solved problem in a lot of cases.
And fifth, you don't seem to appreciate the difficulty of getting everyone to
agree to actually use filesystem metadata to solve any of these problems. This
last is a complete showstopper and I dispair of there ever being significant
progress in this area because of it.

				Ned

To: Jacob Palme <[email-masked]>
Cc: IETF mailing list on MIME and e-mail <[email-masked]>, Lars Enderin <[email-masked]>
Subject: Re: Charset mandatory in unix/linux
From: "Perry E. Metzger" <[email-masked]>
Date: Sun, 12 Mar 2006 13:15:30 -0500
Sender: [email-masked]

Jacob Palme <[email-masked]> writes:
> Would it be possible to get the people responsible for the
> file systems in Unix and Linux to add a mandatory charset
> attribute to all text files?

Not very likely at all I would say. The idea of "file attributes" is
pretty alien to Unix in general, and there is no central control over
all the various scattered forms of Unix, which makes it even harder.

Perry

Date: Sun, 12 Mar 2006 11:31:50 +0100
To: IETF mailing list on MIME and e-mail  <[email-masked]>
From: Jacob Palme <[email-masked]>
Subject: Charset mandatory in unix/linux
Cc: Lars Enderin <[email-masked]>
Sender: [email-masked]

The charset parameter is mandatory in the MIME content-type
attribute. However, such a parameter is not mandatory in
Unix or Linux. This is causing more and more problems, when
people have a mixture of files with different charsets,
which you easily get when you download files from the
Internet or receive them via e-mail.

Would it be possible to get the people responsible for the
file systems in Unix and Linux to add a mandatory charset
attribute to all text files? Best is probably to add a
generalized property list to files, so that also other
properties than charset can be added in the future.

The advantage would be that programs which transport files
across the Internet, such as e-mail, ftp and http, would
more often use the correct charset and not munge the files
by giving then an incorrect charset. The commonly occuring
problem with incorrect charset would be reduced. Also local
problems such as text editors would benefit from knowing
the charset of a file.

(Mac OS earlier had a very good feature, you could add to
every file a property list called the "resource fork").
This still works in Mac OS X, but is less and less often
used, since Unix, on which Mac OS X is based, does not have
this facility. In Mac OS X the resource fork is stored in a
separate file whose file name starts with ".", in the same
directory as the file described).
-- 
Jacob Palme <[email-masked]> (Stockholm University and KTH)
for more info see URL: http://www.dsv.su.se/jpalme/