Subject: Byte order marks and character sets

From: Robert Burbidge <[email-masked]>
Date: Mon, 12 Aug 2002 16:40:32 +0100

I find the current state of documentation regarding character sets and byte
order marks somewhat unsatisfactory. 

1. Page 5 of current draft says "EPP use with current character sets other
than UTF-8 is not described in the document". Is this meant to imply that
the current draft assumes UTF-8 for all XML document interchange? If so it
would be preferable to say so in explicit terms rather than the double
negative at present.

2. Byte order marks are only considered in passing on page 76 in reference
to RFC3023. This RFC is relevant and useful if an underlying transport uses
MIME headers for passing EPP documents around. However, I don't think it
offers any useful advice fo transports that don't have a MIME type or
similar - such as the current TCP binding specification. Our EPP server
interprets incoming BOMs if the client provides them, and if no BOM is
provided UTF-8 encoding is assumed. But when we generate responses, should
we provide the UTF-8 BOM or not? In an ideal world all clients would have a
conforming XML parser and it would not matter for UTF8 data. However, having
seen some of the published EPP clients it looks like no one handles BOM
correctly for server responses. [Note: if you have written a client that
does handle BOM correctly please accept my apologies!]

3. TCP bindings section 5 "does not introduce or present any
internationalization or localization issues". If we don't make the BOM
situation explicit then we absolutely will be introducing long term i18n
issues for unicode (etc.) contact or domain names.

This may also have some bearing on the current discussions surrounding
international postal addresses, although this is not my field of expertise
... I'll leave it for others to consider.

Rob Burbidge

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Mon, 12 Aug 2002 12:30:14 -0400

> would be preferable to say so in explicit terms rather than the double

This is addressed in more detail in section 5, though perhaps a "UTF-8 is
RECOMMENDED" would make things more clear.  I don't think it wise to say
that only UTF-8 can be used as the protocol may clearly be useful in
localized environments where something other than UTF-8 or UTF-16 might be
more useful.

> does handle BOM correctly please accept my apologies!]

So what's your suggested improvement?  Section 4.3.3 of the XML rec (a
normative reference) already says that "XML processors must be able to use
this character to differentiate between UTF-8 and UTF-16 encoded documents",
and there's no mention of BOMs in RFC 2279 (UTF-8) so I don't know what you
mean by "the UTF-8 BOM".

> issues for unicode (etc.) contact or domain names.

Sorry, I don't see BOMs as a transport issue.  It's an encoding issue that
should be addressed in the core document, and as far as I can tell it
already is through the normative reference to the W3C XML recommendation.

-Scott-

---

From: Robert Burbidge <[email-masked]>
Date: Mon, 12 Aug 2002 17:51:33 +0100

>localized environments where something other than UTF-8 or UTF-16 might be

In that case perhaps you can add "UTF-8 is RECOMMENDED" in next draft.

> and there's no mention of BOMs in RFC 2279 (UTF-8) so I don't know what
you

See http://www.w3.org/TR/REC-xml#sec-guessing which mentions BOM for UTF-8.
My understanding of this is that a BOM does exist for UTF-8, although the
document is discussing non-normative interferencing of character encodings.
The .NET framework for example will generate the EF BB BF UTF-8 BOM. A few
minutes with Google also found
http://www.unicode.org/unicode/faq/utf_bom.html#25.

My problem seems to be that no clients read the UTF-8 BOM. I think they
should understand it even if it's optional; that's what a conforming XML
parser should do. If I'm right, then the problem is not strictly a problem
with the EPP spec but with client implementations which is outside the
strict scope of this discussion group but hey....

> already is through the normative reference to the W3C XML recommendation.

I think you are correct on this point. I just felt that it was
insufficiently clear taking the document set as a whole.

Rob Burbidge

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Mon, 12 Aug 2002 13:47:42 -0400

> strict scope of this discussion group but hey....

Hmm, OK, now I see what you're saying.  Sigh, if folks used XML declarations
this wouldn't be an issue...

This is an XML issue, not directly an EPP issue, but it could be described
in the EPP document since the normative references aren't quite clear.
Given that it's not a normative part of the XML recommendation and it's not
mentioned in RFC 2279 I'm not sure of the best way to address this at the
moment.  I'm going to ping some of the folks I worked with when writing my
IETF XML guidelines I-D to see what they have to say.

FWIW, I know of at least one XML parser that seems to digest a UTF-8 BOM
just fine: Xerces-J v2.0.2.  I just ran a quick test and it worked properly.

-Scott-

---

From: Robert Burbidge <[email-masked]>
Date: Tue, 13 Aug 2002 09:52:52 +0100

Getting advice from the XML folks seems like the right thing to do.
Rob

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Wed, 14 Aug 2002 09:32:05 -0400

> strict scope of this discussion group but hey....

OK, investigation complete.  Use of the BOM to identify UTF-8 is covered in
the errata for the XML 1.0 second edition recommendation:

http://www.w3.org/XML/xml-V10-2e-errata#E22

While it MAY be used it's certainly not necessary and most people I talked
to strongly suggested that it's use should be discouraged.  Thus, this is
what I intend to add to Section 2 of the EPP document to address the issue:

"Normative section 4.3.3 and non-normative appendix F of [XML] describe
use of a byte order mark (BOM) to identify the character encoding in
the absence of an XML declaration or encapsulating headers.  Appendix F
includes a BOM to represent UTF-8 encoding, though section 4.3.3
notes that a BOM is not needed to identify UTF-8 encoding.  Section
4.3.3 was later amended (see [XMLE]) to clarify that a BOM MAY be used
to identify UTF-8 encoding.  EPP clients and servers MUST accept a
UTF-8 BOM if present, though emitting a UTF-8 BOM is NOT RECOMMENDED."