Subject: Re: [ldapext] UTF-8 full support in LDIF / LDIF v2

From: Yves Dorfsman <[email-masked]>
Date: Mon, 01 Jun 2009 16:26:05 -0600

Steven Legg wrote:

>>>>> See http://www.xmled.info/drafts/draft-sciberras-xed-eldif-05.txt

I did look at it, personally I find it difficult for humans, for diff'ing 
etc... XML has its place, but so does pure text.

>> Yes I was wondering about that, do we need multi-line values as work 
>> around because schemas aren't precise enough ?
> 
> No, we need them because sheets of paper, computer screens and RFCs are
> not infinitely wide. :-) Human-readability, line breaks and indenting tend
> to go hand-in-hand.

I've been thinking about this and trying a few things. My conclusion is that 
the best solution would be the good old here document.

objectclass: inetOrgPerson
organizationName:<<EOT
The two line
   company
EOT
sn: Jensen

With the following specifications:
Any of the following characters (or sequence in the case of CR+LF) can be 
used as a separator (<SEP>):

LF (U+000A), CR (U+000D), CR+LF (U+000D followed by U+000A), NEL (U+0085), 
FF (U+000C), LS (U+2028), PS U+2029)

Any sequence of characters can be used instead of EOT, but cannot include a 
separator character. The same sequence has to be used at the begining and 
the end.

Any UTF-8 character, except separators, can be used on each line.

Any separator can be used to separate the lines.

The text start after EOT<SEP>, and finishes with the last character before 
<SEP>EOT. The organization name in the example above is exactly two lines, 
the last separator is not part of the text.

No need or possibility to escape characters, no possibility of folding lines  .

-- 
Yves.
http://www.sollers.ca/

---

From: Yves Dorfsman <[email-masked]>
Date: Wed, 03 Jun 2009 10:10:05 -0600

I have not received any feed back on this message, and am not sure how to 
interpret that.

Is everybody getting tired of the debate ?
Is the idea of a here document syntax too ridiculous ?
Is UTF-8 support in LDIF not that important ?

Am I the only one thinking xml is not a good replacement for LDIF, if so, 
should we help Steven with the xmled RFC ?

Thanks.

Yves Dorfsman wrote:
> I did look at it, personally I find it difficult for humans, for 
> diff'ing etc... XML has its place, but so does pure text.
> 
> 
> I've been thinking about this and trying a few things. My conclusion is 
> that the best solution would be the good old here document.
> 
> Any of the following characters (or sequence in the case of CR+LF) can 
> be used as a separator (<SEP>):
> 
> LF (U+000A), CR (U+000D), CR+LF (U+000D followed by U+000A), NEL 
> (U+0085), FF (U+000C), LS (U+2028), PS U+2029)
> 
> Any sequence of characters can be used instead of EOT, but cannot 
> include a separator character. The same sequence has to be used at the 
> begining and the end.
> 
> The text start after EOT<SEP>, and finishes with the last character 
> before <SEP>EOT. The organization name in the example above is exactly 
> two lines, the last separator is not part of the text.
> 

-- 
Yves.
http://www.sollers.ca/

---

From: Howard Chu <[email-masked]>
Date: Wed, 03 Jun 2009 16:05:02 -0700

> Is the idea of a here document syntax too ridiculous ?

Using the same syntax as the existing Unix shells is problematic, since there 
are already shell scripts that construct LDIF using here-documents. So yes, it 
appears your suggestion is not very well thought out.

This is also the wrong place (or the wrong time) to be discussing multi-line 
values; the purpose of LDIF is to faithfully transmit values stored in a 
directory. As such, whether a value is "multi-line" or not, and how those line 
breaks are represented, is determined solely by the attribute's syntax. 
Anything you discuss here in the LDIF spec is purely cosmetic. E.g., for your 
example below, one can easily write:

organizationName: A Very Long
   Organization name just
   to make a point

but the breaks are immaterial. If you want to talk about actual multi-line 
values with some kind of embedded line-break then you need to define a syntax 
with those properties.

> Is UTF-8 support in LDIF not that important ?

LDIF is just a transfer format, like base64. Any string data carried within it 
is already understood to be in UTF-8. So yes, UTF-8 is important, but it's 
already a given. You're confusing the content with the container.

> should we help Steven with the xmled RFC ?

I don't see how the question is related, but IMO, the less XML exists, the better.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

---

From: Kurt Zeilenga <[email-masked]>
Date: Wed, 3 Jun 2009 17:44:50 -0700

> Is the idea of a here document syntax too ridiculous ?

There are a number of problems with it.  Personally, I think what  
Steven already offered (and likely implemented) is better, though I am  
concerned about line separators.  As Howard comments kind of suggests,  
when you have a value which is multi-lined, it's the syntax that  
controls what line separators are used, not the LDIF.  For instance,  
in some syntaxes, a $ is used to as a line separator.

The problem with your proposal, and Steven's, is that LDIF line  
separators and value line separators are one and the same thing.   
While one might be case occasionally, it cannot be expected to be  
generally the case.

LDIF is first and foremost an interchange format.  Conversion from  
LDAP PDU->LDIF Record->LDAP PDU MUST produce as output the input,  
octet for octet for every "data" component (the DN, every attribute  
description and associated values, etc.).

> Is UTF-8 support in LDIF not that important ?

LDIF being a proper interchange format is important.  UTF-8 support  
(other than being able to interchange values whose syntax is UTF-8  
encoded) is cosmetic.

Adding UTF-8 support does appear to be in support of improving LDIF as  
a proper interchange format.  It seems to be driven by other goals,  
such as trying to make LDIF files displayable.   Given that LDAP does  
not constrain attribute value syntaxes (even directory strings can  
contain arbitrary sequences of Unicode code points), the goal of  
making LDIF files displayable is not terribly feasible.

I note that even today, ASCII LDIF files might not display properly  
without special handling, such as for line separators.  But with  
UTF-8, line separators are only the tip of iceberg of display problems.

I'm not convinced that removing the ASCII restrictions will be a good  
thing.  Not only do I doubt it will have a net positive on  
displayability of LDIF for those who have a displayability goal (I  
don't this goal), I think it will have a net negative impact on  
interoperability and user confusion, such as when the user creates a  
file using one Unicode normalization algorithm, but is trying to set  
values which require a different Unicode normalization value.

> Am I the only one thinking xml is not a good replacement for LDIF,

There already exists a number of XML replacements of LDIF, such as  
DSML...  so I guess at least some do think XML is a good replacement  
for LDIF.

> if so, should we help Steven with the xmled RFC ?

What Steven and Andrew have done is define an extension for LDIF to  
allow XML values to be represented in a human-readable format instead  
of requiring the use use of base64.  Unfortunately his proposal has  
interchange issues (see the I-D's security considerations section).   
This, I think, is a fatal problem with this extension.

-- Kurt

>>>> Yes I was wondering about that, do we need multi-line values as  
>>>> work around because schemas aren't precise enough ?
>>>
>> I've been thinking about this and trying a few things. My  
>> conclusion is that the best solution would be the good old here  
>> document.
>> Any of the following characters (or sequence in the case of CR+LF)  
>> can be used as a separator (<SEP>):
>> LF (U+000A), CR (U+000D), CR+LF (U+000D followed by U+000A), NEL (U 
>> +0085), FF (U+000C), LS (U+2028), PS U+2029)
>> include a separator character. The same sequence has to be used at  
>> the begining and the end.
>> before <SEP>EOT. The organization name in the example above is  
>> exactly two lines, the last separator is not part of the text.
> _______________________________________________
> Ldapext mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ldapext

---

From: Jim Willeke <[email-masked]>
Date: Wed, 3 Jun 2009 20:54:19 -0400

I would agree with Kurt's and Howard's comments.
Perhaps what is really required is a good tool to display and edit the
existing LDIF formatted files that in a view that is deemed acceptable.

-jim
Jim Willeke

On Wed, Jun 3, 2009 at 8:44 PM, Kurt Zeilenga <[email-masked]>wrote:

> There are a number of problems with it.  Personally, I think what Steven
> already offered (and likely implemented) is better, though I am concerned
> about line separators.  As Howard comments kind of suggests, when you have a
> value which is multi-lined, it's the syntax that controls what line
> separators are used, not the LDIF.  For instance, in some syntaxes, a $ is
> used to as a line separator.
>
> The problem with your proposal, and Steven's, is that LDIF line separators
> and value line separators are one and the same thing.  While one might be
> case occasionally, it cannot be expected to be generally the case.
>
> LDIF is first and foremost an interchange format.  Conversion from LDAP
> PDU->LDIF Record->LDAP PDU MUST produce as output the input, octet for octet
> for every "data" component (the DN, every attribute description and
> associated values, etc.).
>
> LDIF being a proper interchange format is important.  UTF-8 support (other
> than being able to interchange values whose syntax is UTF-8 encoded) is
> cosmetic.
>
> Adding UTF-8 support does appear to be in support of improving LDIF as a
> proper interchange format.  It seems to be driven by other goals, such as
> trying to make LDIF files displayable.   Given that LDAP does not constrain
> attribute value syntaxes (even directory strings can contain arbitrary
> sequences of Unicode code points), the goal of making LDIF files displayable
> is not terribly feasible.
>
> I note that even today, ASCII LDIF files might not display properly without
> special handling, such as for line separators.  But with UTF-8, line
> separators are only the tip of iceberg of display problems.
>
> thing.  Not only do I doubt it will have a net positive on displayability of
> LDIF for those who have a displayability goal (I don't this goal), I think
> it will have a net negative impact on interoperability and user confusion,
> such as when the user creates a file using one Unicode normalization
> algorithm, but is trying to set values which require a different Unicode
> normalization value.
>
> There already exists a number of XML replacements of LDIF, such as DSML...
>  so I guess at least some do think XML is a good replacement for LDIF.
>
> What Steven and Andrew have done is define an extension for LDIF to allow
> XML values to be represented in a human-readable format instead of requiring
> the use use of base64.  Unfortunately his proposal has interchange issues
> (see the I-D's security considerations section).  This, I think, is a fatal
> problem with this extension.
>
> -- Kurt
>
>
>
>>
>> Thanks.
>>
>>
>> Yves Dorfsman wrote:
>>
>>> Steven Legg wrote:
>>>
>>> Any sequence of characters can be used instead of EOT, but cannot include
>>> a separator character. The same sequence has to be used at the begining and
>>> the end.
>>> before <SEP>EOT. The organization name in the example above is exactly two
>>> lines, the last separator is not part of the text.

---

From: Steven Legg <[email-masked]>
Date: Thu, 04 Jun 2009 17:29:07 +1000

Yves,

> No need or possibility to escape characters, no possibility of folding 

I would be more amenable to a here-document scheme if there were no
special treatment for line separators. That is, if the UTF-8 character
sequence between the end of the introducer and the beginning of the
terminator is the literal directory attribute value without any modification.
That means that any line separators in the attribute value come out
as is, and may be different from the usual LDIF line separator.
Some tools will display these line breaks nicely and some won't, but even
a misaligned display is more accessible than a base64 representation of
the value.

The downside of a here-document scheme is the need to scan the value
before hand to make sure it doesn't contain the terminator sequence,
which becomes awkward for streaming applications, so I'd be inclined to
keep the current scheme as an alternative.

Regards,
Steven

---

From: Steven Legg <[email-masked]>
Date: Thu, 04 Jun 2009 17:29:27 +1000

Kurt,

Kurt Zeilenga wrote:
> already offered (and likely implemented)

I have.

> is better, though I am 
> concerned about line separators.

Me too, but at least it doesn't matter for XML.

 > As Howard comments kind of suggests,
> controls what line separators are used, not the LDIF.  For instance, in 
> some syntaxes, a $ is used to as a line separator.
> 
> separators and value line separators are one and the same thing.  While 
> one might be case occasionally, it cannot be expected to be generally 
> the case.
> 
> PDU->LDIF Record->LDAP PDU MUST produce as output the input, octet for 
> octet for every "data" component (the DN, every attribute description 
> and associated values, etc.).

That's highly desirable for directory to directory interchange, but LDIF
is also used for composing data from various data sources to put in a
directory and to extract data from a directory to put in other data
sources. The octet-for-octet preservation usually doesn't apply in these
other cases and the need to turn line-based data such as XML documents
into base64 encodings is a serious impediment, hence the reason Andrew
and I wrote the Internet-draft.

> proper interchange format.  It seems to be driven by other goals, such 
> as trying to make LDIF files displayable.   Given that LDAP does not 
> constrain attribute value syntaxes (even directory strings can contain 
> arbitrary sequences of Unicode code points), the goal of making LDIF 
> files displayable is not terribly feasible.
> 
> without special handling, such as for line separators.  But with UTF-8, 
> line separators are only the tip of iceberg of display problems.
> 
> displayability of LDIF for those who have a displayability goal (I don't 
> this goal), I think it will have a net negative impact on 

The user is not going to directly enter a base64 encoded value. They
would use a tool that has those same normalization issues to produce
a UTF-8 character stream that has to be passed to another tool to turn
into base64. The issues exist anyway. It is just a question of where.

BTW, it is not my intent to replace LDIFv1 for pure directory to directory
interchange. I just need the option to produce and consume something more
amenable to human editing and batch processing, where appropriate.

Regards,
Steven

> DSML...  so I guess at least some do think XML is a good replacement for 
> LDIF.
> 
> allow XML values to be represented in a human-readable format instead of 
> requiring the use use of base64.  Unfortunately his proposal has 
>>> I've been thinking about this and trying a few things. My conclusion 
>>> is that the best solution would be the good old here document.

---

From: Michael Ströder <[email-masked]>
Date: Wed, 10 Jun 2009 12:43:14 +0200

> existing LDIF formatted files that in a view that is deemed acceptable.

Whatever "good tool" means to you. People tend to strongly disagree on
user interfaces. ;-)

It would boil down to the same problems you have when displaying
atribute values of an entry or presenting an input form for editing an
entry received via LDAP. This is not feasible without making assumptions
about data and/or looking closely at the subschema. But when solely
looking at a LDIF file you usually don't have the accompanying subschema
subentry available.

In my web2ldap you can switch the input form to LDIF (and back to
template/table form) when editing an entry. It accepts LDIF with UTF-8
values (converted from what the browser sent as charset) as input
unfolding multiple lines according to LDIF rules. Attribute values of
syntax DirectoryString containing non-ASCII chars are displayed
base64-encoded. This all is very simple and I'd not consider it to be
completely what's discussed here but at least it helps copying&pasting
(parts of) an LDIF export.

Ciao, Michael.

---

From: Yves Dorfsman <[email-masked]>
Date: Wed, 10 Jun 2009 23:34:45 -0600

Kurt Zeilenga wrote:
> already offered (and likely implemented) is better, though I am 

My problem with Steven's solution is that it is half LDIF, half XML. As I 
have mentioned earlier, I thing XML has its place, and maybe DSML should be 
fixed or re-invented, but for other application, I find the simplicity of 
LDIF an advantage ; unfortunately, having to base64 encode anything that's 
not 7 bit ASCII takes away some of its simplicity.

> when you have a value which is multi-lined, 

I have never run into the situation where I needed a multi-line value in an 
LDAP directory and was surprised by the need, but Steven brought this up 
earlier in the thread and said that he has a real-world need for it, and 
that the lack of a syntax for it in my proposition for an updated LDIF 
format was an issue.

> one might be case occasionally, it cannot be expected to be generally 

On the contrary, both Steven's solution and mine separate the lines but do 
not impose a line separator. Steven delimits his line with the <item></item> 
syntax, while I let the user choose any line separator out of the half dozen 
that has been used throughout the history of computing.

Our syntaxes are clear enough to let the import process know that those are 
separate lines, and the import process or the LDAP server can choose 
whichever line separator it wants. Making the line separator part of the 
data will create cross-platform issues. The LDAP server or actually the LDAP 
client should choose which line separator to use for its context/platform.

> as trying to make LDIF files displayable.   

Yes and no. My main reason for pushing this is diffing. You run into a 
problem and you want to diff the original and the problematic LDIF export of 
your directory. Having half of your LDIF file base64 encoded makes it a lot 
more difficult to pin point the problem. If you are right, that LDIF is 
purely for exchanging information between applications, never to be looked 
at by humans, then why is the current version so human friendly ?

> values which require a different Unicode normalization value.

How so ?
In the current version, you have to encode your Unicode to UTF-8, and then 
encode it again to base64. With my proposal, you would get the exact same 
UTF-8 strings as you do today, but they would not be (or would not have to 
be) encoded in base64.

This is not a rebuttal of your argument, I am truly interested in 
understanding what you mean here (in the same way I was glad somebody 
brought up the issue of Right To Left characters, as I had not thought about 
it). Maybe it is a problem that we can address ?

> This, I think, is a fatal problem with this extension.

So really this is the issue, should the value of the line separator be part 
of the data, or should everybody (LDIF importers/exporters, LDAP servers, 
LDAP clients) treat multi-line entries as just that, several lines, and 
choose their own line separator ? (in case I wasn't clear earlier, I am in 
favour of the latter).

-- 
Yves.
http://www.sollers.ca/

---

From: Yves Dorfsman <[email-masked]>
Date: Wed, 10 Jun 2009 23:48:11 -0600

> terminator is the literal directory attribute value without any 

That would be easy to implement, just pick one line separators instead of 
allowing several. But, is this the right way of doing things ?

When you deal with multi-line entries, isn't the LDAP client at the end of 
the chain modifying your line separator into a different one, one 
appropriate for its platform ?

If so, how much more difficult would it be to let every element in the chain 
use whichever line separator they want, and translate as needed ?

> keep the current scheme as an alternative.

It should be easy to pick a sufficiently complicated word that you know will 
never appear in any of your entries (e.g.: X**ENDOFTEXTDONOTUSE**X), and you 
can still scan for it from your entries and raise an exception in the 
unlikely case that it does appear (no need to scan ahead, still fool proof). 
This is a slight inconvenient, but the advantages of not having to escape 
characters, and of making the file much easier to read make it, in my 
opinion, worth it.

-- 
Yves.
http://www.sollers.ca/

---

From: Michael Ströder <[email-masked]>
Date: Thu, 11 Jun 2009 17:11:09 +0200

Yves Dorfsman wrote:
> When you deal with multi-line entries,

Let us be more precise here: I guess you meant multi-line attribute values.

> isn't the LDAP client at the end
> of the chain modifying your line separator into a different one, one

The LDAP client is supposed to send on the wire the correct LDAP string
encoding of the attribute value. Local display methods and handling of
user input data within the UI of the client is out-of-scope here.

> If so, how much more difficult would it be to let every element in the
> chain use whichever line separator they want, and translate as needed ?

???

> It should be easy to pick a sufficiently complicated word that you know
> will never appear in any of your entries (e.g.:
> X**ENDOFTEXTDONOTUSE**X),

I'm against the here-document scheme. Use decent LDIF module for your
favourite scripting language.

Ciao, Michael.

---

From: Michael Ströder <[email-masked]>
Date: Thu, 11 Jun 2009 17:12:17 +0200

Yves Dorfsman wrote:
>> such as trying to make LDIF files displayable.   
> 
> problem and you want to diff the original and the problematic LDIF
> export of your directory.

Sorry, but this is a very bad use-case. Given the fact that there are
decent LDIF modules for various programming languages you can implement
a robust delta-generation of LDIF data. Doing that with line-oriented
text tools is bad practice anyway!

> If you are right, that LDIF is purely for exchanging information
> between applications, never to be looked at by humans, then why is
> the current version so human friendly ?

Most parts of LDIF data is human-readable because most data is ASCII and
most consoles or other apps do not have problems displaying ASCII. This
does not say anything about whether to lift the ASCII restriction or not.

> In the current version, you have to encode your Unicode to UTF-8, and
> then encode it again to base64. With my proposal, you would get the
> exact same UTF-8 strings as you do today, but they would not be (or
> would not have to be) encoded in base64.

I agree with Yves here.

> So really this is the issue, should the value of the line separator be
> part of the data, or should everybody (LDIF importers/exporters, LDAP
> servers, LDAP clients) treat multi-line entries as just that, several
> lines, and choose their own line separator ?

Line separators for the attribute value itself is subject of the LDAP
syntax, isn't it? So if e.g. the LDAP syntax DirectoryString allows a
line separator (don't know) it has to be distinguished from the line
separator used for LDIF line folding.

Ciao, Michael.

---

From: Kurt Zeilenga <[email-masked]>
Date: Thu, 11 Jun 2009 19:13:46 -0700

On Jun 11, 2009, at 8:12 AM, Michael Ströder wrote:

> most consoles or other apps do not have problems displaying ASCII.

I do not concur with this statement.  I find that most applications  
can generally deal with one line separator convention in US-ASCII  
files and there are multiple line separator conventions.

> This does not say anything about whether to lift the ASCII  
> restriction or not.

The US-ASCII problems might suggest that use of Net-ASCII might have  
been more appropriate than US-ASCII.

-- Kurt

---

From: Kurt Zeilenga <[email-masked]>
Date: Thu, 11 Jun 2009 20:54:10 -0700

On Jun 10, 2009, at 10:34 PM, Yves Dorfsman wrote:

> My problem with Steven's solution is that it is half LDIF, half XML.  
> As I have mentioned earlier, I thing XML has its place, and maybe  
> DSML should be fixed or re-invented, but for other application, I  
> find the simplicity of LDIF an advantage ; unfortunately, having to  
> base64 encode anything that's not 7 bit ASCII takes away some of its  
> simplicity.
>
>> concerned about line separators.  As Howard comments kind of  
>> suggests, when you have a value which is multi-lined,
>
> I have never run into the situation where I needed a multi-line  
> value in an LDAP directory and was surprised by the need, but Steven  
> brought this up earlier in the thread and said that he has a real- 
> world need for it, and that the lack of a syntax for it in my  
> proposition for an updated LDIF format was an issue.

The problem is that what line separators to use is syntax specific (or  
possible attribute value convention specific).  For instance, an LDAP  
syntax could say multiple lines are to be separated by a particular  
set of code points (such as '$') or it could be simple be a convention  
that an attribute uses a particular set of characters.

To convert a LDIF specific line separator to an attribute value line  
separator requires not only knowledge of the LDAP schema, but  
knowledge of the attribute value conventions not expressed in the LDAP  
schema.

LDIF however was designed to allow a mechanical conversion of LDIF to  
LDAP PDUs without such knowledge.  Requiring implementations to have  
additional knowledge is quite problematic.

> On the contrary, both Steven's solution and mine separate the lines  
> but do not impose a line separator. Steven delimits his line with  
> the <item></item> syntax,

I think you confuse XML elements with line separators in XML data.   
Two very different things.

Steven's proposal represents line separators in the XML data using the  
<SEP> production.

> while I let the user choose any line separator out of the half dozen  

The problem here is how line separators in LDIF relate to line  
separators in the value.

Your approach assumes that whatever line separator the user chooses to  
use in the LDIF file is valid per the LDAP value syntax and any  
attribute type specific restrictions.

>
> Our syntaxes are clear enough to let the import process know that  
> those are separate lines, and the import process or the LDAP server  
> can choose whichever line separator it wants.

That requires LDAP schema and attribute type specific restrictions  
knowledge.

> Making the line separator part of the data will create cross- 
> platform issues.

Yes, but this is what your proposal seems to do.  (See below).

> The LDAP server or actually the LDAP client should choose which line  
> separator to use for its context/platform.

Today, LDIF line separators (<SEP>) are not part of the LDAP value.

That is,

foo: X
  Y

is equivalent to:

foo: XY

That is, the LDAP value is merely wrapped over multiple LDIF lines.

Now maybe you meant:

foo:<<EOT
X
Y
EOT

to also be equivalent to:

foo: XY

I don't see any value in offering yet another way to line wrap an LDAP  
value.

I took your proposal as representing foo attribute value "X<SEP>Y"  
where <SEP> was the sequence of characters used in the LDIF to  
separate X and Y.  This is problematic.

>
>
>> Adding UTF-8 support does appear to be in support of improving LDIF  
>> as a proper interchange format.  It seems to be driven by other  
>> goals, such as trying to make LDIF files displayable.
>
> Yes and no. My main reason for pushing this is diffing.

Diffing requires knowledge of LDAP schema.   One might store "foo" and  
get back "FOO" (or any other equivalent value) [See LDAP's data  
preservation requirements].

> You run into a problem and you want to diff the original and the  
> problematic LDIF export of your directory.  Having half of your LDIF  
> file base64 encoded makes it a lot more difficult to pin point the  
> problem.

As Michael noted, there exists LDIF diffing tools (most of which are  
likely not schema aware, and hence show equivalent values as being  
different).

> the current version so human friendly ?

I never claimed LDIF would not be looked at by humans.

I have stated that LDIF, with ASCII restrictions, already suffers from  
some display/editing issues namely due line separator issues.  Lifting  
the ASCII restriction will make these matters far worse.  (Line  
separators are the tip of the displayability/editability iceberg.)

>
>
>> I'm not convinced that removing the ASCII restrictions will be a  
>> good thing.  Not only do I doubt it will have a net positive on  
>> interoperability and user confusion, such as when the user creates  
>> a file using one Unicode normalization algorithm, but is trying to  
>> set values which require a different Unicode normalization value.
>
> How so ?
> In the current version, you have to encode your Unicode to UTF-8,  
> and then encode it again to base64. With my proposal, you would get  
> the exact same UTF-8 strings as you do today, but they would not be  
> (or would not have to be) encoded in base64.

I see two kinds of problems.

1) This would result in LDIF files which programs designed to display  
UTF-8 encoded Unicode text will not be able to display.  There is a  
user expectation that LDIF files be displayable.  With the current  
LDIF format, we do have some display issues (e.g., line separators),  
but they are limited.  If we remove the ASCII restrictions, we'll run  
into a wide range of display issues.

2) Today we have some separation between (non-ASCII) Unicode LDAP  
attribute values and their LDIF representation.  This separation, I  
think, has some value in that it instills LDIF syntactically  
requirements are LDAP attribute value syntax requirements are  
independent of each other.   Removing this separation, I think, will  
lead to user confusion.

> understanding what you mean here (in the same way I was glad  
> somebody brought up the issue of Right To Left characters, as I had  
> not thought about it). Maybe it is a problem that we can address ?

Keep what separation we do have between LDIF representation of an  
attribute value and the LDAP syntax for the attribute value.

>> allow XML values to be represented in a human-readable format  
>> instead of requiring the use use of base64.  Unfortunately his  
>> proposal has interchange issues (see the I-D's security  
>> considerations section).  This, I think, is a fatal problem with  
>> this extension.
>
> So really this is the issue, should the value of the line separator  
> be part of the data, or should everybody (LDIF importers/exporters,  
> LDAP servers, LDAP clients) treat multi-line entries as just that,  
> several lines, and choose their own line separator ? (in case I  
> wasn't clear earlier, I am in favour of the latter).

In RFC 2849, <SEP> is never part of the LDAP attribute value.

In ELDIF extension, certain <SEP>s are part of the LDAP attribute  
value.  This, I think, is problematic.

---

From: Yves Dorfsman <[email-masked]>
Date: Thu, 11 Jun 2009 21:58:51 -0600

Ok, let's split the two issue:

First, is it worth amending the standard to allow non-base64 UTF-8 ?
(forget about multi-line attribute value for a moment).

Looking back (way back) at this thread, I am not the only one seeing value here:

Michael Ströder wrote:
>> How so ? In the current version, you have to encode your Unicode to
>> UTF-8, and then encode it again to base64. With my proposal, you would
>> get the exact same UTF-8 strings as you do today, but they would not be

Ludovic Poitou wrote:
>> I went back through the mailing list archive, "charset" came up, but I
 >> can't make sense of who started with it.

 > I probably did.
 > In europe there are lots of directory users and administrators that have
 >  non ascii data they need to transform to LDIF.
 > With simple scripts, turning the data to UTF-8 and then to base64 encoded
 > is a pain. Allowing to specify a charset and then letting the tools doing
 > the conversion to UTF-8 automatically could simplify their life.

Can we try to think of what sort of problem non-encoded UTF-8 would create ?
If there is none, than, could we implement a version 2 that does this ?
The people that don't care for it, can carry on using version 1 ?

-- 
Yves.
http://www.sollers.ca/

---

From: Kurt Zeilenga <[email-masked]>
Date: Thu, 11 Jun 2009 22:14:13 -0700

Yves,

I think we need clear problem statements, a proposal for addressing  
each problem, a summary of why the proposal does address the problem  
and a statement of what (known) problems the proposal might introduce.

You have noted that the "diffing problem".  But here it's not clear  
whether a) you are wish to determine how two LDIF files differ, b) you  
wish to determine if two LDIF files represent the same LDAP requests,  
or c) you wish to determine how directory information represented in  
the LDIF represented LDAP requests differ and, if so, how.

For a), one can use file comparison tools to determine how two LDIF  
files differ.

For b), one can use an LDIF comparison tool.  For instance, such a  
tool know that
	a: foo
and
	a:foo
and
	a:f
	 o
	 o
and:
	a::Zm9v

represent the same value for the attribute a.

For c), a schema aware tool is need.  For instance,
	a:foo
and
	b:foo

might be equivalent as b could be an alias for a.

Or
	a:foo
and
	a:FOO

could be different but equivalent values.

Simply having a UTF-8 value encoding option doesn't generally solve  
any diffing problem.  In cases b and c, the LDIF encoding of the value  
doesn't matter.  In case a), you'd be introducing another way for LDIF  
files which represented the same LDAP requests to differ.
It seems to me that you want to use file comparision tools to  
accomplish b, you need to have a single way to represent in LDIF a  
particular LDAP request.  As long as there are multiple ways to  
represent a particular LDAP request in LDIF, file comparison tools are  
inadequate for b.

It may be you are thinking that a human would be better able to  
visually detect certain kinds of differences.  However, this assumes  
that removing the ASCII restriction would produce a readily display  
Unicode text.  That, I believe, is a bad assumption.  For instance,  
say a user diff(1) to LDIF files and get:

% diff -u ?.ldif
--- 1.ldif	2009-06-11 22:08:21.000000000 -0700
+++ 2.ldif	2009-06-11 22:08:47.000000000 -0700
@@ -1 +1 @@
-a: f??
+a:f??

where ? represents character not displayable on the user's screen.   
The user might assume the values here are same when they aren't.

This, I hope, illustrates why general file diff'ing tools, like  
diff(1), are suitable only for case a but not b.

Do you have a more compelling use case?

-- Kurt

---

From: Michael Ströder <[email-masked]>
Date: Fri, 12 Jun 2009 15:38:08 +0200

>>> (or would not have to be) encoded in base64.

Hmm, I agreed on SASL normalization not being an issue.
But you skipped my strong disagreement about your LDIF diffing use-case.

Excerpt from
http://www.ietf.org/mail-archive/web/ldapext/current/msg01759.html

Ciao, Michael.

---

From: Yves Dorfsman <[email-masked]>
Date: Sun, 14 Jun 2009 22:28:14 -0600

> But you skipped my strong disagreement about your LDIF diffing use-case.

I'm sorry, I did not try to put any word in your mouth, all I meant was that 
there is more than just me who see value in having non base64 encoded UTF-8.

I know you disagree with me about diffing, but was under the impression you 
saw value somewhere else.

Sorry for the confusion.

-- 
Yves.
http://www.sollers.ca/

---

From: Yves Dorfsman <[email-masked]>
Date: Sun, 14 Jun 2009 22:46:53 -0600

Kurt Zeilenga wrote:
> 
> I think we need clear problem statements, a proposal for addressing each 
> problem, a summary of why the proposal does address the problem and a 
> statement of what (known) problems the proposal might introduce.
> 
> wish to determine if two LDIF files represent the same LDAP requests, or 
> c) you wish to determine how directory information represented in the 
> LDIF represented LDAP requests differ and, if so, how.
> 

Yes, but because it displays unreadable characters, it makes it slightly 
more complicated. The better case (than simply diffing) I have given in the 
past is:

-the directory is broken
-you export to LDIF
-compare this LDIF with a previous one from when the directory was working.

I personally find that in such a case, being able to read the values makes 
it simpler and faster.

Other case: People have mentioned scripts that build LDIF file from other 
source, and have mentioned that encoding the values in base64 is an overhead 
they could do without.

> 
> Simply having a UTF-8 value encoding option doesn't generally solve any 
> diffing problem.  In cases b and c, the LDIF encoding of the value 

If you always do your export with the same tool, with the same options, then 
this shouldn't be an issue. A narrow case I admit, but this is one specific 
case I was thinking about.

> It may be you are thinking that a human would be better able to visually 
> detect certain kinds of differences.  However, this assumes that 
> removing the ASCII restriction would produce a readily display Unicode 
> text.

On a modern OS setup properly, Unicode text is displayed properly (my 
experience is with UTF-8 on Linux and solaris here).

>  That, I believe, is a bad assumption.  For instance, say a user 
> diff(1) to LDIF files and get:
> 
> % diff -u ?.ldif
> where ? represents character not displayable on the user's screen.  The 
> user might assume the values here are same when they aren't.
> 
> This, I hope, illustrates why general file diff'ing tools, like diff(1), 
> are suitable only for case a but not b.

Ok, but a) is still a valid case.

-- 
Yves.
http://www.sollers.ca/

---

From: Michael Ströder <[email-masked]>
Date: Tue, 16 Jun 2009 12:19:10 +0200

Yves Dorfsman wrote:
> I'm sorry, I did not try to put any word in your mouth, all I meant was
> that there is more than just me who see value in having non base64
> encoded UTF-8.
> 
> I know you disagree with me about diffing,

Yes, because that's better implemented by decent LDIF-aware tools.

> but was under the impression
> you saw value somewhere else.

I see some value. As I said my web2ldap accepts single-line UTF-8 input
in an LDIF editor multi-line input field but does not display UTF-8.

But if we run into more complicated issues like multi-line UTF-8 data
I'm against opening a can of worms with here document scheme or similar
complicated things.

Ciao, Michael.

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 16 Jun 2009 06:46:01 -0700

On Jun 14, 2009, at 9:46 PM, Yves Dorfsman wrote:

>> each problem, a summary of why the proposal does address the  
>> problem and a statement of what (known) problems the proposal might  
>> introduce.
>> whether a) you are wish to determine how two LDIF files differ, b)  
>> you wish to determine if two LDIF files represent the same LDAP  
>> requests, or c) you wish to determine how directory information  
>> represented in the LDIF represented LDAP requests differ and, if  
>> so, how.
> Yes, but because it displays unreadable characters, it makes it  
> slightly more complicated. The better case (than simply diffing) I  
> have given in the past is:
>
> -the directory is broken
> -you export to LDIF

You don't need UTF-8 for this.  A simple text diff tool will tell you  
that the base64 differs.
>
> I personally find that in such a case, being able to read the values  
> makes it simpler and faster.

But now you assume you'll be able to read them.  This is a bad  
assumption.  A simple diff tool might show two DIFFERENT values the  
same way, leading the human to believe there is no difference when  
there is a significant difference.  And then there the issue that a  
UTF-8 encoded Unicode file is not well-formed text, and trying to  
treat it as text will be quite problematic.  (see below)

> Other case: People have mentioned scripts that build LDIF file from  
> other source, and have mentioned that encoding the values in base64  
> is an overhead they could do without.

While base64 data is an additional step, it's an additional step that  
well supported today.  If we lift the ASCII restriction now, we'll  
have some implementations that do support it and some that don't, and  
that will cause interop problems.   I cannot support inducing such  
interop problems without a strong justification.

>> visually detect certain kinds of differences.  However, this  
>> assumes that removing the ASCII restriction would produce a readily  
>> display Unicode text.
>
> On a modern OS setup properly, Unicode text is displayed properly  
> (my experience is with UTF-8 on Linux and solaris here).

The key phrase here is "Unicode text".  And most such display tools  
not only require "well-formed" text, but often cannot display all  
"well-formed" text.  But removing the ASCII restriction does not make  
a LDIF file "Unicode text".  It makes it a series of Unicode code  
points and hence display of it as text will be quite problematic.  And  
even it's displayable, you have the problem that two values might  
display in the same way, making visual diff'ing problematic.

-- Kurt

---

From: Yves Dorfsman <[email-masked]>
Date: Tue, 16 Jun 2009 08:02:01 -0600

> I'm against opening a can of worms with here document scheme or similar

So being able to read/write directly in UTF-8 in the LDIF file will 
eliminate one step in the process ? Can you give more details ?

Listing the case we have so far (please argument against if you think a case 
is not valid):

-comparing at the file level by humans, to fix an issue for example ; this 
imply that the user is using a display properly setup for UTF-8 (Yves Dorfsman).

-simplifying scripts to import/export data form/to other format to/from LDIF 
(Steven Legg. Steven).

-simplifying online editors (Michael Stöder).

Steven:
Can you confirm this is what you were saying, otherwise reword ?

Kurt:
My case was the case a) in you email. In my last email I argued that 1) that 
case has value
2) your argument against it, not being able to display UTF-8 properly, is 
solved in modern environment setup for UTF-8

Michael:
Can you give us more explanation of what the problem is in the case of your 
editor. Would it simplify programing the editor ? Is it impossible to allow 
UTF-8 in your editor today, and this change would allow it ?

-- 
Yves.
http://www.sollers.ca/

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 16 Jun 2009 08:35:28 -0700

On Jun 16, 2009, at 7:02 AM, Yves Dorfsman wrote:

> Listing the case we have so far (please argument against if you  
> think a case is not valid):
>
> -comparing at the file level by humans, to fix an issue for example ;

Again, two UTF-8 strings may display in the same, meaning humans will  
NOT be able to visually detect the difference and hence won't be able  
to fix the difference.

> this imply that the user is using a display properly setup for UTF-8  
> (Yves Dorfsman).

Again, Unicode display programs are typically designed to display well- 
formed text, not arbitrary sequences of Unicode code points.

> -simplifying scripts to import/export data form/to other format to/ 
> from LDIF (Steven Legg. Steven).

As I noted before, Steve's suggestion is problematic in that it  
doesn't preserve Unicode code points in exchange of LDAP values.

> -simplifying online editors (Michael Stöder).

I seriously doubt that removing the ASCII restriction actually  
simplifies anything.  Removing the restriction adds another encoding  
option, which adds complexity and adds interop problems.

> My case was the case a) in you email. In my last email I argued that  
> 1) that case has value
> 2) your argument against it, not being able to display UTF-8  
> properly, is solved in modern environment setup for UTF-8

I think your assumptions about "modern environment setup for UTF-8"  
are flawed.  Good UTF-8 display programs (that display UTF-8 as text)  
generally expect the UTF-8 to be well formed text.   If the input to  
these programs is not well formed text, the display will not be well  
formed (though it may not be obvious to the viewer that it's not well- 
formed text).  And there are lots of "bad" UTF-8 display programs in  
wide use, ones which cannot handle even all well formed text (like  
bidirectional text).  And most every display program suffers from  
incomplete fonts.  And then there are private use code points, and  
various other things that even the best UTF-8 display programs have  
problems with.

Here's a contrived file that illustrates just some of the display  
problems removing the ASCII restriction will lead to.  (apologies if  
my MUA associates the wrong MIME information to this, but that just  
illustrates yet another problem folks will face (wrong MIME  
information) when exchange such files.)  BTW, none of my well setup  
Unicode "text" display programs handled this file well... because it's  
simply not well-formed text.

-- Kurt

---

From: Yves Dorfsman <[email-masked]>
Date: Tue, 16 Jun 2009 09:40:27 -0600

Thanks Kurt, sorry for my asking an answer in another email, it seems we 
were writing at the same time.

> You don't need UTF-8 for this.  A simple text diff tool will tell you 

True, diff will tell you that they are different, and where, but then you 
need to decode base64 to find what the text is in the two file to help you 
understand why they are different.

If there were no base64 encryption, then you would know right away, in most 
case, making it a much faster process.

> assumption.

I still don't understand why you are saying this. Can you give precise 
examples ?

>  A simple diff tool might show two DIFFERENT values the same 
> way, leading the human to believe there is no difference when there is a 
> significant difference.

So, for example, one file contains U+2026 (ellipsis, "…") while the other 
contains three U_002E (three times the full stop character, "..."), and the 
issue of duplication in Unicode 
(http://en.wikipedia.org/wiki/Duplicate_characters_in_Unicode) ?

Well, what do we do today when diff shows us that there is a difference in 
two ascii files, but our eyes can't see it ? We hex dump of the offending 
line(s), and go "ahah, I've got <CR> here, but <CR><LF> there). On the other 
hand what's the percentage of time you diff files and run into this problem ?

Having those base64 encoded will make the fact that they are different more 
obvious, but won't help us when we're trying to understand why they are 
different, we will still have to go base64 --> UTF-8 --> hex values.

Data can be different because it comes from different sources, in which case 
it is likely that people will use different duplicate of Unicode point to 
express the same thing, but data can also be different because it was 
changed, which is the case I am more concerned with (backup of LDAP servers 
etc...), and in this case, I doubt somebody will change the mu symbol to the 
micro symbol or something similar. It might happen for malicious reasons, 
but I expect that to be a small percentage.

> 
> The key phrase here is "Unicode text".  And most such display tools not 
> only require "well-formed" text, but often cannot display all 
> "well-formed" text.  But removing the ASCII restriction does not make a 
> LDIF file "Unicode text".  It makes it a series of Unicode code points 
> and hence display of it as text will be quite problematic.  

I don't understand what you are saying here. Could you give an example of a 
problematic case, or a link to an explanation ?

Why is a series of Unicode point not text ?
What do you mean by "well-formed text", or actually by text that is not 
well-formed ?

> And even 
> it's displayable, you have the problem that two values might display in 
> the same way, making visual diff'ing problematic.

This is the issue above (Unicode duplication) ? Right ?

 >> other source, and have mentioned that encoding the values in base64 is
 >> an overhead they could do without.
 >
 > well supported today.  If we lift the ASCII restriction now, we'll have
 > some implementations that do support it and some that don't, and that
 > will cause interop problems.   I cannot support inducing such interop
 > problems without a strong justification.

Adding a new version of the standard, does not remove the old one. You can 
tell people that you only work with version 1 LDIF files. The advantage of 
having a standard, is that the tools will slowly adopt it and will be able 
to deal with it. If we don't create a standard, we run into the situation 
where everybody creates their own format, and create their own tools to 
transform their format to the current LDIF one. If I remember right, Jim 
mentioned that he already uses his version of extended LDIF.

-- 
Yves.
http://www.sollers.ca/

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 16 Jun 2009 09:48:17 -0700

On Jun 16, 2009, at 8:40 AM, Yves Dorfsman wrote:

>
> Thanks Kurt, sorry for my asking an answer in another email, it  
> seems we were writing at the same time.
>
>
> Kurt Zeilenga wrote:
>>>
>>> -the directory is broken
>>> -you export to LDIF
>> You don't need UTF-8 for this.  A simple text diff tool will tell  
>> you that the base64 differs.
>
> True, diff will tell you that they are different, and where, but  
> then you need to decode base64 to find what the text is in the two  
> file to help you understand why they are different.

You have to compare the values octet by octet to understand why they  
are different.

> If there were no base64 encryption, then you would know right away,  
> in most case, making it a much faster process.

s/encryption/encoding/

I disagree.  Display of UTF-8 as text will hide differences that  
base64 or octet-by-octet comparison won't hide.

> I still don't understand why you are saying this. Can you give  
> precise examples ?

Consider a combining character such as diaeresis (U+0308).  It can  
only be combined with certain base characters.  It cannot be combined  
with arbitrary characters.   If one has an attribute intended to hold  
a single Unicode code point, then stores a diaresis as a value of that  
attribute, in the UTF-8 LDIF, one would end up diaeresis following a  
colon or a space, neither of which is valid text.

This is just the tip of the iceberg.  There are even issues where the  
LDAP values, being well-formed text by themselves, when composed into  
a file will result in the file not being well-formed text, such as in  
values which utilize bi-directional text.

IDNA when through all of this.  They found that they had to place  
significant restrictions on Unicode domain components to ensure that a  
domain name was well-formed Unicode text.

You have not suggested placing similar restrictions on LDIF but simply  
removing the ASCII restriction.

>
>> A simple diff tool might show two DIFFERENT values the same way,  
>> leading the human to believe there is no difference when there is a  
>> significant difference.
>
> So, for example, one file contains U+2026 (ellipsis, "…") while the  
> other contains three U_002E (three times the full stop character,  
> "..."), and the issue of duplication in Unicode (http://en.wikipedia.org/wiki/Duplicate_characters_in_Unicode 
> ) ?

Well, that's one case.  I had a few cases in mind: character  
equivalences and look likes (but non-equivalences).

>
> Well, what do we do today when diff shows us that there is a  
> difference in two ascii files, but our eyes can't see it ? We hex  
> dump of the offending line(s), and go "ahah, I've got <CR> here, but  
> <CR><LF> there).

But there are more subtles involved.  There are cases where the diff  
will show a difference, your eye will think it sees the difference,  
but the actual difference will be hidden.

This is why ldif difference tools are needed and why they have been  
written.  Removing the UTF-8 restriction won't make their job any  
easier.  It will just add another encoding option.

> On the other hand what's the percentage of time you diff files and  
> run into this problem ?

Well, where such problems are less likely, that makes the hidden  
difference problem even worse (as won't be an expected issue).   In  
some situations, the problem will be quite likely.

It was likely enough in IDNA for restrictions (*) to be placed on code  
point sequences used in domain names.  (* many of the INDA Unicode  
restrictions were to address other issues, but there are IDNA  
restrictions in place to ensure that domain components and domain name  
sequences of Unicode code points are well formed text.)

>
> Having those base64 encoded will make the fact that they are  
> different more obvious, but won't help us when we're trying to  
> understand why they are different, we will still have to go base64 -- 
> > UTF-8 --> hex values.

While I don't disagree that to understand the differences, one has to  
consider the value octet by octet, this in no way implies that we need  
another way to encode those octets.

> Data can be different because it comes from different sources, in  
> which case it is likely that people will use different duplicate of  
> Unicode point to express the same thing, but data can also be  
> different because it was changed, which is the case I am more  
> concerned with (backup of LDAP servers etc...), and in this case, I  
> doubt somebody will change the mu symbol to the micro symbol or  
> something similar.

Ah, but someone might apply normalization to the data.  This may well  
be worthy of detection.

> It might happen for malicious reasons, but I expect that to be a  
> small percentage.

Could be.  But could happen for non-malicious reasons.  Things could  
be "broken" because some re-normalized the data using the wrong  
normalization function.   Visual differencing won't detect this sort  
of change, as normalization doesn't change the display of the text (it  
just changes which code points are used to represent the text).

>> "well-formed" text.  But removing the ASCII restriction does not  
>> make a LDIF file "Unicode text".  It makes it a series of Unicode  
>> code points and hence display of it as text will be quite  
>> problematic.
>
> I don't understand what you are saying here. Could you give an  
> example of a problematic case, or a link to an explanation ?

See the bare combining character illustration above.

> Why is a series of Unicode point not text ?

For a series of Unicode code point to be text, certain restrictions  
must be maintained.  For instance, U+0308 by itself is not text.  It  
has to be combined with an appropriate base character to be text.

> What do you mean by "well-formed text", or actually by text that is  
> not well-formed ?

Well, even where a sequence of code points represent a sequence of  
characters and control information, that sequence itself may not be  
well-formed text.

It is also important to note that the sequence composed of two  
sequences of code points each representing well formed text may not be  
well-formed text.  That is, WF(A + B) may be false even where WF(A)  
and WF(B) are true.   BIDI is case where this often happens.

>
>> And even it's displayable, you have the problem that two values  
>> might display in the same way, making visual diff'ing problematic.
>

It's a problem of equivalences, look-a-likes, etc.

> have
> Adding a new version of the standard, does not remove the old one.  
> You can tell people that you only work with version 1 LDIF files.  
> The advantage of having a standard, is that the tools will slowly  
> adopt it and will be able to deal with it.  If we don't create a  
> standard, we run into the situation where everybody creates their  
> own format, and create their own tools to transform their format to  
> the current LDIF one.

When LDIFv1 was introduced, it brought with it interop problems with U- 
Mich LDIF (v0).  The standards community however well justified the  
standardization of LDIF, the changes from U-Mich LDIF.  But to this  
day, we still suffer from some interop problems tied specifically to  
differences between U-Mich LDIF and v1.

> If I remember right, Jim mentioned that he already uses his version  
> of extended LDIF.

There are actually a number of extended versions of LDIF in use  
today... and a number of interop problems exist because of this.   
Standardization won't necessary remove the desire to have extended  
versions of LDIF, as the standard LDIF is never going to met  
everyone's desires for additional capabilities.

In my opinion, in revising Standard Track LDIF, we should focus on  
changes that improve data interchange functionality and  
interoperability.  We should dismiss changes that our outside of the  
primary goal of having an interoperable interchange format, includes  
dismissing changes intended to improve the "human readability" of LDIF.

-- Kurt

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 16 Jun 2009 12:01:41 -0700

Another issue...

Though LDIFv1 doesn't actually allow multi-byte UTF-8 anywhere (except  
in base64 encoded strings), RFC 2849 did say:
   Implementations SHOULD NOT fold lines in the middle of a multi-byte  
UTF-8 character.
(this was a left over, I think, when non-base64 encodings of UTF-8 was  
being considered for LDIFv1.  That is, the IETF seems to have already  
considered this issue when it was developing LDIFv1 but choose to  
require base64 of multi-byte UTF-8 characters.)
Anyways, if we remove the ASCII restriction and leave the above  
SHOULD, that would mean that implementations would have to deal with  
folding in the middle of multi-byte UTF-8 sequences, and that means  
the file as whole might not be valid UTF-8, and that would be  
problematic.
If we do remove the ASCII restriction (which I don't support), I think  
we install a couple of restrictions.
1) The resulting file MUST only be valid UTF-8 (minimally length  
encodings, a start octet must be followed by continuation octets,  
etc.) AND MUST be a valid Unicode code point (U+0000 through U+D7FF, U 
+E000 through U+10FFFF, inclusively) excluding the NUL (U+0000) code  
point.
2) Lines longer that 76 octets MUST be folded.  Folding of lines MUST  
NOT occur in the middle of a multi-byte UTF-8 character.   
Implementations need to take care that both of these requirements are  
met.
Both of these requirements will significantly increase the complexity  
of LDIF encoders.  Though the first is similar to current requirement  
to ensure that values encodes as SAFE-STRING met all of SAFE-STRING  
requirements, these checks are octet by octet.  Checking for valid  
UTF-8 and Unicode requires both an expansion of each UTF-8 sequence  
but the code point represented by that UTF-8 sequence.  And the second  
requires a bit more math, something experience has show implementors  
tend to get wrong.
We'd also need text explaining that format does not ensure the UTF-8  
represents well-formed text, or even text, and hence may not be  
displayable or editable.  And even displayable, users must be careful  
not to rely on visual inspection.
Also, users must be careful to avoid passing LDIF data through systems  
that might apply Unicode normalization or other transformations.  Such  
systems will likely lead to unintended changes to the LDAP data  
represented by the original LDIF (and/or cause the data to be non- 
conforming LDIF).  For instance, when emailing an LDIF file, both the  
sending and receiving user must be careful that their MUAs (and MTAs)  
are not altering the LDIF data.  Unfortunately, many MUAs and MTAs do  
alter text attachments (to conform to various conventions).  While  
such alterations may be desirable for text attachments, it is not  
desirable for LDIF data (which are commonly sent as sent as text  
attachments).
By raising these points, I hope to show that simply removing the ASCII  
restrictions will lead to problems, and even though some of them (such  
as invalid UTF-8) can be address by installing various restrictions,  
the simple fact that LDIF represents LDAP data not text will lead to  
various problems (such as unintended text file conversions, inability  
to use text processing programs, etc.).
-- Kurt

---

From: Howard Chu <[email-masked]>
Date: Tue, 16 Jun 2009 12:32:34 -0700

> to use text processing programs, etc.).

Agreed. Not to mention that to implement this properly will require complete 
schema knowledge at the time the LDIF is generated. (Otherwise, how do you 
distinguish a genuine octetString value, which cannot be safely represented in 
UTF-8, from a directoryString value...)

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 16 Jun 2009 13:25:50 -0700

On Jun 16, 2009, at 12:32 PM, Howard Chu wrote:

> Not to mention that to implement this properly will require complete  
> schema knowledge at the time the LDIF is generated. (Otherwise, how  
> do you distinguish a genuine octetString value, which cannot be  
> safely represented in UTF-8, from a directoryString value...)

Well, one could scan the value to see the octets are a valid UTF-8  
sequence of valid Unicode code points just as today most  
implementations scan data for octets within of SAFE-STRING.  The  
significant difference is the check is straight forward in LDIFv1, as  
it's san octet-by-octet check.  But if we allow UTF-8 sequences of  
valid Unicode points, each octet of the value must be checked to see  
that it's part of a valid UTF-8 sequence, and each UTF-8 sequence  
checked to see if encodes a valid Unicode code point.  And then  
wrapping becomes more complicated, etc..

And even with all of that, LDIF would still not be well-formed Unicode  
text.  And even if we solved that (by even more complex restrictions  
on what Unicode code point sequences can be represented as UTF-8  
instead of base64-encoded UTF-8), we'd have the problem of unintended  
Unicode transformations in transporting LDIF.  (We have this problem  
with LDIFv1, but it's generally limited to end-of-line characters.   
With UTF-8, data will be impacted.  For instance, consider MUAs (or  
the like) that might convert (on send or receive) text to Net-Unicode.)

I've expanded my UTF-8 LDIF with some more goofiness.

-- Kurt

---

From: Steven Legg <[email-masked]>
Date: Wed, 17 Jun 2009 09:54:14 +1000

Yves,

Yves Dorfsman wrote:
[snip]
> Listing the case we have so far (please argument against if you think a 
> case is not valid):
> 
> -simplifying scripts to import/export data form/to other format to/from 
> LDIF (Steven Legg. Steven).
> 

That is my main use case, including where some of the composition is done
by hand.

For the record, although XML documents can be more or less arbitrary UTF-8
character strings, most of the ones I deal with are pure ASCII. It is just
the line separators that require me to base64 encode such documents when
storing them as directory attribute values through LDIFv1.

Also, X.500 directories already lose something in the translation when outputting
as LDIF. For example, the choice in a DirectoryString is lost and if that
choice is teletexString then transcoding wipes out the exact octet encoding.
Most of the ad-hoc LDAP string encodings are lossy in some respect.
Such changes are tolerable because the resulting value in LDIF is the same
as far as the matching rules are concerned. Unicode normalization of the
extended LDIF output is a similar situation.

Regards,
Steven

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 16 Jun 2009 20:12:35 -0700

> extended LDIF output is a similar situation.

The loss you are talking about is inherent in LDAP not LDIF.  That is,  
LDIF does not lose anything (for the LDAP requests it's design to  
represent) in translation to/from LDAP.  I don't it is tolerable for  
an LDAP intermediate format to "lose" LDAP information.

Unicode normalization of the LDIF will be problematic where the  
normalization required by the directory is different.   For instance,  
if the value is expected to be NFD but NFC was applied breakage will  
occur.

-- Kurt

---

From: Steven Legg <[email-masked]>
Date: Thu, 18 Jun 2009 16:57:39 +1000

Kurt,

Kurt Zeilenga wrote:
> represent) in translation to/from LDAP.  I don't it is tolerable for an 
> LDAP intermediate format to "lose" LDAP information.

A "here" document mechanism where the UTF-8 character sequence between
the end of the introducer and the beginning of the terminator is the
literal directory attribute value without any modification would satisfy
the no loss requirement. We could always recommend base64 encoding the
entire file to transfer it across the network, or at least treating it as
a binary file, or suggest using LDIFv1 instead for such purposes.

For me, the convenience of being able to easily view, edit or compose the
content in files I'm uploading outweighs the need to be careful about
which text handling tools I use. If others have similar or overlapping
requirements it would be better for us to use the same specification even
if it never becomes a proposed standard RFC.

Would you feel better if we called it the LDAP Data Composition Format(LDCF)
instead of LDIFv2 or Extended LDIF ?

Regards,
Steven

> normalization required by the directory is different.   For instance, if 
> the value is expected to be NFD but NFC was applied breakage will occur.
> 
> -- Kurt
>

---

From: Michael Ströder <[email-masked]>
Date: Thu, 18 Jun 2009 12:15:14 +0200

Steven Legg wrote:
>>> Also, X.500 directories already lose something in the translation
>>> when outputting as LDIF.  For example, the choice in a
>>> DirectoryString is lost and if that choice is teletexString then
>>> transcoding wipes out the exact octet encoding. Most of the
>>> ad-hoc LDAP string encodings are lossy in some respect. Such
>>> changes are tolerable because the resulting value in LDIF is the 
>>> same as far as the matching rules are concerned. Unicode
>>> normalization of the extended LDIF output is a similar situation.
>>
> the no loss requirement.

Steven, like Kurt I cannot see how the loss of the choice in a
DirectoryString stored in a X.500 directory is prevented by the "here"
document mechanism since it's simply already lost when retrieving the
DirectoryString attribute via LDAP. (Well, one might think about
inventing a scheme with attribute sub-types to keep the choice...)

Ciao, Michael.

---

From: Michael Ströder <[email-masked]>
Date: Thu, 18 Jun 2009 12:31:46 +0200

Howard Chu wrote:
> Agreed. Not to mention that to implement this properly will require
> complete schema knowledge at the time the LDIF is generated. (Otherwise,
> how do you distinguish a genuine octetString value, which cannot be

Well, in the real world applications are already applying schema
knowledge to the LDAP/LDIF data produced or parsed. But not in the
formal way of a DSA or DUA using information from the subschema
subentry. Rather it's more hard-coded a-priori schema knowledge.

A LDIF writer/reader module could be called with data passed in as
Unicode strings vs. binary buffers to let the application decide which
attribute values to treat as textual string (to be encoded as UTF-8) and
which to treat as blob which needs base64-encoding.

Ciao, Michael.

---

From: Michael Ströder <[email-masked]>
Date: Thu, 18 Jun 2009 12:57:05 +0200

> domain name was well-formed Unicode text.

I'd like to learn more about the term "well-formed Unicode text". Do you
have a reference at hand? [NAMEPREP] and/or [STRINGPREP]?

I found

http://www.unicode.org/versions/Unicode5.1.0/#Conformance_Changes

which contains a replacement for the text in Unicode5.0 standard.
(Strange that one cannot simply download the recent version.)

> removing the ASCII restriction.

Would it help to define similar restrictions?

Stepping back a bit from the details of the interesting Unicode issues
posted here I wonder what the general strategy of the IETF regarding
these issues is? I remember discussions on the ietf-pkix mailing list
mentioning problems like these (e.g. when displaying subject names of
X.509 certs) without any real solution.

I think any system which takes (user) input, decodes it to a Unicode
code point sequence and display it to the user is affected by issues
with BIDI, combining characters and duplicate Unicode points.

Ciao, Michael.

---

From: Michael Ströder <[email-masked]>
Date: Thu, 18 Jun 2009 12:59:23 +0200

> I've expanded my UTF-8 LDIF with some more goofiness.

I like it. ;-)

Someone else once posted this look-alike Unicode sequence to a mailing
list (ietf-pkix) during a similar discussion:

Міϲrοѕоft Іnс

(see example LDIF below)

But personally I wonder whether this is a general problem which needs to
be solved within UIs.

Ciao, Michael.

------------------------- snip -------------------------
dn:: z3QnNGWz7Jyzr/RldC+ZnQg0IZu0YEsb3U9VGVzdGluZyxkYz1zdHJvZWRlcixkYz1k
 ZQ==
description: Weird Unicode sample demonstrating potential of name faking
o:: 0JzRls+ycs6/0ZXQvmZ0INCGbtGB
objectClass: organization

------------------------- snip -------------------------

---

From: Kurt Zeilenga <[email-masked]>
Date: Thu, 18 Jun 2009 06:34:50 -0700

On Jun 17, 2009, at 11:57 PM, Steven Legg wrote:

>>> when outputting
>> The loss you are talking about is inherent in LDAP not LDIF.  That  
>> is, LDIF does not lose anything (for the LDAP requests it's design  
>> to represent) in translation to/from LDAP.  I don't it is tolerable  
>> for an LDAP intermediate format to "lose" LDAP information.

First, for loss, let's only talk about loss between the data format  
and LDAP requests.   Loss between the data format and DAP or some  
other protocol ought to be beyond the scope of an LDAP data format.   
Loss of the DirectoryString CHOICE is an LDAP issue, not a LDAP data  
format issue.

So when I respond to your reply, I'm only going to consider loss of  
information between the file and protocol representations of the LDAP  
request.

> literal directory attribute value without any modification would  

Depending on particulars of value and "here" mechanism.   For  
instance, line separators are problematic in the mechanism suggested  
by Yves.

> We could always recommend base64 encoding the
> content in files

Have you tried to view/edit the UTF-8 file I sent out?

> I'm uploading outweighs the need to be careful about
> which text handling tools I use.

You will have to be quite careful in the tools you use.  If they are  
"text" tools, you'll have problems, because the LDIF file is not  
restricted to text.  And the tools encoding of values as text might be  
problematic (wrong normalization on a per value basis), etc..

For instance, any "text" tool which normalizes its output is generally  
unsuitable because that normalization will be incorrect for some set  
of values.

> If others have similar or overlapping

I wouldn't object to experimenting here, just so long as it not  
confused with standard LDIF.

But if one is going to try to solve a particular problem within the  
standard community, they should try to solve it generally.  That is,  
we should dismiss solutions which only work well for limited subsets  
of Unicode.

> instead of LDIFv2 or Extended LDIF ?

Well, if one was to have an experiment to develop a LDAP data format,  
which the whole content is say Net-Unicode, calling it something other  
than LDIF would be good to avoid confusing it with LDIF.  Things to  
tackle would include

- how to represent text values needing to be in different Unicode  
normalization (in the directory) than the required normalization,
- how to represent non-text Unicode values (arbitrary sequences of  
Unicode points),
- if Net-Unicode line separators are to be considered part of the  
value, how to represent values requiring other line separators, and
- bidirectional values.

Note: not requiring a particular normalization in the file is not a  
solution to first item, as that merely leaves it up to tools as to  
which normalization (if any) to apply.  So the first item just  
becomes, how to convey the normalization algorithm required for the  
value when represented in the LDAP request.

These are not simple problems and are unlikely to be solved well on  
the first publication, hence experimental track does seem appropriate  
for this feature.

I would like LDIF standard extensions work to be limited to improving  
data interchange capabilities (such as representation of LDAP  
responses), which is a much more straight forward problem, something  
we are far more likely to get right on the first publication.

-- Kurt

---

From: Kurt Zeilenga <[email-masked]>
Date: Thu, 18 Jun 2009 07:07:30 -0700

> have a reference at hand? [NAMEPREP] and/or [STRINGPREP]?

Unfortunately, I don't have a good reference handy.  The Unicode  
community might actually use different terms.  I tried to loosely  
define the terms in a prior email.  (I'm not a Unicode expert, just  
someone who's been involved Unicode issues (such as with IDNA) for a  
number of years.)

To rephrase MY definitions:

"text" merely implies that the sequence of Unicode code points  
represents a character.  In my ldif example, there is a colon followed  
by a combining code point.  This is an example of a sequence which  
doesn't represent "text".

"well-formed text" implies that not only is the sequence is "text" but  
that various other rules are met.  For instance, the sequence will  
result in proper directional display of bidirectional text.  There are  
some examples in the LDIF which show that introduction of line  
wrapping can break the directional display of the value.

> (Strange that one cannot simply download the recent version.)

You can download each chapter of the current version (each has a front  
page detailing copying restrictions, etc.).

> Would it help to define similar restrictions?

First, I don't see how any of this helps in LDAP data interchange, the  
primary purpose of LDIF.

Second, if one were to say that the resulting file has to be Net- 
Unicode (which I think at least means the file is "text"), you run  
into "data loss" problems due to unintended transformations.

> these issues is?

Punt.

> I remember discussions on the ietf-pkix mailing list

Yes.  The IETF tends to punt such issues to the user interface  
development community.  The IETF tends to restrict itself to design of  
protocols not design user interface (though the IETF does try to  
document user interface issues, especially those with security impact).

I think of LDIF as an alternative encoding of protocol data units,  
used for out-of-band transfer data between protocol peers.  That is, I  
punt the "user" as far as I can.

Others see LDIF as a user display format and user input format for  
LDAP data.  I argue that LDIFv1 didn't handle this well for ASCII and  
that handling this for ASCII (without data loss) is hard.  Solving it  
for Unicode, well, that's very, very hard.

-- Kurt

---

From: Steven Legg <[email-masked]>
Date: Fri, 19 Jun 2009 11:19:01 +1000

Michael,

Michael Ströder wrote:
> document mechanism

It isn't. My comment was in the context of LDAP to LDAP interchange.

 > since it's simply already lost when retrieving the

There is a scheme already. I could create LDIF dumps where every attribute
of DirectoryString syntax is output with ";binary", which would necessarily
cause the values to be base64 encoded. However, I don't think there are
many directory implementations that could cope with that so I accept that
X.500 to LDAP to LDIF is essentially lossy (but not in any way that has
proven to be critical). Consequently I'm not too concerned if LDAP to LDIFv2
isn't perfect either, though the "here" mechanism described above does at
least provide for lossless LDAP to LDIFv2 conversion.

The potential for an inadvertent change of normalization in the LDIFv2 if
it is edited doesn't overly concern me. Stringprep takes care of it for
matching purposes and any client that expects attribute values to be in,
or remain in, a particular normalization form is asking for trouble. The
values could be modified by some other client that changes the normalization
during editing and I wouldn't count on every directory implementation
preserving the exact character sequence it is given (though mine does).
If a client needs the values to be in a particular normalization form it
should do the conversion itself.

Regards,
Steven

---

From: Steven Legg <[email-masked]>
Date: Fri, 19 Jun 2009 11:20:22 +1000

Kurt,

Kurt Zeilenga wrote:
>>> is, LDIF does not lose anything (for the LDAP requests it's design to 
> First, for loss, let's only talk about loss between the data format and 
> LDAP requests.   Loss between the data format and DAP or some other 
> protocol ought to be beyond the scope of an LDAP data format.  Loss of 
> the DirectoryString CHOICE is an LDAP issue, not a LDAP data format issue.
> 
> Depending on particulars of value and "here" mechanism.   For instance, 
> line separators are problematic in the mechanism suggested by Yves.

Which is why I don't like it and why I say above "the literal directory
attribute value". The directory attribute value may have no LDIF line
separators at all which means either ELDIF/LDIFv2/LDCF would have to allow
arbitrarily long "lines" in the "here" document mechanism (my preference)
or fallback to base64 if the value has any "lines" (in the LDIF sense,
which might not correspond to the natural lines in the value) that are
above some specific length.

> Have you tried to view/edit the UTF-8 file I sent out?

Not until now. My LDIF editing tool of choice is vim which doesn't display the
non-ASCII characters sensibly, but that doesn't really matter to me. I'm more
interested in the structure of structured values in GSER or XML format. The
element and attribute names in an XML document are almost certainly in ASCII
and come from an international standard. The syntactically significant
punctuation is also in ASCII. I can see and manipulate the overall structure
of a complex value even if the contained character data is unreadable. If it's
not in English it doesn't mean much to me anyway so I have no particular
purpose in editing it. If it isn't a Latin script it's totally opaque to me
regardless of how vim displays it. What's important is that I can faithfully
cut and paste such unreadable values and write out the file without any
conversions taking place. If I did want to fiddle the character data, then I
would either do that through a directory client after uploading the data or
cut out the text, put it in a separate file, edit that with a proper text
editor then paste it back into the LDIFv2 using vim.

> problematic (wrong normalization on a per value basis), etc..

Which is a problem with the tool rather than a problem with the file
representation.

> unsuitable because that normalization will be incorrect for some set of 
> values.

That problem will exist with LDAP clients doing updates as well. We use
stringprep to mitigate such problems, but the clients have to take some
responsibility for their expectations.

> I wouldn't object to experimenting here, just so long as it not confused 
> with standard LDIF.
> 
> standard community, they should try to solve it generally.  That is, we 
> should dismiss solutions which only work well for limited subsets of 
> Unicode.
> 
>>

That's also a problem for LDAP clients that we don't need to solve
for ELDIF/LDIFv2/LDCF.

> - how to represent non-text Unicode values (arbitrary sequences of 

Literally.

> - if Net-Unicode line separators are to be considered part of the value, 
> how to represent values requiring other line separators, and
> - bidirectional values.

I don't have a need for anything other than the literal UTF-8 attribute
value. It's simple, preserves the LDAP value, and doesn't face problems
we don't already have with normal directory access using LDAP clients.

Regards,
Steven

> solution to first item, as that merely leaves it up to tools as to which 
> normalization (if any) to apply.  So the first item just becomes, how to 
> convey the normalization algorithm required for the value when 
> represented in the LDAP request.
> 
> These are not simple problems and are unlikely to be solved well on the 
> first publication, hence experimental track does seem appropriate for 
> this feature.
> 
> responses), which is a much more straight forward problem, something we 
> are far more likely to get right on the first publication.
> 
> -- Kurt

---

From: Yves Dorfsman <[email-masked]>
Date: Thu, 18 Jun 2009 21:59:54 -0600

> are far more likely to get right on the first publication.

Just when I was about to give up.... Anyway, if we go ahead with this, I am 
assuming we need to move the discussion to a different mailing list ?

Should it be an ietf mailing list, or should it happen somewhere else and 
come back here when it starts to take form ?

-- 
Yves.
http://www.sollers.ca/

---

From: Yves Dorfsman <[email-masked]>
Date: Thu, 18 Jun 2009 22:04:36 -0600

Steven Legg wrote:
> 
> Not until now. My LDIF editing tool of choice is vim which doesn't 
> display the non-ASCII characters sensibly, but that doesn't really matter

For whatever it's worth, vim (and vim, not vi, not nvi) is able to display 
non-latin character, but, you have to:
-have the right font installed on your system
-your environment needs to be setup correctly (LANG needs to be set for 
en_CA.UTF8 for example

-enough pixel. On most setup, you get that by switching xterm to huge font 
(<ctrl>, right click).

On the other hand, vim does not know how to display Right To Left text. 
Funnily enough, gedit handle all of it nicely (see the Ali Baba example I 
had sent a few months ago).

If anything this highlights what Kurt was saying, that in order to display / 
edit Unicode proper tools, editors that know how to, but it also highlights 
the fact that the people who need them tend to have the tools, and vice 
versa, if 7 bit ascii covers all your needs, then chances are you won't make 
a priority of moving to an environment that supports Unicode well, using new 
tools etc...

-- 
Yves.
http://www.sollers.ca/

---

From: Michael Ströder <[email-masked]>
Date: Fri, 19 Jun 2009 09:40:01 +0200

Yves Dorfsman wrote:
> Michael, are you sure about your dn ?
> It does not seem to decode properly...

Sorry, messed it up in my MUA when manually editing line-breaks. ;-)

------------------------- snip -------------------------
dn:: bz3QnNGWz7Jyzr/RldC+ZnQg0IZu0YEsb3U9VGVzdGluZyxkYz1zdHJvZWRlcixkYz
 1kZQ==
description: Weird Unicode sample demonstrating potential of name faking
o:: 0JzRls+ycs6/0ZXQvmZ0INCGbtGB
objectClass: organization

------------------------- snip -------------------------

In Python:

>>> s="""bz3QnNGWz7Jyzr/RldC+ZnQg0IZu0YEsb3U9VGVzdGluZyxkYz1zdHJvZWRlcixkYz
...  1kZQ=="""
>>> s.decode('base64').decode('utf-8')
u'o=\u041c\u0456\u03f2r\u03bf\u0455\u043eft
\u0406n\u0441,ou=Testing,dc=stroeder
,dc=de'

Ciao, Michael.

---

From: Kurt Zeilenga <[email-masked]>
Date: Mon, 22 Jun 2009 18:23:23 -0700

On Jun 18, 2009, at 8:59 PM, Yves Dorfsman wrote:

> Just when I was about to give up.... Anyway, if we go ahead with  
> this, I am assuming we need to move the discussion to a different  
> mailing list ?

We actually need less LDAP lists.  It is quite fine to continue to use  
this list for LDAP extension engineering discussions, whether standard  
track or not.

-- Kurt

---

From: Kurt Zeilenga <[email-masked]>
Date: Mon, 22 Jun 2009 19:06:29 -0700

On Jun 18, 2009, at 6:19 PM, Steven Legg wrote:

> matching purposes

Not for userPassword and the like.

Not for value syntaxes which require a specific normalization to be  
applied else result in a syntax error.

And, end-of-line characters appearing in values are not required to be  
base64'ed or otherwise escaped, there will inadvertent change of end- 
of-line characters to deal with.

LDIFv1 avoided such problems by limiting the characters in values that  
could appear without being base64'ed to a subset of the ASCII subset  
of characters.  These issues haven't gone away since the introduction  
of LDIFv1.

> and any client that expects attribute values to be in,
> or remain in, a particular normalization form is asking for trouble.

If a technical specification says an attribute value is to be in a  
particular Unicode normalization form, then all clients supporting  
that technical specification need to be provide the values of that  
attribute in a particular Unicode normalization form.

> The

If the normalization is specified as part of the LDAP syntax for the  
attribute value syntax, it follows that there would be a requirement  
for directory servers to preserve that normalization.  Or the value  
might be stored in an octet string (like userPassword) and the server  
required to preserve the octets and hence the normalization.

> should do the conversion itself.

We already have one standard attribute, userPassword, where values  
(when text) SHOULD to be provided in a particular Unicode normalization.

-- Kurt

---

From: Yves Dorfsman <[email-masked]>
Date: Tue, 23 Jun 2009 09:00:52 -0600

Kurt Zeilenga wrote:
> 
>> Just when I was about to give up.... Anyway, if we go ahead with this, 
>> I am assuming we need to move the discussion to a different mailing 
>> list ?
> 

Ok. I will have intermittent access to email throughout July, if nothing 
happens here by early august, I will re-start this discussion, focusing on a 
Data Composition Format (to re-use Jim's idea) then.

-- 
Yves.
http://www.sollers.ca/

---

From: Steven Legg <[email-masked]>
Date: Wed, 24 Jun 2009 17:54:16 +1000

Kurt,

> Not for userPassword and the like.

The extended format in the ELDIF specification in its current form can alter
end-of-line characters so I only use it for syntaxes where I know this is
harmless, which basically means XED syntaxes that are known to contain only XML
documents. Since the Octet String syntax doesn't fit this criterion, the
userPassword attribute never uses the extended format. If we generalize the
extended format to allow "here" documents with the unmodified literal LDAP
attribute value, then I would expect the extended format to be limited to
syntaxes that are known to produce exclusively UTF-8 character strings,
which would continue to exclude userPassword.

When I dump userPassword values they are encrypted, so even if the contents
of the octet string were UTF-8 to start with they probably aren't after
the encryption is done with it.

> applied else result in a syntax error.

I'm not aware of any such syntax.

> base64'ed or otherwise escaped, there will inadvertent change of 
> end-of-line characters to deal with.

It's best to be tolerant of such variations anyway since editing by
ordinary LDAP clients could create such inadvertent changes.

> could appear without being base64'ed to a subset of the ASCII subset of 
> characters.  These issues haven't gone away since the introduction of 
> LDIFv1.
> 
> particular Unicode normalization form, then all clients supporting that 
> technical specification need to be provide the values of that attribute 
> in a particular Unicode normalization form.

I don't know of any such specification for an existing syntax that produces
exclusively UTF-8 encodings. It would be most unwise for such a requirement
to be placed on the use of an existing syntax (e.g., Directory String)
because of the installed base of software that just wouldn't honour the
requirement. If it's a new syntax, then it wouldn't be known to existing
ELDIF implementations so attributes of the syntax wouldn't use the
extended format. Depending on the details, when I got around to implementing
it (i.e., making it known) I might explicitly exclude it from using the
extended format and/or have my ELDIF parser renormalize the values it is
importing.

> attribute value syntax, it follows that there would be a requirement for 
> directory servers to preserve that normalization.  Or the value might be
> stored in an octet string (like userPassword) and the server required to 
> preserve the octets and hence the normalization.

Putting it in an octet string would protect it from using the extended format,
as I envisage its applicability.

Regards,
Steven

> We already have one standard attribute, userPassword, where values (when 
> text) SHOULD to be provided in a particular Unicode normalization.
> 
> -- Kurt

---

From: Michael Ströder <[email-masked]>
Date: Wed, 01 Jul 2009 00:13:40 +0200

Kurt Zeilenga wrote:
> I think of LDIF as an alternative encoding of protocol data units, used
> for out-of-band transfer data between protocol peers.  That is, I punt
> the "user" as far as I can.

I'm not sure whether I fully understand the usage of the word "punt".
(Bear in mind I'm not a native English speaker.)

I agree that LDIF is just an alternative encoding of protocol data
units. So lifting the ASCII limitation in LDIF would IMO not introduce
any other problem a LDAP client with user interface does not already
have today (despite the new-line issue).

> Others see LDIF as a user display format

Strictly speaking nobody said that. ;-}

Seriously there's another layer between the LDIF data and the user - the
user interface. So if the IETF does not care about the user interface we
also don't have to care about that regarding LDIF.

Whether a particular LDIF is displayable by any means in some localized
context is not something we have to solve at this LDIF-formatted PDU
level because we also don't do it for normal LDAP PDUs.

> and user input format for LDAP data.

If the user enters data into an input field most times he uses his
normal keyboard and the display shows the input using fonts installed on
his system which matches his language. I don't think that there is
really a new problem introduced except a BIDI problem because the
attribute type names are written left-to-right. Maybe even this could be
solved in a decent user interface.

So I'd lift the ASCII limitation but I'm not keen on having to deal with
the "here" document scheme.

Ciao, Michael.

---

From: Kurt Zeilenga <[email-masked]>
Date: Tue, 30 Jun 2009 15:48:05 -0700

On Jun 30, 2009, at 3:13 PM, Michael Ströder wrote:

>> used

Removing the ASCII restriction will mean that users and systems will  
need to be far more careful in how they transfer LDIF data.   Today,  
one can email LDIF files about, FTP them without putting clients in  
"binary" mode, etc, because of the ASCII restriction.  Without this  
restriction, users and systems will have to be quite careful to ensure  
transfers preserve the LDIF data octet-for-octet.

> also don't have to care about that regarding LDIF.

It's not that the IETF doesn't care about UIs, it's just that UIs are  
typically outside the scope of expertise and areas of work the IETF  
normally undertakes.

> level because we also don't do it for normal LDAP PDUs.

Yes.  But folks here seem to be calling for LDIF to be displayable/ 
editable.

> his system which matches his language.

It's generally presumed that the user is using a client which is aware  
of the application specific semantics and syntaxes of the data he is  
entering, and if not, the user is obligated to do the work on his own  
to ensure application-specific semantics and syntaxes are met.

> I don't think that there is

A decent UI would not display attribute type names, but localized  
strings convening the appropriate semantics.  For instance, a decent  
UI ought not display "SN" (or the OID identifying the SN attribute  
type) but the local equivalent to "Surname", and then ensure valued  
entered met the syntactical requirements of SN attribute type.

> the "here" document scheme.

I would have less problem replacing the current "safe character"  
restriction with a broader set of "safe characters".  However, I  
rather not spend my time arguing over which particular characters are  
"safe enough" (for interoperability in data exchange).

-- Kurt