Re: why do we need utf8 normalization when compare name?

Re: why do we need utf8 normalization when compare name?

[Aleksa Sarai]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

On 2020-03-02, lampahome <pahome.chen@mirlab.org> wrote:
> According to case insensitive since kernel 5.2, d_compare will
> transform string into normalized form and then compare.
>
> But why do we need this normalization function? Could we just compare
> by utf8 string?

The problem is that there are multiple ways to represent the same glyph
in Unicode -- for instance, you can represent Å (the symbol for
angstrom) as both U+212B and U+0041 U+030A (the latin letter "A"
followed by the ring-above symbol "°"). Different software may choose to
represent the same glyphs in different Unicode forms, hence the need for
normalisation.

[1] is the Wikipedia article that describes this problem and what the
different kinds of Unicode normalisation are.

[1]: https://en.wikipedia.org/wiki/Unicode_equivalence

-- 
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
<https://www.cyphar.com/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: why do we need utf8 normalization when compare name?

[Aleksa Sarai]

[-- Attachment #1: Type: text/plain, Size: 1346 bytes --]

On 2020-03-02, Aleksa Sarai <cyphar@cyphar.com> wrote:
> On 2020-03-02, lampahome <pahome.chen@mirlab.org> wrote:
> > According to case insensitive since kernel 5.2, d_compare will
> > transform string into normalized form and then compare.
> >
> > But why do we need this normalization function? Could we just compare
> > by utf8 string?
> 
> The problem is that there are multiple ways to represent the same glyph
> in Unicode -- for instance, you can represent Å (the symbol for
> angstrom) as both U+212B and U+0041 U+030A (the latin letter "A"
> followed by the ring-above symbol "°"). Different software may choose to
> represent the same glyphs in different Unicode forms, hence the need for
> normalisation.

Sorry, a better example would've been "ñ" (U+00F1). You can also
represent it as "n" (U+006E) followed by "◌̃" (U+0303 -- "combining
tilde"). Both forms are defined by Unicode to be canonically equivalent
so it would be incorrect to treat the two Unicode strings differently
(that isn't quite the case for "Å").

> [1] is the Wikipedia article that describes this problem and what the
> different kinds of Unicode normalisation are.
> 
> [1]: https://en.wikipedia.org/wiki/Unicode_equivalence

-- 
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
<https://www.cyphar.com/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: why do we need utf8 normalization when compare name?

[lampahome]

> Sorry, a better example would've been "ñ" (U+00F1). You can also
> represent it as "n" (U+006E) followed by "◌̃" (U+0303 -- "combining
> tilde"). Both forms are defined by Unicode to be canonically equivalent
> so it would be incorrect to treat the two Unicode strings differently
> (that isn't quite the case for "Å").

So utf8-normalize will convert  "ñ" (U+00F1) and "n" (U+006E) followed
by "◌̃" to a utf8 code, and both are the same, right?
Then compare it byte by byte.

Re: why do we need utf8 normalization when compare name?

[lampahome]

> Unicode normalisation will take the strings "ñ" (U+00F1) and "n◌̃"
> (U+006E U+0303) and turn them into the same Unicode string. Note that
> there are four kinds of Unicode normalisation (NFD, NFC, NFKD, NFKC), so
> what precise string you end up with depends on which form you're using.
> Linux uses NFD, I believe.


> And yes, once the strings are normalised and encoded as UTF-8 you then
> do a byte-by-byte comparison (if the comparison is case-insensitive then
> fs/unicode/... will case-fold the Unicode symbols during normalisation).
>

What I'm confused is why encoded as utf-8 after normalize finished?
From above, turn "ñ" (U+00F1) and "n◌̃" (U+006E U+0303) into the same
Unicode string. Then why should we just compare bytes from normalized.

Re: why do we need utf8 normalization when compare name?

[Theodore Y. Ts'o]

On Tue, Mar 03, 2020 at 06:13:56PM +0800, lampahome wrote:
> 
> > And yes, once the strings are normalised and encoded as UTF-8 you then
> > do a byte-by-byte comparison (if the comparison is case-insensitive then
> > fs/unicode/... will case-fold the Unicode symbols during normalisation).
> 
> What I'm confused is why encoded as utf-8 after normalize finished?
> From above, turn "ñ" (U+00F1) and "n◌̃" (U+006E U+0303) into the same
> Unicode string. Then why should we just compare bytes from normalized.

For the same reason why we don't upcase or downcase all of the letters
in a directory with case-folding.  The term for this is
"case-preserving, case-insensitive" matching.  So that means that if
you save a file as "Makefile", ls will return "Makefile", and not
"MAKEFILE" or "makefile".

Of course, if you delete or truncate "makefile", it will affect the
file stored in the directory as "Makefile", and the file system will
not allow a directory with case-folding enabled to contain "makefile"
and "Makefile" at the same time.

Simiarly, with normalization, we preserve the existing utf-8 form
(both the composed and decomposed forms are valid utf-8), but we
compare without taking the composition form into account.

Cheers,

					- Ted

P.S.  Some people may hate this, but if the goal is interoperability
with how Windows and MacOS does things, this is basically what they do
as well.  (Well, mostly; MacOS is a little weird for historical
reasons.)

P.P.S.  And before you comment on it, as one Internationalization
expert once said, I18N *is* complicated.  It truly would be easier to
teach all of the world to speak a single language and use it as the
"Federation Standard" language, ala Star Trek.  For better or for
worse, that's not happening, and so we deal with the world as it is,
not as we would like it to be.  :-)