Re: UTF-8 practically vs. theoretically in the VFS API (was: Re:

Re: UTF-8 practically vs. theoretically in the VFS API (was: Re:

[Nicolas Mailhot]

[-- Attachment #1: Type: text/plain, Size: 5756 bytes --]

|Alex Belit a écrit :
|
|On Mon, 16 Feb 2004, Marc Lehmann wrote:
|
|> > I have never claimed that the kernel really talk s UTF-8, and indeed, I
|> > would say that such a kernel would be terminally and horribly broken.
|>
|> And I'd say such a kernel would be highly useful, as it would standardize
|> the encoding of filenames, just as unix standardizes on "mostly ascii"
|> (i.e. the SuS).
|>
|> However, just as POSIX is a nice but very limited base, (mostly) ASCII
|> is a nice and very limited base. UTF-8 would also be a good base.
|
|  UTF-8 is dependent on Unicode, that is cumbersome [...] Enforcing UTF-8
| will burn the bridges to any other language support infrastructure or 
| encoding, right at the time when such infrastructure is likely to be created.

Quite the contrary. The current UTF-8 migration shows the major showstopper 
when changing filename encodings is right know you don't know what damned 
encoding to convert from.

With a clear policy (for *current* encodings) one can change.

Without one you're reduced to expensive guesswork (ie *humans* have to spend
 *days* checking the conversion worked as expected.)

I happen to hate imperial units. *My* country switched to full metric more than
 two hundred years ago. However I'll take a value in imperial units any day over
some number without explicit unit any day.

Implicit unit/encoding is a damn stupid thing to do. There are numerous examples
 of big expensive projects that failed because of this kind of misunderstanding. 
Many apps and humans need to interpret filenames to perform their job.

(BTW if anyone cares I was raised next to a computer which primary purpose was 
translating to a non-latin language. So I know quite a lot of the recipes for
"getting by" and having worthless archives after a few years)

|> 8-bit bytes as filenames is not a good base, however, since they enforce
|> a difefrent layer of interrpetation between the user and the kernel, and
|> this interpretation cannot be based on the locale nor the filesystem
|> itself, as there is no way to find out what encoding the filename is in.
|
|  This is a matter of GUI implementation. If someone cared about this, he
|would store language metadata with filename, too, however this is clearly
|contrary to the Unix filesystem design.

If you think filename interpretation is GUI-only stuff you're sadly mistaken.
Filename-based processing is widespread.

|> 8-bit bytes is convinient, but not useful for i18n environments. in the
|> past, it was also convinient and nobody cared, since everything was
|> either 8-bit or double-byte, and nobody exchanged files.
|
|  I did, and it worked _fine_. Everyone who is willing to use UTF-8 is
|free to do this right now, and everything will already work great for
|them. Writing software to deliberately enforce UTF-8 is something
|completely different from using UTF-8 for yourself.

|> This, however, is going to change, and the current methodology of "just
|> guess, you might be right" is a hindrance to this.
|
|  This was "going to change" for more than a decade already, or,
|alternatively, already happened if you listen to someone like Martin
|Duerst. The reality is, everything can pass UTF-8 already, yet people use
|other encodings for everything, too, and as long as they don't break,
|things work.

Till a certain point.
Past this point all the heuristics in the world won't help you and people 
suddenly revise their "work" definition.

| Breaking byte-value transparency in any place in the system
|is counterproductive

There is nothing transparent in the system for filename users.
Generalised guesswork is not transparency.

[...]

|> However, just as with URLs (which are byte-streams, too), byte-streams are
|> useless to store text. You need bytestreams + known encoding.
|
|  MIME has a perfectly usable standard for declaring encodings, and huge
|amounts of text (that may include filenames) are distributed by
|MIME-compliant or MIME-like protocols (mail and HTTP, to name two). 

Fine. Just convert all your filenames to garbage at see how great it is their 
contents are still readable because the file formats have encoding info. I'm
pretty sure you'll still miss your nice filenames.

Let me repeat my point :
1. filenames have a meaning
2. the meanings are important
3. they can not be reliably decoded without encoding info

Therefore encoding info needs to be added, using either FS metadata or a clear standard.
And I don't care if the standard is UTF-8, UCS-foo, egyptian hieroglyphs or whatever.
I want a f* standard. Every single person that had to work on the mess that results now
from many users using different incompatible locales on a single FS want a f* standard.

Someone wrote about it being akin to changing read() write() to do encoding conversion 
on the fly. This is blatantly false - filename contents are userspace-level and an app 
isn't expected to read other app files. And an app can use formats that declare file
encoding. But any app *will* need to read files it didn't generate because they happen to
reside in the same directory. And it *won't* be able to specify filename encoding because
the filename format belongs to the kernel so it's the *kernel* job to provide encoding 
info somewhere so app authors can interpret it correctly.

Sorry, we won't do it is not a valid answer.

App writers have solved what they could - file contents (which are encoding-aware now 
thanks to xml and friends). What they can not solve without kernel help is filename 
encoding - because filenames are shared unlike files, and it requires a system-level 
decision.

-- 
Nicolas Mailhot

[-- Attachment #2: Ceci est une partie de message numériquement signée --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: UTF-8 practically vs. theoretically in the VFS API (was: Re:

[Alex Belits]

On Tue, 17 Feb 2004, Nicolas Mailhot wrote:

> |  UTF-8 is dependent on Unicode, that is cumbersome [...] Enforcing UTF-8
> | will burn the bridges to any other language support infrastructure or
> | encoding, right at the time when such infrastructure is likely to be created.
>
> Quite the contrary. The current UTF-8 migration shows the major showstopper
> when changing filename encodings is right know you don't know what damned
> encoding to convert from.
>
> With a clear policy (for *current* encodings) one can change.
>
> Without one you're reduced to expensive guesswork (ie *humans* have to spend
>  *days* checking the conversion worked as expected.)

  Solution: don't convert. It's the user who made the filename, it's also
the user who will read it. If he wants to change the encoding, he
definitely knows which one is he using.


> I happen to hate imperial units. *My* country switched to full metric more than
>  two hundred years ago. However I'll take a value in imperial units any day over
> some number without explicit unit any day.

  Metric system is a real standard, proven by time. Unicode is at most
what Esperanto is to languages -- something that some people invented,
and it still didn't catch on.

> Implicit unit/encoding is a damn stupid thing to do. There are numerous examples
>  of big expensive projects that failed because of this kind of misunderstanding.
> Many apps and humans need to interpret filenames to perform their job.

  This is why people always mention units and charsets.

> (BTW if anyone cares I was raised next to a computer which primary
> purpose was translating to a non-latin language. So I know quite a lot
> of the recipes for "getting by" and having worthless archives after a
> few years)

  And I am Russian, and I have used non-latin language for my whole life.
So?

> |> 8-bit bytes as filenames is not a good base, however, since they enforce
> |> a difefrent layer of interrpetation between the user and the kernel, and
> |> this interpretation cannot be based on the locale nor the filesystem
> |> itself, as there is no way to find out what encoding the filename is in.
> |
> |  This is a matter of GUI implementation. If someone cared about this, he
> |would store language metadata with filename, too, however this is clearly
> |contrary to the Unix filesystem design.
>
> If you think filename interpretation is GUI-only stuff you're sadly mistaken.
> Filename-based processing is widespread.

  Filename-based processing is not simplified by Unicode because
pattern-matching works on byte sequences regardless of the charset.

> |Duerst. The reality is, everything can pass UTF-8 already, yet people use
> |other encodings for everything, too, and as long as they don't break,
> |things work.
>
> Till a certain point.
> Past this point all the heuristics in the world won't help you and people
> suddenly revise their "work" definition.

  There is no NEED for heuristics. It's just when charsets matter, their
names are already available off-band, and when they don't matter, there is
no point doing charset-specific operations.

> | Breaking byte-value transparency in any place in the system
> |is counterproductive
>
> There is nothing transparent in the system for filename users.
> Generalised guesswork is not transparency.

  Byte-value transparency, not "transparency" as a marketing buzzword.
Again, guesswork is unnecessary unless some piece of software does it.
There is absolutely no need to know the charset until the point when the
filename is presented to the user -- and this is beyond the filesystem
functionality.

> [...]
>
> |> However, just as with URLs (which are byte-streams, too), byte-streams are
> |> useless to store text. You need bytestreams + known encoding.
> |
> |  MIME has a perfectly usable standard for declaring encodings, and huge
> |amounts of text (that may include filenames) are distributed by
> |MIME-compliant or MIME-like protocols (mail and HTTP, to name two).
>
> Fine. Just convert all your filenames to garbage at see how great it is their
> contents are still readable because the file formats have encoding info. I'm
> pretty sure you'll still miss your nice filenames.

  When I see filenames in a language that I don't understand, they look
like garbage to me anyway. And so does Unicode.

> Let me repeat my point :
> 1. filenames have a meaning
> 2. the meanings are important
> 3. they can not be reliably decoded without encoding info


  This is great -- but this is a matter of _applications_ and _users_
interpreting them. If users wanted to use Unicode for everything, they
would already willingly adopt it.

> Therefore encoding info needs to be added, using either FS metadata or a clear standard.
> And I don't care if the standard is UTF-8, UCS-foo, egyptian hieroglyphs or whatever.
> I want a f* standard. Every single person that had to work on the mess that results now
> from many users using different incompatible locales on a single FS want a f* standard.

  And some people don't want either of those standards forced on them.
What are you going to do, beat them all up?

> Someone wrote about it being akin to changing read() write() to do encoding conversion
> on the fly. This is blatantly false - filename contents are userspace-level and an app
> isn't expected to read other app files. And an app can use formats that declare file
> encoding. But any app *will* need to read files it didn't generate because they happen to
> reside in the same directory. And it *won't* be able to specify filename encoding because
> the filename format belongs to the kernel so it's the *kernel* job to provide encoding
> info somewhere so app authors can interpret it correctly.

  All that an application does is in userspace. If any encoding-definition
system will be applied to filenames, it will be in userspace as well.

> Sorry, we won't do it is not a valid answer.

  It's a very valid answer -- recruiting kernel for the Unicode crusade is
subverting its nature. I really don't care if it's supposed to be good or
bad, it's just something that it never was supposed to do.

> App writers have solved what they could - file contents (which are encoding-aware now
> thanks to xml and friends).

  No, they are not. XML is heavily unicode-biased because it allows to
supply multiple languages per document yet only one charset per document.
This is wrong, and it is the most common cause for the use of Unicode in
any file format -- they are based on XML, and can't use multiple charsets,
so they have to use Unicode. This is precisely an example of subversion
of a standard to serve the biases of Unicode supporters, there never was a
technical reason why charset should be one per document yet language is
one per tag.

> What they can not solve without kernel help is filename encoding -
> because filenames are shared unlike files, and it requires a
> system-level decision.

  If everyone loved Unicode, there would be no such problem -- there would
be never files in anything else to begin with.

-- 
Alex

Re: UTF-8 practically vs. theoretically in the VFS API (was: Re:

[Robin Rosenberg]

On Tuesday 17 February 2004 19.55, Alex Belits wrote:
> On Tue, 17 Feb 2004, Nicolas Mailhot wrote:
> > Quite the contrary. The current UTF-8 migration shows the major showstopper
> > when changing filename encodings is right know you don't know what damned
> > encoding to convert from.

In most environments you do know, because you have a legacy encoding and a new 
UTF-8 encoding. There is no medicine that solves all compatibility problems, but the worst
symptoms can be alleviated while getting medicine to fix the disease.

> > With a clear policy (for *current* encodings) one can change.
> >
> > Without one you're reduced to expensive guesswork (ie *humans* have to spend
> >  *days* checking the conversion worked as expected.)
> 
>   Solution: don't convert. It's the user who made the filename, it's also
> the user who will read it. If he wants to change the encoding, he
> definitely knows which one is he using.

User's are clueless and you know that :-) They see garbage and blame the 
administrator or creator of the apps or whomever comes in handy.

>   When I see filenames in a language that I don't understand, they look
> like garbage to me anyway. And so does Unicode.
When I see I languages I don't understand, I understand that that's it. It doesn't
mean it actual IS garbage. The problem is when I see garbage simply because
the OS thinks it's unimporant.

> > Let me repeat my point :
> > 1. filenames have a meaning
> > 2. the meanings are important
> > 3. they can not be reliably decoded without encoding info
> 
>   This is great -- but this is a matter of _applications_ and _users_
> interpreting them. If users wanted to use Unicode for everything, they
> would already willingly adopt it.

Again users want characters, not bytes. Here up in the north we are among the
lucky ones that can still read a partially unlegible file name, because enough many
characters are usually just ascii. If a name was encoded in SJIS and you see them
interpreted as UTF-8 you'll a a string of pure garbage and you need to ask a bit
twiddler for help in decoding it simply becase ASCII characters are not likely to be
among the characters.

Note that I assume that the cases where there is a temporary working fix is the
two-locale (legacy+new) situation. And the only reason for the fix is to support 
migration from "old" charset to new charset without requiring martial arts.

For those application where it doesn't matter the point is moot. For most
users everything they see matters and the application cannot know what encoding
is used because there often is no relation between the application that creates and
the one that uses a file, and hence its name. The only place that can reliably store
the encoding of a filename is the file system, either as metdata, standard encoding on
disk or as a convention. It could be =?iso-8859-1?Q?=E5rsfest?=.rtf, fine with me. Not
adding a convention is that application can just work. A convention needs agreeing
upon between all apps. The kernel can just make it work.

-- robin





> > Therefore encoding info needs to be added, using either FS metadata or a clear standard.
> > And I don't care if the standard is UTF-8, UCS-foo, egyptian hieroglyphs or whatever.
> > I want a f* standard. Every single person that had to work on the mess that results now
> > from many users using different incompatible locales on a single FS want a f* standard.
> 
>   And some people don't want either of those standards forced on them.
> What are you going to do, beat them all up?
That's why there are locales, unfortunately these are volatile parameters that get lost. And it
doesn't have to be a a particular character encoding. It can be something per file system although
a fixed standard has the advantage that the filename can be read by anyone. 

> > Sorry, we won't do it is not a valid answer.
Agree.
>   It's a very valid answer -- recruiting kernel for the Unicode crusade is
> subverting its nature. I really don't care if it's supposed to be good or
> bad, it's just something that it never was supposed to do.
I thought the kernel was there to support user space. Ok, I know its' Just For Fun, but still
some usability is nice. Wh

> > App writers have solved what they could - file contents (which are encoding-aware now
> > thanks to xml and friends).
> 
>   No, they are not. XML is heavily unicode-biased because it allows to
> supply multiple languages per document yet only one charset per document.
> This is wrong, and it is the most common cause for the use of Unicode in
> any file format -- they are based on XML, and can't use multiple charsets,
> so they have to use Unicode. This is precisely an example of subversion
> of a standard to serve the biases of Unicode supporters, there never was a
> technical reason why charset should be one per document yet language is
> one per tag.
There's no need for multiple charsets when one can do all. So they added a
compromise that a charset could be specified to simplify creation of XML files
for legacy eight bit apps. I don't thing you understand XML at all. Enforcing
Unicode is helluvalot better than the "enforce" ASCII attitude of the past.

> > What they can not solve without kernel help is filename encoding -
> > because filenames are shared unlike files, and it requires a
> > system-level decision.
> 
>   If everyone loved Unicode, there would be no such problem -- there would
> be never files in anything else to begin with.

No matter what I think of unicode, I simply cannot have it yet everywhere.

-- robin

(Sorry for the suboptimal quotation here).

Re: UTF-8 practically vs. theoretically in the VFS API (was: Re:

[viro]

On Tue, Feb 17, 2004 at 10:56:30PM +0100, Robin Rosenberg wrote:
> 
> Again users want characters, not bytes. Here up in the north we are among the
> lucky ones that can still read a partially unlegible file name, because enough many
> characters are usually just ascii. If a name was encoded in SJIS and you see them
> interpreted as UTF-8 you'll a a string of pure garbage and you need to ask a bit
> twiddler for help in decoding it simply becase ASCII characters are not likely to be
> among the characters.

What I see is a string of pure garbage _here_.  On l-k.  Large part of that
garbage obviously coming from kooks with agenda.  Could we please take that
shite to more appropriate place?  Alt.sex.encodings.byte.byte.byte, whatever.

Re: UTF-8 practically vs. theoretically in the VFS API (was: Re:

[Robin Rosenberg]

On Wednesday 18 February 2004 01.08, viro@parcelfarce.linux.theplanet.co.uk used half a braincel and wrote:
> What I see is a string of pure garbage _here_.  On l-k.  Large part of that
> garbage obviously coming from kooks with agenda.  Could we please take that
> shite to more appropriate place?  Alt.sex.encodings.byte.byte.byte, whatever.

Sic! 

&>/dev/null

-- robin