Re: stty utf8

Re: stty utf8

[Clayton Weaver]

In an application that is taking input in utf-8,
if I want to search, compare, yada yada, I need
to convert to wchar_t first (for iswspace() et al),
then convert back to utf-8 for output (so grep et al
and operations on pathnames are not hosed by embedded
nul bytes in the output).

Why would not terminals do the same thing?

Done that way, Jamie's delete example is
backspace-space-backspace and remove sizeof(wchar_t)
from the input.

Ok, it takes more space than operating on the utf-8
encoding directly, but otherwise why not? All display
characters begin at the same offset from the
character before or after. It's up to the terminal
code to convert to/from utf-8 when talking to the rest
of the kernel.

Regards,

Clayton Weaver
<mailto: cgweav@email.com>

-- 
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm

Re: stty utf8

[Jamie Lokier]

Clayton Weaver wrote:
> In an application that is taking input in utf-8,
> if I want to search, compare, yada yada, I need
> to convert to wchar_t first (for iswspace() et al),

No, most search and compare operations are done directly on UTF-8,
and it's much more efficient to do that than convert to wchar_t.

> Why would not terminals do the same thing?

Because they're made of logic chips which you can't change now, have
ROMs, must be compatible with the VT100 and ANSI escape sequences,
must run over an 8-bit channel...

Mostly because that's the way terminals are.  You could design a _new_
terminal protocol but it's not going to be useful because you want to
talk with the existing terminals.

> Done that way, Jamie's delete example is
> backspace-space-backspace and remove sizeof(wchar_t)
> from the input.

You could store wchar_t in the terminal queue, but what would be the
point?  Removing a UTF-8 character from the input is _trivial_.

> Ok, it takes more space than operating on the utf-8
> encoding directly, but otherwise why not?

Because there's no point.

-- Jamie

Re: stty utf8

[Clayton Weaver]

[Note: I didn't have legacy hardware terminals in mind.
I was thinking more of a software terminal written from
scratch in utf-8 filesystems world.]

 > > Done that way, Jamie's delete example is
> > backspace-space-backspace and remove sizeof(wchar_t)
> > from the input.
 
> You could store wchar_t in the terminal queue, but what would be the
> point?  Removing a UTF-8 character from the input is _trivial_.
 
> > Ok, it takes more space than operating on the utf-8
> > encoding directly, but otherwise why not?
 
> Because there's no point.

Ie the convenience of iterating over a fixed size
character encoding in the terminal queue is negated
by other costs (legacy 8-bit filesystem support,
utf-8 control characters, et al)?

(I caught the discussion further down after posting.)

I'm spiritually with the "abort" people on this issue,
any opportunity to stab locale-overloaded character
values and arcane character encodings with shift
states in the heart is not to be passed up. "Every
character in every language has a unique value
everywhere that it is used, or we are not finished
fixing this yet."

But I agree with Linus that it is an admin policy
issue, not a kernel issue.

Thanks for your time,

Clayton Weaver
<mailto: cgweav@email.com>

-- 
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm

Re: stty utf8

[Markus Kuhn]

Andries reposted on 17 February a UTF-8 mode patch, prepared 5 years ago
by Bruno Haible, to add a new bit to struct termios that switches the
backspace behaviour in cooked mode between what is needed for a
single-byte character set and UTF-8.

This is a change in the API that might warrant a bit more discussion,
because code that uses that bit will have to be added to numerous
applications. In fact, Bruno's patch came originally with various
application patches such as

  ftp://ftp.ilog.fr/pub/Users/haible/utf8/stty.diff
  ftp://ftp.ilog.fr/pub/Users/haible/utf8/telnet.diff
  ftp://ftp.ilog.fr/pub/Users/haible/utf8/xterm.diff
  ftp://ftp.ilog.fr/pub/Users/haible/utf8/xterm2.diff

and numerous other applications will need to set that bit as well.

I would like to suggest that we could alternatively add a 2-bit field to
struct termios that allows us to switch between four modes for how to
execute the ERASE function in ICANON mode:

  0    auto-detect
  1    single-byte encoding (ISO 8859-*, KOI*, etc.)
  2    UTF-8
  3    EUC (AT&T's Extended Unix Code, what East Asia used before UTF-8)

Values 1 and 2 are just what Bruno's patch offers.

The autodetection mode (0) would assume, by default, for each line that
the encoding used is UTF-8, until it encounters a malformed UTF-8
sequence. In that case, it will switch to single-byte mode, for that
single line only.

Background: UTF-8 has the very useful property that ISO 8859-* strings
with non-ASCII characters contain with very high probability (>> 99%)
byte sequences that are not valid UTF-8. In particular, any string with
a single non-ASCII byte surrounded by two ASCII bytes can never be
mistaken for UTF-8. A tested simple (public domain) function to scan
whether a string is correct UTF-8 is now available at

  http://www.cl.cam.ac.uk/~mgk25/ucs/utf8_check.c

I would suggest that, by default, the Linux tty driver should run in
this UTF-8 auto-detection mode. Applications (and users via "stty utf8")
can still decide to hardwire ERASE to a fixed encoding, but the
auto-detection mode will make sure that for all the applications for
which we forget to set the bits correctly, backspace will still work
correctly almost always.

(I perfectly understand that auto-detection can be a potential source of
problems, but in this context, it seems the lesser evil. Backspace is
used only during manual text entry, which is rather error prone anyway.
I would argue that the remaining error rate is far less evil than not
having auto-detection and therefore many people using the wrong setting
all the time.)

There are already a number of applications out there ("talk" for
instance), where UTF-8 was added by auto-detection, as the protocol
provides no way of negotiation character sets. And it works rather well.

Executive summary: Dedicate two bits instead of just one in struct
termios to signal what backspace needs to know about the encoding.

Markus

-- 
Markus Kuhn, Computer Laboratory, University of Cambridge
http://www.cl.cam.ac.uk/~mgk25/ || CB3 0FD, Great Britain

Re: stty utf8

[Andries.Brouwer]

    > When you edit a line with the kernel's terminal line editor, when you
    > press the Delete key, it writes backspace-space-backspace and removes
    > one byte from the input.  That fails to do the right thing on UTF-8
    > terminals.

    Yes. I looked at that a year ago, and it should be pretty easy to make the 
    backspace code look more like the "delete word" code - except the "word" 
    is just a utf character.

    I didn't care enough to really bother fixing it - the fact is, that people 
    who care about UTF-8 tend to have to be in graphics mode anyway, and there 
    is something to be said for keeping the text console simple even if it 
    means it lacks functionality.

    But if somebody cares more than I do (hint, hint ;), I do think it should 
    be fixed.

    > There is no fancy environment setting which corrects this problem.
    > The kernel needs to know it's dealing with a UTF-8 terminal for basic
    > line editing to work.

    Yes. And I'd happily take patches for it.

OK - this sounds like a good moment to come with such patches.
The below is essentially a five-year-old patch by Bruno Haible.
It introduces utf8 mode, and the effect it has on erasing input characters.
(For the console more is needed, there are a few more patches there.)

Andries


P.S. Is there a reason for the different definition of
IMAXBEL in asm-parisc/termbits.h, or was that a typo?
If there is a reason, I think a comment is in order.

-----
diff -uprN -X /linux/dontdiff a/drivers/char/n_tty.c b/drivers/char/n_tty.c
--- a/drivers/char/n_tty.c	2003-12-18 03:58:04.000000000 +0100
+++ b/drivers/char/n_tty.c	2004-02-17 23:25:36.000000000 +0100
@@ -172,6 +172,16 @@ ssize_t n_tty_chars_in_buffer(struct tty
 	return n;
 }
 
+static inline int is_utf8_continuation(unsigned char c)
+{
+	return (c & 0xc0) == 0x80;
+}
+
+static inline int is_continuation(unsigned char c, struct tty_struct *tty)
+{
+	return I_IUTF8(tty) && is_utf8_continuation(c);
+}
+
 /*
  * Perform OPOST processing.  Returns -1 when the output device is
  * full and the character must be retried.
@@ -226,7 +236,7 @@ static int opost(unsigned char c, struct
 		default:
 			if (O_OLCUC(tty))
 				c = toupper(c);
-			if (!iscntrl(c))
+			if (!iscntrl(c) && !is_continuation(c, tty))
 				tty->column++;
 			break;
 		}
@@ -330,7 +340,7 @@ static inline void finish_erasing(struct
 static void eraser(unsigned char c, struct tty_struct *tty)
 {
 	enum { ERASE, WERASE, KILL } kill_type;
-	int head, seen_alnums;
+	int head, seen_alnums, cnt;
 	unsigned long flags;
 
 	if (tty->read_head == tty->canon_head) {
@@ -368,8 +378,18 @@ static void eraser(unsigned char c, stru
 
 	seen_alnums = 0;
 	while (tty->read_head != tty->canon_head) {
-		head = (tty->read_head - 1) & (N_TTY_BUF_SIZE-1);
-		c = tty->read_buf[head];
+		head = tty->read_head;
+
+		/* erase a single possibly multibyte character */
+		do {
+			head = (head - 1) & (N_TTY_BUF_SIZE-1);
+			c = tty->read_buf[head];
+		} while (is_continuation(c, tty) && head != tty->canon_head);
+
+		/* do not partially erase */
+		if (is_continuation(c, tty))
+			break;
+
 		if (kill_type == WERASE) {
 			/* Equivalent to BSD's ALTWERASE. */
 			if (isalnum(c) || c == '_')
@@ -377,9 +397,10 @@ static void eraser(unsigned char c, stru
 			else if (seen_alnums)
 				break;
 		}
+		cnt = (tty->read_head - head) & (N_TTY_BUF_SIZE-1);
 		spin_lock_irqsave(&tty->read_lock, flags);
 		tty->read_head = head;
-		tty->read_cnt--;
+		tty->read_cnt -= cnt;
 		spin_unlock_irqrestore(&tty->read_lock, flags);
 		if (L_ECHO(tty)) {
 			if (L_ECHOPRT(tty)) {
@@ -388,7 +409,12 @@ static void eraser(unsigned char c, stru
 					tty->column++;
 					tty->erasing = 1;
 				}
+				/* if cnt > 1, output a multi-byte character */
 				echo_char(c, tty);
+				while (--cnt > 0) {
+					head = (head+1) & (N_TTY_BUF_SIZE-1);
+					put_char(tty->read_buf[head], tty);
+				}
 			} else if (kill_type == ERASE && !L_ECHOE(tty)) {
 				echo_char(ERASE_CHAR(tty), tty);
 			} else if (c == '\t') {
@@ -403,7 +429,7 @@ static void eraser(unsigned char c, stru
 					else if (iscntrl(c)) {
 						if (L_ECHOCTL(tty))
 							col += 2;
-					} else
+					} else if (!is_continuation(c, tty))
 						col++;
 					tail = (tail+1) & (N_TTY_BUF_SIZE-1);
 				}
diff -uprN -X /linux/dontdiff a/include/asm-alpha/termbits.h b/include/asm-alpha/termbits.h
--- a/include/asm-alpha/termbits.h	2003-12-18 03:59:05.000000000 +0100
+++ b/include/asm-alpha/termbits.h	2004-02-17 23:18:21.000000000 +0100
@@ -56,12 +56,10 @@ struct termios {
 #define ICRNL	0000400
 #define IXON	0001000
 #define IXOFF	0002000
-#if !defined(KERNEL) || defined(__USE_BSD)
-  /* POSIX.1 doesn't want these... */
-# define IXANY		0004000
-# define IUCLC		0010000
-# define IMAXBEL	0020000
-#endif
+#define IXANY	0004000
+#define IUCLC	0010000
+#define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-arm/termbits.h b/include/asm-arm/termbits.h
--- a/include/asm-arm/termbits.h	2003-12-18 04:00:00.000000000 +0100
+++ b/include/asm-arm/termbits.h	2004-02-17 23:12:29.000000000 +0100
@@ -49,6 +49,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-arm26/termbits.h b/include/asm-arm26/termbits.h
--- a/include/asm-arm26/termbits.h	2003-12-18 03:59:04.000000000 +0100
+++ b/include/asm-arm26/termbits.h	2004-02-17 23:22:31.000000000 +0100
@@ -49,6 +49,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-cris/termbits.h b/include/asm-cris/termbits.h
--- a/include/asm-cris/termbits.h	2003-12-18 03:58:16.000000000 +0100
+++ b/include/asm-cris/termbits.h	2004-02-17 23:23:05.000000000 +0100
@@ -53,6 +53,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-h8300/termbits.h b/include/asm-h8300/termbits.h
--- a/include/asm-h8300/termbits.h	2003-12-18 03:59:58.000000000 +0100
+++ b/include/asm-h8300/termbits.h	2004-02-17 23:21:34.000000000 +0100
@@ -52,6 +52,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-i386/termbits.h b/include/asm-i386/termbits.h
--- a/include/asm-i386/termbits.h	2003-12-18 03:58:40.000000000 +0100
+++ b/include/asm-i386/termbits.h	2004-02-17 23:20:59.000000000 +0100
@@ -51,6 +51,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-ia64/termbits.h b/include/asm-ia64/termbits.h
--- a/include/asm-ia64/termbits.h	2004-02-05 19:55:21.000000000 +0100
+++ b/include/asm-ia64/termbits.h	2004-02-17 23:10:43.000000000 +0100
@@ -60,6 +60,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-m68k/termbits.h b/include/asm-m68k/termbits.h
--- a/include/asm-m68k/termbits.h	2003-12-18 03:59:46.000000000 +0100
+++ b/include/asm-m68k/termbits.h	2004-02-17 23:11:54.000000000 +0100
@@ -52,6 +52,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-mips/termbits.h b/include/asm-mips/termbits.h
--- a/include/asm-mips/termbits.h	2003-12-18 03:58:08.000000000 +0100
+++ b/include/asm-mips/termbits.h	2004-02-17 23:10:02.000000000 +0100
@@ -77,6 +77,7 @@ struct termios {
 #define IXANY	0004000		/* Any character will restart after stop.  */
 #define IXOFF	0010000		/* Enable start/stop input control.  */
 #define IMAXBEL	0020000		/* Ring bell when input queue is full.  */
+#define IUTF8	0040000		/* Input is UTF8 */
 
 /* c_oflag bits */
 #define OPOST	0000001		/* Perform output processing.  */
diff -uprN -X /linux/dontdiff a/include/asm-parisc/termbits.h b/include/asm-parisc/termbits.h
--- a/include/asm-parisc/termbits.h	2003-12-18 03:58:56.000000000 +0100
+++ b/include/asm-parisc/termbits.h	2004-02-17 23:07:35.000000000 +0100
@@ -52,6 +52,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0040000
+#define IUTF8	0100000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-ppc/termbits.h b/include/asm-ppc/termbits.h
--- a/include/asm-ppc/termbits.h	2003-12-18 03:58:57.000000000 +0100
+++ b/include/asm-ppc/termbits.h	2004-02-17 23:11:14.000000000 +0100
@@ -58,6 +58,7 @@ struct termios {
 #define IXANY		0004000
 #define IUCLC		0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-ppc64/termbits.h b/include/asm-ppc64/termbits.h
--- a/include/asm-ppc64/termbits.h	2003-12-18 03:59:04.000000000 +0100
+++ b/include/asm-ppc64/termbits.h	2004-02-17 23:09:02.000000000 +0100
@@ -66,6 +66,7 @@ struct termios {
 #define IXANY	0004000
 #define IUCLC	0010000
 #define IMAXBEL	0020000
+#define	IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-s390/termbits.h b/include/asm-s390/termbits.h
--- a/include/asm-s390/termbits.h	2003-12-18 03:59:29.000000000 +0100
+++ b/include/asm-s390/termbits.h	2004-02-17 23:20:26.000000000 +0100
@@ -59,6 +59,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-sh/termbits.h b/include/asm-sh/termbits.h
--- a/include/asm-sh/termbits.h	2003-12-18 03:58:18.000000000 +0100
+++ b/include/asm-sh/termbits.h	2004-02-17 23:23:40.000000000 +0100
@@ -51,6 +51,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-sparc/termbits.h b/include/asm-sparc/termbits.h
--- a/include/asm-sparc/termbits.h	2003-12-18 03:59:20.000000000 +0100
+++ b/include/asm-sparc/termbits.h	2004-02-17 23:13:13.000000000 +0100
@@ -78,6 +78,7 @@ struct termios {
 #define IXANY	0x00000800
 #define IXOFF	0x00001000
 #define IMAXBEL	0x00002000
+#define IUTF8   0x00004000
 
 /* c_oflag bits */
 #define OPOST	0x00000001
diff -uprN -X /linux/dontdiff a/include/asm-sparc64/termbits.h b/include/asm-sparc64/termbits.h
--- a/include/asm-sparc64/termbits.h	2003-12-18 03:58:18.000000000 +0100
+++ b/include/asm-sparc64/termbits.h	2004-02-17 23:22:06.000000000 +0100
@@ -80,6 +80,7 @@ struct termios {
 #define IXANY	0x00000800
 #define IXOFF	0x00001000
 #define IMAXBEL	0x00002000
+#define IUTF8	0x00004000
 
 /* c_oflag bits */
 #define OPOST	0x00000001
diff -uprN -X /linux/dontdiff a/include/asm-v850/termbits.h b/include/asm-v850/termbits.h
--- a/include/asm-v850/termbits.h	2003-12-18 03:58:04.000000000 +0100
+++ b/include/asm-v850/termbits.h	2004-02-17 23:23:25.000000000 +0100
@@ -52,6 +52,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/asm-x86_64/termbits.h b/include/asm-x86_64/termbits.h
--- a/include/asm-x86_64/termbits.h	2003-12-18 03:58:08.000000000 +0100
+++ b/include/asm-x86_64/termbits.h	2004-02-17 23:06:05.000000000 +0100
@@ -51,6 +51,7 @@ struct termios {
 #define IXANY	0004000
 #define IXOFF	0010000
 #define IMAXBEL	0020000
+#define IUTF8	0040000
 
 /* c_oflag bits */
 #define OPOST	0000001
diff -uprN -X /linux/dontdiff a/include/linux/tty.h b/include/linux/tty.h
--- a/include/linux/tty.h	2003-12-18 03:58:49.000000000 +0100
+++ b/include/linux/tty.h	2004-02-17 23:04:31.000000000 +0100
@@ -200,6 +200,7 @@ struct tty_flip_buffer {
 #define I_IXANY(tty)	_I_FLAG((tty),IXANY)
 #define I_IXOFF(tty)	_I_FLAG((tty),IXOFF)
 #define I_IMAXBEL(tty)	_I_FLAG((tty),IMAXBEL)
+#define I_IUTF8(tty)	_I_FLAG((tty),IUTF8)
 
 #define O_OPOST(tty)	_O_FLAG((tty),OPOST)
 #define O_OLCUC(tty)	_O_FLAG((tty),OLCUC)

Re: stty utf8

[Linus Torvalds]

On Tue, 17 Feb 2004 Andries.Brouwer@cwi.nl wrote:
> 
> OK - this sounds like a good moment to come with such patches.
> The below is essentially a five-year-old patch by Bruno Haible.
> It introduces utf8 mode, and the effect it has on erasing input characters.
> (For the console more is needed, there are a few more patches there.)

Heh. Looks fine and "obviously correct" by me. Can people verify that it
works for them? (I'd rather not put it in now, but I can put in the first
2.6.4-rc kernel).

> P.S. Is there a reason for the different definition of
> IMAXBEL in asm-parisc/termbits.h, or was that a typo?
> If there is a reason, I think a comment is in order.

I think it's a typo. But I've never used HP-UX, maybe it tries to be 
binary compatible and there's some other HP-UX bit in between those 
values?

		Linus

Re: JFS default behavior

[Robin Rosenberg]

On Saturday 14 February 2004 16.40, you wrote:
> The same goes for file names.  Filename is a sequence of bytes, no more and
> no less.  Anything beyond that belongs to applications.

Should be a sequence of characters since humans are supposed to use them and
it should be the same characters wheneve possible regardless of user's locale.

The  "sequence of bytes" idea is a legacy from prehistoric times when byte == character
was true. That is no longer the case and actually hasn't been for quite a while in
some parts of the world. Interchange is important. The application cannot handle
this since it cannot know what characters a byte string represents. Fixing it in the
kernel is the simple solution since it knows the locale. Its also a small change I
believe. Having an iocharset options for all file systems make it backward compatible
and creates a migration path to UTF-8 as system default locale.

-- robin

Re: JFS default behavior

[viro]

On Sun, Feb 15, 2004 at 12:06:23AM +0100, Robin Rosenberg wrote:
> On Saturday 14 February 2004 16.40, you wrote:
> > The same goes for file names.  Filename is a sequence of bytes, no more and
> > no less.  Anything beyond that belongs to applications.
> 
> Should be a sequence of characters since humans are supposed to use them and
> it should be the same characters wheneve possible regardless of user's locale.
 
> The  "sequence of bytes" idea is a legacy from prehistoric times when byte == character
> was true.

Bullshit.  It has _nothing_ to characters, wide or not.  For system filenames
are opaque.  The only things that have special meanings are:
	octet 0x2f ('/') splits the pathname into components
	"." as a component has a special meaning
	".." as a component has a special meaning.
That's it.  The rest is never interpreted by the kernel.

> Having an iocharset options for all file systems make it backward compatible
> and creates a migration path to UTF-8 as system default locale.

Try to realize that different users CAN HAVE DIFFERENT LOCALES.  On the same
system.  And have files on the same fs.  Moreover, homedirs that used to be
on different filesystems can end up one the same fs.  What iocharset would
you use, then?  Sigh...

Again, there is no such thing as iocharset of filesystem - it varies between
users and users can and do share filesystems.  Think of /home; think of /tmp.

It isn't feasible.  At all.  Just as timezone doesn't belong in kernel, locales
have no place there.

Re: JFS default behavior

[Robin Rosenberg]

On Sunday 15 February 2004 00.29, you wrote:
> On Sun, Feb 15, 2004 at 12:06:23AM +0100, Robin Rosenberg wrote:
> > The  "sequence of bytes" idea is a legacy from prehistoric times when byte == character
> > was true.
> 
> Bullshit.  It has _nothing_ to characters, wide or not.  For system filenames
> are opaque.  The only things that have special meanings are:
> 	octet 0x2f ('/') splits the pathname into components
> 	"." as a component has a special meaning
> 	".." as a component has a special meaning.
> That's it.  The rest is never interpreted by the kernel.
I know how it is (to some degree), and its wrong. The user sees inside the filename
and sees a string of characters, not a byte sequence.

> Try to realize that different users CAN HAVE DIFFERENT LOCALES.  On the same
> system.  And have files on the same fs.  Moreover, homedirs that used to be
> on different filesystems can end up one the same fs.  What iocharset would
> you use, then?  Sigh...
Ok, I've got the iocharset option wrong, god knows why. The problem 
however remains.

It seems you simply don't want to understand the problem, which is that users 
CAN HAVE DIFFERENT LOCALES on the same system and on different system. 
Sigh...

I less concerned with which solution than that a solution should be found. So it
seems no file system has a solution today. Still an iocharset option would relieve
the problem for removable media and muli-boot systems. Most linux machines
are essentially single user and have either the same locale for all users or all
users are using UTF-8 with their locale. It's not the locale, but the charset used
for encoding the locale. The rest cannot be helped.

-- robin

Re: JFS default behavior

[Linus Torvalds]

On Sun, 15 Feb 2004, Robin Rosenberg wrote:
> > 
> > Bullshit.  It has _nothing_ to characters, wide or not.  For system filenames
> > are opaque.  The only things that have special meanings are:
> > 	octet 0x2f ('/') splits the pathname into components
> > 	"." as a component has a special meaning
> > 	".." as a component has a special meaning.
> > That's it.  The rest is never interpreted by the kernel.
>
> I know how it is (to some degree), and its wrong. The user sees inside the filename
> and sees a string of characters, not a byte sequence.

Yes, the user sees a string of characters, but the octet 0x2f ('/') and 
the terminating NUL character '\0' are still perfectly normal characters 
and there is no confusion.

The reason: UTF-8. It's the only sane encoding (apart from a pure extended
ASCII setup, which is also sane, but is obviously unacceptable for a large
portion of the world).

If some misguided person has told you about UCS-2 and horrors like UTF-9,
just ignore them. They are crazy and deluded, and - perhaps more
importantly - stupid.

In short: the kernel talks bytestreams, and that implies that if you want 
to talk to the kernel, you HAVE TO USE UTF-8.

At which point there are no locale issues any more. The only locale issue 
you can have is user space mistaking a stream of bytes as extended ASCII, 
which will cause all your pretty UTF-8 characters to be shown as strange 
latin1 (or other) squiggles.

> It seems you simply don't want to understand the problem, which is that users 
> CAN HAVE DIFFERENT LOCALES on the same system and on different system. 
> Sigh...

People understand the problem. And UTF-8 is the solution.

It's getting there. I think even Microsoft has seen the light, and is
phasing out their crapola (UCS-2LE? Whatever). 

> I less concerned with which solution than that a solution should be found. So it
> seems no file system has a solution today. Still an iocharset option would relieve
> the problem for removable media and muli-boot systems.

No. Things like "iocharset" are not the solution. They are literally the
_problem_. The solution is to use something that not only acts as ASCII,
but also has a wide enough range to cover the whole required space (UCS-2
fails _both_ of these fundamental tests). At which point "iocharset" makes 
no sense any more, and only exists as a way to translate legacy crap into 
the one true format.

And that one true format is UTF-8. End of story. If you try to talk to the 
kernel in UCS-2 or anything else, you _will_ fail.

			Linus

stty utf8

[Jamie Lokier]

Linus Torvalds wrote:
> People understand the problem. And UTF-8 is the solution.

Linus, I agree 100%.
My own filesystems have UTF-8 file names, of course.

There are still practical problems, two of which stand out:

1. Just because you hope a filesystem is UTF-8, does not preclude
   readdir() from returning non-UTF-8 names.  (These are far too
   easy to create by accident).

   Because of that, programs which interpret the result of
   readdir() as text, yet are expected to handle any name without
   silently rejecting them or aborting, are forced into strange
   compromises which break basic expectations.

   Spot the bug in this perl script:

     perl -e 'for (glob "*") { rename $_, "ņi-".$_ or die "rename: $!\n"; }'

   (NB: The prefix string is N WITH CEDILLA followed by "i-").

   (Hint: it mangles perfectly fine non-ASCII file names).

   Perl has no perfect behaviour to offer, because what should that
   behaviour be if readdir() might return a non-UTF-8 byte sequence
   as a name?


2. Terminals are not all UTF-8, and some never will be.

   So when someone types something like this on a non-UTF-8
   terminal, they get non-UTF-8 filename:

       vi el-niño.txt

   It isn't just a problem of display.  Now you have created a
   filename which isn't valid UTF-8, and GUI programs may complain,
   perhaps refusing to let you select the file.

   Furthermore, how exactly do you expect a user to use UTF-8 on
   the filesystem when their terminal is not (or sometimes is not)
   using UTF-8?


==> This problem would be very nicely solved with an additional
    terminal flag.  We have "stty ocrnl", "onlcr", "igncr" etc. to
    translate between terminal line endings and the unix convention of
    LF at the end of each line.  Why not create "stty utf8" so that
    non-UTF-8 terminals and UTF-8 terminals alike can work with a
    Linux convention that all programs enter and display UTF-8?  It
    would simplify a lot of things.


-- Jamie

Re: stty utf8

[Gerd Knorr]

Jamie Lokier <jamie@shareable.org> writes:

> 2. Terminals are not all UTF-8, and some never will be.

> ==> This problem would be very nicely solved with an additional
>     terminal flag.  We have "stty ocrnl", "onlcr", "igncr" etc. to
>     translate between terminal line endings and the unix convention of
>     LF at the end of each line.  Why not create "stty utf8" so that
>     non-UTF-8 terminals and UTF-8 terminals alike can work with a
>     Linux convention that all programs enter and display UTF-8?  It
>     would simplify a lot of things.

It's probably possible to extend luit doing that too.  luit comes with
recent xfree86 releases and does utf-8 <=> locale conversion.  Right
now it does just the opposite:  let people use non-utf8 locales in a
utf-8 xterm.

  Gerd

-- 
Es geht darum, daß ein Haufen Scriptkiddies gerade dabei sind, USENET in
Bunt neu zu erfinden, und sie derzeit einen Haufen Fehler neu machen,
die schon seit 20 Jahren nicht mehr Gegenstand der Forschung sind.
	-- Kristian Köhntopp über blogs und blogger

Re: stty utf8

[Jamie Lokier]

Jamie Lokier wrote:
> Why not create "stty utf8" so that non-UTF-8 terminals and UTF-8
> terminals alike can work with a Linux convention that all programs
> enter and display UTF-8?  It would simplify a lot of things.

I little thought and an experiment later, and I discovered:

When you edit a line with the kernel's terminal line editor, when you
press the Delete key, it writes backspace-space-backspace and removes
one byte from the input.  That fails to do the right thing on UTF-8
terminals.

For example, in a UTF-8 xterm or Gnome terminal, or even on the Linux
console after running "unicode_start", run the command "cat" by
itself, then type "ééé", then hit DEL twice - it will show one
accented letter(*).  Press enter, and cat will echo the line
containing _two_ letters.

There is no fancy environment setting which corrects this problem.
The kernel needs to know it's dealing with a UTF-8 terminal for basic
line editing to work.

(*) The text in quotes is three E WITH ACUTE letters, in case that
doesn't show properly in your mailer.

-- Jamie

Re: stty utf8

[Linus Torvalds]

On Mon, 16 Feb 2004, Jamie Lokier wrote:
> 
> I little thought and an experiment later, and I discovered:
> 
> When you edit a line with the kernel's terminal line editor, when you
> press the Delete key, it writes backspace-space-backspace and removes
> one byte from the input.  That fails to do the right thing on UTF-8
> terminals.

Yes. I looked at that a year ago, and it should be pretty easy to make the 
backspace code look more like the "delete word" code - except the "word" 
is just a utf character.

(Btw, that's one of the things I like about UTF-8, and shows how _well_ 
designed it is - it's trivial to find the beginning of a UTF-8 character, 
even when just doing a stupid scan backwards).

I didn't care enough to really bother fixing it - the fact is, that people 
who care about UTF-8 tend to have to be in graphics mode anyway, and there 
is something to be said for keeping the text console simple even if it 
means it lacks functionality.

But if somebody cares more than I do (hint, hint ;), I do think it should 
be fixed.

> There is no fancy environment setting which corrects this problem.
> The kernel needs to know it's dealing with a UTF-8 terminal for basic
> line editing to work.

Yes. And I'd happily take patches for it.

		Linus

Re: stty utf8

[Jamie Lokier]

Jamie Lokier wrote:
>      perl -e 'for (glob "*") { rename $_, "ņi-".$_ or die "rename: $!\n"; }'
> 
>    (NB: The prefix string is N WITH CEDILLA followed by "i-").
>    (Hint: it mangles perfectly fine non-ASCII file names).
>
>    Perl has no perfect behaviour to offer, because what should that
>    behaviour be if readdir() might return a non-UTF-8 byte sequence
>    as a name?

I've had someone point out that the perl script mangles non-UTF-8
filenames, and there is no correct behaviour for that case.

In fact the _real_ bug is that it mangles perfectly fine UTF-8 filenames.

It's a Perl quirk, but the behaviour is like that for compatibility
with non-UTF-8 filesystems.  I wanted to show how just using UTF-8 for
filenames isn't _yet_ as straightforward as it should be.

-- Jamie