[RFC] potential ancient bug (since 1.99.14) at char/pty

[Mauro Carvalho Chehab <mchehab@redhat.com>]

Hi Alan/Ts'o,

There's an open bugzilla at Red Hat for the pty driver that I'm not sure
what would be the better fix. It tracks to kernel 1.99.14:
	http://www.linuxhq.com/kernel/v1.99/13/drivers/char/ChangeLog
	http://www.linuxhq.com/kernel/v1.99/13/drivers/char/pty.c

The code is still present upstream.

The pty changes added two lines to pty_close(): tty->packet = 0 and
tty->link->packet = 0.

The bug is at:
	https://bugzilla.redhat.com/show_bug.cgi?id=504703

The result of the bug is a race condition where telnetd may not properly
recognize the logout message (as it assumes that the device is in packet mode,
where the driver already changed its state to non-packet mode).

By removing tty->link->packet = 0, it fixes the issue, with a patch like:

--- a/drivers/char/pty.c
+++ b/drivers/char/pty.c
@@ -51,7 +51,6 @@ static void pty_close(struct tty_struct *tty, struct file *filp)
 	tty->packet = 0;
 	if (!tty->link)
 		return;
-	tty->link->packet = 0;
 	set_bit(TTY_OTHER_CLOSED, &tty->link->flags);
 	wake_up_interruptible(&tty->link->read_wait);
 	wake_up_interruptible(&tty->link->write_wait);

However, I'm not sure if this is the better fix, since it means that, after
closing both devices and opening again, link->packet will be 1, and this seems
wrong.

So, maybe the better way would be to add an open counter to pty and changing
link->packet to zero only if the open usage count is zero.

Another point to be considered is that a change like that on a very old code can
cause regressions on other parts.

Comments?

Cheers,
Mauro