PROBLEM: serial port driver grabs occupied port

[columbus@hit.handshake.de (Christoph Bartelmus)]

Hi,

there's a problem with linux/drivers/char/serial.c that appears when using  
the LIRC serial port drivers (see http://www.lirc.org). Here's how to  
reproduce the problem:

1. Load the lirc_serial module (included in the LIRC package). This module  
uses request_region() for the used serial port's IO space.

2. Load the kernel serial module. In rs_init() the available serial ports  
are initialized. For the port already claimed by the lirc_serial module  
check_region() fails and state->type for this port stays PORT_UNKNOWN.

Now the problem:
3. Open the /dev/ttySx for the port that is occupied by the lirc_serial  
module. rs_open() does not check the state of the serial port and returns  
no error. Now you can use setserial to hijack the port... lirc_serial  
stops working.

The patch below fixes the problem for me (tested with 2.2.19 and 2.2.20).  
There's one problem with this patch though. Once state->type ist  
PORT_UNKNOWN for a port there's no way to reclaim the port with e.g.  
setserial unless you remove the serial port module and load it again.

Christoph

PS: I'm not subscribed, cc me if replying.

--- Schnipp ---
--- linux/drivers/char/serial.c	Sun Mar 25 18:37:31 2001
+++ serial.c	Tue Jan 15 18:20:59 2002
@@ -928,6 +928,11 @@
 	unsigned short ICP;
 #endif

+	if(state->type == PORT_UNKNOWN)
+	{
+		return -ENODEV;
+	}
+
 	page = get_free_page(GFP_KERNEL);
 	if (!page)
 		return -ENOMEM;
@@ -1738,7 +1743,10 @@
 	info->xmit_fifo_size = state->xmit_fifo_size =
 		new_serial.xmit_fifo_size;

-	release_region(state->port,8);
+	if(old_state.type != PORT_UNKNOWN)
+	{
+		release_region(state->port,8);
+	}
 	if (change_port || change_irq) {
 		/*
 		 * We need to shutdown the serial port at the old
@@ -1750,8 +1758,14 @@
 		info->hub6 = state->hub6 = new_serial.hub6;
 	}
 	if (state->type != PORT_UNKNOWN)
+	{
+		retval=check_region(state->port,8);
+		if(retval<0)
+		{
+			return retval;
+		}
 		request_region(state->port,8,"serial(set)");
-
+	}
 	
 check_and_exit:
 	if (!state->port || !state->type)
@@ -3586,3 +3600,4 @@
 	return kmem_start;
 }
 #endif
+
--- Schnipp ---