[PATCH] fix mmap random address range on x86

[PATCH] fix mmap random address range on x86

[Ludwig Nussel]

On x86 casting the unsigned int result of get_random_int() to long
may result in a negative value. On x86 the range of mmap_rnd()
therefore was -255 to 255. The 32bit mode on x86_64 used 0 to 255 as
intended.

The bug was introduced by commit 675a081 in January 2008.

Signed-off-by: Ludwig Nussel <ludwig.nussel@suse.de>
---
 arch/x86/mm/mmap.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c
index 1dab519..f927429 100644
--- a/arch/x86/mm/mmap.c
+++ b/arch/x86/mm/mmap.c
@@ -87,9 +87,9 @@ static unsigned long mmap_rnd(void)
 	*/
 	if (current->flags & PF_RANDOMIZE) {
 		if (mmap_is_ia32())
-			rnd = (long)get_random_int() % (1<<8);
+			rnd = get_random_int() % (1<<8);
 		else
-			rnd = (long)(get_random_int() % (1<<28));
+			rnd = get_random_int() % (1<<28);
 	}
 	return rnd << PAGE_SHIFT;
 }
-- 
1.7.1

Re: [Security] [PATCH] fix mmap random address range on x86

[Andrew Morton]

On Fri, 18 Feb 2011 14:15:57 +0100
Ludwig Nussel <ludwig.nussel@suse.de> wrote:

> On x86 casting the unsigned int result of get_random_int() to long
> may result in a negative value. On x86 the range of mmap_rnd()
> therefore was -255 to 255. The 32bit mode on x86_64 used 0 to 255 as
> intended.
> 
> The bug was introduced by commit 675a081 in January 2008.
> 
> Signed-off-by: Ludwig Nussel <ludwig.nussel@suse.de>
> ---
>  arch/x86/mm/mmap.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c
> index 1dab519..f927429 100644
> --- a/arch/x86/mm/mmap.c
> +++ b/arch/x86/mm/mmap.c
> @@ -87,9 +87,9 @@ static unsigned long mmap_rnd(void)
>  	*/
>  	if (current->flags & PF_RANDOMIZE) {
>  		if (mmap_is_ia32())
> -			rnd = (long)get_random_int() % (1<<8);
> +			rnd = get_random_int() % (1<<8);
>  		else
> -			rnd = (long)(get_random_int() % (1<<28));
> +			rnd = get_random_int() % (1<<28);
>  	}
>  	return rnd << PAGE_SHIFT;
>  }

The changelog didn't describe the user-visible consequences of this
bug, so readers must try to work this out for themselves.  That's not a
very desirable or efficient thing, so please do prepare more complete
changelogs.

afacit the effects are very small: the mmap base may fall slightly
below MIN_GAP, but that won't really affect anything?

Re: [Security] [PATCH] fix mmap random address range on x86

[Ludwig Nussel]

Andrew Morton wrote:
> On Fri, 18 Feb 2011 14:15:57 +0100
> Ludwig Nussel <ludwig.nussel@suse.de> wrote:
> 
> > On x86 casting the unsigned int result of get_random_int() to long
> > may result in a negative value. On x86 the range of mmap_rnd()
> > therefore was -255 to 255. The 32bit mode on x86_64 used 0 to 255 as
> > intended.
> > 
> > The bug was introduced by commit 675a081 in January 2008.
> > 
> > Signed-off-by: Ludwig Nussel <ludwig.nussel@suse.de>
> > ---
> >  arch/x86/mm/mmap.c |    4 ++--
> >  1 files changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c
> > index 1dab519..f927429 100644
> > --- a/arch/x86/mm/mmap.c
> > +++ b/arch/x86/mm/mmap.c
> > @@ -87,9 +87,9 @@ static unsigned long mmap_rnd(void)
> >  	*/
> >  	if (current->flags & PF_RANDOMIZE) {
> >  		if (mmap_is_ia32())
> > -			rnd = (long)get_random_int() % (1<<8);
> > +			rnd = get_random_int() % (1<<8);
> >  		else
> > -			rnd = (long)(get_random_int() % (1<<28));
> > +			rnd = get_random_int() % (1<<28);
> >  	}
> >  	return rnd << PAGE_SHIFT;
> >  }
> 
> The changelog didn't describe the user-visible consequences of this
> bug, so readers must try to work this out for themselves.  That's not a
> very desirable or efficient thing, so please do prepare more complete
> changelogs.

The consequence of the bug is that (pie)programs and libraries are
mapped to 511 possible addresses on i586. The intention of the code
is to use only 256 addresses though ("8 bits of randomness"). That's
also what the code did on x86_64.

> afacit the effects are very small: the mmap base may fall slightly
> below MIN_GAP, but that won't really affect anything?

Apparently it has no bad effect otherwise the bug would have been
discovered earlier I guess. So given that the bigger address range
worked unintentionally I wonder whether it would make sense to
increase the range explicitly.

cu
Ludwig

-- 
 (o_   Ludwig Nussel
 //\   
 V_/_  http://www.suse.de/
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg)