changing __PAGE_OFFSET on x86?

changing __PAGE_OFFSET on x86?

[Josh Fryman]

hi,

i posted a query last week about a problem we're having with some PCI devices.
we have a P4 system with 512M of RAM and two PCI cards, each having another
256MB of RAM.  we want our device driver to expose the entire memory region
for the pci cards.

when adding all this up, we're exceeding the 3G/1G split of the linux kernel.
as dan maas pointed out, there should be a way to fix this in the kernel...
so, after some grepping and code browsing, my new question is:

to fix this, if we change the __PAGE_OFFSET in include/asm-i386/page.h from
0xc0000000 to 0xb000000, are there any hidden dependencies?  is there anything 
else we need to worry about?  (does the __PAGE_OFFSET need to lie on a 1G
boundary?)

i haven't seen any, but them i'm not willing to say i understand all the 
implications from chaging such a fundamental #define ....

thanks,

josh

Re: changing __PAGE_OFFSET on x86?

[Martin J. Bligh]

> to fix this, if we change the __PAGE_OFFSET in include/asm-i386/page.h 
> from 0xc0000000 to 0xb000000, are there any hidden dependencies?  is 
> there anything else we need to worry about?  

Change arch/i386/vmlinux.lds as well (or the equiv on your machine).

> (does the __PAGE_OFFSET need to lie on a 1G boundary?)

Only if you're running PAE (64Gb support).

M.

PS. the -aa tree (Andrea's) has a config option to change this.

Re: changing __PAGE_OFFSET on x86?

[Robert Love]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]

On Tue, 2002-05-28 at 07:15, Josh Fryman wrote:

> to fix this, if we change the __PAGE_OFFSET in include/asm-i386/page.h from
> 0xc0000000 to 0xb000000, are there any hidden dependencies?

Yes, you need to edit arch/i386/vmlinux.lds as well.  Specifically, this
line:

      . = 0xC0000000 + 0x100000;	/* edit the 0xc0000000 */

to make it easier, there is a patch to let you configure a kernel /
user-space memory split at compile-time during configure.  It is by
Andrea - I have attached it.  This is a local copy in which I have added
some configure help.

> (does the __PAGE_OFFSET need to lie on a 1G boundary?)

If you are using PAE, yes - otherwise no.

	Robert Love

[-- Attachment #2: config-address-space-rml-2.4.18-1.patch --]
[-- Type: text/x-patch, Size: 12035 bytes --]

diff -urN linux-2.4.18-preempt-lb/Documentation/Configure.help linux/Documentation/Configure.help
--- linux-2.4.18-preempt-lb/Documentation/Configure.help	Sat Apr 20 16:59:13 2002
+++ linux/Documentation/Configure.help	Tue Apr 23 15:34:18 2002
@@ -399,6 +399,49 @@
   Select this if you have a 32-bit processor and more than 4
   gigabytes of physical RAM.
 
+# Choice: address_size
+User address space size
+CONFIG_1GB
+  This option allows you to set the partitioning of the logical address
+  space between user and kernel memory. The Linux address space is 4GB
+  and the default partition is to give 3GB to user and 1GB to
+  kernel-space. This is abbreviated "3GB/1GB".
+
+  Some specialized workloads require a larger user or kernel memory
+  space. For example, specialized network processing often benefits
+  from a larger kernel address space than default.  This options lets
+  you select:
+
+	3GB user / 1GB kernel		(default)
+	2GB user / 2GB kernel
+	1GB user / 3GB kernel
+	3.5GB user / 0.5GB kernel	(non-PAE only)
+
+  Note shrinking the user address space may adversely affect certain
+  applications that assume the user address space is 3GB or otherwise
+  that certain regions of memory exist.  It is recommended that you
+  are sure of your system's compatibility with a given partition scheme
+  before you deploy a system with a partition other than 3GB/1GB.
+
+  If uncertain, select "3GB/1GB".  Note this does not affect the amount
+  of physical memory accessible by Linux but only the addressable
+  per-process memory space.
+
+2GB
+CONFIG_2GB
+  Parition the addressable memory space into 2GB user and 2GB kernel.
+
+3GB
+CONFIG_3GB
+  Partion the addressable memory space into 3GB user and 1GB kernel.
+  This is the default.
+
+3.5GB
+CONFIG_05GB
+  Partition the addressable memory space into 3.5GB user and 0.5GB
+  kernel.  This is only valid if PAE (CONFIG_HIGHMEM64G) is not
+  enabled, since segments under PAE must lie on 1GB boundaries.
+
 Normal floppy disk support
 CONFIG_BLK_DEV_FD
   If you want to use the floppy disk drive(s) of your PC under Linux,
diff -urN linux-2.4.18-preempt-lb/Rules.make linux/Rules.make
--- linux-2.4.18-preempt-lb/Rules.make	Sat Apr 20 16:58:06 2002
+++ linux/Rules.make	Tue Apr 23 14:46:47 2002
@@ -214,6 +214,7 @@
 #
 # Added the SMP separator to stop module accidents between uniprocessor
 # and SMP Intel boxes - AC - from bits by Michael Chastain
+# Added separator for different PAGE_OFFSET memory models - Ingo.
 #
 
 ifdef CONFIG_SMP
@@ -222,6 +223,22 @@
 	genksyms_smp_prefix := 
 endif
 
+ifdef CONFIG_2GB
+ifdef CONFIG_SMP
+	genksyms_smp_prefix := -p smp_2gig_
+else
+	genksyms_smp_prefix := -p 2gig_
+endif
+endif
+
+ifdef CONFIG_3GB
+ifdef CONFIG_SMP
+	genksyms_smp_prefix := -p smp_3gig_
+else
+	genksyms_smp_prefix := -p 3gig_
+endif
+endif
+
 $(MODINCL)/%.ver: %.c
 	@if [ ! -r $(MODINCL)/$*.stamp -o $(MODINCL)/$*.stamp -ot $< ]; then \
 		echo '$(CC) $(CFLAGS) $(EXTRA_CFLAGS) -E -D__GENKSYMS__ $<'; \
diff -urN linux-2.4.18-preempt-lb/arch/i386/Makefile linux/arch/i386/Makefile
--- linux-2.4.18-preempt-lb/arch/i386/Makefile	Sat Apr 20 16:58:56 2002
+++ linux/arch/i386/Makefile	Tue Apr 23 14:46:47 2002
@@ -106,6 +106,9 @@
 
 MAKEBOOT = $(MAKE) -C arch/$(ARCH)/boot
 
+arch/i386/vmlinux.lds: arch/i386/vmlinux.lds.S FORCE
+	$(CPP) -C -P -I$(HPATH) -imacros $(HPATH)/asm-i386/page_offset.h -Ui386 arch/i386/vmlinux.lds.S >arch/i386/vmlinux.lds
+
 vmlinux: arch/i386/vmlinux.lds
 
 FORCE: ;
@@ -142,6 +145,7 @@
 	@$(MAKEBOOT) clean
 
 archmrproper:
+	rm -f arch/i386/vmlinux.lds
 
 archdep:
 	@$(MAKEBOOT) dep
diff -urN linux-2.4.18-preempt-lb/arch/i386/config.in linux/arch/i386/config.in
--- linux-2.4.18-preempt-lb/arch/i386/config.in	Sat Apr 20 16:58:56 2002
+++ linux/arch/i386/config.in	Tue Apr 23 14:46:47 2002
@@ -174,12 +174,23 @@
 	"off    CONFIG_NOHIGHMEM \
 	 4GB    CONFIG_HIGHMEM4G \
 	 64GB   CONFIG_HIGHMEM64G" off
-if [ "$CONFIG_HIGHMEM4G" = "y" ]; then
+if [ "$CONFIG_HIGHMEM4G" = "y" -o "$CONFIG_HIGHMEM64G" = "y" ]; then
    define_bool CONFIG_HIGHMEM y
+else
+   define_bool CONFIG_HIGHMEM n
 fi
 if [ "$CONFIG_HIGHMEM64G" = "y" ]; then
-   define_bool CONFIG_HIGHMEM y
    define_bool CONFIG_X86_PAE y
+   choice 'User address space size' \
+	"3GB		CONFIG_1GB \
+	 2GB		CONFIG_2GB \
+	 1GB		CONFIG_3GB" 3GB
+else
+   choice 'User address space size' \
+	"3GB		CONFIG_1GB \
+	 2GB		CONFIG_2GB \
+	 1GB		CONFIG_3GB \
+	 3.5GB		CONFIG_05GB" 3GB
 fi
 
 bool 'Math emulation' CONFIG_MATH_EMULATION
diff -urN linux-2.4.18-preempt-lb/arch/i386/vmlinux.lds linux/arch/i386/vmlinux.lds
--- linux-2.4.18-preempt-lb/arch/i386/vmlinux.lds	Sat Apr 20 16:58:56 2002
+++ linux/arch/i386/vmlinux.lds	Wed Dec 31 19:00:00 1969
@@ -1,82 +0,0 @@
-/* ld script to make i386 Linux kernel
- * Written by Martin Mares <mj@atrey.karlin.mff.cuni.cz>;
- */
-OUTPUT_FORMAT("elf32-i386", "elf32-i386", "elf32-i386")
-OUTPUT_ARCH(i386)
-ENTRY(_start)
-SECTIONS
-{
-  . = 0xC0000000 + 0x100000;
-  _text = .;			/* Text and read-only data */
-  .text : {
-	*(.text)
-	*(.fixup)
-	*(.gnu.warning)
-	} = 0x9090
-
-  _etext = .;			/* End of text section */
-
-  .rodata : { *(.rodata) *(.rodata.*) }
-  .kstrtab : { *(.kstrtab) }
-
-  . = ALIGN(16);		/* Exception table */
-  __start___ex_table = .;
-  __ex_table : { *(__ex_table) }
-  __stop___ex_table = .;
-
-  __start___ksymtab = .;	/* Kernel symbol table */
-  __ksymtab : { *(__ksymtab) }
-  __stop___ksymtab = .;
-
-  .data : {			/* Data */
-	*(.data)
-	CONSTRUCTORS
-	}
-
-  _edata = .;			/* End of data section */
-
-  . = ALIGN(8192);		/* init_task */
-  .data.init_task : { *(.data.init_task) }
-
-  . = ALIGN(4096);		/* Init code and data */
-  __init_begin = .;
-  .text.init : { *(.text.init) }
-  .data.init : { *(.data.init) }
-  . = ALIGN(16);
-  __setup_start = .;
-  .setup.init : { *(.setup.init) }
-  __setup_end = .;
-  __initcall_start = .;
-  .initcall.init : { *(.initcall.init) }
-  __initcall_end = .;
-  . = ALIGN(4096);
-  __init_end = .;
-
-  . = ALIGN(4096);
-  .data.page_aligned : { *(.data.idt) }
-
-  . = ALIGN(32);
-  .data.cacheline_aligned : { *(.data.cacheline_aligned) }
-
-  __bss_start = .;		/* BSS */
-  .bss : {
-	*(.bss)
-	}
-  _end = . ;
-
-  /* Sections to be discarded */
-  /DISCARD/ : {
-	*(.text.exit)
-	*(.data.exit)
-	*(.exitcall.exit)
-	}
-
-  /* Stabs debugging sections.  */
-  .stab 0 : { *(.stab) }
-  .stabstr 0 : { *(.stabstr) }
-  .stab.excl 0 : { *(.stab.excl) }
-  .stab.exclstr 0 : { *(.stab.exclstr) }
-  .stab.index 0 : { *(.stab.index) }
-  .stab.indexstr 0 : { *(.stab.indexstr) }
-  .comment 0 : { *(.comment) }
-}
diff -urN linux-2.4.18-preempt-lb/arch/i386/vmlinux.lds.S linux/arch/i386/vmlinux.lds.S
--- linux-2.4.18-preempt-lb/arch/i386/vmlinux.lds.S	Wed Dec 31 19:00:00 1969
+++ linux/arch/i386/vmlinux.lds.S	Tue Apr 23 14:46:47 2002
@@ -0,0 +1,82 @@
+/* ld script to make i386 Linux kernel
+ * Written by Martin Mares <mj@atrey.karlin.mff.cuni.cz>;
+ */
+OUTPUT_FORMAT("elf32-i386", "elf32-i386", "elf32-i386")
+OUTPUT_ARCH(i386)
+ENTRY(_start)
+SECTIONS
+{
+  . = PAGE_OFFSET_RAW + 0x100000;
+  _text = .;			/* Text and read-only data */
+  .text : {
+	*(.text)
+	*(.fixup)
+	*(.gnu.warning)
+	} = 0x9090
+
+  _etext = .;			/* End of text section */
+
+  .rodata : { *(.rodata) *(.rodata.*) }
+  .kstrtab : { *(.kstrtab) }
+
+  . = ALIGN(16);		/* Exception table */
+  __start___ex_table = .;
+  __ex_table : { *(__ex_table) }
+  __stop___ex_table = .;
+
+  __start___ksymtab = .;	/* Kernel symbol table */
+  __ksymtab : { *(__ksymtab) }
+  __stop___ksymtab = .;
+
+  .data : {			/* Data */
+	*(.data)
+	CONSTRUCTORS
+	}
+
+  _edata = .;			/* End of data section */
+
+  . = ALIGN(8192);		/* init_task */
+  .data.init_task : { *(.data.init_task) }
+
+  . = ALIGN(4096);		/* Init code and data */
+  __init_begin = .;
+  .text.init : { *(.text.init) }
+  .data.init : { *(.data.init) }
+  . = ALIGN(16);
+  __setup_start = .;
+  .setup.init : { *(.setup.init) }
+  __setup_end = .;
+  __initcall_start = .;
+  .initcall.init : { *(.initcall.init) }
+  __initcall_end = .;
+  . = ALIGN(4096);
+  __init_end = .;
+
+  . = ALIGN(4096);
+  .data.page_aligned : { *(.data.idt) }
+
+  . = ALIGN(32);
+  .data.cacheline_aligned : { *(.data.cacheline_aligned) }
+
+  __bss_start = .;		/* BSS */
+  .bss : {
+	*(.bss)
+	}
+  _end = . ;
+
+  /* Sections to be discarded */
+  /DISCARD/ : {
+	*(.text.exit)
+	*(.data.exit)
+	*(.exitcall.exit)
+	}
+
+  /* Stabs debugging sections.  */
+  .stab 0 : { *(.stab) }
+  .stabstr 0 : { *(.stabstr) }
+  .stab.excl 0 : { *(.stab.excl) }
+  .stab.exclstr 0 : { *(.stab.exclstr) }
+  .stab.index 0 : { *(.stab.index) }
+  .stab.indexstr 0 : { *(.stab.indexstr) }
+  .comment 0 : { *(.comment) }
+}
diff -urN linux-2.4.18-preempt-lb/include/asm-i386/page.h linux/include/asm-i386/page.h
--- linux-2.4.18-preempt-lb/include/asm-i386/page.h	Sat Apr 20 16:57:57 2002
+++ linux/include/asm-i386/page.h	Tue Apr 23 14:46:47 2002
@@ -78,7 +78,9 @@
  * and CONFIG_HIGHMEM64G options in the kernel configuration.
  */
 
-#define __PAGE_OFFSET		(0xC0000000)
+#include <asm/page_offset.h>
+
+#define __PAGE_OFFSET		(PAGE_OFFSET_RAW)
 
 /*
  * This much address space is reserved for vmalloc() and iomap()
diff -urN linux-2.4.18-preempt-lb/include/asm-i386/page_offset.h linux/include/asm-i386/page_offset.h
--- linux-2.4.18-preempt-lb/include/asm-i386/page_offset.h	Wed Dec 31 19:00:00 1969
+++ linux/include/asm-i386/page_offset.h	Tue Apr 23 14:46:47 2002
@@ -0,0 +1,10 @@
+#include <linux/config.h>
+#ifdef CONFIG_05GB
+#define PAGE_OFFSET_RAW 0xE0000000
+#elif defined(CONFIG_1GB)
+#define PAGE_OFFSET_RAW 0xC0000000
+#elif defined(CONFIG_2GB)
+#define PAGE_OFFSET_RAW 0x80000000
+#elif defined(CONFIG_3GB)
+#define PAGE_OFFSET_RAW 0x40000000
+#endif
diff -urN linux-2.4.18-preempt-lb/include/asm-i386/processor.h linux/include/asm-i386/processor.h
--- linux-2.4.18-preempt-lb/include/asm-i386/processor.h	Sat Apr 20 16:57:57 2002
+++ linux/include/asm-i386/processor.h	Tue Apr 23 14:46:47 2002
@@ -270,7 +270,11 @@
 /* This decides where the kernel will search for a free chunk of vm
  * space during mmap's.
  */
+#ifndef CONFIG_05GB
 #define TASK_UNMAPPED_BASE	(TASK_SIZE / 3)
+#else
+#define TASK_UNMAPPED_BASE	(TASK_SIZE / 16)
+#endif
 
 /*
  * Size of io_bitmap in longwords: 32 is ports 0-0x3ff.
diff -urN linux-2.4.18-preempt-lb/mm/memory.c linux/mm/memory.c
--- linux-2.4.18-preempt-lb/mm/memory.c	Sat Apr 20 16:57:55 2002
+++ linux/mm/memory.c	Tue Apr 23 14:46:47 2002
@@ -106,8 +106,7 @@
 
 static inline void free_one_pgd(pgd_t * dir)
 {
-	int j;
-	pmd_t * pmd;
+	pmd_t * pmd, * md, * emd;
 
 	if (pgd_none(*dir))
 		return;
@@ -118,9 +117,23 @@
 	}
 	pmd = pmd_offset(dir, 0);
 	pgd_clear(dir);
-	for (j = 0; j < PTRS_PER_PMD ; j++) {
-		prefetchw(pmd+j+(PREFETCH_STRIDE/16));
-		free_one_pmd(pmd+j);
+
+	/*
+	 * Beware if changing the loop below.  It once used int j,
+	 *	for (j = 0; j < PTRS_PER_PMD; j++)
+	 *		free_one_pmd(pmd+j);
+	 * but some older i386 compilers (e.g. egcs-2.91.66, gcc-2.95.3)
+	 * terminated the loop with a _signed_ address comparison
+	 * using "jle", when configured for HIGHMEM64GB (X86_PAE).
+	 * If also configured for 3GB of kernel virtual address space,
+	 * if page at physical 0x3ffff000 virtual 0x7ffff000 is used as
+	 * a pmd, when that mm exits the loop goes on to free "entries"
+	 * found at 0x80000000 onwards.  The loop below compiles instead
+	 * to be terminated by unsigned address comparison using "jb".
+	 */
+	for (md = pmd, emd = pmd + PTRS_PER_PMD; md < emd; md++) {
+		prefetchw(md+(PREFETCH_STRIDE/16));
+		free_one_pmd(md);
 	}
 	pmd_free(pmd);
 }

Re: changing __PAGE_OFFSET on x86?

[Albert D. Cahalan]

Josh Fryman writes:

> when adding all this up, we're exceeding the 3G/1G split of the linux kernel.
> as dan maas pointed out, there should be a way to fix this in the kernel...
> so, after some grepping and code browsing, my new question is:
>
> to fix this, if we change the __PAGE_OFFSET in include/asm-i386/page.h from
> 0xc0000000 to 0xb000000, are there any hidden dependencies?  is there anything 
> else we need to worry about?  (does the __PAGE_OFFSET need to lie on a 1G
> boundary?)
>
> i haven't seen any, but them i'm not willing to say i understand all the 
> implications from chaging such a fundamental #define ....

At least at one time, the kernel side had to be a power of 2.
So you'd have to go to an even split if that's still the case.