Re: HIGHMEM4G config for 1GB RAM on desktop?

Re: HIGHMEM4G config for 1GB RAM on desktop?

[linux]

[Resend because my fingers STILL type "@vger.rutgers.edu" if I'm not
paying attention...]

> In practice, I suspect 2/2 will do exactly what you want ... and what 
> 99.9% of people want actually ;-) We could add more options, but be sure
> to mark anything that's not 1GB aligned as not suitable for PAE (as the
> 0.5 split was).

But if you're using PAE, you've got > 4G of RAM, so there's no need to
be clever trying to avoid HIGHMEM options.

Unfortunately, I just had a server with Con's patch detonate overnight
(Oops in interrupt -> panic; details in separate e-mail), and wli tells
me that there are additional places in the code that need fixing.

I notice that all previous patches had the kernel range a power of 2 in size.
Is this required somewhere?  I thought it was just that the kernel
had to start at a PGD boundary (4M on normal x86, 1G on PAE).

If 128M is always enough, a split at 0xb800000 seems possible, but
giving it an extra 128M seems like a nice bit of safety for PCI devices.

[PATCH] (was Re: HIGHMEM4G config for 1GB RAM on desktop?)

[Roland Dreier]

    > wli tells me that there are additional places in the code that
    > need fixing.

Looks like arch/i386/kernel/doublefault.c is one place in the code
that hardcodes the assumption that PAGE_OFFSET == 0xC0000000.  Here's
a patch that fixes that.

(Of course this doesn't really fix anything except debugging output)

 - Roland

Signed-off-by: Roland Dreier <roland@topspin.com>

Index: linux-2.6.8-rc1/arch/i386/kernel/doublefault.c
===================================================================
--- linux-2.6.8-rc1.orig/arch/i386/kernel/doublefault.c
+++ linux-2.6.8-rc1/arch/i386/kernel/doublefault.c
@@ -13,7 +13,7 @@
 static unsigned long doublefault_stack[DOUBLEFAULT_STACKSIZE];
 #define STACK_START (unsigned long)(doublefault_stack+DOUBLEFAULT_STACKSIZE)
 
-#define ptr_ok(x) ((x) > 0xc0000000 && (x) < 0xc1000000)
+#define ptr_ok(x) ((x) > PAGE_OFFSET && (x) < PAGE_OFFSET + 0x1000000)
 
 static void doublefault_fn(void)
 {

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 493 bytes --]

On Fri, 06 Aug 2004 12:52:36 -0000, linux@horizon.com said:

> But if you're using PAE, you've got > 4G of RAM, so there's no need to
> be clever trying to avoid HIGHMEM options.

Poking around in arch/i386/mm/init.c finds this:

#ifdef CONFIG_X86_PAE
int nx_enabled = 0;

static void __init set_nx(void)
{
        unsigned int v[4], l, h;

        if (cpu_has_pae && (cpuid_eax(0x80000000) > 0x80000001)) {

I could certainly see somebody wanting to use the NX stuff with only 2G of RAM.....

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Matt Heler]

[-- Attachment #1: Type: text/plain, Size: 89 bytes --]

Tim,

Try this patch , it seems to help with my 3ware 7000-2 controller card. 

matt h.


[-- Attachment #2: 3ware-64-lun.patch --]
[-- Type: text/x-diff, Size: 758 bytes --]

diff -urpN linux-2.6.9-base/drivers/scsi/3w-xxxx.h linux-2.6.9/drivers/scsi/3w-xxxx.h
--- linux-2.6.9-base/drivers/scsi/3w-xxxx.h	2004-08-28 22:03:22.000000000 -0700
+++ linux-2.6.9/drivers/scsi/3w-xxxx.h	2004-09-01 01:25:21.166428080 -0700
@@ -214,7 +214,7 @@ static unsigned char tw_sense_table[][4]
 #define TW_MAX_PCI_BUSES		      255
 #define TW_MAX_RESET_TRIES		      3
 #define TW_UNIT_INFORMATION_TABLE_BASE	      0x300
-#define TW_MAX_CMDS_PER_LUN		      254 /* 254 for io, 1 for
+#define TW_MAX_CMDS_PER_LUN		      64 /* 64 for io, 1 for
                                                      chrdev ioctl, one for
                                                      internal aen post */
 #define TW_BLOCK_SIZE			      0x200 /* 512-byte blocks */

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Miquel van Smoorenburg]

On Wed, 01 Sep 2004 00:46:27, Timothy Miller wrote:
> Miquel van Smoorenburg wrote:
> > In article <41336CB1.6030105@techsource.com>,
> > Timothy Miller  <miller@techsource.com> wrote:
> > 
> >>Timothy Miller wrote:
> >>
> >>>Hey, that rings a bell.  I have a 3ware 7000-2 controller with two 
> >>>WD1200JB drives in RAID1.  I find that if I dd from the disk, I get 
> >>>exactly the read throughput that is the max for the drives (47MB/sec). 
> >>>However, if I do a WRITE test, the performance is miserable.
> > 
> > Try setting /sys/block/sda/queue/nr_requests to twice the number
> > in /sys/block/sda/device/queue_depth
> 
> This will improve write performance?

You won't know before you try it ofcourse. It helps on my 85xx controllers.
The problem is that the internal queue size of some 3ware controllers
(queue_depth) is larger than the I/O schedulers nr_requests so that
the I/O scheduler doesn't get much chance to properly order and merge
the requests.

I've sent patches to 3ware a couple of times to make queue_depth
writable so that you can tune that as well, but they were refused
for no good reason AFAICS. Very unfortunate - if you have 8 JBOD
disks attached, you want to set queue_depth for each of them to
(max_controller_queue_depth / 8) to prevent one disk from starving
the other ones, but oh well.

> And if this helps, how do I make 
> it permanent?

Can't say, depends on your distribution. For recent Debian at
least you can use /etc/sysctl.conf

Mike.

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Timothy Miller]

Miquel van Smoorenburg wrote:
> In article <41336CB1.6030105@techsource.com>,
> Timothy Miller  <miller@techsource.com> wrote:
> 
>>Timothy Miller wrote:
>>
>>>Hey, that rings a bell.  I have a 3ware 7000-2 controller with two 
>>>WD1200JB drives in RAID1.  I find that if I dd from the disk, I get 
>>>exactly the read throughput that is the max for the drives (47MB/sec). 
>>>However, if I do a WRITE test, the performance is miserable.
>>>
>>>I have been going back and forth with 3ware for months, and what's odd 
>>>is that my drives with my controller in any machine other than the 
>>>primary box get great write throughput, BUT on my main box with 1G of 
>>>RAM, I get MISERABLE write throughput.  When I should be getting 
>>>36MB/sec or faster, I get 8 to 12 MB/sec.
>>>
>>>Now, I have tried limiting the memory with a mem= boot option, but that 
>>>doesn't change the performance any.
>>
>>Scratch all this.  Even if I physically remove half the memory, I STILL 
>>get the performance problem.
> 
> 
> 3ware eh?
> 
> Try setting /sys/block/sda/queue/nr_requests to twice the number
> in /sys/block/sda/device/queue_depth


This will improve write performance?  And if this helps, how do I make 
it permanent?

Thanks!

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Timothy Miller]

Timothy Miller wrote:
> 
> 
> Andrew Morton wrote:
> 
>>
>>
>> The 896M/128M split has a bit of a problem now each zone has its own LRU:
>> the size of the highmem zone is less than the amount of memory which is
>> described by the default /proc/sys/vm/dirty_ratio.  So it is easy to
>> completely fill highmem with dirty pages.  This causes a fairly large
>> amount of writeback via vmscan.c's writepage().  This causes poor I/O
>> submission patterns.  This causes a simple large, linear `dd' write to 
>> run
>> at only 50-70% of disk bandwidth.  (This was 6-12 months ago - it 
>> might be
>> a bit better now)
>>
> 
> 
> Hey, that rings a bell.  I have a 3ware 7000-2 controller with two 
> WD1200JB drives in RAID1.  I find that if I dd from the disk, I get 
> exactly the read throughput that is the max for the drives (47MB/sec). 
> However, if I do a WRITE test, the performance is miserable.
> 
> I have been going back and forth with 3ware for months, and what's odd 
> is that my drives with my controller in any machine other than the 
> primary box get great write throughput, BUT on my main box with 1G of 
> RAM, I get MISERABLE write throughput.  When I should be getting 
> 36MB/sec or faster, I get 8 to 12 MB/sec.
> 
> Now, I have tried limiting the memory with a mem= boot option, but that 
> doesn't change the performance any.
>

Scratch all this.  Even if I physically remove half the memory, I STILL 
get the performance problem.

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Miquel van Smoorenburg]

In article <41336CB1.6030105@techsource.com>,
Timothy Miller  <miller@techsource.com> wrote:
>Timothy Miller wrote:
>> Hey, that rings a bell.  I have a 3ware 7000-2 controller with two 
>> WD1200JB drives in RAID1.  I find that if I dd from the disk, I get 
>> exactly the read throughput that is the max for the drives (47MB/sec). 
>> However, if I do a WRITE test, the performance is miserable.
>> 
>> I have been going back and forth with 3ware for months, and what's odd 
>> is that my drives with my controller in any machine other than the 
>> primary box get great write throughput, BUT on my main box with 1G of 
>> RAM, I get MISERABLE write throughput.  When I should be getting 
>> 36MB/sec or faster, I get 8 to 12 MB/sec.
>> 
>> Now, I have tried limiting the memory with a mem= boot option, but that 
>> doesn't change the performance any.
>
>Scratch all this.  Even if I physically remove half the memory, I STILL 
>get the performance problem.

3ware eh?

Try setting /sys/block/sda/queue/nr_requests to twice the number
in /sys/block/sda/device/queue_depth

Mike.
-- 
"In times of universal deceit, telling the truth becomes
 a revolutionary act." -- George Orwell.

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Timothy Miller]

Andrew Morton wrote:

> 
> 
> The 896M/128M split has a bit of a problem now each zone has its own LRU:
> the size of the highmem zone is less than the amount of memory which is
> described by the default /proc/sys/vm/dirty_ratio.  So it is easy to
> completely fill highmem with dirty pages.  This causes a fairly large
> amount of writeback via vmscan.c's writepage().  This causes poor I/O
> submission patterns.  This causes a simple large, linear `dd' write to run
> at only 50-70% of disk bandwidth.  (This was 6-12 months ago - it might be
> a bit better now)
> 


Hey, that rings a bell.  I have a 3ware 7000-2 controller with two 
WD1200JB drives in RAID1.  I find that if I dd from the disk, I get 
exactly the read throughput that is the max for the drives (47MB/sec). 
However, if I do a WRITE test, the performance is miserable.

I have been going back and forth with 3ware for months, and what's odd 
is that my drives with my controller in any machine other than the 
primary box get great write throughput, BUT on my main box with 1G of 
RAM, I get MISERABLE write throughput.  When I should be getting 
36MB/sec or faster, I get 8 to 12 MB/sec.

Now, I have tried limiting the memory with a mem= boot option, but that 
doesn't change the performance any.

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Martin J. Bligh]

>     Martin> I had a patch for a config option, ported forward by
>     Martin> someone at IBM (I forget who, possibly Dave) from Andrea's
>     Martin> original. I think we finally decided (in consultation with
>     Martin> Andrea) we could drop the complicated stuff he had in the
>     Martin> asm code, so it's pretty simple ... something like this:
> 
> Am I just being dense, or is this patch solving a different problem
> from "do I really have to turn on HIGHMEM4G just to get the last 128MB
> of my 1GB of RAM?"
> 
> It seems to me that none of the PAGE_OFFSET values offered (the patch
> in allows PAGE_OFFSET to be set to 0xe0000000, 0xc0000000, 0x80000000
> or 0x40000000, in addition to the 4G/4G value of 0x02000000) are
> exactly what someone with 1 GB of RAM wants.  They'd be forced to go
> down to a 2G/2G split which cheats userspace of quite a bit of address
> space (admittedly with only 1 GB of RAM, a process bigger than 2 GB is
> a bit of a stretch).  In any case a 2.75G/1.25G split as suggested
> earlier works fine with 1 GB of RAM.

In practice, I suspect 2/2 will do exactly what you want ... and what 
99.9% of people want actually ;-) We could add more options, but be sure
to mark anything that's not 1GB aligned as not suitable for PAE (as the
0.5 split was).
 
> Also I notice that Con's patch modifies vmlinux.ld.S to update the
> kernel base address, while this patch doesn't.  Is that intentional or
> is does the patch depend on some other patches that use the defines in
> page.h somehow as controlled by the following change?
> 
> +AFLAGS_vmlinux.lds.o += -include $(TOPDIR)/include/asm-i386/page.h

Dunno ... we should test it really ... it's been a while. IIRC, vmlinux.ld.S
was hardcoded, so I don't see how it'd work without those mods.

M.

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Martin J. Bligh]

--On Wednesday, August 04, 2004 12:51:29 -0700 Andrew Morton <akpm@osdl.org> wrote:

> "Martin J. Bligh" <mbligh@aracnet.com> wrote:
>> 
>> I had a patch for a config option, ported forward by someone at IBM (I forget
>>  who, possibly Dave) from Andrea's original. I think we finally decided 
>>  (in consultation with Andrea) we could drop the complicated stuff he had
>>  in the asm code, so it's pretty simple ... something like this:
> 
> I sent such a patch to the boss many moons ago and he said "go away, this
> is a vendor-only thing". 

Yeah, I know. But then he hates 4/4 split even more, and going to 2/2 is
a trivial way to solve the 64GB machines ;-)

Maybe I should embed it in an ioctl patch for a mouse driver?
/me runs like hell.

M.

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Roland Dreier]

    Martin> I had a patch for a config option, ported forward by
    Martin> someone at IBM (I forget who, possibly Dave) from Andrea's
    Martin> original. I think we finally decided (in consultation with
    Martin> Andrea) we could drop the complicated stuff he had in the
    Martin> asm code, so it's pretty simple ... something like this:

Am I just being dense, or is this patch solving a different problem
from "do I really have to turn on HIGHMEM4G just to get the last 128MB
of my 1GB of RAM?"

It seems to me that none of the PAGE_OFFSET values offered (the patch
in allows PAGE_OFFSET to be set to 0xe0000000, 0xc0000000, 0x80000000
or 0x40000000, in addition to the 4G/4G value of 0x02000000) are
exactly what someone with 1 GB of RAM wants.  They'd be forced to go
down to a 2G/2G split which cheats userspace of quite a bit of address
space (admittedly with only 1 GB of RAM, a process bigger than 2 GB is
a bit of a stretch).  In any case a 2.75G/1.25G split as suggested
earlier works fine with 1 GB of RAM.

Also I notice that Con's patch modifies vmlinux.ld.S to update the
kernel base address, while this patch doesn't.  Is that intentional or
is does the patch depend on some other patches that use the defines in
page.h somehow as controlled by the following change?

+AFLAGS_vmlinux.lds.o += -include $(TOPDIR)/include/asm-i386/page.h

Thanks,
  Roland

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Andrew Morton]

"Martin J. Bligh" <mbligh@aracnet.com> wrote:
>
> I had a patch for a config option, ported forward by someone at IBM (I forget
>  who, possibly Dave) from Andrea's original. I think we finally decided 
>  (in consultation with Andrea) we could drop the complicated stuff he had
>  in the asm code, so it's pretty simple ... something like this:

I sent such a patch to the boss many moons ago and he said "go away, this
is a vendor-only thing". 

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Martin J. Bligh]

> The 896M/128M split has a bit of a problem now each zone has its own LRU:
> the size of the highmem zone is less than the amount of memory which is
> described by the default /proc/sys/vm/dirty_ratio.  So it is easy to
> completely fill highmem with dirty pages.  This causes a fairly large
> amount of writeback via vmscan.c's writepage().  This causes poor I/O
> submission patterns.  This causes a simple large, linear `dd' write to run
> at only 50-70% of disk bandwidth.  (This was 6-12 months ago - it might be
> a bit better now)
> 
> But I seem to be the only person who has noticed this yet ;) A workaround
> is to decrease dirty_ratio and dirty_background_ratio.
> 
> Decreasing PAGE_OFFSET as above is attractive, but I believe 0xc0000000 is
> part of the ABI, and although we know (from the 4g/4g and other such
> patches) that everything will work OK, I wonder if it's really worth doing,
> especially as it's a compile-time thing.
> 
> But hey, if someone can identify specific benefits from it then perhaps
> sneaking in a config option, or maintaining an external patch would be
> worthwhile.

I had a patch for a config option, ported forward by someone at IBM (I forget
who, possibly Dave) from Andrea's original. I think we finally decided 
(in consultation with Andrea) we could drop the complicated stuff he had
in the asm code, so it's pretty simple ... something like this:

diff -purN -X /home/mbligh/.diff.exclude 200-config_hz/arch/i386/Kconfig 210-config_page_offset/arch/i386/Kconfig
--- 200-config_hz/arch/i386/Kconfig	2004-03-14 09:48:36.000000000 -0800
+++ 210-config_page_offset/arch/i386/Kconfig	2004-03-14 09:49:04.000000000 -0800
@@ -763,6 +763,44 @@ config HIGHMEM64G
 
 endchoice
 
+choice
+	help
+	  On i386, a process can only virtually address 4GB of memory.  This
+	  lets you select how much of that virtual space you would like to 
+	  devoted to userspace, and how much to the kernel.
+
+	  Some userspace programs would like to address as much as possible and 
+	  have few demands of the kernel other than it get out of the way.  These
+	  users may opt to use the 3.5GB option to give their userspace program 
+	  as much room as possible.  Due to alignment issues imposed by PAE, 
+	  the "3.5GB" option is unavailable if "64GB" high memory support is 
+	  enabled.
+
+	  Other users (especially those who use PAE) may be running out of
+	  ZONE_NORMAL memory.  Those users may benefit from increasing the
+	  kernel's virtual address space size by taking it away from userspace, 
+	  which may not need all of its space.  An indicator that this is 
+	  happening is when /proc/Meminfo's "LowFree:" is a small percentage of
+	  "LowTotal:" while "HighFree:" is very large.
+
+	  If unsure, say "3GB"
+	prompt "User address space size"
+        default 1GB
+	
+config	05GB
+	bool "3.5 GB"
+	depends on !HIGHMEM64G
+	
+config	1GB
+	bool "3 GB"
+	
+config	2GB
+	bool "2 GB"
+	
+config	3GB
+	bool "1 GB"
+endchoice
+
 config HIGHMEM
 	bool
 	depends on HIGHMEM64G || HIGHMEM4G
diff -purN -X /home/mbligh/.diff.exclude 200-config_hz/arch/i386/Makefile 210-config_page_offset/arch/i386/Makefile
--- 200-config_hz/arch/i386/Makefile	2004-03-12 11:06:23.000000000 -0800
+++ 210-config_page_offset/arch/i386/Makefile	2004-03-14 09:49:04.000000000 -0800
@@ -114,6 +114,7 @@ drivers-$(CONFIG_PM)			+= arch/i386/powe
 
 CFLAGS += $(mflags-y)
 AFLAGS += $(mflags-y)
+AFLAGS_vmlinux.lds.o += -include $(TOPDIR)/include/asm-i386/page.h
 
 boot := arch/i386/boot
 
diff -purN -X /home/mbligh/.diff.exclude 200-config_hz/include/asm-i386/page.h 210-config_page_offset/include/asm-i386/page.h
--- 200-config_hz/include/asm-i386/page.h	2004-03-12 11:07:27.000000000 -0800
+++ 210-config_page_offset/include/asm-i386/page.h	2004-03-14 09:49:04.000000000 -0800
@@ -97,9 +97,20 @@ typedef struct { unsigned long pgprot; }
 #ifdef CONFIG_X86_4G_VM_LAYOUT
 #define __PAGE_OFFSET		(0x02000000)
 #define TASK_SIZE		(0xff000000)
-#else
+#elif defined(CONFIG_05GB)
+#define __PAGE_OFFSET		(0xe0000000)
+#define TASK_SIZE		(0xe0000000)
+#elif defined(CONFIG_1GB)
 #define __PAGE_OFFSET		(0xc0000000)
 #define TASK_SIZE		(0xc0000000)
+#elif defined(CONFIG_2GB)
+#define __PAGE_OFFSET		(0x80000000)
+#define TASK_SIZE		(0x80000000)
+#elif defined(CONFIG_3GB)
+#define __PAGE_OFFSET		(0x40000000)
+#define TASK_SIZE		(0x40000000)
+#else
+#error I have no idea what VM layout to use
 #endif
 
 /*
diff -purN -X /home/mbligh/.diff.exclude 200-config_hz/include/asm-i386/processor.h 210-config_page_offset/include/asm-i386/processor.h
--- 200-config_hz/include/asm-i386/processor.h	2004-03-12 11:07:47.000000000 -0800
+++ 210-config_page_offset/include/asm-i386/processor.h	2004-03-14 09:49:04.000000000 -0800
@@ -294,7 +294,11 @@ extern unsigned int mca_pentium_flag;
 /* This decides where the kernel will search for a free chunk of vm
  * space during mmap's.
  */
+#ifdef CONFIG_05GB
+#define TASK_UNMAPPED_BASE	(PAGE_ALIGN(TASK_SIZE / 16))
+#else
 #define TASK_UNMAPPED_BASE	(PAGE_ALIGN(TASK_SIZE / 3))
+#endif
 
 /*
  * Size of io_bitmap, covering ports 0 to 0x3ff.

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Marc-Christian Petersen]

On Wednesday 04 August 2004 21:06, Andrew Morton wrote:

Hi Andrew,

> The 896M/128M split has a bit of a problem now each zone has its own LRU:
> the size of the highmem zone is less than the amount of memory which is
> described by the default /proc/sys/vm/dirty_ratio.  So it is easy to
> completely fill highmem with dirty pages.  This causes a fairly large
> amount of writeback via vmscan.c's writepage().  This causes poor I/O
> submission patterns.  This causes a simple large, linear `dd' write to run
> at only 50-70% of disk bandwidth.  (This was 6-12 months ago - it might be
> a bit better now)
> But I seem to be the only person who has noticed this yet ;) A workaround
> is to decrease dirty_ratio and dirty_background_ratio.

hmm, never tested to change the split with 2.6.x, but on 2.4 I didn't notice 
any disk i/o regressions. Maybe due to a different VM ;)


> Decreasing PAGE_OFFSET as above is attractive, but I believe 0xc0000000 is
> part of the ABI, and although we know (from the 4g/4g and other such
> patches) that everything will work OK, I wonder if it's really worth doing,
> especially as it's a compile-time thing.

> But hey, if someone can identify specific benefits from it then perhaps
> sneaking in a config option, or maintaining an external patch would be
> worthwhile.

Maybe we can introduce something like 3.5GB patch like 2.4-aa and 2.4-wolk 
has? For reference: 

http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.23aa3/00_3.5G-address-space-5

Let me know and I'll cook up a 2.6 version.

Grmpf, that reminds me of my Documentation cleanup patches ;(

ciao, Marc

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Andrew Morton]

Jens Axboe <axboe@suse.de> wrote:
>
> > > > -#define __PAGE_OFFSET		(0xC0000000)
>  > > > +#define __PAGE_OFFSET		(0xB0000000)
>  > > >  #else
>  > > > -#define __PAGE_OFFSET		(0xC0000000UL)
>  > > > +#define __PAGE_OFFSET		(0xB0000000UL)
>  > > >  #endif
>  > >
>  > > Yup precisely. I agree that there probably isn't a whole lot of
>  > > performance hit on a 1GB, it just seems silly that we need highmem on
>  > > such a standard memory configuration these days. Especially when just
>  > > moving the offset slightly removes that need.
>  > 
>  > As a desktop user with 1024MB ram I agree that HIMEM has a silly threshold and 
>  > should not need to be enabled in this case. Its becoming common, especially 
>  > with dual channel memory systems to use 2x512MB sticks. On a hunch I bet 
>  > 2x512 is more common that 1x512 and 1x256 so why not merge this up? Who would 
>  > we submit this patch to?
> 
>  One way would be to ask Andrew what he thinks?

The 896M/128M split has a bit of a problem now each zone has its own LRU:
the size of the highmem zone is less than the amount of memory which is
described by the default /proc/sys/vm/dirty_ratio.  So it is easy to
completely fill highmem with dirty pages.  This causes a fairly large
amount of writeback via vmscan.c's writepage().  This causes poor I/O
submission patterns.  This causes a simple large, linear `dd' write to run
at only 50-70% of disk bandwidth.  (This was 6-12 months ago - it might be
a bit better now)

But I seem to be the only person who has noticed this yet ;) A workaround
is to decrease dirty_ratio and dirty_background_ratio.

Decreasing PAGE_OFFSET as above is attractive, but I believe 0xc0000000 is
part of the ABI, and although we know (from the 4g/4g and other such
patches) that everything will work OK, I wonder if it's really worth doing,
especially as it's a compile-time thing.

But hey, if someone can identify specific benefits from it then perhaps
sneaking in a config option, or maintaining an external patch would be
worthwhile.

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Jens Axboe]

On Wed, Aug 04 2004, Eric Bambach wrote:
> On Wednesday 04 August 2004 01:06 am, Jens Axboe wrote:
> > On Wed, Aug 04 2004, Con Kolivas wrote:
> > > Jens Axboe wrote:
> > > >On Mon, Aug 02 2004, Barry K. Nathan wrote:
> > > >>On Mon, Aug 02, 2004 at 04:02:34PM -0500, Steve Snyder wrote:
> > > >>>There seems to be a controversy about the use of the CONFIG_HIGHMEM4G
> > > >>>kernel configuration.  After reading many posts on the subject, I
> > > >>> still don't know which setting is best for me.
> > >
> > > No idea what the performance hit is of highmem these days - it seems
> > > insignificant compared to 2.4 so I've had it enabled for 1Gb ram.
> > >
> > > >There's also the option of moving the mapping only slightly, so that all
> > > >of the 1G fits in low memory. That's the best option for 1G desktop
> > > >machines, imho. Changing PAGE_OFFSET from 0xc0000000 to 0xb0000000 would
> > > >probably be enough.
> > > >
> > > >Then you can have your cake and eat it too.
> > >
> > > Something like this attached patch? Seems to work nicely. Thanks!
> > >
> > > Cheers,
> > > Con
> > >
> > > Index: linux-2.6.8-rc2-mm2/arch/i386/kernel/vmlinux.lds.S
> > > ===================================================================
> > > --- linux-2.6.8-rc2-mm2.orig/arch/i386/kernel/vmlinux.lds.S	2004-05-23
> > > 12:54:46.000000000 +1000 +++
> > > linux-2.6.8-rc2-mm2/arch/i386/kernel/vmlinux.lds.S	2004-08-04
> > > 00:20:02.219462913 +1000 @@ -11,7 +11,7 @@
> > >  jiffies = jiffies_64;
> > >  SECTIONS
> > >  {
> > > -  . = 0xC0000000 + 0x100000;
> > > +  . = 0xB0000000 + 0x100000;
> > >    /* read-only */
> > >    _text = .;			/* Text and read-only data */
> > >    .text : {
> > > Index: linux-2.6.8-rc2-mm2/include/asm-i386/page.h
> > > ===================================================================
> > > --- linux-2.6.8-rc2-mm2.orig/include/asm-i386/page.h	2004-08-03
> > > 01:29:28.000000000 +1000 +++
> > > linux-2.6.8-rc2-mm2/include/asm-i386/page.h	2004-08-03 23:58:16.000000000
> > > +1000 @@ -123,9 +123,9 @@
> > >  #endif /* __ASSEMBLY__ */
> > >
> > >  #ifdef __ASSEMBLY__
> > > -#define __PAGE_OFFSET		(0xC0000000)
> > > +#define __PAGE_OFFSET		(0xB0000000)
> > >  #else
> > > -#define __PAGE_OFFSET		(0xC0000000UL)
> > > +#define __PAGE_OFFSET		(0xB0000000UL)
> > >  #endif
> >
> > Yup precisely. I agree that there probably isn't a whole lot of
> > performance hit on a 1GB, it just seems silly that we need highmem on
> > such a standard memory configuration these days. Especially when just
> > moving the offset slightly removes that need.
> 
> As a desktop user with 1024MB ram I agree that HIMEM has a silly threshold and 
> should not need to be enabled in this case. Its becoming common, especially 
> with dual channel memory systems to use 2x512MB sticks. On a hunch I bet 
> 2x512 is more common that 1x512 and 1x256 so why not merge this up? Who would 
> we submit this patch to?

One way would be to ask Andrew what he thinks?

-- 
Jens Axboe

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Eric Bambach]

On Wednesday 04 August 2004 01:06 am, Jens Axboe wrote:
> On Wed, Aug 04 2004, Con Kolivas wrote:
> > Jens Axboe wrote:
> > >On Mon, Aug 02 2004, Barry K. Nathan wrote:
> > >>On Mon, Aug 02, 2004 at 04:02:34PM -0500, Steve Snyder wrote:
> > >>>There seems to be a controversy about the use of the CONFIG_HIGHMEM4G
> > >>>kernel configuration.  After reading many posts on the subject, I
> > >>> still don't know which setting is best for me.
> >
> > No idea what the performance hit is of highmem these days - it seems
> > insignificant compared to 2.4 so I've had it enabled for 1Gb ram.
> >
> > >There's also the option of moving the mapping only slightly, so that all
> > >of the 1G fits in low memory. That's the best option for 1G desktop
> > >machines, imho. Changing PAGE_OFFSET from 0xc0000000 to 0xb0000000 would
> > >probably be enough.
> > >
> > >Then you can have your cake and eat it too.
> >
> > Something like this attached patch? Seems to work nicely. Thanks!
> >
> > Cheers,
> > Con
> >
> > Index: linux-2.6.8-rc2-mm2/arch/i386/kernel/vmlinux.lds.S
> > ===================================================================
> > --- linux-2.6.8-rc2-mm2.orig/arch/i386/kernel/vmlinux.lds.S	2004-05-23
> > 12:54:46.000000000 +1000 +++
> > linux-2.6.8-rc2-mm2/arch/i386/kernel/vmlinux.lds.S	2004-08-04
> > 00:20:02.219462913 +1000 @@ -11,7 +11,7 @@
> >  jiffies = jiffies_64;
> >  SECTIONS
> >  {
> > -  . = 0xC0000000 + 0x100000;
> > +  . = 0xB0000000 + 0x100000;
> >    /* read-only */
> >    _text = .;			/* Text and read-only data */
> >    .text : {
> > Index: linux-2.6.8-rc2-mm2/include/asm-i386/page.h
> > ===================================================================
> > --- linux-2.6.8-rc2-mm2.orig/include/asm-i386/page.h	2004-08-03
> > 01:29:28.000000000 +1000 +++
> > linux-2.6.8-rc2-mm2/include/asm-i386/page.h	2004-08-03 23:58:16.000000000
> > +1000 @@ -123,9 +123,9 @@
> >  #endif /* __ASSEMBLY__ */
> >
> >  #ifdef __ASSEMBLY__
> > -#define __PAGE_OFFSET		(0xC0000000)
> > +#define __PAGE_OFFSET		(0xB0000000)
> >  #else
> > -#define __PAGE_OFFSET		(0xC0000000UL)
> > +#define __PAGE_OFFSET		(0xB0000000UL)
> >  #endif
>
> Yup precisely. I agree that there probably isn't a whole lot of
> performance hit on a 1GB, it just seems silly that we need highmem on
> such a standard memory configuration these days. Especially when just
> moving the offset slightly removes that need.

As a desktop user with 1024MB ram I agree that HIMEM has a silly threshold and 
should not need to be enabled in this case. Its becoming common, especially 
with dual channel memory systems to use 2x512MB sticks. On a hunch I bet 
2x512 is more common that 1x512 and 1x256 so why not merge this up? Who would 
we submit this patch to?
-- 

-EB

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Jens Axboe]

On Wed, Aug 04 2004, Con Kolivas wrote:
> Jens Axboe wrote:
> >On Mon, Aug 02 2004, Barry K. Nathan wrote:
> >
> >>On Mon, Aug 02, 2004 at 04:02:34PM -0500, Steve Snyder wrote:
> >>
> >>>There seems to be a controversy about the use of the CONFIG_HIGHMEM4G  
> >>>kernel configuration.  After reading many posts on the subject, I still 
> >>>don't know which setting is best for me.
> 
> No idea what the performance hit is of highmem these days - it seems 
> insignificant compared to 2.4 so I've had it enabled for 1Gb ram.
> 
> >There's also the option of moving the mapping only slightly, so that all
> >of the 1G fits in low memory. That's the best option for 1G desktop
> >machines, imho. Changing PAGE_OFFSET from 0xc0000000 to 0xb0000000 would
> >probably be enough.
> >
> >Then you can have your cake and eat it too.
> 
> Something like this attached patch? Seems to work nicely. Thanks!
> 
> Cheers,
> Con

> Index: linux-2.6.8-rc2-mm2/arch/i386/kernel/vmlinux.lds.S
> ===================================================================
> --- linux-2.6.8-rc2-mm2.orig/arch/i386/kernel/vmlinux.lds.S	2004-05-23 12:54:46.000000000 +1000
> +++ linux-2.6.8-rc2-mm2/arch/i386/kernel/vmlinux.lds.S	2004-08-04 00:20:02.219462913 +1000
> @@ -11,7 +11,7 @@
>  jiffies = jiffies_64;
>  SECTIONS
>  {
> -  . = 0xC0000000 + 0x100000;
> +  . = 0xB0000000 + 0x100000;
>    /* read-only */
>    _text = .;			/* Text and read-only data */
>    .text : {
> Index: linux-2.6.8-rc2-mm2/include/asm-i386/page.h
> ===================================================================
> --- linux-2.6.8-rc2-mm2.orig/include/asm-i386/page.h	2004-08-03 01:29:28.000000000 +1000
> +++ linux-2.6.8-rc2-mm2/include/asm-i386/page.h	2004-08-03 23:58:16.000000000 +1000
> @@ -123,9 +123,9 @@
>  #endif /* __ASSEMBLY__ */
>  
>  #ifdef __ASSEMBLY__
> -#define __PAGE_OFFSET		(0xC0000000)
> +#define __PAGE_OFFSET		(0xB0000000)
>  #else
> -#define __PAGE_OFFSET		(0xC0000000UL)
> +#define __PAGE_OFFSET		(0xB0000000UL)
>  #endif

Yup precisely. I agree that there probably isn't a whole lot of
performance hit on a 1GB, it just seems silly that we need highmem on
such a standard memory configuration these days. Especially when just
moving the offset slightly removes that need.

-- 
Jens Axboe

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Con Kolivas]

[-- Attachment #1.1: Type: text/plain, Size: 838 bytes --]

Jens Axboe wrote:
> On Mon, Aug 02 2004, Barry K. Nathan wrote:
> 
>>On Mon, Aug 02, 2004 at 04:02:34PM -0500, Steve Snyder wrote:
>>
>>>There seems to be a controversy about the use of the CONFIG_HIGHMEM4G  
>>>kernel configuration.  After reading many posts on the subject, I still 
>>>don't know which setting is best for me.

No idea what the performance hit is of highmem these days - it seems 
insignificant compared to 2.4 so I've had it enabled for 1Gb ram.

> There's also the option of moving the mapping only slightly, so that all
> of the 1G fits in low memory. That's the best option for 1G desktop
> machines, imho. Changing PAGE_OFFSET from 0xc0000000 to 0xb0000000 would
> probably be enough.
> 
> Then you can have your cake and eat it too.

Something like this attached patch? Seems to work nicely. Thanks!

Cheers,
Con

[-- Attachment #1.2: 1g_lowmem_i386.diff --]
[-- Type: text/x-patch, Size: 1038 bytes --]

Index: linux-2.6.8-rc2-mm2/arch/i386/kernel/vmlinux.lds.S
===================================================================
--- linux-2.6.8-rc2-mm2.orig/arch/i386/kernel/vmlinux.lds.S	2004-05-23 12:54:46.000000000 +1000
+++ linux-2.6.8-rc2-mm2/arch/i386/kernel/vmlinux.lds.S	2004-08-04 00:20:02.219462913 +1000
@@ -11,7 +11,7 @@
 jiffies = jiffies_64;
 SECTIONS
 {
-  . = 0xC0000000 + 0x100000;
+  . = 0xB0000000 + 0x100000;
   /* read-only */
   _text = .;			/* Text and read-only data */
   .text : {
Index: linux-2.6.8-rc2-mm2/include/asm-i386/page.h
===================================================================
--- linux-2.6.8-rc2-mm2.orig/include/asm-i386/page.h	2004-08-03 01:29:28.000000000 +1000
+++ linux-2.6.8-rc2-mm2/include/asm-i386/page.h	2004-08-03 23:58:16.000000000 +1000
@@ -123,9 +123,9 @@
 #endif /* __ASSEMBLY__ */
 
 #ifdef __ASSEMBLY__
-#define __PAGE_OFFSET		(0xC0000000)
+#define __PAGE_OFFSET		(0xB0000000)
 #else
-#define __PAGE_OFFSET		(0xC0000000UL)
+#define __PAGE_OFFSET		(0xB0000000UL)
 #endif
 
 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Prakash K. Cheemplavam]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jens Axboe wrote:
| On Mon, Aug 02 2004, Barry K. Nathan wrote:
|
|>On Mon, Aug 02, 2004 at 04:02:34PM -0500, Steve Snyder wrote:
|>
|>>There seems to be a controversy about the use of the CONFIG_HIGHMEM4G
|>>kernel configuration.  After reading many posts on the subject, I still
|>>don't know which setting is best for me.
|>
|>On my own desktop system with 1GB RAM, any highmem slowdown seems to be
|>outweighed by the fact that more disk data stays cached in RAM (so I hit
|>the disk much less often).
|
|
| There's also the option of moving the mapping only slightly, so that all
| of the 1G fits in low memory. That's the best option for 1G desktop
| machines, imho. Changing PAGE_OFFSET from 0xc0000000 to 0xb0000000 would
| probably be enough.
|
| Then you can have your cake and eat it too.

This works nicely for me. I wonder why this doesn't become standard
behaviour in kernel. At least a lot of people would be happy about it.

Prakash
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBD52UxU2n/+9+t5gRArjwAKDhLKcV2C42O++Eqd7yFOQoURtoxgCgyt/m
5dDyjJtoF7WDzelrGzk7MGM=
=fdFY
-----END PGP SIGNATURE-----

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Jens Axboe]

On Mon, Aug 02 2004, Barry K. Nathan wrote:
> On Mon, Aug 02, 2004 at 04:02:34PM -0500, Steve Snyder wrote:
> > There seems to be a controversy about the use of the CONFIG_HIGHMEM4G  
> > kernel configuration.  After reading many posts on the subject, I still 
> > don't know which setting is best for me.
> 
> On my own desktop system with 1GB RAM, any highmem slowdown seems to be
> outweighed by the fact that more disk data stays cached in RAM (so I hit
> the disk much less often).

There's also the option of moving the mapping only slightly, so that all
of the 1G fits in low memory. That's the best option for 1G desktop
machines, imho. Changing PAGE_OFFSET from 0xc0000000 to 0xb0000000 would
probably be enough.

Then you can have your cake and eat it too.

-- 
Jens Axboe

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Barry K. Nathan]

On Mon, Aug 02, 2004 at 04:02:34PM -0500, Steve Snyder wrote:
> There seems to be a controversy about the use of the CONFIG_HIGHMEM4G  
> kernel configuration.  After reading many posts on the subject, I still 
> don't know which setting is best for me.

On my own desktop system with 1GB RAM, any highmem slowdown seems to be
outweighed by the fact that more disk data stays cached in RAM (so I hit
the disk much less often).

Everyone else I know has also found the extra RAM to greatly outweigh
the highmem slowdown, although those people are running clusters &
servers, not desktops, with this much RAM.

-Barry K. Nathan <barryn@pobox.com>

Re: HIGHMEM4G config for 1GB RAM on desktop?

[Bart Alewijnse]

Last time I checked (which is already a somewhat older kernel) without
the 4G option you can only only address nine hundred something out of
of your thousand and twenty four megs.

And I believe the highmem thing creates some overhead.

So if you can live without the ~100mb, keep it off. If you want it,
turn it on. Other people will probably tell you te exact effects of
highmem. Probably in hard to decipher english too:)

I kept it off for ages, myself, basically because I rarely use all my
1GB. I think it's on now.

--Bart

HIGHMEM4G config for 1GB RAM on desktop?

[Steve Snyder]

There seems to be a controversy about the use of the CONFIG_HIGHMEM4G  
kernel configuration.  After reading many posts on the subject, I still 
don't know which setting is best for me.

My x86 system has 1.0GB of installed memory and is primarily used as a 
desktop environment.  I don't have any SCSI devices that might require a 
high-memory buffer.  Should I enable the CONFIG_HIGHMEM4G config for this 
environment or not?

Thanks.