* Kernel memory requirements and BK
@ 2004-11-05 14:46 Art Haas
2004-11-05 14:55 ` Grzegorz Kulewski
0 siblings, 1 reply; 4+ messages in thread
From: Art Haas @ 2004-11-05 14:46 UTC (permalink / raw)
To: linux-kernel
Hi.
I've been having problems with 'bk pull' execution when using kernels
after the 2.6.8/2.6.8.1 releases. My machine has 192M of memory and 100M
of swap, so I believe that the memory requirements for using BK to keep
up with the kernel is sufficient, and when the machine is running with a
2.6.8.1 kernel I can 'bk pull' even if X windows is running. With the
2.6.9 and 2.6.10-rc kernels, BK bombs out with out-of-memory errors once
the repository checking begins. I've run the 'bk pull' under the newer
kernels without X running, as well as shutting down various daemons, and
still things fail with memory errors.
Here's a snippet of /var/log/messages under 2.6.10-rc1 when a
'bk pull' fails:
kernel: oom-killer: gfp_mask=0xd2
kernel: DMA per-cpu:
kernel: cpu 0 hot: low 2, high 6, batch 1
kernel: cpu 0 cold: low 0, high 2, batch 1
kernel: Normal per-cpu:
kernel: cpu 0 hot: low 22, high 66, batch 11
kernel: cpu 0 cold: low 0, high 22, batch 11
kernel: HighMem per-cpu: empty
kernel:
kernel: Free pages: 420kB (0kB HighMem)
kernel: Active:45877 inactive:64 dirty:0 writeback:0 unstable:0 free:105 slab:1451 mapped:45846 pagetables:195
kernel: DMA free:36kB min:36kB low:72kB high:108kB active:13684kB inactive:0kB present:16384kB pages_scanned:14695 all_unreclaimable? yes
kernel: protections[]: 0 0 0
kernel: Normal free:384kB min:400kB low:800kB high:1200kB active:169824kB inactive:256kB present:180224kB pages_scanned:214215 all_unreclaimable? yes
kernel: protections[]: 0 0 0
kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
kernel: protections[]: 0 0 0
kernel: DMA: 1*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36kB
kernel: Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 384kB
kernel: HighMem: empty
kernel: Swap cache: add 110618, delete 108152, find 38132/44306, race 0+1
The machine used to have only 128M of memory, and 'bk pull' worked
albiet _very_ slowly as the machine went heavily into swap space usage,
but it seemed that once the number of files went over 20,000 or so I had
to add more memory, and then once again 'bk pull' worked without any
problem.
I realize that there have been many, many changes to the kernel between
2.6.8 and the current BK, but my configuration for my builds has
remained mostly the same, so I am at somewhat of a loss to figure out
just where the extra memory usage is coming from.
Is there a known set of changes that explain the additional memory
requirements?
Art Haas
--
Man once surrendering his reason, has no remaining guard against absurdities
the most monstrous, and like a ship without rudder, is the sport of every wind.
-Thomas Jefferson to James Smith, 1822
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Kernel memory requirements and BK
2004-11-05 14:46 Kernel memory requirements and BK Art Haas
@ 2004-11-05 14:55 ` Grzegorz Kulewski
2004-11-05 15:28 ` Art Haas
0 siblings, 1 reply; 4+ messages in thread
From: Grzegorz Kulewski @ 2004-11-05 14:55 UTC (permalink / raw)
To: Art Haas; +Cc: linux-kernel
On Fri, 5 Nov 2004, Art Haas wrote:
> Hi.
>
> I've been having problems with 'bk pull' execution when using kernels
> after the 2.6.8/2.6.8.1 releases. My machine has 192M of memory and 100M
> of swap, so I believe that the memory requirements for using BK to keep
> up with the kernel is sufficient, and when the machine is running with a
> 2.6.8.1 kernel I can 'bk pull' even if X windows is running. With the
> 2.6.9 and 2.6.10-rc kernels, BK bombs out with out-of-memory errors once
> the repository checking begins. I've run the 'bk pull' under the newer
> kernels without X running, as well as shutting down various daemons, and
> still things fail with memory errors.
Maybe you have some kernel debuging options set? Some of them can eat your
RAM very fast with fs heavy load.
Grzegorz Kulewski
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Kernel memory requirements and BK
2004-11-05 14:55 ` Grzegorz Kulewski
@ 2004-11-05 15:28 ` Art Haas
0 siblings, 0 replies; 4+ messages in thread
From: Art Haas @ 2004-11-05 15:28 UTC (permalink / raw)
To: Grzegorz Kulewski; +Cc: linux-kernel
On Fri, Nov 05, 2004 at 03:55:53PM +0100, Grzegorz Kulewski wrote:
> On Fri, 5 Nov 2004, Art Haas wrote:
>
> >Hi.
> >
> >I've been having problems with 'bk pull' execution when using kernels
> >after the 2.6.8/2.6.8.1 releases. My machine has 192M of memory and 100M
> >of swap, so I believe that the memory requirements for using BK to keep
> >up with the kernel is sufficient, and when the machine is running with a
> >2.6.8.1 kernel I can 'bk pull' even if X windows is running. With the
> >2.6.9 and 2.6.10-rc kernels, BK bombs out with out-of-memory errors once
> >the repository checking begins. I've run the 'bk pull' under the newer
> >kernels without X running, as well as shutting down various daemons, and
> >still things fail with memory errors.
>
> Maybe you have some kernel debuging options set? Some of them can eat your
> RAM very fast with fs heavy load.
My current configuration does have a few debug options, but these options
are also in my configure for the 2.6.8.1 kernel as well:
$ grep -i debug ./art_config_2610
CONFIG_PNP_DEBUG=y
CONFIG_NETFILTER_DEBUG=y
CONFIG_SND_DEBUG=y
CONFIG_SND_DEBUG_MEMORY=y
CONFIG_SND_DEBUG_DETECT=y
# CONFIG_JBD_DEBUG is not set
# CONFIG_DEBUG_KERNEL is not set
$ grep -i debug ./art_config_2681
CONFIG_PNP_DEBUG=y
CONFIG_NETFILTER_DEBUG=y
CONFIG_SND_DEBUG=y
CONFIG_SND_DEBUG_MEMORY=y
CONFIG_SND_DEBUG_DETECT=y
# CONFIG_JBD_DEBUG is not set
# CONFIG_DEBUG_KERNEL is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
$
I believe that the PnP, Netfilter, and sound debug statements have been
in my configuration for a long time, but I no longer have older copies
of my '.config' files to confirm that.
Here are the differences between the configuration files. Aside from a
few new configuration options only found in the new kernel, nothing
jumps out at me as being an option that would significantly increase the
memory usage.
$ diff -u ./art_config_2681 ./art_config_2610
--- ./art_config_2681 2004-08-20 08:20:02.000000000 -0500
+++ ./art_config_2610 2004-10-28 20:19:29.000000000 -0500
@@ -1,10 +1,13 @@
#
# Automatically generated make config: don't edit
+# Linux kernel version: 2.6.10-rc1-ajh
+# Thu Oct 28 20:19:01 2004
#
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
+CONFIG_GENERIC_IOMAP=y
#
# Code maturity level options
@@ -13,10 +16,12 @@
# CONFIG_CLEAN_COMPILE is not set
CONFIG_BROKEN=y
CONFIG_BROKEN_ON_SMP=y
+CONFIG_LOCK_KERNEL=y
#
# General setup
#
+CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
@@ -26,17 +31,20 @@
# CONFIG_AUDIT is not set
CONFIG_LOG_BUF_SHIFT=14
CONFIG_HOTPLUG=y
+CONFIG_KOBJECT_UEVENT=y
# CONFIG_IKCONFIG is not set
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_FUTEX=y
CONFIG_EPOLL=y
-CONFIG_IOSCHED_NOOP=y
-CONFIG_IOSCHED_AS=y
-CONFIG_IOSCHED_DEADLINE=y
-CONFIG_IOSCHED_CFQ=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
+CONFIG_SHMEM=y
+CONFIG_CC_ALIGN_FUNCTIONS=0
+CONFIG_CC_ALIGN_LABELS=0
+CONFIG_CC_ALIGN_LOOPS=0
+CONFIG_CC_ALIGN_JUMPS=0
+# CONFIG_TINY_SHMEM is not set
#
# Loadable module support
@@ -46,6 +54,7 @@
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
# CONFIG_MODVERSIONS is not set
+# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
#
@@ -74,6 +83,7 @@
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
+# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
@@ -127,7 +137,7 @@
# ACPI (Advanced Configuration and Power Interface) Support
#
# CONFIG_ACPI is not set
-CONFIG_ACPI_BOOT=y
+CONFIG_ACPI_BLACKLIST_YEAR=0
#
# CPU Frequency scaling
@@ -144,7 +154,6 @@
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
-CONFIG_PCI_MMCONFIG=y
# CONFIG_PCI_LEGACY_PROC is not set
CONFIG_PCI_NAMES=y
CONFIG_ISA=y
@@ -153,9 +162,13 @@
# CONFIG_SCx200 is not set
#
-# PCMCIA/CardBus support
+# PCCARD (PCMCIA/CardBus) support
+#
+# CONFIG_PCCARD is not set
+
+#
+# PC-card bridges
#
-# CONFIG_PCMCIA is not set
CONFIG_PCMCIA_PROBE=y
#
@@ -227,7 +240,17 @@
# CONFIG_BLK_DEV_SX8 is not set
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_SIZE=4096
+CONFIG_INITRAMFS_SOURCE=""
# CONFIG_LBD is not set
+# CONFIG_CDROM_PKTCDVD is not set
+
+#
+# IO Schedulers
+#
+CONFIG_IOSCHED_NOOP=y
+CONFIG_IOSCHED_AS=y
+CONFIG_IOSCHED_DEADLINE=y
+CONFIG_IOSCHED_CFQ=y
#
# ATA/ATAPI/MFM/RLL support
@@ -246,7 +269,6 @@
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_IDE_TASK_IOCTL is not set
-CONFIG_IDE_TASKFILE_IO=y
#
# IDE chipset support/bugfixes
@@ -264,7 +286,6 @@
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
-CONFIG_BLK_DEV_ADMA=y
# CONFIG_BLK_DEV_AEC62XX is not set
CONFIG_BLK_DEV_ALI15X3=y
# CONFIG_WDC_ALI15X3 is not set
@@ -349,17 +370,13 @@
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
+CONFIG_INET_TUNNEL=m
#
# IP: Virtual Server Configuration
#
# CONFIG_IP_VS is not set
-CONFIG_IPV6=m
-CONFIG_IPV6_PRIVACY=y
-CONFIG_INET6_AH=m
-CONFIG_INET6_ESP=m
-CONFIG_INET6_IPCOMP=m
-CONFIG_IPV6_TUNNEL=m
+# CONFIG_IPV6 is not set
CONFIG_NETFILTER=y
CONFIG_NETFILTER_DEBUG=y
@@ -367,6 +384,9 @@
# IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=m
+# CONFIG_IP_NF_CT_ACCT is not set
+# CONFIG_IP_NF_CONNTRACK_MARK is not set
+# CONFIG_IP_NF_CT_PROTO_SCTP is not set
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_TFTP=m
@@ -391,8 +411,16 @@
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_OWNER=m
+CONFIG_IP_NF_MATCH_ADDRTYPE=m
+CONFIG_IP_NF_MATCH_REALM=m
+# CONFIG_IP_NF_MATCH_SCTP is not set
+CONFIG_IP_NF_MATCH_COMMENT=m
+# CONFIG_IP_NF_MATCH_HASHLIMIT is not set
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
+CONFIG_IP_NF_TARGET_LOG=m
+CONFIG_IP_NF_TARGET_ULOG=m
+CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
@@ -411,23 +439,12 @@
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_CLASSIFY=m
-CONFIG_IP_NF_TARGET_LOG=m
-CONFIG_IP_NF_TARGET_ULOG=m
-CONFIG_IP_NF_TARGET_TCPMSS=m
+# CONFIG_IP_NF_RAW is not set
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
# CONFIG_IP_NF_COMPAT_IPCHAINS is not set
# CONFIG_IP_NF_COMPAT_IPFWADM is not set
-# CONFIG_IP_NF_RAW is not set
-CONFIG_IP_NF_MATCH_ADDRTYPE=m
-CONFIG_IP_NF_MATCH_REALM=m
-
-#
-# IPv6: Netfilter Configuration
-#
-# CONFIG_IP6_NF_QUEUE is not set
-# CONFIG_IP6_NF_IPTABLES is not set
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
@@ -447,7 +464,6 @@
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
-# CONFIG_NET_HW_FLOWCONTROL is not set
#
# QoS and/or fair queueing
@@ -522,7 +538,6 @@
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
-# CONFIG_VIA_VELOCITY is not set
# CONFIG_NET_POCKET is not set
#
@@ -536,6 +551,7 @@
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SK98LIN is not set
+# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
#
@@ -563,7 +579,7 @@
# CONFIG_PLIP is not set
CONFIG_PPP=m
# CONFIG_PPP_MULTILINK is not set
-# CONFIG_PPP_FILTER is not set
+CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
# CONFIG_PPP_SYNC_TTY is not set
CONFIG_PPP_DEFLATE=m
@@ -611,6 +627,7 @@
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
+# CONFIG_SERIO_RAW is not set
#
# Input Device Drivers
@@ -657,7 +674,6 @@
# CONFIG_PRINTER is not set
# CONFIG_PPDEV is not set
# CONFIG_TIPAR is not set
-# CONFIG_QIC02_TAPE is not set
#
# IPMI
@@ -797,6 +813,7 @@
#
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
+# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
@@ -839,6 +856,8 @@
# USB support
#
# CONFIG_USB is not set
+CONFIG_USB_ARCH_HAS_HCD=y
+CONFIG_USB_ARCH_HAS_OHCI=y
#
# USB Gadget Support
@@ -876,7 +895,8 @@
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=m
-# CONFIG_UDF_FS is not set
+CONFIG_UDF_FS=m
+CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
@@ -896,6 +916,7 @@
# CONFIG_DEVFS_FS is not set
# CONFIG_DEVPTS_FS_XATTR is not set
CONFIG_TMPFS=y
+# CONFIG_TMPFS_XATTR is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y
@@ -934,6 +955,7 @@
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_RPCSEC_GSS_KRB5=m
+# CONFIG_RPCSEC_GSS_SPKM3 is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
@@ -999,14 +1021,14 @@
# Kernel hacking
#
# CONFIG_DEBUG_KERNEL is not set
-CONFIG_EARLY_PRINTK=y
-# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_FRAME_POINTER is not set
+CONFIG_EARLY_PRINTK=y
CONFIG_4KSTACKS=y
#
# Security options
#
+# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
#
@@ -1020,6 +1042,7 @@
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
+CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_TWOFISH=m
@@ -1043,5 +1066,7 @@
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=m
CONFIG_ZLIB_DEFLATE=m
+CONFIG_GENERIC_HARDIRQS=y
+CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y
Art Haas
--
Man once surrendering his reason, has no remaining guard against absurdities
the most monstrous, and like a ship without rudder, is the sport of every wind.
-Thomas Jefferson to James Smith, 1822
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Kernel memory requirements and BK
@ 2004-11-07 5:02 Chuck Ebbert
0 siblings, 0 replies; 4+ messages in thread
From: Chuck Ebbert @ 2004-11-07 5:02 UTC (permalink / raw)
To: Art Haas; +Cc: linux-kernel
Art Haas wrote:
> With the 2.6.9 and 2.6.10-rc kernels, BK bombs out with
> out-of-memory errors once the repository checking begins.
There's nothing wrong with BK's performance that can't be
solved by a terahertz processor and a terabyte of RAM. ;)
But these patches might help:
===============================================================================
# vm_pages_scanned_active_list.patch
#
# Stop kswapd from looping.
#
# Patch by Nick Piggin 24 Oct 2004
# Signed-off-by: Andrew Morton <akpm@osdl.org>
# Signed-off-by: Linus Torvalds <torvalds@osdl.org>
# Status: in 2.6.10
#
--- 2.6.9/mm/vmscan.c
+++ 2.6.9.1/mm/vmscan.c
@@ -574,7 +574,6 @@ static void shrink_cache(struct zone *zo
nr_taken++;
}
zone->nr_inactive -= nr_taken;
- zone->pages_scanned += nr_taken;
spin_unlock_irq(&zone->lru_lock);
if (nr_taken == 0)
@@ -675,6 +674,7 @@ refill_inactive_zone(struct zone *zone,
}
pgscanned++;
}
+ zone->pages_scanned += pgscanned;
zone->nr_active -= pgmoved;
spin_unlock_irq(&zone->lru_lock);
===============================================================================
# spurious_oomkill.patch
#
# Prevent spurious out of memory process kills.
# Reported to work by testers on lkml.
#
# Patch by Rik van Riel <riel@redhat.com>
# Status: NOT in 2.6.10
#
--- 2.6.9/mm/vmscan.c
+++ 2.6.9.1/mm/vmscan.c
@@ -379,7 +379,7 @@
referenced = page_referenced(page, 1);
/* In active use or really unfreeable? Activate it. */
- if (referenced && page_mapping_inuse(page))
+ if (referenced && sc->priority && page_mapping_inuse(page))
goto activate_locked;
#ifdef CONFIG_SWAP
@@ -715,7 +715,7 @@
if (page_mapped(page)) {
if (!reclaim_mapped ||
(total_swap_pages == 0 && PageAnon(page)) ||
- page_referenced(page, 0)) {
+ (page_referenced(page, 0) && sc->priority)) {
list_add(&page->lru, &l_active);
continue;
}
===============================================================================
--Chuck Ebbert 06-Nov-04 23:55:23
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2004-11-07 5:05 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2004-11-05 14:46 Kernel memory requirements and BK Art Haas
2004-11-05 14:55 ` Grzegorz Kulewski
2004-11-05 15:28 ` Art Haas
2004-11-07 5:02 Chuck Ebbert
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®