(Found?) Re: Hard Hang with __alloc_pages: 0-order allocation failed (gfp=0x20/1) - Not out of memory

(Found?) Re: Hard Hang with __alloc_pages: 0-order allocation failed (gfp=0x20/1) - Not out of memory

[Roger Larsson]

Hi,

Since I read linux-kernel via achieves, this might be found already - but 
anyway... (CONFIG_SMP problem? Oh, noticed that this is 2.4.21...)

decode "(gfp=0x20/1)"  and you find that
	current->flags is PF_MEMALLOC 
	gfp is __GFP_HIGH (== GFP_ATOMIC)
and the backtrace says we are in_interrupt()

So I guess this is the key line...
	if (current->flags & PF_MEMALLOC && !in_interrupt()) {

* Less than (.min >> 2) of memory left.
* In interrupt
=> Not allowed to allocate anything more

Does the caller understand that repeating requests will not help?

In e1000_main.c
		skb = dev_alloc_skb(adapter->rx_buffer_len + reserve_len);

___skbuff.h___
static inline struct sk_buff *dev_alloc_skb(unsigned int length)
{
	return __dev_alloc_skb(length, GFP_ATOMIC);
}

static inline struct sk_buff *__dev_alloc_skb(unsigned int length,
					      int gfp_mask)
{
	struct sk_buff *skb;

	skb = alloc_skb(length+16, gfp_mask);
	if (skb)
		skb_reserve(skb,16);
	return skb;
}

And the rather big alloc_skb calls kmalloc

void * kmalloc (size_t size, int flags)
{
	cache_sizes_t *csizep = cache_sizes;

	for (; csizep->cs_size; csizep++) {
		if (size > csizep->cs_size)
			continue;
		return __kmem_cache_alloc(flags & GFP_DMA ?
			 csizep->cs_dmacachep : csizep->cs_cachep, flags);
	}
	return NULL;
}

- - - now take a close look at the try_again path - - -

static inline void * __kmem_cache_alloc (kmem_cache_t *cachep, int flags)
{
	unsigned long save_flags;
	void* objp;

	kmem_cache_alloc_head(cachep, flags);
try_again:
	local_irq_save(save_flags);

#ifdef CONFIG_SMP
	{
		cpucache_t *cc = cc_data(cachep);

		if (cc) {
			if (cc->avail) {
				STATS_INC_ALLOCHIT(cachep);
				objp = cc_entry(cc)[--cc->avail];
			} else {
				STATS_INC_ALLOCMISS(cachep);
				objp = kmem_cache_alloc_batch(cachep,cc,flags);
				if (!objp)
					goto alloc_new_slab_nolock;

- - -
alloc_new_slab_nolock:
#endif
	local_irq_restore(save_flags);
	if (kmem_cache_grow(cachep, flags))
		/* Someone may have stolen our objs.  Doesn't matter, we'll
		 * just come back here again.
		 */
		goto try_again;

But kmem_cache_grow will return failed...
	-> kmem_getpages ->  _get_free_pages -> alloc_pages

Or have I missed something?

/RogerL

-- 
Roger Larsson
Skellefteå
Sweden

Re: Hard Hang with __alloc_pages: 0-order allocation failed (gfp=0x20/1) - Not out of memory

[Roger Larsson]

Modifying the header so you find it. And write a better summary...

* When you have little memory left __alloc_pages wont allow
   allocations from interrupt context (even if PF_MEMALLOC)

	if (current->flags & PF_MEMALLOC && !in_interrupt()) {

* But in the callchain there is a loop in __kmem_cache_alloc when running SMP

> static inline void * __kmem_cache_alloc (kmem_cache_t *cachep, int flags)
> {
> 	unsigned long save_flags;
> 	void* objp;
>
> 	kmem_cache_alloc_head(cachep, flags);
> try_again:
> 	local_irq_save(save_flags);
>
> #ifdef CONFIG_SMP
> 	{
> 		cpucache_t *cc = cc_data(cachep);
>
> 		if (cc) {
> 			if (cc->avail) {
> 				STATS_INC_ALLOCHIT(cachep);
> 				objp = cc_entry(cc)[--cc->avail];
> 			} else {
> 				STATS_INC_ALLOCMISS(cachep);
> 				objp = kmem_cache_alloc_batch(cachep,cc,flags);
> 				if (!objp)
> 					goto alloc_new_slab_nolock;
>
> - - - snip - - -
> alloc_new_slab_nolock:
> #endif
> 	local_irq_restore(save_flags);
> 	if (kmem_cache_grow(cachep, flags))
> 		/* Someone may have stolen our objs.  Doesn't matter, we'll
> 		 * just come back here again.
> 		 */
> 		goto try_again;
>
> But kmem_cache_grow will return failed...
> 	-> kmem_getpages ->  _get_free_pages -> alloc_pages

But alloc_pages WILL fail every time... => infinite loop in interrupt
context... It probably should not try again in this case...

	if (!in_interrupt())
		goto try_again;

Or it should check if GFP_WAIT is set... A _busy_ wait is also a wait...

/RogerL

-- 
Roger Larsson
Skellefteå
Sweden

Re: PROBLEM: e100 or e1000 on SMP kernel freeze system (ipx+ncp)

[Roger Larsson]

Might be related to subject

"Re: Hard Hang with __alloc_pages: 0-order allocation failed (gfp=0x20/1) - 
Not out of memory"?

e1000 and SMP...

/RogerL

-- 
Roger Larsson
Skellefteå
Sweden

Re: PROBLEM: e100 or e1000 on SMP kernel freeze system (ipx+ncp)

[Andrew Shirrayev]

> Might be related to subject
>
> "Re: Hard Hang with __alloc_pages: 0-order allocation failed (gfp=0x20/1) -
> Not out of memory"?
Ok, thanks.

My problem very different, but maybe have common source.

My problem can repeat on simple system (some kernel
thread and one bash... 1Gb RAM) and very quiet... :-(

Yes, SMP and e10x required :-)

Infinit loop?

>
> e1000 and SMP...
>
> /RogerL

PROBLEM: e100 or e1000 on SMP kernel freeze system (ipx+ncp)

[Andrew Shirrayev]

Test on last driver e1000 5.2.52 (from intel.com)...
And get same result....

NAPI enable / disable not important...

"ip link set eth0 down" may be replace Instead of "connect cable to NIC".
And get same result... (w/o message "NIC Link is Up").

On Tue, 25 May 2004, Andrew Shirrayev wrote:

>
> #If run this script on start-up,
> #kernel hung with high risk...
>
> echo disconnect eth0-cable and press Enter
> read
> insmod e100
> # or
> #insmod e1000
> ip link set eth0 up
> ipx_interface add -p eth0 802.2 111
> #run ncpmount 2 times.
> ncpmount -S srv -U usr -n /mnt
> ncpmount -S srv -U usr -n /mnt
> echo insert eth0-cable to NIC
> echo and wait 3-4 seconds until
> echo message \"NIC Link is Up\"
> echo and press Enter
> while ! read -t 1; do echo  -n .;done
> echo Don't hung? Great...
> ==============================
> if add "sleep 1" beetwen ncpmount
> risk of hung decrease, and <1%...
>
> ncpmount 2.2.0.18 have dalay approximately 1 second
> before print error message... Same result?
>
> ncpmount 2.2.4 print error message very fast...
> ==============================
> Result table:
> 1)
>   Kernel: 2.4.21 or 2.4.26 (SMP, acpi on or off)
>   system: 2xP4Xeon (HT on or off) on ServerWorks chipset
>   NIC: dual e1000
>   ncpfs+ipx: 2.2.4 (Debian)
>   Result: Freeze after message "NIC Link is Up"
>   Risk of hung: 99%
> 2)
>   Kernel: 2.4.21 (SMP, acpi off)
>   system: 2xP4Xeon (HT on or off) on ServerWorks chipset
>   NIC: dual e1000
>   ncpfs+ipx: 2.2.0.18 (Debian)
>   Result: Don't freeze after message "NIC Link is Up"
>   Risk of hung: less 10%
> 3)
>   Kernel: 2.4.26 (nonSMP, acpi off)
>   system: 2xP4Xeon (HT on or off) on ServerWorks chipset
>   NIC: dual e1000
>   ncpfs+ipx: 2.2.4 (Debian)
>   Result: Don't freeze after message "NIC Link is Up"
>   Risk of hung: less 1% :-)
> 4)
>   Kernel: 2.4.26 (SMP, acpi on or off)
>   system: P4 on intel chipset
>   NIC: e100
>   ncpfs+ipx: 2.2.0.18 (Debian)
>   Result: Some times freeze after message "NIC Link is Up"
>   Risk of hung: >10%
> 5)
>   Kernel: 2.4.26 (SMP, acpi on or off)
>   system: P4 on intel chipset
>   NIC: e1000
>   ncpfs+ipx: 2.2.0.18 (Debian)
>   Result: Some times freeze after message "NIC Link is Up"
>   Risk of hung: >10%
> 6)
>   Kernel: 2.4.26 (nonSMP, acpi off)
>   system: P4 on intel chipset
>   NIC: e1000 or e100
>   ncpfs+ipx: 2.2.0.18 (Debian)
>   Result: Don't freeze after message "NIC Link is Up"
>   Risk of hung: less 1%

PROBLEM: e100 or e1000 on SMP kernel freeze system (ipx+ncp)

[Andrew Shirrayev]

#If run this script on start-up,
#kernel hung with high risk...

echo disconnect eth0-cable and press Enter
read
insmod e100
# or
#insmod e1000
ip link set eth0 up
ipx_interface add -p eth0 802.2 111
#run ncpmount 2 times.
ncpmount -S srv -U usr -n /mnt
ncpmount -S srv -U usr -n /mnt
echo insert eth0-cable to NIC
echo and wait 3-4 seconds until
echo message \"NIC Link is Up\"
echo and press Enter
while ! read -t 1; do echo  -n .;done
echo Don't hung? Great...
==============================
if add "sleep 1" beetwen ncpmount
risk of hung decrease, and <1%...

ncpmount 2.2.0.18 have dalay approximately 1 second
before print error message... Same result?

ncpmount 2.2.4 print error message very fast...
==============================
Result table:
1)
  Kernel: 2.4.21 or 2.4.26 (SMP, acpi on or off)
  system: 2xP4Xeon (HT on or off) on ServerWorks chipset
  NIC: dual e1000
  ncpfs+ipx: 2.2.4 (Debian)
  Result: Freeze after message "NIC Link is Up"
  Risk of hung: 99%
2)
  Kernel: 2.4.21 (SMP, acpi off)
  system: 2xP4Xeon (HT on or off) on ServerWorks chipset
  NIC: dual e1000
  ncpfs+ipx: 2.2.0.18 (Debian)
  Result: Don't freeze after message "NIC Link is Up"
  Risk of hung: less 10%
3)
  Kernel: 2.4.26 (nonSMP, acpi off)
  system: 2xP4Xeon (HT on or off) on ServerWorks chipset
  NIC: dual e1000
  ncpfs+ipx: 2.2.4 (Debian)
  Result: Don't freeze after message "NIC Link is Up"
  Risk of hung: less 1% :-)
4)
  Kernel: 2.4.26 (SMP, acpi on or off)
  system: P4 on intel chipset
  NIC: e100
  ncpfs+ipx: 2.2.0.18 (Debian)
  Result: Some times freeze after message "NIC Link is Up"
  Risk of hung: >10%
5)
  Kernel: 2.4.26 (SMP, acpi on or off)
  system: P4 on intel chipset
  NIC: e1000
  ncpfs+ipx: 2.2.0.18 (Debian)
  Result: Some times freeze after message "NIC Link is Up"
  Risk of hung: >10%
6)
  Kernel: 2.4.26 (nonSMP, acpi off)
  system: P4 on intel chipset
  NIC: e1000 or e100
  ncpfs+ipx: 2.2.0.18 (Debian)
  Result: Don't freeze after message "NIC Link is Up"
  Risk of hung: less 1%