"Trying to free nonexistent swap-page" error message.

"Trying to free nonexistent swap-page" error message.

[Johan Simon Seland]

[-- Attachment #1: Type: text/plain, Size: 1540 bytes --]

Hello,

I have searched the archives for this error message before, but no one
seems to have given a good answer. (Though the question has been
posted before.) I am not sure if this is a kernel problem, a hardware
problem or a Oracle problem. (Or a combination of them.)

One one of our Linux Oracle servers the following messages has started
to appear : 

Jun 29 07:16:32 blanco kernel: swap_free: Trying to free nonexistent swap-page
Jun 29 07:16:32 blanco kernel: swap_free: Trying to free nonexistent swap-page

They seem to always come in pairs, and usually with about three hours
between them. 

The database had to be restored from backup because of massive table
corruption recently, but these messages also appeared before we had to
restore it. (But we believe they might have caused the corruption.)

I also find some of these:

Jun 29 06:25:01 blanco kernel: EXT2-fs error (device sd(8,10)): ext2_readdir: bad entry in directory #172258: rec_len %% 4 != 0 - offset=192, inode=812610409, rec_len=11833, name_len=115
Jun 29 06:25:32 blanco kernel: EXT2-fs error (device sd(8,10)): ext2_readdir: bad entry in directory #172258: rec_len %% 4 != 0 - offset=192, inode=812610409, rec_len=11833, name_len=115

Machine is a 2x933MhZ P3 with 2GB of memory. Kernel version is now
2.2.19, but the same problem appeared with 2.2.18 as well. The
database is in moderate to heavy use 24/7 and with a lot (~ 500 - 3000)
processes during business hours.

The machine has only 128MB of swap, is this to little since it has a
full 2GB of memory?


[-- Attachment #2: dmesg output from machine --]
[-- Type: application/octet-stream, Size: 7675 bytes --]

emory: 2009512k/2031616k available (1196k kernel code, 420k reserved, 20416k data, 72k init)
Dentry hash table entries: 262144 (order 9, 2048k)
Buffer cache hash table entries: 524288 (order 9, 2048k)
Page cache hash table entries: 524288 (order 9, 2048k)
CPU serial number disabled.
256K L2 cache (8 way)
CPU: L2 Cache: 256K
Checking 386/387 coupling... OK, FPU using exception 16 error reporting.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
CPU serial number disabled.
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
256K L2 cache (8 way)
CPU: L2 Cache: 256K
per-CPU timeslice cutoff: 49.99 usecs.
CPU0: Intel Pentium III (Coppermine) stepping 03
calibrating APIC timer ... 
..... CPU clock speed is 935.4725 MHz.
..... system bus clock speed is 133.6387 MHz.
Booting processor 1 eip 2000
Calibrating delay loop... 1867.77 BogoMIPS
CPU serial number disabled.
Intel machine check reporting enabled on CPU#1.
256K L2 cache (8 way)
CPU: L2 Cache: 256K
OK.
CPU1: Intel Pentium III (Coppermine) stepping 03
Total of 2 processors activated (3728.99 BogoMIPS).
enabling symmetric IO mode... ...done.
ENABLING IO-APIC IRQs
init IO_APIC IRQs
 IO-APIC (apicid-pin) 2-0, 2-5, 2-9, 2-10, 2-11, 2-20, 2-21, 2-22, 2-23 not connected.
number of MP IRQ sources: 21.
number of IO-APIC #2 registers: 24.
testing the IO APIC.......................

IO APIC #2......
.... register #00: 02000000
.......    : physical APIC id: 02
.... register #01: 00170011
.......     : max redirection entries: 0017
.......     : IO APIC version: 0011
.... register #02: 00000000
.......     : arbitration: 00
.... IRQ redirection table:
 NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
 00 000 00  1    0    0   0   0    0    0    00
 01 000 00  0    0    0   0   0    1    1    59
 02 0FF 0F  0    0    0   0   0    1    1    51
 03 000 00  0    0    0   0   0    1    1    61
 04 000 00  0    0    0   0   0    1    1    69
 05 000 00  1    0    0   0   0    0    0    00
 06 000 00  0    0    0   0   0    1    1    71
 07 000 00  0    0    0   0   0    1    1    79
 08 000 00  0    0    0   0   0    1    1    81
 09 000 00  1    0    0   0   0    0    0    00
 0a 000 00  1    0    0   0   0    0    0    00
 0b 000 00  1    0    0   0   0    0    0    00
 0c 000 00  0    0    0   0   0    1    1    89
 0d 000 00  1    0    0   0   0    0    0    00
 0e 000 00  0    0    0   0   0    1    1    91
 0f 000 00  0    0    0   0   0    1    1    99
 10 0FF 0F  1    1    0   1   0    1    1    A1
 11 0FF 0F  1    1    0   1   0    1    1    A9
 12 0FF 0F  1    1    0   1   0    1    1    B1
 13 0FF 0F  1    1    0   1   0    1    1    B9
 14 000 00  1    0    0   0   0    0    0    00
 15 000 00  1    0    0   0   0    0    0    00
 16 000 00  1    0    0   0   0    0    0    00
 17 000 00  1    0    0   0   0    0    0    00
IRQ to pin mappings:
IRQ0 -> 2
IRQ1 -> 1
IRQ3 -> 3
IRQ4 -> 4
IRQ6 -> 6
IRQ7 -> 7
IRQ8 -> 8
IRQ12 -> 12
IRQ13 -> 13
IRQ14 -> 14
IRQ15 -> 15
IRQ16 -> 16
IRQ17 -> 17
IRQ18 -> 18
IRQ19 -> 19
.................................... done.
checking TSC synchronization across CPUs: passed.
PCI: PCI BIOS revision 2.10 entry at 0xfb2d0
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI->APIC IRQ transform: (B0,I7,P3) -> 19
PCI->APIC IRQ transform: (B0,I7,P3) -> 19
PCI->APIC IRQ transform: (B0,I7,P2) -> 18
PCI->APIC IRQ transform: (B0,I12,P0) -> 18
PCI->APIC IRQ transform: (B0,I15,P0) -> 17
PCI->APIC IRQ transform: (B0,I15,P1) -> 18
PCI->APIC IRQ transform: (B0,I17,P0) -> 19
PCI->APIC IRQ transform: (B1,I0,P0) -> 16
Linux NET4.0 for Linux 2.2
Based upon Swansea University Computer Society NET3.039
NET4: Unix domain sockets 1.0 for Linux NET4.0.
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
TCP: Hash tables configured (ehash 524288 bhash 65536)
Starting kswapd v 1.5 
Detected PS/2 Mouse Port.
Serial driver version 4.27 with no serial options enabled
ttyS00 at 0x03f8 (irq = 4) is a 16550A
ttyS01 at 0x02f8 (irq = 3) is a 16550A
pty: 256 Unix98 ptys configured
VP_IDE: IDE controller on PCI bus 00 dev 39
VP_IDE: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xb000-0xb007, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xb008-0xb00f, BIOS settings: hdc:pio, hdd:pio
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
(scsi0) <Adaptec AIC-7899 Ultra 160/m SCSI host adapter> found at PCI 0/15/0
(scsi0) Wide Channel A, SCSI ID=7, 32/255 SCBs
(scsi0) Downloading sequencer code... 396 instructions downloaded
(scsi1) <Adaptec AIC-7899 Ultra 160/m SCSI host adapter> found at PCI 0/15/1
(scsi1) Wide Channel B, SCSI ID=7, 32/255 SCBs
(scsi1) Downloading sequencer code... 396 instructions downloaded
scsi0 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.33/3.2.4
       <Adaptec AIC-7899 Ultra 160/m SCSI host adapter>
scsi1 : Adaptec AHA274x/284x/294x (EISA/VLB/PCI-Fast SCSI) 5.1.33/3.2.4
       <Adaptec AIC-7899 Ultra 160/m SCSI host adapter>
scsi : 2 hosts.
  Vendor: IBM       Model: DDYS-T09170N      Rev: S96H
  Type:   Direct-Access                      ANSI SCSI revision: 03
Detected scsi disk sda at scsi0, channel 0, id 6, lun 0
(scsi0:0:6:1) Synchronous at 160.0 Mbyte/sec, offset 63.
  Vendor: IBM       Model: DDYS-T09170N      Rev: S96H
  Type:   Direct-Access                      ANSI SCSI revision: 03
Detected scsi disk sdb at scsi0, channel 0, id 10, lun 0
(scsi0:0:10:1) Synchronous at 160.0 Mbyte/sec, offset 63.
  Vendor: IBM       Model: DDYS-T09170N      Rev: S96H
  Type:   Direct-Access                      ANSI SCSI revision: 03
Detected scsi disk sdc at scsi1, channel 0, id 6, lun 0
(scsi1:0:6:1) Synchronous at 160.0 Mbyte/sec, offset 63.
  Vendor: IBM       Model: DDYS-T09170N      Rev: S96H
  Type:   Direct-Access                      ANSI SCSI revision: 03
Detected scsi disk sdd at scsi1, channel 0, id 10, lun 0
(scsi1:0:10:1) Synchronous at 160.0 Mbyte/sec, offset 63.
scsi : detected 4 SCSI disks total.
SCSI device sda: hdwr sector= 512 bytes. Sectors= 17916240 [8748 MB] [8.7 GB]
SCSI device sdb: hdwr sector= 512 bytes. Sectors= 17916240 [8748 MB] [8.7 GB]
SCSI device sdc: hdwr sector= 512 bytes. Sectors= 17916240 [8748 MB] [8.7 GB]
SCSI device sdd: hdwr sector= 512 bytes. Sectors= 17916240 [8748 MB] [8.7 GB]
3c59x.c 18Feb01 Donald Becker and others http://www.scyld.com/network/vortex.html
eth0: 3Com 3c905C Tornado at 0xe800,  00:01:02:df:f9:6a, IRQ 19
  8K byte-wide RAM 5:3 Rx:Tx split, autoselect/Autonegotiate interface.
  MII transceiver found at address 24, status 782d.
  Enabling bus-master transmits and whole-frame receives.
Partition check:
 sda: sda1 < sda5 sda6 sda7 sda8 sda9 sda10 sda11 >
 sdb: sdb1
 sdc: sdc1
 sdd: sdd1
VFS: Mounted root (ext2 filesystem) readonly.
Freeing unused kernel memory: 72k freed
Adding Swap: 120452k swap-space (priority -1)
eth0: Initial media type Autonegotiate.
eth0: MII #24 status 782d, link partner capability 41e1, setting full-duplex.
nfs warning: mount version older than kernel
nfs warning: mount version older than kernel
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
EXT2-fs error (device sd(8,10)): ext2_readdir: bad entry in directory #172258: rec_len % 4 != 0 - offset=192, inode=812610409, rec_len=11833, name_len=115
EXT2-fs error (device sd(8,10)): ext2_readdir: bad entry in directory #172258: rec_len % 4 != 0 - offset=192, inode=812610409, rec_len=11833, name_len=115
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page


[-- Attachment #3: Type: text/plain, Size: 442 bytes --]


The kernel is stock 2.2.19 with the following patch applied:

--- include/linux/tasks.h~	Wed Jan 17 14:45:54 2001
+++ include/linux/tasks.h	Wed Jan 17 14:46:39 2001
@@ -11,7 +11,7 @@
 #define NR_CPUS 1
 #endif
 
-#define NR_TASKS	512	/* On x86 Max about 4000 */
+#define NR_TASKS	4000	/* On x86 Max about 4000 */
 
 #define MAX_TASKS_PER_USER (NR_TASKS/2)
 #define MIN_TASKS_LEFT_FOR_ROOT 4

--
Regards
Johan Seland
Programmer
Net Fonds ASA

Re: "Trying to free nonexistent swap-page" error message.

[Andreas Dilger]

Johan Seland
> One one of our Linux Oracle servers the following messages has started
> to appear : 
> 
> Jun 29 07:16:32 blanco kernel: swap_free: Trying to free nonexistent swap-page
> Jun 29 07:16:32 blanco kernel: swap_free: Trying to free nonexistent swap-page
> 
> I also find some of these:
> 
> Jun 29 06:25:01 blanco kernel: EXT2-fs error (device sd(8,10)): ext2_readdir: bad entry in directory #172258: rec_len %% 4 != 0 - offset=192, inode=812610409, rec_len=11833, name_len=115
> Jun 29 06:25:32 blanco kernel: EXT2-fs error (device sd(8,10)): ext2_readdir: bad entry in directory #172258: rec_len %% 4 != 0 - offset=192, inode=812610409, rec_len=11833, name_len=115
> 
> Machine is a 2x933MhZ P3 with 2GB of memory. Kernel version is now
> 2.2.19, but the same problem appeared with 2.2.18 as well.

My first guess would be some sort of hardware/software problem with your
SCSI controller, cables, disk, etc.  I'm not sure about the swap problem,
but the ext2 problems are caused by corruption of the disk or memory.

It is not just a single-bit error either, because rec_len % 4 != 0 AND it
is larger than a page size, so the value is totally bogus, as is the inode
number.  Interestingly, converting the above ext2 numbers into ascii gives:

0x69 0x73 0x6f 0x30 0x39 0x2e 0x73 => iso09.s

(in the order they are layed out in ext2_dir_entry_2).  Coincidence or bug?
I would suggest a full fsck for the filesystem, as it is likely that there
are other problems.

Now when you say "servers" do you mean you have the same problem on
multiple machines?  Are they identical, or different?

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: "Trying to free nonexistent swap-page" error message.

[Troy Benjegerdes]

On Fri, Jun 29, 2001 at 03:02:03AM -0600, Andreas Dilger wrote:
> Johan Seland
> > One one of our Linux Oracle servers the following messages has started
> > to appear : 
> > 
> > Jun 29 07:16:32 blanco kernel: swap_free: Trying to free nonexistent swap-page
> > Jun 29 07:16:32 blanco kernel: swap_free: Trying to free nonexistent swap-page
> > 
> > I also find some of these:
> > 
> > Jun 29 06:25:01 blanco kernel: EXT2-fs error (device sd(8,10)): ext2_readdir: bad entry in directory #172258: rec_len %% 4 != 0 - offset=192, inode=812610409, rec_len=11833, name_len=115
> > Jun 29 06:25:32 blanco kernel: EXT2-fs error (device sd(8,10)): ext2_readdir: bad entry in directory #172258: rec_len %% 4 != 0 - offset=192, inode=812610409, rec_len=11833, name_len=115
> > 
> > Machine is a 2x933MhZ P3 with 2GB of memory. Kernel version is now
> > 2.2.19, but the same problem appeared with 2.2.18 as well.
> 
> My first guess would be some sort of hardware/software problem with your
> SCSI controller, cables, disk, etc.  I'm not sure about the swap problem,
> but the ext2 problems are caused by corruption of the disk or memory.

My first guess would be hardware also, except in this case I've seen
similiar things on three different dual processor G4 systems running 2.2,
and they work fine with 2.4.

Does Oracle for Linux us pthreads or the 'clone()' system call?

Can you try running the included pthreads program on an 2.2.19 SMP system
(but make it's idle, since if this is a genric 2.2 SMP bug it will
probably crash the system)

Compile with the following command:
gcc -o pt pthread-test.c -lpthread 

Run it repeatedly:
I=0; while [ $? -eq 0 ] ; let I=I+1; do ./pt ; done ; echo $I

on 2.2.19 on a mac dual G4, the pthreads program will sometimes get 
segfaults and illegal instructions, and if I run it long enough, I will 
eventually get the following in dmesg:

swap_free: Trying to free nonexistent swap-page
swap_free: offset exceeds max
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: offset exceeds max
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: offset exceeds max
swap_free: offset exceeds max
swap_free: offset exceeds max
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: Trying to free nonexistent swap-page
swap_free: offset exceeds max
swap_free: offset exceeds max
swap_free: offset exceeds max

If I keep running it, I will eventually wind up with a kernel panic on an 
illegal instruction. Something is corrupting memory, and in my case, the 
kernel panics are caused by a '0x00000008' being written over a random 
location in the kernel code.


#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
 
#define NTH 8
 
void
thread(void *arg)
{
        printf("THREAD: pid %d\n", getpid());
 
        return;
}

int
main(int argc, char **argv)
{
        int i;
        pthread_t t[NTH];
 
        for (i=0; i<NTH; i++)
           pthread_create(t+i, NULL, thread, NULL);
 
        for (i=0; i<NTH; i++)
           pthread_join(t[i], NULL);
 
        return 0;
}



-- 
Troy Benjegerdes | master of mispeeling | 'da hozer' |  hozer@drgw.net
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's 
why I draw cartoons. It's my life." -- Charles Shulz

Re: "Trying to free nonexistent swap-page" error message.

[Johan Simon Seland]

Troy Benjegerdes <hozer@drgw.net> writes:

> My first guess would be hardware also, except in this case I've seen
> similiar things on three different dual processor G4 systems running 2.2,
> and they work fine with 2.4.

I have replaced all the memory with fresh 4x512MB REGISTRED ECC RAM,
and it has been running stable for 8 days now. I also replaced the
SCSI cables. 2.4 is not an option.

> Does Oracle for Linux us pthreads or the 'clone()' system call?

I am not really sure. How do I find out?

> Can you try running the included pthreads program on an 2.2.19 SMP system
> (but make it's idle, since if this is a genric 2.2 SMP bug it will
> probably crash the system)

Sorry, I am not going to risk bringing down the database with your
program. Its to critical for our business. (The database contains all
stock quotes for Oslo, Stockholm, Frankfurt, NYSE, AMEX and NASDAQ
stock exchanges and they are updated 24/7. I can only bring it down in
weekends, and only if it really important.)

However I have access to a few other dual boxen. (2x550, 2x350, 2x166
and a dual SparcStation 20). I can run your program on them with stock
2.2.19 SMP if you want me to.

-- 
Med vennlig hilsen 
Johan Seland
for Net Fonds ASA

Re: "Trying to free nonexistent swap-page" error message.

[Troy Benjegerdes]

On Wed, Jul 11, 2001 at 12:58:12PM +0200, Johan Simon Seland wrote:
> Troy Benjegerdes <hozer@drgw.net> writes:
> 
> > My first guess would be hardware also, except in this case I've seen
> > similiar things on three different dual processor G4 systems running 2.2,
> > and they work fine with 2.4.
> 
> I have replaced all the memory with fresh 4x512MB REGISTRED ECC RAM,
> and it has been running stable for 8 days now. I also replaced the
> SCSI cables. 2.4 is not an option.

If you have several identical machines running oracle, and the machine is 
stable after changing the RAM & scsi cables, my guess it was a RAM 
or cable problem. 

> > Does Oracle for Linux us pthreads or the 'clone()' system call?
> 
> I am not really sure. How do I find out?
> 
> > Can you try running the included pthreads program on an 2.2.19 SMP system
> > (but make it's idle, since if this is a genric 2.2 SMP bug it will
> > probably crash the system)
> 
> Sorry, I am not going to risk bringing down the database with your
> program. Its to critical for our business. (The database contains all
> stock quotes for Oslo, Stockholm, Frankfurt, NYSE, AMEX and NASDAQ
> stock exchanges and they are updated 24/7. I can only bring it down in
> weekends, and only if it really important.)
> 
> However I have access to a few other dual boxen. (2x550, 2x350, 2x166
> and a dual SparcStation 20). I can run your program on them with stock
> 2.2.19 SMP if you want me to.

I don't think it will show anything.. I've manged to find a dual PPro to 
run on and didn't see any problems. I think it's a PPC specific problem.

Thanks though.

-- 
Troy Benjegerdes | master of mispeeling | 'da hozer' |  hozer@drgw.net
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's 
why I draw cartoons. It's my life." -- Charles Shulz