ibmvstgt/aio broken with 2.6.20-rc6 powerpc

ibmvstgt/aio broken with 2.6.20-rc6 powerpc

[Bastian Blank]

Hi folks

I'm not really sure if this is a ibmvstgt or a generic aio problem.

I tried ibmvstgt on my openpower and the kernel shows the following
warning several times:
| Jan 28 15:11:15 waldi kernel: Badness at arch/powerpc/mm/slb.c:95
| Jan 28 15:11:15 waldi kernel: Call Trace:
| Jan 28 15:11:15 waldi kernel: [C00000000282B750] [C00000000000EC94] .show_stack+0x68/0x1b0 (unreliable)
| Jan 28 15:11:15 waldi kernel: [C00000000282B7F0] [C00000000015ED64] .report_bug+0x94/0xe8
| Jan 28 15:11:15 waldi kernel: [C00000000282B880] [C0000000000228DC] .program_check_exception+0x140/0x54c
| Jan 28 15:11:15 waldi kernel: [C00000000282B920] [C000000000004774] program_check_common+0xf4/0x100
| Jan 28 15:11:15 waldi kernel: --- Exception: 700 at .slb_flush_and_rebolt+0xc/0xe0
| Jan 28 15:11:15 waldi kernel:     LR = .switch_slb+0xac/0x1a0
| Jan 28 15:11:15 waldi kernel: [C00000000282BC10] [FFFFFFFFFFFFFFFF] 0xffffffffffffffff (unreliable)
| Jan 28 15:11:15 waldi kernel: [C00000000282BCB0] [C0000000000E1750] .aio_kick_handler+0x128/0x278
| Jan 28 15:11:15 waldi kernel: [C00000000282BD60] [C000000000066380] .run_workqueue+0xec/0x1d4
| Jan 28 15:11:15 waldi kernel: [C00000000282BE00] [C000000000066F2C] .worker_thread+0x164/0x1bc
| Jan 28 15:11:15 waldi kernel: [C00000000282BEE0] [C00000000006B6E8] .kthread+0x120/0x170
| Jan 28 15:11:15 waldi kernel: [C00000000282BF90] [C000000000024204] .kernel_thread+0x4c/0x68

One possibility:
aio_kick_handler calls use_mm. use_mm calls switch_mm. It seems that
90aef12e6dd609e1ad7fb70044eedc78ca55ee5e changed that from activate_mm.

activate_mm on powerpc disables interupts and calls swtich_mm. So
switch_mm is now called with interrupts enabled and runs into the
WARN_ON.

Bastian

-- 
She won' go Warp 7, Cap'n!  The batteries are dead!

Re: ibmvstgt/aio broken with 2.6.20-rc6 powerpc

[FUJITA Tomonori]

From: Bastian Blank <bastian@waldi.eu.org>
Subject: ibmvstgt/aio broken with 2.6.20-rc6 powerpc
Date: Sun, 28 Jan 2007 19:52:58 +0100

> I'm not really sure if this is a ibmvstgt or a generic aio problem.

You use 2.6.20-rc6 with the aio-epoll-wait patch, right?

I think that this is due to the aio-epoll-wait patch because without
using ibmvstgt target driver, simple aio workload crashes 2.6.20-rc6
with the aio-epoll-wait patch on my POWER box. I cannot recall since
when the patch doesn't work on POWER.

If you are interested in only ibmvstgt target driver, use the
following patch. It uses epoll instead of AIO for event
notification. ibmvstgt target driver (and most of the kernel-space
target drivers) uses mmapped I/O and does not need AIO. The user-space
target drivers (like iSCSI) need the aio-epoll-wait patch though.


Index: usr/tgtd.c
===================================================================
--- usr/tgtd.c	(revision 771)
+++ usr/tgtd.c	(working copy)
@@ -202,9 +202,6 @@
 	int nevent, i, err;
 	struct epoll_event events[1024];
 	struct tgt_event *tev;
-	struct iocb iocbs[1], *iocb;
-	struct io_event aioevents[2048];
-	struct timespec timeout = {1, 0};
 
 	err = io_queue_init(2048, &ctx);
 	if (err) {
@@ -212,13 +209,8 @@
 		return;
 	}
 
-	iocb = iocbs;
-	io_prep_epoll_wait(iocb, ep_fd, events, ARRAY_SIZE(events), -1);
-	err = io_submit(ctx, 1, &iocb);
-
 retry:
-	nevent = io_getevents(ctx, 1, ARRAY_SIZE(aioevents), aioevents, &timeout);
-
+	nevent = epoll_wait(ep_fd, events, ARRAY_SIZE(events), -1);
 	if (nevent < 0) {
 		if (errno != EINTR) {
 			eprintf("%m\n");
@@ -226,18 +218,8 @@
 		}
 	} else if (nevent) {
 		for (i = 0; i < nevent; i++) {
-			if (iocb == aioevents[i].obj) {
-				int j;
-				for (j = 0; j < aioevents[i].res; j++) {
-					tev = (struct tgt_event *) events[j].data.ptr;
-					tev->handler(tev->fd, events[j].events, tev->data);
-				}
-
-				err = io_submit(ctx, 1, &iocb);
-			} else {
-				/* FIXME */
-				target_cmd_io_done(aioevents[i].data, 0);
-			}
+			tev = (struct tgt_event *) events[i].data.ptr;
+			tev->handler(tev->fd, events[i].events, tev->data);
 		}
 	} else
 		schedule();