2.6.5-rc1-mm2 very slow

2.6.5-rc1-mm2 very slow

[Holger Kiehl]

Hello

I am testing 2.6.5-rc1-mm2 and find it very slow when I do a bonnie
test, also the system itself feels very sluggish. Looking at dmesg
I get the following:

   Badness in elv_remove_request at drivers/block/elevator.c:249
   Call Trace:
    [<c028b28f>] elv_remove_request+0x8d/0x8f
    [<c02b6a4e>] scsi_request_fn+0x289/0x333
    [<c028b174>] elv_next_request+0x3d/0xcb
    [<c028c8da>] generic_unplug_device+0x43/0x45
    [<c02f1bfc>] md_unplug_mddev+0x32/0x34
    [<c02f07e4>] make_request+0x22/0x29f
    [<c01a5523>] ext3_get_branch+0x4d/0xca
    [<c028ddc7>] generic_make_request+0xf2/0x165
    [<c0138a41>] mempool_alloc+0x63/0x114
    [<c011d42b>] autoremove_wake_function+0x0/0x38
    [<c028dea6>] submit_bio+0x6c/0x11b
    [<c0154c42>] bio_alloc+0xba/0x182
    [<c015355e>] block_read_full_page+0x19d/0x2a2
    [<c01a5d77>] ext3_get_block+0x0/0x9e
    [<c0154cb3>] bio_alloc+0x12b/0x182
    [<c01a5dcb>] ext3_get_block+0x54/0x9e
    [<c016d08e>] mpage_alloc+0x25/0x8a
    [<c016d3b8>] do_mpage_readpage+0x24b/0x314
    [<c0246894>] radix_tree_node_alloc+0x10/0x4d
    [<c0246b00>] radix_tree_insert+0xdd/0xfd
    [<c0135dfa>] add_to_page_cache+0x5a/0xd5
    [<c016d59b>] mpage_readpages+0x11a/0x15e
    [<c01a5d77>] ext3_get_block+0x0/0x9e
    [<c01a6a59>] ext3_readpages+0x0/0x15
    [<c013bb8d>] read_pages+0x136/0x13f
    [<c01a5d77>] ext3_get_block+0x0/0x9e
    [<c0139a98>] buffered_rmqueue+0xf6/0x1d3
    [<c0139c20>] __alloc_pages+0xab/0x2e0
    [<c013be48>] do_page_cache_readahead+0xfc/0x165
    [<c013c039>] page_cache_readahead+0x188/0x1d7
    [<c01364f0>] do_generic_mapping_read+0xc9/0x38e
    [<c01367b5>] file_read_actor+0x0/0xca
    [<c0136a4c>] __generic_file_aio_read+0x1cd/0x1ff
    [<c01367b5>] file_read_actor+0x0/0xca
    [<c0136ac5>] generic_file_aio_read+0x47/0x61
    [<c015031e>] do_sync_read+0x6f/0xa4
    [<c011a214>] load_balance+0x3c/0x156
    [<c01503fc>] vfs_read+0xa9/0xf5
    [<c0150619>] sys_read+0x38/0x59
    [<c034b8a6>] sysenter_past_esp+0x43/0x65

The system is a dual Xeon with HT enabled and 2GB memory. Bonnie is
running on a software raid10 (six SCSI disks) with ext3.

It is compiled with CONFIG_SCHED_SMT and CONFIG_HIGHMEM4G. CONFIG_PREEMPT
is not set. I can also post the full .config if necessary.

I get more of these badness messages when I boot.

Under 2.6.3 there are no problems. Also have tried 2.6.4-mm2 but that would
oops during boot (when starting the SW raid).

Holger

Re: 2.6.5-rc1-mm2 very slow

[Jens Axboe]

On Fri, Mar 19 2004, Holger Kiehl wrote:
> Hello
> 
> I am testing 2.6.5-rc1-mm2 and find it very slow when I do a bonnie
> test, also the system itself feels very sluggish. Looking at dmesg
> I get the following:
> 
>    Badness in elv_remove_request at drivers/block/elevator.c:249
>    Call Trace:
>     [<c028b28f>] elv_remove_request+0x8d/0x8f
>     [<c02b6a4e>] scsi_request_fn+0x289/0x333
>     [<c028b174>] elv_next_request+0x3d/0xcb
>     [<c028c8da>] generic_unplug_device+0x43/0x45

Does it still complain with this patch?

--- drivers/block/elevator.c~	2004-03-18 10:56:34.494431670 +0100
+++ drivers/block/elevator.c	2004-03-18 11:05:00.115063157 +0100
@@ -153,7 +153,7 @@
 	 * it already went through dequeue, we need to decrement the
 	 * in_flight count again
 	 */
-	if (blk_rq_started(rq)) {
+	if (blk_account_rq(rq)) {
 		WARN_ON(q->in_flight == 0);
 		q->in_flight--;
 	}
@@ -244,7 +244,7 @@
 	 * driver has seen (REQ_STARTED set), to avoid false accounting
 	 * for request-request merges
 	 */
-	if (blk_rq_started(rq)) {
+	if (blk_account_rq(rq)) {
 		q->in_flight++;
 		WARN_ON(q->in_flight > 2 * q->nr_requests);
 	}
@@ -341,7 +341,7 @@
 	/*
 	 * request is released from the driver, io must be done
 	 */
-	if (blk_rq_started(rq)) {
+	if (blk_account_rq(rq)) {
 		WARN_ON(q->in_flight == 0);
 		q->in_flight--;
 	}
--- include/linux/blkdev.h~	2004-03-18 11:03:59.431584757 +0100
+++ include/linux/blkdev.h	2004-03-18 11:05:16.980250506 +0100
@@ -381,6 +381,8 @@
 #define blk_noretry_request(rq)	((rq)->flags & REQ_FAILFAST)
 #define blk_rq_started(rq)	((rq)->flags & REQ_STARTED)
 
+#define blk_account_rq(rq)	(blk_rq_started(rq) && blk_fs_request(rq))
+
 #define blk_pm_suspend_request(rq)	((rq)->flags & REQ_PM_SUSPEND)
 #define blk_pm_resume_request(rq)	((rq)->flags & REQ_PM_RESUME)
 #define blk_pm_request(rq)	\


-- 
Jens Axboe

Re: 2.6.5-rc1-mm2 very slow

[Holger Kiehl]

On Fri, 19 Mar 2004, Jens Axboe wrote:

> On Fri, Mar 19 2004, Holger Kiehl wrote:
> > Hello
> > 
> > I am testing 2.6.5-rc1-mm2 and find it very slow when I do a bonnie
> > test, also the system itself feels very sluggish. Looking at dmesg
> > I get the following:
> > 
> >    Badness in elv_remove_request at drivers/block/elevator.c:249
> >    Call Trace:
> >     [<c028b28f>] elv_remove_request+0x8d/0x8f
> >     [<c02b6a4e>] scsi_request_fn+0x289/0x333
> >     [<c028b174>] elv_next_request+0x3d/0xcb
> >     [<c028c8da>] generic_unplug_device+0x43/0x45
> 
> Does it still complain with this patch?
> 
No, this also fixes the slowness. Performance is back to normal.

That was the quickest patch I have ever received :)

Many thanks!

Holger