[PATCH] blktrace: Fix uaf in blk_trace access after removing by sysfs

[PATCH] blktrace: Fix uaf in blk_trace access after removing by sysfs

[Zhihao Cheng]

There is an use-after-free problem triggered by following process:

      P1(sda)				P2(sdb)
			echo 0 > /sys/block/sdb/trace/enable
			  blk_trace_remove_queue
			    synchronize_rcu
			    blk_trace_free
			      relay_close
rcu_read_lock
__blk_add_trace
  trace_note_tsk
  (Iterate running_trace_list)
			        relay_close_buf
				  relay_destroy_buf
				    kfree(buf)
    trace_note(sdb's bt)
      relay_reserve
        buf->offset <- nullptr deference (use-after-free) !!!
rcu_read_unlock

[  502.714379] BUG: kernel NULL pointer dereference, address:
0000000000000010
[  502.715260] #PF: supervisor read access in kernel mode
[  502.715903] #PF: error_code(0x0000) - not-present page
[  502.716546] PGD 103984067 P4D 103984067 PUD 17592b067 PMD 0
[  502.717252] Oops: 0000 [#1] SMP
[  502.720308] RIP: 0010:trace_note.isra.0+0x86/0x360
[  502.732872] Call Trace:
[  502.733193]  __blk_add_trace.cold+0x137/0x1a3
[  502.733734]  blk_add_trace_rq+0x7b/0xd0
[  502.734207]  blk_add_trace_rq_issue+0x54/0xa0
[  502.734755]  blk_mq_start_request+0xde/0x1b0
[  502.735287]  scsi_queue_rq+0x528/0x1140
...
[  502.742704]  sg_new_write.isra.0+0x16e/0x3e0
[  502.747501]  sg_ioctl+0x466/0x1100

Reproduce method:
  ioctl(/dev/sda, BLKTRACESETUP, blk_user_trace_setup[buf_size=127])
  ioctl(/dev/sda, BLKTRACESTART)
  ioctl(/dev/sdb, BLKTRACESETUP, blk_user_trace_setup[buf_size=127])
  ioctl(/dev/sdb, BLKTRACESTART)

  echo 0 > /sys/block/sdb/trace/enable &
  // Add delay(mdelay/msleep) before kernel enters blk_trace_free()

  ioctl$SG_IO(/dev/sda, SG_IO, ...)
  // Enters trace_note_tsk() after blk_trace_free() returned
  // Use mdelay in rcu region rather than msleep(which may schedule out)

Don't remove blk_trace by sysfs when blk_trace is at Blktrace_running
state, just like function __blk_trace_remove() does. The state change
process and blk_trace_remove_queue() are protected and by mutex lock
'q->debugfs_mutex', so the sequence of stopping blk_trace first and
then removing it will be ensured.

Fixes: c71a896154119f ("blktrace: add ftrace plugin")
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com>
---
 kernel/trace/blktrace.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index c221e4c3f625..7fe29bb9746f 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -1821,8 +1821,17 @@ static ssize_t sysfs_blk_trace_attr_store(struct device *dev,
 		}
 		if (value)
 			ret = blk_trace_setup_queue(q, bdev);
-		else
-			ret = blk_trace_remove_queue(q);
+		else {
+			/*
+			 * Don't remove blk_trace under running state, in
+			 * case triggering use-after-free in function
+			 * __blk_add_trace().
+			 */
+			if (bt->trace_state != Blktrace_running)
+				ret = blk_trace_remove_queue(q);
+			else
+				ret = -EBUSY;
+		}
 		goto out_unlock_bdev;
 	}
 
-- 
2.31.1

Re: [PATCH] blktrace: Fix uaf in blk_trace access after removing by sysfs

[Zhihao Cheng]

在 2021/9/10 17:21, Zhihao Cheng 写道:
friendly ping
> There is an use-after-free problem triggered by following process:
>
>        P1(sda)				P2(sdb)
> 			echo 0 > /sys/block/sdb/trace/enable
> 			  blk_trace_remove_queue
> 			    synchronize_rcu
> 			    blk_trace_free
> 			      relay_close
> rcu_read_lock
> __blk_add_trace
>    trace_note_tsk
>    (Iterate running_trace_list)
> 			        relay_close_buf
> 				  relay_destroy_buf
> 				    kfree(buf)
>      trace_note(sdb's bt)
>        relay_reserve
>          buf->offset <- nullptr deference (use-after-free) !!!
> rcu_read_unlock
>
> [  502.714379] BUG: kernel NULL pointer dereference, address:
> 0000000000000010
> [  502.715260] #PF: supervisor read access in kernel mode
> [  502.715903] #PF: error_code(0x0000) - not-present page
> [  502.716546] PGD 103984067 P4D 103984067 PUD 17592b067 PMD 0
> [  502.717252] Oops: 0000 [#1] SMP
> [  502.720308] RIP: 0010:trace_note.isra.0+0x86/0x360
> [  502.732872] Call Trace:
> [  502.733193]  __blk_add_trace.cold+0x137/0x1a3
> [  502.733734]  blk_add_trace_rq+0x7b/0xd0
> [  502.734207]  blk_add_trace_rq_issue+0x54/0xa0
> [  502.734755]  blk_mq_start_request+0xde/0x1b0
> [  502.735287]  scsi_queue_rq+0x528/0x1140
> ...
> [  502.742704]  sg_new_write.isra.0+0x16e/0x3e0
> [  502.747501]  sg_ioctl+0x466/0x1100
>
> Reproduce method:
>    ioctl(/dev/sda, BLKTRACESETUP, blk_user_trace_setup[buf_size=127])
>    ioctl(/dev/sda, BLKTRACESTART)
>    ioctl(/dev/sdb, BLKTRACESETUP, blk_user_trace_setup[buf_size=127])
>    ioctl(/dev/sdb, BLKTRACESTART)
>
>    echo 0 > /sys/block/sdb/trace/enable &
>    // Add delay(mdelay/msleep) before kernel enters blk_trace_free()
>
>    ioctl$SG_IO(/dev/sda, SG_IO, ...)
>    // Enters trace_note_tsk() after blk_trace_free() returned
>    // Use mdelay in rcu region rather than msleep(which may schedule out)
>
> Don't remove blk_trace by sysfs when blk_trace is at Blktrace_running
> state, just like function __blk_trace_remove() does. The state change
> process and blk_trace_remove_queue() are protected and by mutex lock
> 'q->debugfs_mutex', so the sequence of stopping blk_trace first and
> then removing it will be ensured.
>
> Fixes: c71a896154119f ("blktrace: add ftrace plugin")
> Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com>
> ---
>   kernel/trace/blktrace.c | 13 +++++++++++--
>   1 file changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
> index c221e4c3f625..7fe29bb9746f 100644
> --- a/kernel/trace/blktrace.c
> +++ b/kernel/trace/blktrace.c
> @@ -1821,8 +1821,17 @@ static ssize_t sysfs_blk_trace_attr_store(struct device *dev,
>   		}
>   		if (value)
>   			ret = blk_trace_setup_queue(q, bdev);
> -		else
> -			ret = blk_trace_remove_queue(q);
> +		else {
> +			/*
> +			 * Don't remove blk_trace under running state, in
> +			 * case triggering use-after-free in function
> +			 * __blk_add_trace().
> +			 */
> +			if (bt->trace_state != Blktrace_running)
> +				ret = blk_trace_remove_queue(q);
> +			else
> +				ret = -EBUSY;
> +		}
>   		goto out_unlock_bdev;
>   	}
>   

Re: [PATCH] blktrace: Fix uaf in blk_trace access after removing by sysfs

[Christoph Hellwig]

On Fri, Sep 10, 2021 at 05:21:20PM +0800, Zhihao Cheng wrote:
> +		else {
> +			/*
> +			 * Don't remove blk_trace under running state, in
> +			 * case triggering use-after-free in function
> +			 * __blk_add_trace().
> +			 */
> +			if (bt->trace_state != Blktrace_running)
> +				ret = blk_trace_remove_queue(q);
> +			else
> +				ret = -EBUSY;
> +		}
>  		goto out_unlock_bdev;

So who is going to eventually call blk_trace_free in this case?

Also Having the check in blk_trace_remove_queue would seem a little
cleaner.

Re: [PATCH] blktrace: Fix uaf in blk_trace access after removing by sysfs

[Zhihao Cheng]

在 2021/9/22 13:21, Christoph Hellwig 写道:
Hi Christoph,
> On Fri, Sep 10, 2021 at 05:21:20PM +0800, Zhihao Cheng wrote:
>> +		else {
>> +			/*
>> +			 * Don't remove blk_trace under running state, in
>> +			 * case triggering use-after-free in function
>> +			 * __blk_add_trace().
>> +			 */
>> +			if (bt->trace_state != Blktrace_running)
>> +				ret = blk_trace_remove_queue(q);
>> +			else
>> +				ret = -EBUSY;
>> +		}
>>   		goto out_unlock_bdev;
> So who is going to eventually call blk_trace_free in this case?
Agree. How about removing blk_trace from running_list and stopping it 
before calling blk_trace_free()?
>
> Also Having the check in blk_trace_remove_queue would seem a little
> cleaner.
Oh right, will move the check in v2.