[PATCH] blk-mq: quiesce queue while reallocating hctxs

[PATCH] blk-mq: quiesce queue while reallocating hctxs

[Yu Kuai]

From: Yu Kuai <yukuai3@huawei.com>

commit 8237c01f1696 ("blk-mq: use quiesced elevator switch when
reinitializing queues") add quiesce queue while switching elevator,
however, if old elevator is none, queue is still not quiesced. Hence
reallocating hctxs can concurrent with run queue. Fix it by also
quiesce queue in the beginning of __blk_mq_update_nr_hw_queues().

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
---
 block/blk-mq.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index d3494a796ba8..fb44ef0dff8a 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -4691,8 +4691,10 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
 	if (set->nr_maps == 1 && nr_hw_queues == set->nr_hw_queues)
 		return;
 
-	list_for_each_entry(q, &set->tag_list, tag_set_list)
+	list_for_each_entry(q, &set->tag_list, tag_set_list) {
 		blk_mq_freeze_queue(q);
+		blk_mq_quiesce_queue(q);
+	}
 	/*
 	 * Switch IO scheduler to 'none', cleaning up the data associated
 	 * with the previous scheduler. We will switch back once we are done
@@ -4741,8 +4743,10 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
 	list_for_each_entry(q, &set->tag_list, tag_set_list)
 		blk_mq_elv_switch_back(&head, q);
 
-	list_for_each_entry(q, &set->tag_list, tag_set_list)
+	list_for_each_entry(q, &set->tag_list, tag_set_list) {
+		blk_mq_unquiesce_queue(q);
 		blk_mq_unfreeze_queue(q);
+	}
 }
 
 void blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues)
-- 
2.31.1

Re: [PATCH] blk-mq: quiesce queue while reallocating hctxs

[Yu Kuai]

Hi,

friendly ping ...

Thanks,
Kuai

在 2023/02/21 17:24, Yu Kuai 写道:
> From: Yu Kuai <yukuai3@huawei.com>
> 
> commit 8237c01f1696 ("blk-mq: use quiesced elevator switch when
> reinitializing queues") add quiesce queue while switching elevator,
> however, if old elevator is none, queue is still not quiesced. Hence
> reallocating hctxs can concurrent with run queue. Fix it by also
> quiesce queue in the beginning of __blk_mq_update_nr_hw_queues().
> 
> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
> ---
>   block/blk-mq.c | 8 ++++++--
>   1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index d3494a796ba8..fb44ef0dff8a 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -4691,8 +4691,10 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
>   	if (set->nr_maps == 1 && nr_hw_queues == set->nr_hw_queues)
>   		return;
>   
> -	list_for_each_entry(q, &set->tag_list, tag_set_list)
> +	list_for_each_entry(q, &set->tag_list, tag_set_list) {
>   		blk_mq_freeze_queue(q);
> +		blk_mq_quiesce_queue(q);
> +	}
>   	/*
>   	 * Switch IO scheduler to 'none', cleaning up the data associated
>   	 * with the previous scheduler. We will switch back once we are done
> @@ -4741,8 +4743,10 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
>   	list_for_each_entry(q, &set->tag_list, tag_set_list)
>   		blk_mq_elv_switch_back(&head, q);
>   
> -	list_for_each_entry(q, &set->tag_list, tag_set_list)
> +	list_for_each_entry(q, &set->tag_list, tag_set_list) {
> +		blk_mq_unquiesce_queue(q);
>   		blk_mq_unfreeze_queue(q);
> +	}
>   }
>   
>   void blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues)
> 

Re: [PATCH] blk-mq: quiesce queue while reallocating hctxs

[Keith Busch]

On Tue, Feb 21, 2023 at 05:24:36PM +0800, Yu Kuai wrote:
> From: Yu Kuai <yukuai3@huawei.com>
> 
> commit 8237c01f1696 ("blk-mq: use quiesced elevator switch when
> reinitializing queues") add quiesce queue while switching elevator,
> however, if old elevator is none, queue is still not quiesced. Hence
> reallocating hctxs can concurrent with run queue. Fix it by also
> quiesce queue in the beginning of __blk_mq_update_nr_hw_queues().

Is this actually fixing anything? The quiesced elevator switch was to prevent
use-after-free from an elevator being torn down, but if you are not switching
elevators, then what resource does quiescing protect?

Re: [PATCH] blk-mq: quiesce queue while reallocating hctxs

[Yu Kuai]

Hi,

在 2023/03/03 9:24, Keith Busch 写道:
> On Tue, Feb 21, 2023 at 05:24:36PM +0800, Yu Kuai wrote:
>> From: Yu Kuai <yukuai3@huawei.com>
>>
>> commit 8237c01f1696 ("blk-mq: use quiesced elevator switch when
>> reinitializing queues") add quiesce queue while switching elevator,
>> however, if old elevator is none, queue is still not quiesced. Hence
>> reallocating hctxs can concurrent with run queue. Fix it by also
>> quiesce queue in the beginning of __blk_mq_update_nr_hw_queues().
> 
> Is this actually fixing anything? The quiesced elevator switch was to prevent
> use-after-free from an elevator being torn down, but if you are not switching
> elevators, then what resource does quiescing protect?
> .
> 

What I can think of is hctx itself. run queue can fetch a hctx that will
be exited, I did't found any real problems yet, but I don't think it
is not good to run work for a exited or reused hctx.

Thanks,
Kuai