[PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[Yihang Li]

This series [1] reducing the kmalloc() minimum alignment on arm64 to 8
(from 128). In libsas, this will cause SMP requests to be 8-byte-aligned
through kmalloc() allocation. However, for the hisi_sas hardware, all
commands address must be 16-byte-aligned. Otherwise, the commands fail to
be executed.

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations, so use ARCH_DMA_MINALIGN as the alignment for SMP request.

Link: https://lkml.kernel.org/r/20230612153201.554742-1-catalin.marinas@arm.com [1]
Signed-off-by: Yihang Li <liyihang9@huawei.com>
---
Changes since v1:
- Directly modify alloc_smp_req() instead of using handler callback.
---
 drivers/scsi/libsas/sas_expander.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
index a2204674b680..941abc7298df 100644
--- a/drivers/scsi/libsas/sas_expander.c
+++ b/drivers/scsi/libsas/sas_expander.c
@@ -135,7 +135,10 @@ static int smp_execute_task(struct domain_device *dev, void *req, int req_size,
 
 static inline void *alloc_smp_req(int size)
 {
-	u8 *p = kzalloc(size, GFP_KERNEL);
+	u8 *p;
+
+	size = ALIGN(size, ARCH_DMA_MINALIGN);
+	p = kzalloc(size, GFP_KERNEL);
 	if (p)
 		p[0] = SMP_REQUEST;
 	return p;
-- 
2.33.0

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[Damien Le Moal]

On 3/26/24 21:43, Yihang Li wrote:
> This series [1] reducing the kmalloc() minimum alignment on arm64 to 8
> (from 128). In libsas, this will cause SMP requests to be 8-byte-aligned
> through kmalloc() allocation. However, for the hisi_sas hardware, all
> commands address must be 16-byte-aligned. Otherwise, the commands fail to
> be executed.
> 
> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> operations, so use ARCH_DMA_MINALIGN as the alignment for SMP request.
> 
> Link: https://lkml.kernel.org/r/20230612153201.554742-1-catalin.marinas@arm.com [1]
> Signed-off-by: Yihang Li <liyihang9@huawei.com>
> ---
> Changes since v1:
> - Directly modify alloc_smp_req() instead of using handler callback.
> ---
>  drivers/scsi/libsas/sas_expander.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
> index a2204674b680..941abc7298df 100644
> --- a/drivers/scsi/libsas/sas_expander.c
> +++ b/drivers/scsi/libsas/sas_expander.c
> @@ -135,7 +135,10 @@ static int smp_execute_task(struct domain_device *dev, void *req, int req_size,
>  
>  static inline void *alloc_smp_req(int size)
>  {
> -	u8 *p = kzalloc(size, GFP_KERNEL);
> +	u8 *p;
> +
> +	size = ALIGN(size, ARCH_DMA_MINALIGN);
> +	p = kzalloc(size, GFP_KERNEL);

Nit: why not:

	p = kzalloc(ALIGN(size, ARCH_DMA_MINALIGN), GFP_KERNEL);

>  	if (p)
>  		p[0] = SMP_REQUEST;
>  	return p;

Otherwise looks OK to me.

John,

Unrelated to this patch, but I wonder if the GFP_KERNEL used here shouldn't be
GFP_NOIO... Is this ever called in the IO path or error recovery ?

-- 
Damien Le Moal
Western Digital Research

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[John Garry]

On 26/03/2024 13:14, Damien Le Moal wrote:
> On 3/26/24 21:43, Yihang Li wrote:
>> This series [1] reducing the kmalloc() minimum alignment on arm64 to 8
>> (from 128). In libsas, this will cause SMP requests to be 8-byte-aligned
>> through kmalloc() allocation. However, for the hisi_sas hardware, all
>> commands address must be 16-byte-aligned. 
> Otherwise, the commands fail to
>> be executed.
>>
>> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
>> operations, so use ARCH_DMA_MINALIGN as the alignment for SMP request.
>>
>> Link: https://lkml.kernel.org/r/20230612153201.554742-1-catalin.marinas@arm.com [1]
>> Signed-off-by: Yihang Li <liyihang9@huawei.com>
>> ---
>> Changes since v1:
>> - Directly modify alloc_smp_req() instead of using handler callback.
>> ---
>>   drivers/scsi/libsas/sas_expander.c | 5 ++++-
>>   1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
>> index a2204674b680..941abc7298df 100644
>> --- a/drivers/scsi/libsas/sas_expander.c
>> +++ b/drivers/scsi/libsas/sas_expander.c
>> @@ -135,7 +135,10 @@ static int smp_execute_task(struct domain_device *dev, void *req, int req_size,
>>   
>>   static inline void *alloc_smp_req(int size)
>>   {
>> -	u8 *p = kzalloc(size, GFP_KERNEL);
>> +	u8 *p;
>> +
>> +	size = ALIGN(size, ARCH_DMA_MINALIGN);


If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and 
not 16B as minimum alignment?

Or are we really talking about an arch requirement?

>> +	p = kzalloc(size, GFP_KERNEL);
> 
> Nit: why not:
> 
> 	p = kzalloc(ALIGN(size, ARCH_DMA_MINALIGN), GFP_KERNEL);
> 
>>   	if (p)
>>   		p[0] = SMP_REQUEST;
>>   	return p;
> 
> Otherwise looks OK to me.
> 
> John,
> 
> Unrelated to this patch, but I wonder if the GFP_KERNEL used here shouldn't be
> GFP_NOIO... Is this ever called in the IO path or error recovery ?
> 

These should not be called in the IO path - as they are management 
functions. But I am quite confident that they can be called in SCSI 
error handling (for libsas).

Thanks,
John

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[Damien Le Moal]

On 3/26/24 22:32, John Garry wrote:
>> John,
>>
>> Unrelated to this patch, but I wonder if the GFP_KERNEL used here shouldn't be
>> GFP_NOIO... Is this ever called in the IO path or error recovery ?
>>
> 
> These should not be called in the IO path - as they are management 
> functions. But I am quite confident that they can be called in SCSI 
> error handling (for libsas).

So it sounds like GFP_NOIO would be a lot safer...

> 
> Thanks,
> John

-- 
Damien Le Moal
Western Digital Research

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[Christoph Hellwig]

On Tue, Mar 26, 2024 at 01:32:09PM +0000, John Garry wrote:
> > > +	u8 *p;
> > > +
> > > +	size = ALIGN(size, ARCH_DMA_MINALIGN);
> 
> 
> If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and not
> 16B as minimum alignment?
> 
> Or are we really talking about an arch requirement?

One thing is that we should never allocate unaligned memory for
anything DMA mapped, or data will be corrupted by non-coherent DMA.
So ARCH_DMA_MINALIGN needs to be here.  If specific hardware has
further requirements we'll need to communicated it through a field
or op vector.

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[Yihang Li]

On 2024/3/28 14:35, Christoph Hellwig wrote:
> On Tue, Mar 26, 2024 at 01:32:09PM +0000, John Garry wrote:
>>>> +	u8 *p;
>>>> +
>>>> +	size = ALIGN(size, ARCH_DMA_MINALIGN);
>>
>>
>> If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and not
>> 16B as minimum alignment?
>>
>> Or are we really talking about an arch requirement?
> 
> One thing is that we should never allocate unaligned memory for
> anything DMA mapped, or data will be corrupted by non-coherent DMA.
> So ARCH_DMA_MINALIGN needs to be here.  If specific hardware has
> further requirements we'll need to communicated it through a field
> or op vector.

Got it. Looks like it's still going to be aligned to ARCH_DMA_MINALIGN.

Thanks,
Yihang

> 
> 
> .
> 

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[Damien Le Moal]

On 3/28/24 15:59, Yihang Li wrote:
> 
> 
> On 2024/3/28 14:35, Christoph Hellwig wrote:
>> On Tue, Mar 26, 2024 at 01:32:09PM +0000, John Garry wrote:
>>>>> +	u8 *p;
>>>>> +
>>>>> +	size = ALIGN(size, ARCH_DMA_MINALIGN);
>>>
>>>
>>> If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and not
>>> 16B as minimum alignment?
>>>
>>> Or are we really talking about an arch requirement?
>>
>> One thing is that we should never allocate unaligned memory for
>> anything DMA mapped, or data will be corrupted by non-coherent DMA.
>> So ARCH_DMA_MINALIGN needs to be here.  If specific hardware has
>> further requirements we'll need to communicated it through a field
>> or op vector.
> 
> Got it. Looks like it's still going to be aligned to ARCH_DMA_MINALIGN.

But I thought that the original issue was that some arch have ARCH_DMA_MINALIGN
down to 8B but hisi driver needs at least 16 ?

So in the end, you need something like:

	size = ALIGN(size, max(16, ARCH_DMA_MINALIGN));

no ?

And define a macro for the "16" value to document it.
Something like:

#define SAS_SMP_REQ_ALIGN	16

Not sure about the name... Naming is hard :)

> 
> Thanks,
> Yihang
> 
>>
>>
>> .
>>

-- 
Damien Le Moal
Western Digital Research

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[Christoph Hellwig]

On Thu, Mar 28, 2024 at 04:23:22PM +0900, Damien Le Moal wrote:
> But I thought that the original issue was that some arch have ARCH_DMA_MINALIGN
> down to 8B but hisi driver needs at least 16 ?
> 
> So in the end, you need something like:
> 
> 	size = ALIGN(size, max(16, ARCH_DMA_MINALIGN));
> 
> no ?

I don't think we ever have an 8 byte dma minalign.  With 8-byte
aligned addresses dma_mapping_error could run into problems.

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[Damien Le Moal]

On 3/28/24 16:29, Christoph Hellwig wrote:
> On Thu, Mar 28, 2024 at 04:23:22PM +0900, Damien Le Moal wrote:
>> But I thought that the original issue was that some arch have ARCH_DMA_MINALIGN
>> down to 8B but hisi driver needs at least 16 ?
>>
>> So in the end, you need something like:
>>
>> 	size = ALIGN(size, max(16, ARCH_DMA_MINALIGN));
>>
>> no ?
> 
> I don't think we ever have an 8 byte dma minalign.  With 8-byte
> aligned addresses dma_mapping_error could run into problems.

My bad: it is kmalloc() that can return something aligned to 8B...
So "size = ALIGN(size, ARCH_DMA_MINALIGN);" is the right thing to do.

-- 
Damien Le Moal
Western Digital Research

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[Christoph Hellwig]

On Thu, Mar 28, 2024 at 04:36:12PM +0900, Damien Le Moal wrote:
> My bad: it is kmalloc() that can return something aligned to 8B...

Yes, that's new on arm64, and possibly soon riscv.

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[Yihang Li]

On 2024/3/28 15:45, Christoph Hellwig wrote:
> On Thu, Mar 28, 2024 at 04:36:12PM +0900, Damien Le Moal wrote:
>> My bad: it is kmalloc() that can return something aligned to 8B...
> 
> Yes, that's new on arm64, and possibly soon riscv.

Thanks for the discussion, I will still aligned to ARCH_DMA_MINALIGN in v4.