mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path
@ 2024-04-07  1:14 Lu Baolu
  2024-04-08  3:52 ` Tian, Kevin
                   ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: Lu Baolu @ 2024-04-07  1:14 UTC (permalink / raw)
  To: iommu
  Cc: Jason Gunthorpe, Kevin Tian, Joerg Roedel, Will Deacon,
	Robin Murphy, linux-kernel, Lu Baolu

Commit 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
devices") adds all devices probed by the iommu driver in a rbtree
indexed by the source ID of each device. It assumes that each device
has a unique source ID. This assumption is incorrect and the VT-d
spec doesn't state this requirement either.

The reason for using a rbtree to track devices is to look up the device
with PCI bus and devfunc in the paths of handling ATS invalidation time
out error and the PRI I/O page faults. Both are PCI ATS feature related.

Only track the devices that have PCI ATS capabilities in the rbtree to
avoid unnecessary WARN_ON in the iommu probe path. Otherwise, on some
platforms below kernel splat will be displayed and the iommu probe results
in failure.

 WARNING: CPU: 3 PID: 166 at drivers/iommu/intel/iommu.c:158 intel_iommu_probe_device+0x319/0xd90
 Call Trace:
  <TASK>
  ? __warn+0x7e/0x180
  ? intel_iommu_probe_device+0x319/0xd90
  ? report_bug+0x1f8/0x200
  ? handle_bug+0x3c/0x70
  ? exc_invalid_op+0x18/0x70
  ? asm_exc_invalid_op+0x1a/0x20
  ? intel_iommu_probe_device+0x319/0xd90
  ? debug_mutex_init+0x37/0x50
  __iommu_probe_device+0xf2/0x4f0
  iommu_probe_device+0x22/0x70
  iommu_bus_notifier+0x1e/0x40
  notifier_call_chain+0x46/0x150
  blocking_notifier_call_chain+0x42/0x60
  bus_notify+0x2f/0x50
  device_add+0x5ed/0x7e0
  platform_device_add+0xf5/0x240
  mfd_add_devices+0x3f9/0x500
  ? preempt_count_add+0x4c/0xa0
  ? up_write+0xa2/0x1b0
  ? __debugfs_create_file+0xe3/0x150
  intel_lpss_probe+0x49f/0x5b0
  ? pci_conf1_write+0xa3/0xf0
  intel_lpss_pci_probe+0xcf/0x110 [intel_lpss_pci]
  pci_device_probe+0x95/0x120
  really_probe+0xd9/0x370
  ? __pfx___driver_attach+0x10/0x10
  __driver_probe_device+0x73/0x150
  driver_probe_device+0x19/0xa0
  __driver_attach+0xb6/0x180
  ? __pfx___driver_attach+0x10/0x10
  bus_for_each_dev+0x77/0xd0
  bus_add_driver+0x114/0x210
  driver_register+0x5b/0x110
  ? __pfx_intel_lpss_pci_driver_init+0x10/0x10 [intel_lpss_pci]
  do_one_initcall+0x57/0x2b0
  ? kmalloc_trace+0x21e/0x280
  ? do_init_module+0x1e/0x210
  do_init_module+0x5f/0x210
  load_module+0x1d37/0x1fc0
  ? init_module_from_file+0x86/0xd0
  init_module_from_file+0x86/0xd0
  idempotent_init_module+0x17c/0x230
  __x64_sys_finit_module+0x56/0xb0
  do_syscall_64+0x6e/0x140
  entry_SYSCALL_64_after_hwframe+0x71/0x79

Fixes: 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed devices")
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
---
 drivers/iommu/intel/iommu.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index 50eb9aed47cc..a7ecd90303dc 100644
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -4299,9 +4299,11 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
 	}
 
 	dev_iommu_priv_set(dev, info);
-	ret = device_rbtree_insert(iommu, info);
-	if (ret)
-		goto free;
+	if (pdev && pci_ats_supported(pdev)) {
+		ret = device_rbtree_insert(iommu, info);
+		if (ret)
+			goto free;
+	}
 
 	if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev)) {
 		ret = intel_pasid_alloc_table(dev);
@@ -4336,7 +4338,8 @@ static void intel_iommu_release_device(struct device *dev)
 	struct intel_iommu *iommu = info->iommu;
 
 	mutex_lock(&iommu->iopf_lock);
-	device_rbtree_remove(info);
+	if (dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)))
+		device_rbtree_remove(info);
 	mutex_unlock(&iommu->iopf_lock);
 
 	if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev) &&
-- 
2.34.1


^ permalink raw reply	[flat|nested] 10+ messages in thread

* RE: [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path
  2024-04-07  1:14 [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path Lu Baolu
@ 2024-04-08  3:52 ` Tian, Kevin
  2024-04-08  7:15   ` Baolu Lu
  2024-04-11 12:29   ` Ethan Zhao
  2024-04-11  2:47 ` Baolu Lu
  2026-11-17 18:01 ` Bjorn Helgaas
  2 siblings, 2 replies; 10+ messages in thread
From: Tian, Kevin @ 2024-04-08  3:52 UTC (permalink / raw)
  To: Lu Baolu, iommu
  Cc: Jason Gunthorpe, Joerg Roedel, Will Deacon, Robin Murphy, linux-kernel

> From: Lu Baolu <baolu.lu@linux.intel.com>
> Sent: Sunday, April 7, 2024 9:14 AM
> 
> Commit 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
> devices") adds all devices probed by the iommu driver in a rbtree
> indexed by the source ID of each device. It assumes that each device
> has a unique source ID. This assumption is incorrect and the VT-d
> spec doesn't state this requirement either.
> 
> The reason for using a rbtree to track devices is to look up the device
> with PCI bus and devfunc in the paths of handling ATS invalidation time
> out error and the PRI I/O page faults. Both are PCI ATS feature related.
> 
> Only track the devices that have PCI ATS capabilities in the rbtree to
> avoid unnecessary WARN_ON in the iommu probe path. Otherwise, on some
> platforms below kernel splat will be displayed and the iommu probe results
> in failure.

Just be curious. What about two ATS capable devices putting behind
a PCI-to-PCIe bridge?

> 
>  WARNING: CPU: 3 PID: 166 at drivers/iommu/intel/iommu.c:158
> intel_iommu_probe_device+0x319/0xd90
>  Call Trace:
>   <TASK>
>   ? __warn+0x7e/0x180
>   ? intel_iommu_probe_device+0x319/0xd90
>   ? report_bug+0x1f8/0x200
>   ? handle_bug+0x3c/0x70
>   ? exc_invalid_op+0x18/0x70
>   ? asm_exc_invalid_op+0x1a/0x20
>   ? intel_iommu_probe_device+0x319/0xd90
>   ? debug_mutex_init+0x37/0x50
>   __iommu_probe_device+0xf2/0x4f0
>   iommu_probe_device+0x22/0x70
>   iommu_bus_notifier+0x1e/0x40
>   notifier_call_chain+0x46/0x150
>   blocking_notifier_call_chain+0x42/0x60
>   bus_notify+0x2f/0x50
>   device_add+0x5ed/0x7e0
>   platform_device_add+0xf5/0x240
>   mfd_add_devices+0x3f9/0x500
>   ? preempt_count_add+0x4c/0xa0
>   ? up_write+0xa2/0x1b0
>   ? __debugfs_create_file+0xe3/0x150
>   intel_lpss_probe+0x49f/0x5b0
>   ? pci_conf1_write+0xa3/0xf0
>   intel_lpss_pci_probe+0xcf/0x110 [intel_lpss_pci]
>   pci_device_probe+0x95/0x120
>   really_probe+0xd9/0x370
>   ? __pfx___driver_attach+0x10/0x10
>   __driver_probe_device+0x73/0x150
>   driver_probe_device+0x19/0xa0
>   __driver_attach+0xb6/0x180
>   ? __pfx___driver_attach+0x10/0x10
>   bus_for_each_dev+0x77/0xd0
>   bus_add_driver+0x114/0x210
>   driver_register+0x5b/0x110
>   ? __pfx_intel_lpss_pci_driver_init+0x10/0x10 [intel_lpss_pci]
>   do_one_initcall+0x57/0x2b0
>   ? kmalloc_trace+0x21e/0x280
>   ? do_init_module+0x1e/0x210
>   do_init_module+0x5f/0x210
>   load_module+0x1d37/0x1fc0
>   ? init_module_from_file+0x86/0xd0
>   init_module_from_file+0x86/0xd0
>   idempotent_init_module+0x17c/0x230
>   __x64_sys_finit_module+0x56/0xb0
>   do_syscall_64+0x6e/0x140
>   entry_SYSCALL_64_after_hwframe+0x71/0x79
> 
> Fixes: 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
> devices")
> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
> ---
>  drivers/iommu/intel/iommu.c | 11 +++++++----
>  1 file changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
> index 50eb9aed47cc..a7ecd90303dc 100644
> --- a/drivers/iommu/intel/iommu.c
> +++ b/drivers/iommu/intel/iommu.c
> @@ -4299,9 +4299,11 @@ static struct iommu_device
> *intel_iommu_probe_device(struct device *dev)
>  	}
> 
>  	dev_iommu_priv_set(dev, info);
> -	ret = device_rbtree_insert(iommu, info);
> -	if (ret)
> -		goto free;
> +	if (pdev && pci_ats_supported(pdev)) {
> +		ret = device_rbtree_insert(iommu, info);
> +		if (ret)
> +			goto free;
> +	}

probably replace device_rbtree with ats_rbtree?

> 
>  	if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev)) {
>  		ret = intel_pasid_alloc_table(dev);
> @@ -4336,7 +4338,8 @@ static void intel_iommu_release_device(struct
> device *dev)
>  	struct intel_iommu *iommu = info->iommu;
> 
>  	mutex_lock(&iommu->iopf_lock);
> -	device_rbtree_remove(info);
> +	if (dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)))
> +		device_rbtree_remove(info);
>  	mutex_unlock(&iommu->iopf_lock);
> 
>  	if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev) &&
> --
> 2.34.1


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path
  2024-04-08  3:52 ` Tian, Kevin
@ 2024-04-08  7:15   ` Baolu Lu
  2024-04-08  9:12     ` Tian, Kevin
  2024-04-11 12:29   ` Ethan Zhao
  1 sibling, 1 reply; 10+ messages in thread
From: Baolu Lu @ 2024-04-08  7:15 UTC (permalink / raw)
  To: Tian, Kevin, iommu
  Cc: baolu.lu, Jason Gunthorpe, Joerg Roedel, Will Deacon,
	Robin Murphy, linux-kernel

On 2024/4/8 11:52, Tian, Kevin wrote:
>> From: Lu Baolu<baolu.lu@linux.intel.com>
>> Sent: Sunday, April 7, 2024 9:14 AM
>>
>> Commit 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
>> devices") adds all devices probed by the iommu driver in a rbtree
>> indexed by the source ID of each device. It assumes that each device
>> has a unique source ID. This assumption is incorrect and the VT-d
>> spec doesn't state this requirement either.
>>
>> The reason for using a rbtree to track devices is to look up the device
>> with PCI bus and devfunc in the paths of handling ATS invalidation time
>> out error and the PRI I/O page faults. Both are PCI ATS feature related.
>>
>> Only track the devices that have PCI ATS capabilities in the rbtree to
>> avoid unnecessary WARN_ON in the iommu probe path. Otherwise, on some
>> platforms below kernel splat will be displayed and the iommu probe results
>> in failure.
> Just be curious. What about two ATS capable devices putting behind
> a PCI-to-PCIe bridge?

I don't think ATS capable device putting behind a PCI-to-PCIe bridge is
a right configuration for the ATS service. The PCIe spec requires this
in section 10.1.1, that

"
ATS requires the following:
- ATS capable components must interoperate with [PCIe-1.1] compliant
   components.
...
"

My understanding is that PCI-to-PCIe bridge is not a PCIe compliant
device.

> 
>>   WARNING: CPU: 3 PID: 166 at drivers/iommu/intel/iommu.c:158
>> intel_iommu_probe_device+0x319/0xd90
>>   Call Trace:
>>    <TASK>
>>    ? __warn+0x7e/0x180
>>    ? intel_iommu_probe_device+0x319/0xd90
>>    ? report_bug+0x1f8/0x200
>>    ? handle_bug+0x3c/0x70
>>    ? exc_invalid_op+0x18/0x70
>>    ? asm_exc_invalid_op+0x1a/0x20
>>    ? intel_iommu_probe_device+0x319/0xd90
>>    ? debug_mutex_init+0x37/0x50
>>    __iommu_probe_device+0xf2/0x4f0
>>    iommu_probe_device+0x22/0x70
>>    iommu_bus_notifier+0x1e/0x40
>>    notifier_call_chain+0x46/0x150
>>    blocking_notifier_call_chain+0x42/0x60
>>    bus_notify+0x2f/0x50
>>    device_add+0x5ed/0x7e0
>>    platform_device_add+0xf5/0x240
>>    mfd_add_devices+0x3f9/0x500
>>    ? preempt_count_add+0x4c/0xa0
>>    ? up_write+0xa2/0x1b0
>>    ? __debugfs_create_file+0xe3/0x150
>>    intel_lpss_probe+0x49f/0x5b0
>>    ? pci_conf1_write+0xa3/0xf0
>>    intel_lpss_pci_probe+0xcf/0x110 [intel_lpss_pci]
>>    pci_device_probe+0x95/0x120
>>    really_probe+0xd9/0x370
>>    ? __pfx___driver_attach+0x10/0x10
>>    __driver_probe_device+0x73/0x150
>>    driver_probe_device+0x19/0xa0
>>    __driver_attach+0xb6/0x180
>>    ? __pfx___driver_attach+0x10/0x10
>>    bus_for_each_dev+0x77/0xd0
>>    bus_add_driver+0x114/0x210
>>    driver_register+0x5b/0x110
>>    ? __pfx_intel_lpss_pci_driver_init+0x10/0x10 [intel_lpss_pci]
>>    do_one_initcall+0x57/0x2b0
>>    ? kmalloc_trace+0x21e/0x280
>>    ? do_init_module+0x1e/0x210
>>    do_init_module+0x5f/0x210
>>    load_module+0x1d37/0x1fc0
>>    ? init_module_from_file+0x86/0xd0
>>    init_module_from_file+0x86/0xd0
>>    idempotent_init_module+0x17c/0x230
>>    __x64_sys_finit_module+0x56/0xb0
>>    do_syscall_64+0x6e/0x140
>>    entry_SYSCALL_64_after_hwframe+0x71/0x79
>>
>> Fixes: 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
>> devices")
>> Signed-off-by: Lu Baolu<baolu.lu@linux.intel.com>
>> ---
>>   drivers/iommu/intel/iommu.c | 11 +++++++----
>>   1 file changed, 7 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
>> index 50eb9aed47cc..a7ecd90303dc 100644
>> --- a/drivers/iommu/intel/iommu.c
>> +++ b/drivers/iommu/intel/iommu.c
>> @@ -4299,9 +4299,11 @@ static struct iommu_device
>> *intel_iommu_probe_device(struct device *dev)
>>   	}
>>
>>   	dev_iommu_priv_set(dev, info);
>> -	ret = device_rbtree_insert(iommu, info);
>> -	if (ret)
>> -		goto free;
>> +	if (pdev && pci_ats_supported(pdev)) {
>> +		ret = device_rbtree_insert(iommu, info);
>> +		if (ret)
>> +			goto free;
>> +	}
> probably replace device_rbtree with ats_rbtree?

It makes sense. Probably I need a follow-up patch to do such cleanup.

Best regards,
baolu

^ permalink raw reply	[flat|nested] 10+ messages in thread

* RE: [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path
  2024-04-08  7:15   ` Baolu Lu
@ 2024-04-08  9:12     ` Tian, Kevin
  0 siblings, 0 replies; 10+ messages in thread
From: Tian, Kevin @ 2024-04-08  9:12 UTC (permalink / raw)
  To: Baolu Lu, iommu
  Cc: Jason Gunthorpe, Joerg Roedel, Will Deacon, Robin Murphy,
	linux-kernel, Bjorn Helgaas

+Bjorn

> From: Baolu Lu <baolu.lu@linux.intel.com>
> Sent: Monday, April 8, 2024 3:15 PM
> 
> On 2024/4/8 11:52, Tian, Kevin wrote:
> >> From: Lu Baolu<baolu.lu@linux.intel.com>
> >> Sent: Sunday, April 7, 2024 9:14 AM
> >>
> >> Commit 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
> >> devices") adds all devices probed by the iommu driver in a rbtree
> >> indexed by the source ID of each device. It assumes that each device
> >> has a unique source ID. This assumption is incorrect and the VT-d
> >> spec doesn't state this requirement either.
> >>
> >> The reason for using a rbtree to track devices is to look up the device
> >> with PCI bus and devfunc in the paths of handling ATS invalidation time
> >> out error and the PRI I/O page faults. Both are PCI ATS feature related.
> >>
> >> Only track the devices that have PCI ATS capabilities in the rbtree to
> >> avoid unnecessary WARN_ON in the iommu probe path. Otherwise, on
> some
> >> platforms below kernel splat will be displayed and the iommu probe
> results
> >> in failure.
> > Just be curious. What about two ATS capable devices putting behind
> > a PCI-to-PCIe bridge?
> 
> I don't think ATS capable device putting behind a PCI-to-PCIe bridge is
> a right configuration for the ATS service. The PCIe spec requires this
> in section 10.1.1, that
> 
> "
> ATS requires the following:
> - ATS capable components must interoperate with [PCIe-1.1] compliant
>    components.
> ...
> "
> 
> My understanding is that PCI-to-PCIe bridge is not a PCIe compliant
> device.
> 

what is the pci core policy on such device? pci_enable_ats() doesn't
prevent such device and I saw various places in pci core do handle
the PCI_EXP_TYPE_PCIE_BRIDGE type...

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path
  2024-04-07  1:14 [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path Lu Baolu
  2024-04-08  3:52 ` Tian, Kevin
@ 2024-04-11  2:47 ` Baolu Lu
  2024-04-11  6:53   ` Tian, Kevin
  2026-11-17 18:01 ` Bjorn Helgaas
  2 siblings, 1 reply; 10+ messages in thread
From: Baolu Lu @ 2024-04-11  2:47 UTC (permalink / raw)
  To: iommu
  Cc: baolu.lu, Jason Gunthorpe, Kevin Tian, Joerg Roedel, Will Deacon,
	Robin Murphy, linux-kernel

On 4/7/24 9:14 AM, Lu Baolu wrote:
> Commit 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
> devices") adds all devices probed by the iommu driver in a rbtree
> indexed by the source ID of each device. It assumes that each device
> has a unique source ID. This assumption is incorrect and the VT-d
> spec doesn't state this requirement either.
> 
> The reason for using a rbtree to track devices is to look up the device
> with PCI bus and devfunc in the paths of handling ATS invalidation time
> out error and the PRI I/O page faults. Both are PCI ATS feature related.
> 
> Only track the devices that have PCI ATS capabilities in the rbtree to
> avoid unnecessary WARN_ON in the iommu probe path. Otherwise, on some
> platforms below kernel splat will be displayed and the iommu probe results
> in failure.
> 
>   WARNING: CPU: 3 PID: 166 at drivers/iommu/intel/iommu.c:158 intel_iommu_probe_device+0x319/0xd90
>   Call Trace:
>    <TASK>
>    ? __warn+0x7e/0x180
>    ? intel_iommu_probe_device+0x319/0xd90
>    ? report_bug+0x1f8/0x200
>    ? handle_bug+0x3c/0x70
>    ? exc_invalid_op+0x18/0x70
>    ? asm_exc_invalid_op+0x1a/0x20
>    ? intel_iommu_probe_device+0x319/0xd90
>    ? debug_mutex_init+0x37/0x50
>    __iommu_probe_device+0xf2/0x4f0
>    iommu_probe_device+0x22/0x70
>    iommu_bus_notifier+0x1e/0x40
>    notifier_call_chain+0x46/0x150
>    blocking_notifier_call_chain+0x42/0x60
>    bus_notify+0x2f/0x50
>    device_add+0x5ed/0x7e0
>    platform_device_add+0xf5/0x240
>    mfd_add_devices+0x3f9/0x500
>    ? preempt_count_add+0x4c/0xa0
>    ? up_write+0xa2/0x1b0
>    ? __debugfs_create_file+0xe3/0x150
>    intel_lpss_probe+0x49f/0x5b0
>    ? pci_conf1_write+0xa3/0xf0
>    intel_lpss_pci_probe+0xcf/0x110 [intel_lpss_pci]
>    pci_device_probe+0x95/0x120
>    really_probe+0xd9/0x370
>    ? __pfx___driver_attach+0x10/0x10
>    __driver_probe_device+0x73/0x150
>    driver_probe_device+0x19/0xa0
>    __driver_attach+0xb6/0x180
>    ? __pfx___driver_attach+0x10/0x10
>    bus_for_each_dev+0x77/0xd0
>    bus_add_driver+0x114/0x210
>    driver_register+0x5b/0x110
>    ? __pfx_intel_lpss_pci_driver_init+0x10/0x10 [intel_lpss_pci]
>    do_one_initcall+0x57/0x2b0
>    ? kmalloc_trace+0x21e/0x280
>    ? do_init_module+0x1e/0x210
>    do_init_module+0x5f/0x210
>    load_module+0x1d37/0x1fc0
>    ? init_module_from_file+0x86/0xd0
>    init_module_from_file+0x86/0xd0
>    idempotent_init_module+0x17c/0x230
>    __x64_sys_finit_module+0x56/0xb0
>    do_syscall_64+0x6e/0x140
>    entry_SYSCALL_64_after_hwframe+0x71/0x79
> 
> Fixes: 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed devices")
> Signed-off-by: Lu Baolu<baolu.lu@linux.intel.com>
> ---
>   drivers/iommu/intel/iommu.c | 11 +++++++----
>   1 file changed, 7 insertions(+), 4 deletions(-)

This patch closes a boot bug described here.

Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/10689

I will queue it for v6.9-rc if no objection.

Best regards,
baolu

^ permalink raw reply	[flat|nested] 10+ messages in thread

* RE: [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path
  2024-04-11  2:47 ` Baolu Lu
@ 2024-04-11  6:53   ` Tian, Kevin
  0 siblings, 0 replies; 10+ messages in thread
From: Tian, Kevin @ 2024-04-11  6:53 UTC (permalink / raw)
  To: Baolu Lu, iommu
  Cc: Jason Gunthorpe, Joerg Roedel, Will Deacon, Robin Murphy, linux-kernel

> From: Baolu Lu <baolu.lu@linux.intel.com>
> Sent: Thursday, April 11, 2024 10:48 AM
> 
> On 4/7/24 9:14 AM, Lu Baolu wrote:
> > Commit 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
> > devices") adds all devices probed by the iommu driver in a rbtree
> > indexed by the source ID of each device. It assumes that each device
> > has a unique source ID. This assumption is incorrect and the VT-d
> > spec doesn't state this requirement either.
> >
> > The reason for using a rbtree to track devices is to look up the device
> > with PCI bus and devfunc in the paths of handling ATS invalidation time
> > out error and the PRI I/O page faults. Both are PCI ATS feature related.
> >
> > Only track the devices that have PCI ATS capabilities in the rbtree to
> > avoid unnecessary WARN_ON in the iommu probe path. Otherwise, on
> some
> > platforms below kernel splat will be displayed and the iommu probe results
> > in failure.
> >
> >   WARNING: CPU: 3 PID: 166 at drivers/iommu/intel/iommu.c:158
> intel_iommu_probe_device+0x319/0xd90
> >   Call Trace:
> >    <TASK>
> >    ? __warn+0x7e/0x180
> >    ? intel_iommu_probe_device+0x319/0xd90
> >    ? report_bug+0x1f8/0x200
> >    ? handle_bug+0x3c/0x70
> >    ? exc_invalid_op+0x18/0x70
> >    ? asm_exc_invalid_op+0x1a/0x20
> >    ? intel_iommu_probe_device+0x319/0xd90
> >    ? debug_mutex_init+0x37/0x50
> >    __iommu_probe_device+0xf2/0x4f0
> >    iommu_probe_device+0x22/0x70
> >    iommu_bus_notifier+0x1e/0x40
> >    notifier_call_chain+0x46/0x150
> >    blocking_notifier_call_chain+0x42/0x60
> >    bus_notify+0x2f/0x50
> >    device_add+0x5ed/0x7e0
> >    platform_device_add+0xf5/0x240
> >    mfd_add_devices+0x3f9/0x500
> >    ? preempt_count_add+0x4c/0xa0
> >    ? up_write+0xa2/0x1b0
> >    ? __debugfs_create_file+0xe3/0x150
> >    intel_lpss_probe+0x49f/0x5b0
> >    ? pci_conf1_write+0xa3/0xf0
> >    intel_lpss_pci_probe+0xcf/0x110 [intel_lpss_pci]
> >    pci_device_probe+0x95/0x120
> >    really_probe+0xd9/0x370
> >    ? __pfx___driver_attach+0x10/0x10
> >    __driver_probe_device+0x73/0x150
> >    driver_probe_device+0x19/0xa0
> >    __driver_attach+0xb6/0x180
> >    ? __pfx___driver_attach+0x10/0x10
> >    bus_for_each_dev+0x77/0xd0
> >    bus_add_driver+0x114/0x210
> >    driver_register+0x5b/0x110
> >    ? __pfx_intel_lpss_pci_driver_init+0x10/0x10 [intel_lpss_pci]
> >    do_one_initcall+0x57/0x2b0
> >    ? kmalloc_trace+0x21e/0x280
> >    ? do_init_module+0x1e/0x210
> >    do_init_module+0x5f/0x210
> >    load_module+0x1d37/0x1fc0
> >    ? init_module_from_file+0x86/0xd0
> >    init_module_from_file+0x86/0xd0
> >    idempotent_init_module+0x17c/0x230
> >    __x64_sys_finit_module+0x56/0xb0
> >    do_syscall_64+0x6e/0x140
> >    entry_SYSCALL_64_after_hwframe+0x71/0x79
> >
> > Fixes: 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
> devices")
> > Signed-off-by: Lu Baolu<baolu.lu@linux.intel.com>
> > ---
> >   drivers/iommu/intel/iommu.c | 11 +++++++----
> >   1 file changed, 7 insertions(+), 4 deletions(-)
> 
> This patch closes a boot bug described here.
> 
> Closes: https://gitlab.freedesktop.org/drm/intel/-/issues/10689
> 
> I will queue it for v6.9-rc if no objection.
> 

let's do it. the issue which I raised is rare and can be tackled when
it arises.

Reviewed-by: Kevin Tian <kevin.tian@intel.com>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path
  2024-04-08  3:52 ` Tian, Kevin
  2024-04-08  7:15   ` Baolu Lu
@ 2024-04-11 12:29   ` Ethan Zhao
  1 sibling, 0 replies; 10+ messages in thread
From: Ethan Zhao @ 2024-04-11 12:29 UTC (permalink / raw)
  To: Tian, Kevin, Lu Baolu, iommu
  Cc: Jason Gunthorpe, Joerg Roedel, Will Deacon, Robin Murphy, linux-kernel

On 4/8/2024 11:52 AM, Tian, Kevin wrote:
>> From: Lu Baolu <baolu.lu@linux.intel.com>
>> Sent: Sunday, April 7, 2024 9:14 AM
>>
>> Commit 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
>> devices") adds all devices probed by the iommu driver in a rbtree
>> indexed by the source ID of each device. It assumes that each device
>> has a unique source ID. This assumption is incorrect and the VT-d
>> spec doesn't state this requirement either.
>>
>> The reason for using a rbtree to track devices is to look up the device
>> with PCI bus and devfunc in the paths of handling ATS invalidation time
>> out error and the PRI I/O page faults. Both are PCI ATS feature related.
>>
>> Only track the devices that have PCI ATS capabilities in the rbtree to
>> avoid unnecessary WARN_ON in the iommu probe path. Otherwise, on some
>> platforms below kernel splat will be displayed and the iommu probe results
>> in failure.
> Just be curious. What about two ATS capable devices putting behind
> a PCI-to-PCIe bridge?

No PCI-to-PCIe bridges such device AFAIK, ATS TLP couldn't be
routed via PCIe-to-PCI bridge unmodified. ATS is based on PCIe.

But the device-rbtree is possible holding mutiple device nodes behind
PCIe-to-PCI bridge indexed by the same requester-ID.

Thanks,
Ethan

>>   WARNING: CPU: 3 PID: 166 at drivers/iommu/intel/iommu.c:158
>> intel_iommu_probe_device+0x319/0xd90
>>   Call Trace:
>>    <TASK>
>>    ? __warn+0x7e/0x180
>>    ? intel_iommu_probe_device+0x319/0xd90
>>    ? report_bug+0x1f8/0x200
>>    ? handle_bug+0x3c/0x70
>>    ? exc_invalid_op+0x18/0x70
>>    ? asm_exc_invalid_op+0x1a/0x20
>>    ? intel_iommu_probe_device+0x319/0xd90
>>    ? debug_mutex_init+0x37/0x50
>>    __iommu_probe_device+0xf2/0x4f0
>>    iommu_probe_device+0x22/0x70
>>    iommu_bus_notifier+0x1e/0x40
>>    notifier_call_chain+0x46/0x150
>>    blocking_notifier_call_chain+0x42/0x60
>>    bus_notify+0x2f/0x50
>>    device_add+0x5ed/0x7e0
>>    platform_device_add+0xf5/0x240
>>    mfd_add_devices+0x3f9/0x500
>>    ? preempt_count_add+0x4c/0xa0
>>    ? up_write+0xa2/0x1b0
>>    ? __debugfs_create_file+0xe3/0x150
>>    intel_lpss_probe+0x49f/0x5b0
>>    ? pci_conf1_write+0xa3/0xf0
>>    intel_lpss_pci_probe+0xcf/0x110 [intel_lpss_pci]
>>    pci_device_probe+0x95/0x120
>>    really_probe+0xd9/0x370
>>    ? __pfx___driver_attach+0x10/0x10
>>    __driver_probe_device+0x73/0x150
>>    driver_probe_device+0x19/0xa0
>>    __driver_attach+0xb6/0x180
>>    ? __pfx___driver_attach+0x10/0x10
>>    bus_for_each_dev+0x77/0xd0
>>    bus_add_driver+0x114/0x210
>>    driver_register+0x5b/0x110
>>    ? __pfx_intel_lpss_pci_driver_init+0x10/0x10 [intel_lpss_pci]
>>    do_one_initcall+0x57/0x2b0
>>    ? kmalloc_trace+0x21e/0x280
>>    ? do_init_module+0x1e/0x210
>>    do_init_module+0x5f/0x210
>>    load_module+0x1d37/0x1fc0
>>    ? init_module_from_file+0x86/0xd0
>>    init_module_from_file+0x86/0xd0
>>    idempotent_init_module+0x17c/0x230
>>    __x64_sys_finit_module+0x56/0xb0
>>    do_syscall_64+0x6e/0x140
>>    entry_SYSCALL_64_after_hwframe+0x71/0x79
>>
>> Fixes: 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
>> devices")
>> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
>> ---
>>   drivers/iommu/intel/iommu.c | 11 +++++++----
>>   1 file changed, 7 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
>> index 50eb9aed47cc..a7ecd90303dc 100644
>> --- a/drivers/iommu/intel/iommu.c
>> +++ b/drivers/iommu/intel/iommu.c
>> @@ -4299,9 +4299,11 @@ static struct iommu_device
>> *intel_iommu_probe_device(struct device *dev)
>>   	}
>>
>>   	dev_iommu_priv_set(dev, info);
>> -	ret = device_rbtree_insert(iommu, info);
>> -	if (ret)
>> -		goto free;
>> +	if (pdev && pci_ats_supported(pdev)) {
>> +		ret = device_rbtree_insert(iommu, info);
>> +		if (ret)
>> +			goto free;
>> +	}
> probably replace device_rbtree with ats_rbtree?
>
>>   	if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev)) {
>>   		ret = intel_pasid_alloc_table(dev);
>> @@ -4336,7 +4338,8 @@ static void intel_iommu_release_device(struct
>> device *dev)
>>   	struct intel_iommu *iommu = info->iommu;
>>
>>   	mutex_lock(&iommu->iopf_lock);
>> -	device_rbtree_remove(info);
>> +	if (dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)))
>> +		device_rbtree_remove(info);
>>   	mutex_unlock(&iommu->iopf_lock);
>>
>>   	if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev) &&
>> --
>> 2.34.1
>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path
  2024-04-07  1:14 [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path Lu Baolu
  2024-04-08  3:52 ` Tian, Kevin
  2024-04-11  2:47 ` Baolu Lu
@ 2026-11-17 18:01 ` Bjorn Helgaas
  2026-11-18  7:49   ` Baolu Lu
  2 siblings, 1 reply; 10+ messages in thread
From: Bjorn Helgaas @ 2026-11-17 18:01 UTC (permalink / raw)
  To: Lu Baolu
  Cc: iommu, Kevin Tian, Joerg Roedel, Will Deacon, Robin Murphy,
	Danny Gonzalez, David Decotigny, Sheena Mohan, Jason Gunthorpe,
	Pranjal Shrivastava, Nicolin Chen, linux-pci, linux-kernel

[+cc Danny and others because this is related to ongoing work at
https://lore.kernel.org/all/20261020181053.2646782-1-digonzal@google.com/]

On Sun, Apr 07, 2024 at 09:14:29AM +0800, Lu Baolu wrote:
> Commit 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
> devices") adds all devices probed by the iommu driver in a rbtree
> indexed by the source ID of each device. It assumes that each device
> has a unique source ID. This assumption is incorrect and the VT-d
> spec doesn't state this requirement either.
> ...

> +++ b/drivers/iommu/intel/iommu.c
> @@ -4299,9 +4299,11 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
>       }
>  
>       dev_iommu_priv_set(dev, info);
> -     ret = device_rbtree_insert(iommu, info);
> -     if (ret)
> -             goto free;
> +     if (pdev && pci_ats_supported(pdev)) {
> +             ret = device_rbtree_insert(iommu, info);
> +             if (ret)
> +                     goto free;
> +     }

This change is upstream as 89436f4f5412 ("iommu/vt-d: Fix WARN_ON in
iommu probe path"), but I think there's a problem here.  Previously
the rbtree insertion was unconditional, so the unconditional remove in
the error path was correct.

After 89436f4f5412, the rbtree insertion is conditional.  If
intel_pasid_alloc_table() fails, we try to remove it even though it
may never have been inserted:

>       if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev)) {
>               ret = intel_pasid_alloc_table(dev);
                if (ret) {
                        dev_err(dev, "PASID table allocation failed\n");
                        goto clear_rbtree;
                }
    ...
    clear_rbtree:
        device_rbtree_remove(info);
    free:
        kfree(info);

> @@ -4336,7 +4338,8 @@ static void intel_iommu_release_device(struct device *dev)
>       struct intel_iommu *iommu = info->iommu;
>  
>       mutex_lock(&iommu->iopf_lock);
> -     device_rbtree_remove(info);
> +     if (dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)))
> +             device_rbtree_remove(info);

This is also somewhat problematic because it relies on the assumption
that pci_ats_supported(X) always returns the same value it returned in
intel_iommu_probe_device().  That may be true today, but I don't think
you should rely on it.

I think both issues could be solved if intel/iommu.c kept track of the
rbtree insertion itself, e.g., with a bit in struct
device_domain_info.

>       mutex_unlock(&iommu->iopf_lock);
>  
>       if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev) &&
> -- 
> 2.34.1
> 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path
  2026-11-17 18:01 ` Bjorn Helgaas
@ 2026-11-18  7:49   ` Baolu Lu
  2026-11-18 15:49     ` Bjorn Helgaas
  0 siblings, 1 reply; 10+ messages in thread
From: Baolu Lu @ 2026-11-18  7:49 UTC (permalink / raw)
  To: Bjorn Helgaas
  Cc: baolu.lu, iommu, Kevin Tian, Joerg Roedel, Will Deacon,
	Robin Murphy, Danny Gonzalez, David Decotigny, Sheena Mohan,
	Jason Gunthorpe, Pranjal Shrivastava, Nicolin Chen, linux-pci,
	linux-kernel

On 11/18/2026 2:01 AM, Bjorn Helgaas wrote:
> [+cc Danny and others because this is related to ongoing work at
> https://lore.kernel.org/all/20261020181053.2646782-1-digonzal@google.com/]
> 
> On Sun, Apr 07, 2024 at 09:14:29AM +0800, Lu Baolu wrote:
>> Commit 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
>> devices") adds all devices probed by the iommu driver in a rbtree
>> indexed by the source ID of each device. It assumes that each device
>> has a unique source ID. This assumption is incorrect and the VT-d
>> spec doesn't state this requirement either.
>> ...
> 
>> +++ b/drivers/iommu/intel/iommu.c
>> @@ -4299,9 +4299,11 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
>>        }
>>   
>>        dev_iommu_priv_set(dev, info);
>> -     ret = device_rbtree_insert(iommu, info);
>> -     if (ret)
>> -             goto free;
>> +     if (pdev && pci_ats_supported(pdev)) {
>> +             ret = device_rbtree_insert(iommu, info);
>> +             if (ret)
>> +                     goto free;
>> +     }
> 
> This change is upstream as 89436f4f5412 ("iommu/vt-d: Fix WARN_ON in
> iommu probe path"), but I think there's a problem here.  Previously
> the rbtree insertion was unconditional, so the unconditional remove in
> the error path was correct.
> 
> After 89436f4f5412, the rbtree insertion is conditional.  If
> intel_pasid_alloc_table() fails, we try to remove it even though it
> may never have been inserted:

info->node is initialized with RB_CLEAR_NODE() before the insertion, and
the remove helper checks RB_EMPTY_NODE() before erasing. If insertion
was skipped, removal is a no-op, so this error path does not need
another state check.

> 
>>        if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev)) {
>>                ret = intel_pasid_alloc_table(dev);
>                  if (ret) {
>                          dev_err(dev, "PASID table allocation failed\n");
>                          goto clear_rbtree;
>                  }
>      ...
>      clear_rbtree:
>          device_rbtree_remove(info);
>      free:
>          kfree(info);
> 
>> @@ -4336,7 +4338,8 @@ static void intel_iommu_release_device(struct device *dev)
>>        struct intel_iommu *iommu = info->iommu;
>>   
>>        mutex_lock(&iommu->iopf_lock);
>> -     device_rbtree_remove(info);
>> +     if (dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)))
>> +             device_rbtree_remove(info);
> 
> This is also somewhat problematic because it relies on the assumption
> that pci_ats_supported(X) always returns the same value it returned in
> intel_iommu_probe_device().  That may be true today, but I don't think
> you should rely on it.

Agreed. The release path should not re-evaluate pci_ats_supported().

> I think both issues could be solved if intel/iommu.c kept track of the
> rbtree insertion itself, e.g., with a bit in struct
> device_domain_info.

Sure. I will address this with a follow-up patch.

> 
>>        mutex_unlock(&iommu->iopf_lock);
>>   
>>        if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev) &&
>> -- 
>> 2.34.1
>>

Thanks,
baolu


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path
  2026-11-18  7:49   ` Baolu Lu
@ 2026-11-18 15:49     ` Bjorn Helgaas
  0 siblings, 0 replies; 10+ messages in thread
From: Bjorn Helgaas @ 2026-11-18 15:49 UTC (permalink / raw)
  To: Baolu Lu
  Cc: iommu, Kevin Tian, Joerg Roedel, Will Deacon, Robin Murphy,
	Danny Gonzalez, David Decotigny, Sheena Mohan, Jason Gunthorpe,
	Pranjal Shrivastava, Nicolin Chen, linux-pci, linux-kernel

On Wed, Nov 18, 2026 at 03:49:30PM +0800, Baolu Lu wrote:
> On 11/18/2026 2:01 AM, Bjorn Helgaas wrote:
> > [+cc Danny and others because this is related to ongoing work at
> > https://lore.kernel.org/all/20261020181053.2646782-1-digonzal@google.com/]
> > 
> > On Sun, Apr 07, 2024 at 09:14:29AM +0800, Lu Baolu wrote:
> > > Commit 1a75cc710b95 ("iommu/vt-d: Use rbtree to track iommu probed
> > > devices") adds all devices probed by the iommu driver in a rbtree
> > > indexed by the source ID of each device. It assumes that each device
> > > has a unique source ID. This assumption is incorrect and the VT-d
> > > spec doesn't state this requirement either.
> > > ...
> > 
> > > +++ b/drivers/iommu/intel/iommu.c
> > > @@ -4299,9 +4299,11 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
> > >        }
> > >        dev_iommu_priv_set(dev, info);
> > > -     ret = device_rbtree_insert(iommu, info);
> > > -     if (ret)
> > > -             goto free;
> > > +     if (pdev && pci_ats_supported(pdev)) {
> > > +             ret = device_rbtree_insert(iommu, info);
> > > +             if (ret)
> > > +                     goto free;
> > > +     }
> > 
> > This change is upstream as 89436f4f5412 ("iommu/vt-d: Fix WARN_ON in
> > iommu probe path"), but I think there's a problem here.  Previously
> > the rbtree insertion was unconditional, so the unconditional remove in
> > the error path was correct.
> > 
> > After 89436f4f5412, the rbtree insertion is conditional.  If
> > intel_pasid_alloc_table() fails, we try to remove it even though it
> > may never have been inserted:
> 
> info->node is initialized with RB_CLEAR_NODE() before the insertion, and
> the remove helper checks RB_EMPTY_NODE() before erasing. If insertion
> was skipped, removal is a no-op, so this error path does not need
> another state check.

Seems like RB_CLEAR_NODE() and RB_EMPTY_NODE() are lacking
documentation: RB_CLEAR_NODE() isn't used at all by lib/rbtree.c and
Documentation/core-api/rbtree.rst doesn't mention it.

But I agree, this does look safe as-is, though requiring a little
inside knowledge of rbtree internals.

> > >        if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev)) {
> > >                ret = intel_pasid_alloc_table(dev);
> >                  if (ret) {
> >                          dev_err(dev, "PASID table allocation failed\n");
> >                          goto clear_rbtree;
> >                  }
> >      ...
> >      clear_rbtree:
> >          device_rbtree_remove(info);
> >      free:
> >          kfree(info);
> > 
> > > @@ -4336,7 +4338,8 @@ static void intel_iommu_release_device(struct device *dev)
> > >        struct intel_iommu *iommu = info->iommu;
> > >        mutex_lock(&iommu->iopf_lock);
> > > -     device_rbtree_remove(info);
> > > +     if (dev_is_pci(dev) && pci_ats_supported(to_pci_dev(dev)))
> > > +             device_rbtree_remove(info);
> > 
> > This is also somewhat problematic because it relies on the assumption
> > that pci_ats_supported(X) always returns the same value it returned in
> > intel_iommu_probe_device().  That may be true today, but I don't think
> > you should rely on it.
> 
> Agreed. The release path should not re-evaluate pci_ats_supported().
> 
> > I think both issues could be solved if intel/iommu.c kept track of the
> > rbtree insertion itself, e.g., with a bit in struct
> > device_domain_info.
> 
> Sure. I will address this with a follow-up patch.

Great, thank you!

Bjorn

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2026-11-18 15:49 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-07  1:14 [PATCH 1/1] iommu/vt-d: Fix WARN_ON in iommu probe path Lu Baolu
2024-04-08  3:52 ` Tian, Kevin
2024-04-08  7:15   ` Baolu Lu
2024-04-08  9:12     ` Tian, Kevin
2024-04-11 12:29   ` Ethan Zhao
2024-04-11  2:47 ` Baolu Lu
2024-04-11  6:53   ` Tian, Kevin
2026-11-17 18:01 ` Bjorn Helgaas
2026-11-18  7:49   ` Baolu Lu
2026-11-18 15:49     ` Bjorn Helgaas

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®