[PATCH v2 0/2] kunit: enable hardware virtualization

[PATCH v2 0/2] kunit: enable hardware virtualization

[Tamir Duberstein]

This series implements feature detection of hardware virtualization on
Linux and macOS; the latter being my primary use case.

This yields approximately a 6x improvement using HVF on M3 Pro.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
Changes in v2:
- Use QEMU accelerator fallback (Alyssa Ross, Thomas Weißschuh).
- Link to v1: https://lore.kernel.org/r/20241025-kunit-qemu-accel-macos-v1-0-2f30c26192d4@gmail.com

---
Tamir Duberstein (2):
      kunit: add fallback for os.sched_getaffinity
      kunit: enable hardware acceleration when available

 tools/testing/kunit/kunit.py              | 11 ++++++++++-
 tools/testing/kunit/kunit_kernel.py       |  3 +++
 tools/testing/kunit/qemu_configs/arm64.py |  2 +-
 3 files changed, 14 insertions(+), 2 deletions(-)
---
base-commit: 81983758430957d9a5cb3333fe324fd70cf63e7e
change-id: 20241025-kunit-qemu-accel-macos-2840e4c2def5

Best regards,
-- 
Tamir Duberstein <tamird@gmail.com>

[PATCH v2 1/2] kunit: add fallback for os.sched_getaffinity

[Tamir Duberstein]

Python 3.13 added os.process_cpu_count as a cross-platform alternative
for the Linux-only os.sched_getaffinity. Use it when it's available and
provide a fallback when it's not.

This allows kunit to run on macOS.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 tools/testing/kunit/kunit.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index bc74088c458aee20b1a21fdeb9f3cb01ab20fec4..3a8cbb868ac559f68d047e38be92f7c64a3314ea 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -303,7 +303,16 @@ def massage_argv(argv: Sequence[str]) -> Sequence[str]:
 	return list(map(massage_arg, argv))
 
 def get_default_jobs() -> int:
-	return len(os.sched_getaffinity(0))
+	if sys.version_info >= (3, 13):
+		if (ncpu := os.process_cpu_count()) is not None:
+			return ncpu
+		raise RuntimeError("os.process_cpu_count() returned None")
+	 # See https://github.com/python/cpython/blob/b61fece/Lib/os.py#L1175-L1186.
+	if sys.platform != "darwin":
+		return len(os.sched_getaffinity(0))
+	if (ncpu := os.cpu_count()) is not None:
+		return ncpu
+	raise RuntimeError("os.cpu_count() returned None")
 
 def add_common_opts(parser: argparse.ArgumentParser) -> None:
 	parser.add_argument('--build_dir',

-- 
2.47.0

[PATCH v2 2/2] kunit: enable hardware acceleration when available

[Tamir Duberstein]

Use KVM or HVF if supported by the QEMU binary and available on the
system.

This produces a nice improvement on my Apple M3 Pro running macOS 14.7:

Before:
./tools/testing/kunit/kunit.py exec --arch arm64
[HH:MM:SS] Elapsed time: 10.145s

After:
./tools/testing/kunit/kunit.py exec --arch arm64
[HH:MM:SS] Elapsed time: 1.773s

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 tools/testing/kunit/kunit_kernel.py       | 3 +++
 tools/testing/kunit/qemu_configs/arm64.py | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
 				'-append', ' '.join(params + [self._kernel_command_line]),
 				'-no-reboot',
 				'-nographic',
+				'-accel', 'kvm',
+				'-accel', 'hvf',
+				'-accel', 'tcg',
 				'-serial', self._serial] + self._extra_qemu_params
 		# Note: shlex.join() does what we want, but requires python 3.8+.
 		print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command))
diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py
index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644
--- a/tools/testing/kunit/qemu_configs/arm64.py
+++ b/tools/testing/kunit/qemu_configs/arm64.py
@@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''',
 			   qemu_arch='aarch64',
 			   kernel_path='arch/arm64/boot/Image.gz',
 			   kernel_command_line='console=ttyAMA0',
-			   extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
+			   extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])

-- 
2.47.0

Re: [PATCH v2 1/2] kunit: add fallback for os.sched_getaffinity

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 1703 bytes --]

On Sat, 2 Nov 2024 at 20:10, Tamir Duberstein <tamird@gmail.com> wrote:
>
> Python 3.13 added os.process_cpu_count as a cross-platform alternative
> for the Linux-only os.sched_getaffinity. Use it when it's available and
> provide a fallback when it's not.
>
> This allows kunit to run on macOS.
>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---

Looks plausible enough to me. Thanks very much!

Reviewed-by: David Gow <davidgow@google.com>

Cheers,
-- David

>  tools/testing/kunit/kunit.py | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
> index bc74088c458aee20b1a21fdeb9f3cb01ab20fec4..3a8cbb868ac559f68d047e38be92f7c64a3314ea 100755
> --- a/tools/testing/kunit/kunit.py
> +++ b/tools/testing/kunit/kunit.py
> @@ -303,7 +303,16 @@ def massage_argv(argv: Sequence[str]) -> Sequence[str]:
>         return list(map(massage_arg, argv))
>
>  def get_default_jobs() -> int:
> -       return len(os.sched_getaffinity(0))
> +       if sys.version_info >= (3, 13):
> +               if (ncpu := os.process_cpu_count()) is not None:
> +                       return ncpu
> +               raise RuntimeError("os.process_cpu_count() returned None")
> +        # See https://github.com/python/cpython/blob/b61fece/Lib/os.py#L1175-L1186.
> +       if sys.platform != "darwin":
> +               return len(os.sched_getaffinity(0))
> +       if (ncpu := os.cpu_count()) is not None:
> +               return ncpu
> +       raise RuntimeError("os.cpu_count() returned None")
>
>  def add_common_opts(parser: argparse.ArgumentParser) -> None:
>         parser.add_argument('--build_dir',
>
> --
> 2.47.0
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5294 bytes --]

Re: [PATCH v2 2/2] kunit: enable hardware acceleration when available

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 2552 bytes --]

On Sat, 2 Nov 2024 at 20:10, Tamir Duberstein <tamird@gmail.com> wrote:
>
> Use KVM or HVF if supported by the QEMU binary and available on the
> system.
>
> This produces a nice improvement on my Apple M3 Pro running macOS 14.7:
>
> Before:
> ./tools/testing/kunit/kunit.py exec --arch arm64
> [HH:MM:SS] Elapsed time: 10.145s
>
> After:
> ./tools/testing/kunit/kunit.py exec --arch arm64
> [HH:MM:SS] Elapsed time: 1.773s
>
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---

Thanks a lot.

I finally managed to dig up an arm64 machine and test this, and I can
reproduce the performance improvement.

Reviewed-by: David Gow <davidgow@google.com>

Cheers,
-- David

>  tools/testing/kunit/kunit_kernel.py       | 3 +++
>  tools/testing/kunit/qemu_configs/arm64.py | 2 +-
>  2 files changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
> index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644
> --- a/tools/testing/kunit/kunit_kernel.py
> +++ b/tools/testing/kunit/kunit_kernel.py
> @@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
>                                 '-append', ' '.join(params + [self._kernel_command_line]),
>                                 '-no-reboot',
>                                 '-nographic',
> +                               '-accel', 'kvm',
> +                               '-accel', 'hvf',
> +                               '-accel', 'tcg',
>                                 '-serial', self._serial] + self._extra_qemu_params
>                 # Note: shlex.join() does what we want, but requires python 3.8+.
>                 print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command))
> diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py
> index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644
> --- a/tools/testing/kunit/qemu_configs/arm64.py
> +++ b/tools/testing/kunit/qemu_configs/arm64.py
> @@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''',
>                            qemu_arch='aarch64',
>                            kernel_path='arch/arm64/boot/Image.gz',
>                            kernel_command_line='console=ttyAMA0',
> -                          extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
> +                          extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])
>
> --
> 2.47.0
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5294 bytes --]

Re: [PATCH v2 2/2] kunit: enable hardware acceleration when available

[Kristina Martsenko]

On 02/11/2024 12:09, Tamir Duberstein wrote:
> Use KVM or HVF if supported by the QEMU binary and available on the
> system.
> 
> This produces a nice improvement on my Apple M3 Pro running macOS 14.7:
> 
> Before:
> ./tools/testing/kunit/kunit.py exec --arch arm64
> [HH:MM:SS] Elapsed time: 10.145s
> 
> After:
> ./tools/testing/kunit/kunit.py exec --arch arm64
> [HH:MM:SS] Elapsed time: 1.773s
> 
> Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> ---
>  tools/testing/kunit/kunit_kernel.py       | 3 +++
>  tools/testing/kunit/qemu_configs/arm64.py | 2 +-
>  2 files changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
> index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644
> --- a/tools/testing/kunit/kunit_kernel.py
> +++ b/tools/testing/kunit/kunit_kernel.py
> @@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
>  				'-append', ' '.join(params + [self._kernel_command_line]),
>  				'-no-reboot',
>  				'-nographic',
> +				'-accel', 'kvm',
> +				'-accel', 'hvf',
> +				'-accel', 'tcg',
>  				'-serial', self._serial] + self._extra_qemu_params
>  		# Note: shlex.join() does what we want, but requires python 3.8+.
>  		print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command))
> diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py
> index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644
> --- a/tools/testing/kunit/qemu_configs/arm64.py
> +++ b/tools/testing/kunit/qemu_configs/arm64.py
> @@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''',
>  			   qemu_arch='aarch64',
>  			   kernel_path='arch/arm64/boot/Image.gz',
>  			   kernel_command_line='console=ttyAMA0',
> -			   extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
> +			   extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])

Would it be possible to keep 'pauth-impdef=on' for TCG emulation? Otherwise
performance regresses by about 20%.

Before this patch:
./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux-
[11:03:38] Elapsed time: 15.494s

After this patch:
./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux-
[11:10:47] Elapsed time: 19.099s

Thanks,
Kristina

Re: [PATCH v2 2/2] kunit: enable hardware acceleration when available

[Tamir Duberstein]

On Tue, Nov 5, 2024 at 8:36 AM Kristina Martsenko
<kristina.martsenko@arm.com> wrote:
>
> On 02/11/2024 12:09, Tamir Duberstein wrote:
> > Use KVM or HVF if supported by the QEMU binary and available on the
> > system.
> >
> > This produces a nice improvement on my Apple M3 Pro running macOS 14.7:
> >
> > Before:
> > ./tools/testing/kunit/kunit.py exec --arch arm64
> > [HH:MM:SS] Elapsed time: 10.145s
> >
> > After:
> > ./tools/testing/kunit/kunit.py exec --arch arm64
> > [HH:MM:SS] Elapsed time: 1.773s
> >
> > Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> > ---
> >  tools/testing/kunit/kunit_kernel.py       | 3 +++
> >  tools/testing/kunit/qemu_configs/arm64.py | 2 +-
> >  2 files changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
> > index 61931c4926fd6645f2c62dd13f9842a432ec4167..3146acb884ecf0bcff94d5938535aabd4486fe82 100644
> > --- a/tools/testing/kunit/kunit_kernel.py
> > +++ b/tools/testing/kunit/kunit_kernel.py
> > @@ -123,6 +123,9 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
> >                               '-append', ' '.join(params + [self._kernel_command_line]),
> >                               '-no-reboot',
> >                               '-nographic',
> > +                             '-accel', 'kvm',
> > +                             '-accel', 'hvf',
> > +                             '-accel', 'tcg',
> >                               '-serial', self._serial] + self._extra_qemu_params
> >               # Note: shlex.join() does what we want, but requires python 3.8+.
> >               print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command))
> > diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py
> > index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644
> > --- a/tools/testing/kunit/qemu_configs/arm64.py
> > +++ b/tools/testing/kunit/qemu_configs/arm64.py
> > @@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''',
> >                          qemu_arch='aarch64',
> >                          kernel_path='arch/arm64/boot/Image.gz',
> >                          kernel_command_line='console=ttyAMA0',
> > -                        extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
> > +                        extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])
>
> Would it be possible to keep 'pauth-impdef=on' for TCG emulation? Otherwise
> performance regresses by about 20%.
>
> Before this patch:
> ./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux-
> [11:03:38] Elapsed time: 15.494s
>
> After this patch:
> ./tools/testing/kunit/kunit.py exec --arch=arm64 --cross_compile=aarch64-linux-
> [11:10:47] Elapsed time: 19.099s
>
> Thanks,
> Kristina

Hi Kristina, thanks for pointing that out. I'm able to reproduce the
regression. I poked around and I can't find a way to enable
`pauth-impdef` only for TCG, and the problem is that enabling it
globally produces:

tools/testing/kunit/kunit.py exec --arch arm64 --make_options LLVM=1
--raw_output=all
[15:34:05] Starting KUnit Kernel (1/1)...
Running tests with:
$ qemu-system-aarch64 -nodefaults -m 1024 -kernel
.kunit/arch/arm64/boot/Image.gz -append 'kunit.enable=1
console=ttyAMA0 kunit_shutdown=reboot' -no-reboot -nographic -accel
kvm -accel hvf -accel tcg -serial stdio -machine virt -cpu
max,pauth-impdef=on
qemu-system-aarch64: -accel kvm: invalid accelerator kvm
qemu-system-aarch64: falling back to HVF
qemu-system-aarch64: can't apply global max-arm-cpu.pauth-impdef=on:
Property 'max-arm-cpu.pauth-impdef' not found

This behavior is at least somewhat intentional[0]. I have filed a bug
with qemu[1]. If someone can conceive of a way to achieve this, I'd be
delighted to send a v3.

Link: https://gitlab.com/qemu-project/qemu/-/commit/92d6528dbb20c6aec4022dfd63c7ffee44f19f77
[0]
Link: https://gitlab.com/qemu-project/qemu/-/issues/2656 [1]

Re: [PATCH v2 2/2] kunit: enable hardware acceleration when available

[Tamir Duberstein]

On Tue, Nov 5, 2024 at 2:51 PM Tamir Duberstein <tamird@gmail.com> wrote:
>
> This behavior is at least somewhat intentional[0]. I have filed a bug
> with qemu[1]. If someone can conceive of a way to achieve this, I'd be
> delighted to send a v3.
>
> [cut]
>
> Link: https://gitlab.com/qemu-project/qemu/-/issues/2656 [1]

There hasn't been movement on this matter here or on the upstream bug.
We can either proceed with this patch as-is or restore the manual
feature detection that was present in v1
(https://lore.kernel.org/all/20241025-kunit-qemu-accel-macos-v1-2-2f30c26192d4@gmail.com/)
in order to retain `pauth-impdef` for TCG. Could I get some guidance
from maintainers please?

Tamir

Re: [PATCH v2 1/2] kunit: add fallback for os.sched_getaffinity

[Tamir Duberstein]

On Tue, Nov 5, 2024 at 3:36 AM David Gow <davidgow@google.com> wrote:
>
> On Sat, 2 Nov 2024 at 20:10, Tamir Duberstein <tamird@gmail.com> wrote:
> >
> > Python 3.13 added os.process_cpu_count as a cross-platform alternative
> > for the Linux-only os.sched_getaffinity. Use it when it's available and
> > provide a fallback when it's not.
> >
> > This allows kunit to run on macOS.
> >
> > Signed-off-by: Tamir Duberstein <tamird@gmail.com>
> > ---
>
> Looks plausible enough to me. Thanks very much!
>
> Reviewed-by: David Gow <davidgow@google.com>
>
> Cheers,
> -- David

Thanks David! While the next patch is still plausibly undergoing
discussion, would it be possible to pick this one up? Without it
kunit.py is not usable on macOS.

Re: [PATCH v2 1/2] kunit: add fallback for os.sched_getaffinity

[Tamir Duberstein]

+Shuah Khan <skhan@linuxfoundation.org>

On Sat, Dec 7, 2024 at 1:52 PM Tamir Duberstein <tamird@gmail.com> wrote:
>
> Thanks David! While the next patch is still plausibly undergoing
> discussion, would it be possible to pick this one up? Without it
> kunit.py is not usable on macOS.

Shuah, could you pick this if you agree?

Cheers,
Tamir