[PATCH 0/2] kunit: enable hardware virtualization

[PATCH 0/2] kunit: enable hardware virtualization

[Tamir Duberstein]

This series implements feature detection of hardware virtualization on
Linux and macOS; the latter being my primary use case.

This yields approximately a 6x improvement using HVF on M3 Pro.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
Tamir Duberstein (2):
      kunit: add fallback for os.sched_getaffinity
      kunit: enable hardware acceleration when available

 tools/testing/kunit/kunit.py              | 11 ++++++++++-
 tools/testing/kunit/kunit_kernel.py       | 26 +++++++++++++++++++++++++-
 tools/testing/kunit/qemu_configs/arm64.py |  2 +-
 3 files changed, 36 insertions(+), 3 deletions(-)
---
base-commit: ae90f6a6170d7a7a1aa4fddf664fbd093e3023bc
change-id: 20241025-kunit-qemu-accel-macos-2840e4c2def5

Best regards,
-- 
Tamir Duberstein <tamird@gmail.com>

[PATCH 1/2] kunit: add fallback for os.sched_getaffinity

[Tamir Duberstein]

Python 3.13 added os.process_cpu_count as a cross-platform alternative
for the Linux-only os.sched_getaffinity. Use it when it's available and
provide a fallback when it's not.

This allows kunit to run on macOS.

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 tools/testing/kunit/kunit.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index bc74088c458aee20b1a21fdeb9f3cb01ab20fec4..3a8cbb868ac559f68d047e38be92f7c64a3314ea 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -303,7 +303,16 @@ def massage_argv(argv: Sequence[str]) -> Sequence[str]:
 	return list(map(massage_arg, argv))
 
 def get_default_jobs() -> int:
-	return len(os.sched_getaffinity(0))
+	if sys.version_info >= (3, 13):
+		if (ncpu := os.process_cpu_count()) is not None:
+			return ncpu
+		raise RuntimeError("os.process_cpu_count() returned None")
+	 # See https://github.com/python/cpython/blob/b61fece/Lib/os.py#L1175-L1186.
+	if sys.platform != "darwin":
+		return len(os.sched_getaffinity(0))
+	if (ncpu := os.cpu_count()) is not None:
+		return ncpu
+	raise RuntimeError("os.cpu_count() returned None")
 
 def add_common_opts(parser: argparse.ArgumentParser) -> None:
 	parser.add_argument('--build_dir',

-- 
2.47.0

[PATCH 2/2] kunit: enable hardware acceleration when available

[Tamir Duberstein]

Use KVM or HVF if supported by the QEMU binary and available on the
system.

This produces a nice improvement on my Apple M3 Pro running macOS 14.7:

Before:
./tools/testing/kunit/kunit.py exec --arch arm64
[HH:MM:SS] Elapsed time: 10.145s

After:
./tools/testing/kunit/kunit.py exec --arch arm64
[HH:MM:SS] Elapsed time: 1.773s

Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
 tools/testing/kunit/kunit_kernel.py       | 26 +++++++++++++++++++++++++-
 tools/testing/kunit/qemu_configs/arm64.py |  2 +-
 2 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
index 61931c4926fd6645f2c62dd13f9842a432ec4167..10cacf5a3c443bacd6c074647e4bddfc31599cf8 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -116,7 +116,8 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
 
 	def start(self, params: List[str], build_dir: str) -> subprocess.Popen:
 		kernel_path = os.path.join(build_dir, self._kernel_path)
-		qemu_command = ['qemu-system-' + self._qemu_arch,
+		qemu_binary = 'qemu-system-' + self._qemu_arch
+		qemu_command = [qemu_binary,
 				'-nodefaults',
 				'-m', '1024',
 				'-kernel', kernel_path,
@@ -124,6 +125,29 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
 				'-no-reboot',
 				'-nographic',
 				'-serial', self._serial] + self._extra_qemu_params
+		accelerators = {
+			line.strip()
+			for line in subprocess.check_output([qemu_binary, "-accel", "help"], text=True).splitlines()
+			if line and line.islower()
+		}
+		if 'kvm' in accelerators:
+			try:
+				with open('/dev/kvm', 'rb+'):
+					qemu_command.extend(['-accel', 'kvm'])
+			except OSError as e:
+				print(e)
+		elif 'hvf' in accelerators:
+			try:
+				for line in subprocess.check_output(['sysctl', 'kern.hv_support'], text=True).splitlines():
+					if not line:
+						continue
+					key, value = line.split(':')
+					if key == 'kern.hv_support' and bool(value):
+						qemu_command.extend(['-accel', 'hvf'])
+						break
+			except subprocess.CalledProcessError as e:
+				print(e)
+
 		# Note: shlex.join() does what we want, but requires python 3.8+.
 		print('Running tests with:\n$', ' '.join(shlex.quote(arg) for arg in qemu_command))
 		return subprocess.Popen(qemu_command,
diff --git a/tools/testing/kunit/qemu_configs/arm64.py b/tools/testing/kunit/qemu_configs/arm64.py
index d3ff27024755411441f910799be30399295c9541..5c44d3a87e6dd2cd6b086138186a277a1473585b 100644
--- a/tools/testing/kunit/qemu_configs/arm64.py
+++ b/tools/testing/kunit/qemu_configs/arm64.py
@@ -9,4 +9,4 @@ CONFIG_SERIAL_AMBA_PL011_CONSOLE=y''',
 			   qemu_arch='aarch64',
 			   kernel_path='arch/arm64/boot/Image.gz',
 			   kernel_command_line='console=ttyAMA0',
-			   extra_qemu_params=['-machine', 'virt', '-cpu', 'max,pauth-impdef=on'])
+			   extra_qemu_params=['-machine', 'virt', '-cpu', 'max'])

-- 
2.47.0

Re: [PATCH 2/2] kunit: enable hardware acceleration when available

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

On Fri, Oct 25, 2024 at 05:03:54PM -0400, Tamir Duberstein wrote:
> @@ -124,6 +125,29 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
>  				'-no-reboot',
>  				'-nographic',
>  				'-serial', self._serial] + self._extra_qemu_params
> +		accelerators = {
> +			line.strip()
> +			for line in subprocess.check_output([qemu_binary, "-accel", "help"], text=True).splitlines()
> +			if line and line.islower()
> +		}
> +		if 'kvm' in accelerators:
> +			try:
> +				with open('/dev/kvm', 'rb+'):
> +					qemu_command.extend(['-accel', 'kvm'])
> +			except OSError as e:
> +				print(e)
> +		elif 'hvf' in accelerators:
> +			try:
> +				for line in subprocess.check_output(['sysctl', 'kern.hv_support'], text=True).splitlines():
> +					if not line:
> +						continue
> +					key, value = line.split(':')
> +					if key == 'kern.hv_support' and bool(value):
> +						qemu_command.extend(['-accel', 'hvf'])
> +						break
> +			except subprocess.CalledProcessError as e:
> +				print(e)
> +

QEMU supports falling back if one accelerator is not available, if you
specify multiple like -accel kvm:tcg.  Couldn't you rely on that rather
than re-implementing the availability checks here?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 2/2] kunit: enable hardware acceleration when available

[Tamir Duberstein]

On Fri, Nov 1, 2024, 09:52 Alyssa Ross <hi@alyssa.is> wrote:
>
> On Fri, Oct 25, 2024 at 05:03:54PM -0400, Tamir Duberstein wrote:
> > @@ -124,6 +125,29 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
> >                               '-no-reboot',
> >                               '-nographic',
> >                               '-serial', self._serial] + self._extra_qemu_params
> > +             accelerators = {
> > +                     line.strip()
> > +                     for line in subprocess.check_output([qemu_binary, "-accel", "help"], text=True).splitlines()
> > +                     if line and line.islower()
> > +             }
> > +             if 'kvm' in accelerators:
> > +                     try:
> > +                             with open('/dev/kvm', 'rb+'):
> > +                                     qemu_command.extend(['-accel', 'kvm'])
> > +                     except OSError as e:
> > +                             print(e)
> > +             elif 'hvf' in accelerators:
> > +                     try:
> > +                             for line in subprocess.check_output(['sysctl', 'kern.hv_support'], text=True).splitlines():
> > +                                     if not line:
> > +                                             continue
> > +                                     key, value = line.split(':')
> > +                                     if key == 'kern.hv_support' and bool(value):
> > +                                             qemu_command.extend(['-accel', 'hvf'])
> > +                                             break
> > +                     except subprocess.CalledProcessError as e:
> > +                             print(e)
> > +
>
> QEMU supports falling back if one accelerator is not available, if you
> specify multiple like -accel kvm:tcg.  Couldn't you rely on that rather
> than re-implementing the availability checks here?

Have you ever used that? Here's what I get when I try:

tamird@Tamirs-MacBook-Pro linux % tools/testing/kunit/kunit.py run
--arch arm64 --make_options LLVM=1 --raw_output=all
[10:45:03] Configuring KUnit Kernel ...
[10:45:03] Building KUnit Kernel ...
Populating config with:
$ make ARCH=arm64 O=.kunit olddefconfig LLVM=1
Building with:
$ make all compile_commands.json ARCH=arm64 O=.kunit --jobs=12 LLVM=1
[10:45:07] Starting KUnit Kernel (1/1)...
Running tests with:
$ qemu-system-aarch64 -nodefaults -m 1024 -kernel
.kunit/arch/arm64/boot/Image.gz -append 'kunit.enable=1
console=ttyAMA0 kunit_shutdown=reboot' -no-reboot -nographic -serial
stdio -accel kvm:tcg -machine virt -cpu max
qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg

The same thing happens with hvf:kvm:tcg or just hvf:tcg.

I also can't find this in the documentation.
Tamir

Re: [PATCH 2/2] kunit: enable hardware acceleration when available

[Thomas Weißschuh]

On 2024-11-01 10:49:36-0400, Tamir Duberstein wrote:
> On Fri, Nov 1, 2024, 09:52 Alyssa Ross <hi@alyssa.is> wrote:
> >
> > On Fri, Oct 25, 2024 at 05:03:54PM -0400, Tamir Duberstein wrote:
> > > @@ -124,6 +125,29 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
> > >                               '-no-reboot',
> > >                               '-nographic',
> > >                               '-serial', self._serial] + self._extra_qemu_params
> > > +             accelerators = {
> > > +                     line.strip()
> > > +                     for line in subprocess.check_output([qemu_binary, "-accel", "help"], text=True).splitlines()
> > > +                     if line and line.islower()
> > > +             }
> > > +             if 'kvm' in accelerators:
> > > +                     try:
> > > +                             with open('/dev/kvm', 'rb+'):
> > > +                                     qemu_command.extend(['-accel', 'kvm'])
> > > +                     except OSError as e:
> > > +                             print(e)
> > > +             elif 'hvf' in accelerators:
> > > +                     try:
> > > +                             for line in subprocess.check_output(['sysctl', 'kern.hv_support'], text=True).splitlines():
> > > +                                     if not line:
> > > +                                             continue
> > > +                                     key, value = line.split(':')
> > > +                                     if key == 'kern.hv_support' and bool(value):
> > > +                                             qemu_command.extend(['-accel', 'hvf'])
> > > +                                             break
> > > +                     except subprocess.CalledProcessError as e:
> > > +                             print(e)
> > > +
> >
> > QEMU supports falling back if one accelerator is not available, if you
> > specify multiple like -accel kvm:tcg.  Couldn't you rely on that rather
> > than re-implementing the availability checks here?
> 
> Have you ever used that? Here's what I get when I try:
> 
> tamird@Tamirs-MacBook-Pro linux % tools/testing/kunit/kunit.py run
> --arch arm64 --make_options LLVM=1 --raw_output=all
> [10:45:03] Configuring KUnit Kernel ...
> [10:45:03] Building KUnit Kernel ...
> Populating config with:
> $ make ARCH=arm64 O=.kunit olddefconfig LLVM=1
> Building with:
> $ make all compile_commands.json ARCH=arm64 O=.kunit --jobs=12 LLVM=1
> [10:45:07] Starting KUnit Kernel (1/1)...
> Running tests with:
> $ qemu-system-aarch64 -nodefaults -m 1024 -kernel
> .kunit/arch/arm64/boot/Image.gz -append 'kunit.enable=1
> console=ttyAMA0 kunit_shutdown=reboot' -no-reboot -nographic -serial
> stdio -accel kvm:tcg -machine virt -cpu max
> qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg
> 
> The same thing happens with hvf:kvm:tcg or just hvf:tcg.

That syntax is for -machine accel=hvf:kvm:tcg.
But you can also specify -accel multiple times.

> I also can't find this in the documentation.

-accel name[,prop=value[,...]]
      This  is used to enable an accelerator. [..]
      If there is more than one accelerator specified,
      the next one is used if the previous one fails to initialize.

-machine [type=]name[,prop=value[,...]]
      Supported machine properties are:

      accel=accels1[:accels2[:...]]
	     [..]
	     If there is more than one accelerator specified,
	     the next one is used if the  previous one fails to initialize.


Thomas

Re: [PATCH 2/2] kunit: enable hardware acceleration when available

[Tamir Duberstein]

On Fri, Nov 1, 2024 at 11:23 AM Thomas Weißschuh <thomas@t-8ch.de> wrote:
>
> On 2024-11-01 10:49:36-0400, Tamir Duberstein wrote:
> > On Fri, Nov 1, 2024, 09:52 Alyssa Ross <hi@alyssa.is> wrote:
> > >
> > > On Fri, Oct 25, 2024 at 05:03:54PM -0400, Tamir Duberstein wrote:
> > > > @@ -124,6 +125,29 @@ class LinuxSourceTreeOperationsQemu(LinuxSourceTreeOperations):
> > > >                               '-no-reboot',
> > > >                               '-nographic',
> > > >                               '-serial', self._serial] + self._extra_qemu_params
> > > > +             accelerators = {
> > > > +                     line.strip()
> > > > +                     for line in subprocess.check_output([qemu_binary, "-accel", "help"], text=True).splitlines()
> > > > +                     if line and line.islower()
> > > > +             }
> > > > +             if 'kvm' in accelerators:
> > > > +                     try:
> > > > +                             with open('/dev/kvm', 'rb+'):
> > > > +                                     qemu_command.extend(['-accel', 'kvm'])
> > > > +                     except OSError as e:
> > > > +                             print(e)
> > > > +             elif 'hvf' in accelerators:
> > > > +                     try:
> > > > +                             for line in subprocess.check_output(['sysctl', 'kern.hv_support'], text=True).splitlines():
> > > > +                                     if not line:
> > > > +                                             continue
> > > > +                                     key, value = line.split(':')
> > > > +                                     if key == 'kern.hv_support' and bool(value):
> > > > +                                             qemu_command.extend(['-accel', 'hvf'])
> > > > +                                             break
> > > > +                     except subprocess.CalledProcessError as e:
> > > > +                             print(e)
> > > > +
> > >
> > > QEMU supports falling back if one accelerator is not available, if you
> > > specify multiple like -accel kvm:tcg.  Couldn't you rely on that rather
> > > than re-implementing the availability checks here?
> >
> > Have you ever used that? Here's what I get when I try:
> >
> > tamird@Tamirs-MacBook-Pro linux % tools/testing/kunit/kunit.py run
> > --arch arm64 --make_options LLVM=1 --raw_output=all
> > [10:45:03] Configuring KUnit Kernel ...
> > [10:45:03] Building KUnit Kernel ...
> > Populating config with:
> > $ make ARCH=arm64 O=.kunit olddefconfig LLVM=1
> > Building with:
> > $ make all compile_commands.json ARCH=arm64 O=.kunit --jobs=12 LLVM=1
> > [10:45:07] Starting KUnit Kernel (1/1)...
> > Running tests with:
> > $ qemu-system-aarch64 -nodefaults -m 1024 -kernel
> > .kunit/arch/arm64/boot/Image.gz -append 'kunit.enable=1
> > console=ttyAMA0 kunit_shutdown=reboot' -no-reboot -nographic -serial
> > stdio -accel kvm:tcg -machine virt -cpu max
> > qemu-system-aarch64: -accel kvm:tcg: invalid accelerator kvm:tcg
> >
> > The same thing happens with hvf:kvm:tcg or just hvf:tcg.
>
> That syntax is for -machine accel=hvf:kvm:tcg.
> But you can also specify -accel multiple times.
>
> > I also can't find this in the documentation.
>
> -accel name[,prop=value[,...]]
>       This  is used to enable an accelerator. [..]
>       If there is more than one accelerator specified,
>       the next one is used if the previous one fails to initialize.
>
> -machine [type=]name[,prop=value[,...]]
>       Supported machine properties are:
>
>       accel=accels1[:accels2[:...]]
>              [..]
>              If there is more than one accelerator specified,
>              the next one is used if the  previous one fails to initialize.
>
>
> Thomas

Nice, thank you. These details seem to appear only in the docs[0] and
not in the output of `--help`. I'll make that change in v2.

Link: https://www.qemu.org/docs/master/system/invocation.html [0]


Tamir